p-hacking and science with an agenda

I recently read this post about p-hacking (see also: data dredging, fishing, snooping). Two things that I found to be noteworthy were an interactive example of how p-hacking works, and a description of an experiment where different research teams analyzed the same data set:

 

“Twenty-nine teams with a total of 61 analysts took part. The researchers used a wide variety of methods, ranging — for those of you interested in the methodological gore — from simple linear regression techniques to complex multilevel regressions and Bayesian approaches. They also made different decisions about which secondary variables to use in their analyses.

 

Despite analyzing the same data, the researchers got a variety of results. Twenty teams concluded that soccer referees gave more red cards to dark-skinned players, and nine teams found no significant relationship between skin color and red cards.”

 

To reiterate, all of the methods used were justifiable. There wasn’t any fudging or fabricating data. A group of skilled analysts sat down and came up with 29 defensible methods for analyzing the same data that gave different answers. To me, this is the stuff of existential crises. To quote the article, “[e]very result is a temporary truth”. Which I think is pretty concerning if you’re working in a situation where temporary truths don’t cut it.

 

Joshua Tewksbury is a biologist who spent 10 years as a professor at the University of Washington before moving to a position with the World Wildlife Fund. About a year ago, he wrote a post about transitioning to an NGO position where, he writes, “[s]cience shows up as just another wrench in the toolkit.” A deeply malleable tool, apparently. On the one hand, it’s troubling to think about making decisions with temporary truths. On the other hand, and this strikes me as almost heretical to type, if you deeply believe in your cause, maybe it’s not so bad to (ethically and with full disclosure) make subjective decisions in how you analyze your data to advance your cause.

 

After thinking about it for a while, I’m still not sure how bad my crisis should be. In the first post, one of the project leaders is quoted as saying:

 

“On the one hand, our study shows that results are heavily reliant on analytic choices,” Uhlmann told me. “On the other hand, it also suggests there’s a there there. It’s hard to look at that data and say there’s no bias against dark-skinned players.”

 

At first pass, this didn’t help me. As somebody who takes comfort in certainty (and don’t most scientists?) the “squint at it” method of assessing data is an endless source of frustration. But I’ve also realized that we might feel confident about one other thing from the soccer data set. No groups concluded that lighter skinned players received more red cards. Maybe there are some relatively permanent truths, it’s just that they don’t answer the question we set out to answer.

In the field, eh?

Traveling internationally is good for the brain. I’m not sure which synapses are “sparking” differently, but I’ve been in Canada since last Wednesday and I’m pretty sure I think different up north than I do down south. Simple things are not simple anymore and I devote thinking time to things like speed limits and temperatures and how metric-illiterate I am. I also have to look at money very closely, wondering which size and color coin is which amount. I wait for my change in the grocery store but never get it because the country “rounds” the tab — nobody gets pennies back. The end result is that when I’m forced to think about little things I ordinarily wouldn’t pay attention to, I also start thinking about work things in a way that I wouldn’t ordinarily do.

I’m being hosted by Bill Nelson‘s lab at Queen’s University as part of a research exchange program. Bill is a biologist/mathematician who also crosses into the field-work world to look at Daphnia populations in local lakes.

Week one: I was thinking in math. Something I would ordinarily not consider myself to be doing. And when I inevitably had to email Bill after running into troubles, I received in return one of the best email responses ever written: a picture of a hand written math solution.

FullSizeRender

The simple translated email was: “You’re using an incorrect equation. Use this one instead.”

I would love if more emails were hand written. I miss seeing what hand writing looks like.

Week two: I worked in a boat. I’ve never worked in a boat before.

Field work in a boat: sampling gear laid out on the bench as we anchor in to the first field site.

Field work in a boat: sampling gear laid out on the bench as we anchor in at the first field site.

photo 2(2)

Bill samples water for Daphnia from six lakes. The only way to get to sampling locations is via rowing.

I learned science while learning how to row. The rowing learning curve was a bit steeper than expected. Sitting between the oars with Bill in the back and Shelley sitting in the bow, I swerved too far left, distracted by loons, then overcorrected right, swerving back and forth before we got to our buoy-marked location in the deepest part of the lake.

Loons. I must be in Canada.

Loons. I must be in Canada.

From there it was more non-stop learning: new biology jargon, new facts about the layers in water bodies and new terms describing aquatic species and communities.

Tomorrow, week two of learning continues! This time off-boat.

Multiverse?

I spend a lot of time thinking about stochasticity (i.e. randomness) — what causes it, when is it important, and how does it impact data?  So I was really interested in a recent tv episode that discussed some of the different theories of what stochasticity is.  One that I thought was particularly interesting was the theory that stochasticity is an illusion.

Before I go on, a disclaimer.  I am a biologist.  I know very little about quantum physics.  Everything beyond this sentence should be viewed with a lot of skepticism.

To summarize in a way that is almost certainly oversimplifying (and wrong), many particles in our universe exist in multiple states at once.  When these particles are observed, they must “choose” a single state, which results in a single outcome.  If they had “chosen” a different state, it could have resulted in a different outcome.  The theory presented in this show that fascinated me was that no decision ever actually gets made, but instead, the universe splits at each decision such that every outcome occurs, but only one in each parallel universe.  So to the observer, it looks like stochasticity exists, but in actuality it doesn’t.  In practice, this results in infinite (or effectively infinite) parallel universes.

My problem with this theory is that by my reasoning it seems like it must be either useless, or wrong.  If multiple universes existed, but it is impossible to travel or communicate between them, then the theory is useless, because it has no impact on my reality, and it never could.  On the other hand, if it were possible to travel or communicate across these universes, and there are infinite of them, then there must be universes where people developed the technology to travel to other universes.  And in these universes where they developed the ability to travel, in a further subset of these universes, the people must have thought that it would be a good idea to spread their technology to other universes.  In still a further subset, the people from some of these universes that shared the travel technology must have thought that it would be a good idea to recruit other universes to help them spread the technology further.  This would create a multiplicative effect such that the technology would be spread to all universes.  So the fact that we don’t currently have the technology to travel between universes suggests that if there were infinite other universes, it must be impossible to travel between them.

Admittedly, my math might be wrong –I find it hard to think in infinites.  Alternatively, one might suggest that there hasn’t been enough time yet for the technology to spread, and that I should just be patient.  But that seems highly unlikely.  If there were infinite universes, sentient beings would have evolved and developed this travel ability on at least some of them long ago — long enough ago to spread it I would imagine.  A more interesting alternative to me is that there are infinite universes, and people trying to spread this technology, but there are also universes that don’t want the travel technology to get out, and so they travel from universe to universe destroying the people who are trying to spread the technology.

Field trials

I’m currently in Cote d’Ivoire practicing my patient waiting skills. To pass the time, I started reading this book on how to conduct field trials for public health interventions. I’m currently on Chapter 11: Randomization, Blinding, and Coding. In addition to being a useful reference, it’s turning out to be a surprisingly interesting read.

Part of what I’ve learned so far is the real nuts and bolts of designing a field trial. For example, how to calculate trial size, how to stratify treatment arms, various methods for randomizing people or groups. I now know how to code individuals and their relationships in a polygamous household for census purposes.

There is also a section on ethical considerations, where I learned that it’s considered unethical to provide a standard of care that cannot be maintained after the end of the trial. This is something that seemed perfectly reasonable after I read it, but I probably wouldn’t have thought about it otherwise.

A lot of the blinding discussion is like this as well. I know blinding is important, and I know double blinding is the gold standard, but I haven’t really thought a lot about why it matters or how exactly to do it. For example there’s a lot of emphasis on making sure the placebo and treatment are as similar looking as possible (obviously this isn’t always possible, for example with interventions that are behavioral or structural). I wouldn’t think it matters so much, as long as nobody knew which pill was the active one. But apparently, different colored placebo and treatment pills can lead to problems, even if the study participants don’t know which color is which, just because one color is perceived to be more effective.

Blinding is not necessarily possible even when the treatment and placebo are identical looking pills. When people received ivermectin (an anti-helminthic drug) in a study testing the drug for use against river blindness, people were able to guess that they received the anti-helminthic because they started to pass non-target worms in their poop.

I’m looking forward to Chapter 12: Outcome Measures and Case Definition, which I’m sure I will have plenty of time to read in the upcoming week.

A PSA for PSU folks

larger

 

 

 

 

This a “public service announcement”.

State College happens to have some fun things going on this summer and it seems that sometimes being a full time scientist doesn’t leave a lot of room for extracurriculars. Or, if you do get some free time maybe you don’t keep a pulse on local happenings and might miss out on some summer opportunities. That’s right, I just used FOMO to get you to go out and do something fun. You’re welcome.

Here are a few highlights in the area for those that felt the first chill of fall in the air yesterday and are thinking about making the most of these last few days of nice weather. There is a strong bias toward what interests me (music and local food), so please add your own favorites as comments. Let’s make plans to meet up and enjoy some summer fun before students return!

Thursday nights until September there is Wingfest at Tussey Mountain with music at 5:30pm (and hikes in Rothrock anytime), First Friday at Downtown State College every FridayFriday night concerts 7:30pm on the Lemont Green, Farmer’s markets all over the place most days of the week (including Tuesdays  2-6 at the Boalsburg Military Museum and even through the winter indoors, and Wednesdays in Lemont 3-7pm), Farmfest this weekend in nearby Centre Hall, music on Sundays at Webster’s cafe, Shaver’s creek raptor center is cool and had and nearby hikes, and lots of other fun places to hike, birdwatch, or swim.

I hear work-life balance is important…

Training the bee “nose”

A few weeks ago I was hiking in Velebit National Park, Croatia. My sister and I had roadtripped there in a cheap $10/day rental car from the capital city of Zagreb to climb peaks, trail run and get muddy  (a trail runner’s favorite thing to do). We drove up to the National Park visitor center to get a trail map and were given a map similar to this one. In red are areas with suspected land mines.

I’d never been given a land-mine map in prep for hiking before. Most U.S. maps have warnings for more benign things like a dry spring or a sink hole. Here it seemed a bit more important to not venture off-trail.

IMG_2797

The land mines in Croatia are relics of the Yugoslav wars that took place from 1991 – 1995. Twenty years later, land mines are still a danger to civilians and tourists in regions that, on the surface, appear to be recovered.

Who would expect a land mine in a place that looks like this?

IMG_3056

The Croatian plan for land mine removal involves training honey bees to smell out mines.  In many parts of Afghanistan, they use dogs. In Cambodia and Mozambique, they use giant rats (how giant? these rats are over 3 ft in length from nose-tip to tail-tip). I’ve seen trained dogs and trained rats but I had never heard of a trained insect. Can insects be trained?

Insects, apparently, can be trained and have been for a variety of purposes. Mosquitoes have been trained to change landing preferences when certain lands were paired with electric shock. Cockroaches have been trained to “carry” backpacks for search and rescue missions. Wasps have been trained to detect a variety of drugs and pathogens. Maybe it’s not surprising then that bees can sniff out mines.

It seems like scientists are getting good at training insects to find and do things. Now maybe we can work on the reverse: can we train mosquitoes to not find me?