Some of you may have been watching, or even participating in, a debate at the moment around pre-registration of scientific studies. I don’t want to do a long post on this, mainly as other people have already written excellent blogs on the topic, but I did want to write a quick note about why I’m strongly in favour of preregistering studies.
Essentially, preregistration would require study authors to publish their planned analyses before they conduct the study. The idea is that this will cut down on spurious findings, because authors will have already specified what their analysis will look like. They can’t then engage in “fishing expeditions”, where you change your analysis to try and find a significant result. The aim would be to make research more reliable and robust, because authors can’t go digging around for unexpected findings in order to find something interesting to write a paper about.
There have been strong criticisms of this approach though, and I wanted to focus on just one here: the idea that this approach would kill off exploratory science, and that we would never ‘discover’ anything again. Because if you didn’t plan to look for unexpected Effect X, then you have to ignore that Effect X has occurred, as you didn’t say in your analyses that Effect X was on your list of things to look for. But what if Effect X is something really, really important??
I think this is daft. Obviously, if someone finds something interesting, then they can still report it and comment on it – we don’t have to ignore unexpected findings or discoveries. The only difference is that is would be transparent that this is what had happened – ie. it makes clear to any readers that they didn’t expect to find that effect, and this is an exploratory finding. This might mean that we’d be a bit more cautious about drawing conclusions based on that finding, or that we focus on the need for Effect X to be replicated. I think both of those are good things.
On a more personal note, I find the discussion about stumbling across magical, unheard of effects quite odd, in that it doesn’t chime with my experience of conducting research at all. I have found that a focus on drawing out consistent results through synthesis, and careful hypothesis testing, have a much stronger relationship with generating interesting findings. However, that might be because of the field I’m in and it may be different in other disciplines (for example, it’s worth noting that in health research clinical trials are already expected to be preregistered.)
There are a lot of other interesting issues around preregistration – how it relates to the current hot topic (especially in psychology) of replication, how ‘fishing’ itself is implicitly encouraged by journals which publish only significant findings, and how preregistration relates to efforts to increase the transparency of science. It’s a fascinating issue, and I look forward to seeing how the debate develops. You can follow it on twitter with the #prereg hashtag, and do check out the comments so far which have been storified by Pete Etchells.