This post is a response to “Pre-Registration of Analysis of Experiments is Dangerous for Science” by Mel Slater (2016). Preregistration is stating what you’re going to do and how you’re going to do it before you collect data (for more detail, read this). Slater gives a few examples of hypothetical (but highly plausible) experiments and explains why preregistering the analyses of the studies (not preregistration of the studies themselves) would not have worked. I will reply to his comments and attempt to show why he is wrong.

Slater describes an experiment where they are conducting a between groups experimental design, with 2 conditions (experimental & control), 1 response variable, and no covariates. You find the expected result but it’s not exactly as you predicted. It turns out the result is totally explained by the gender of the participants (a variable you weren’t initially analysing but was balanced by chance). So its gone from a 2 group analysis to a 2×2 analysis (with the experimental & control conditions as one factor and male & female being the other).

Slater then argues that (according to preregistration) you must preregister and conduct a new experimental design because you have not preregistered those new analyses (which analyse the role of gender). The example steadily gets more detailed (with other covariates discovered along the way) until the final analysis is very different from what you initially expected. He states that you would need to throw out the data each time and start again every time you find a new covariate or factor because it wasn’t initially preregistered. The reason you would need to restart your experiment is because doing a “post hoc analysis is not supposed to be valid in the classical statistical framework”. So because you didn’t preregister the analyses you now want to perform, you need to restart the whole process. This can result in wrong conclusions being drawn as it could lead to complex (but non-predicted) relationships being missed as the original finding will be published (as often it’s too expensive or time consuming or not even possible to run the experiment again with the new analyses) and the role of gender (and the other covariates) won’t be explored.

This is, however, a fundamental misunderstanding of what preregistration of analyses is. If you perform any new analyses on your data that weren’t preregistered, you don’t need to set up another study. You can perform these new analyses (which you didn’t predict before the experiment began) but you have to be explicit in the Results section that this was the case (Chambers, Feredoes, Muthukumaraswamy, & Etchells; 2014). And post hoc analyses of the data is very common (Ioannidis, 2005) and preregistration is directly trying to counter this.

Later in the post, he argues “discovery is out the window” because this occurs when “you get results that are not those that were predicted by the experiment.” Preregistration would therefore stifle discovery as you have to conduct a new study for each new analysis you want to perform. He states “Registrationists” argue for an ‘input-output’ model of science where “thought is eliminated”.

This is a fair concern, but it has already been answered by the FAQ page for Registered Reports (link here) and many other places. To summarise, discovery will not be stifled because you can perform the non-predicted analyses but you have to clearly state they weren’t predicted. The only thing you aren’t allowed to do is pretend you were going to conduct that analysis initially which is called HARKing, or hypothesising after results known (Kerr, 1998).

Slater argues that because data in the life and social sciences is so messy (as compared with physics) it is much harder to make the distinction between ‘exploratory’ and ‘confirmatory’ experiments. He implies preregistration requires a harsh divide between them so confirmatory experiments cannot become exploratory (which often happens in the real world) because they weren’t preregistered. Whilst there would be a clearer divide between exploratory and confirmatory experiments, preregistration does not forbid the latter becoming the former (merely that you are open about what you’ve done). Having a clear divide between the two is very important for maintaining the value of both types of experiments (de Groot, 2014).

He argues that (due to the pressure to publish positive results) researchers could “run their experiment, fish for some result, and then register it”. But this is not “possible without committing fraud” (Chambers, Feredoes, Muthukumaraswamy, & Etchells; 2014). You have to share time-stamped raw data files that were used in the study so you can see when the data was collected. This will help reduce the chance of fraud and ensure they are performed properly.

He argues that currently there is not enough thought put into the analysis process. He states this based on the fact results sections start with F-tests and t-tests rather than presenting the data in tables and graphs and discussing it. Researchers look straight for the result they were expecting and only focus on those, potentially missing other important aspects. Preregistration, he believes, would exacerbate this problem.

Whilst I agree there is an over-emphasis on getting P<0.05 in the literature, preregistration will not make this problem worse. If anything, preregistration could help reduce the collective obsession with P<0.05 because if a study is preregistered and agreed for publication (based on the quality of the methods) then it won’t rely on a significant value to be published (see here for a diagram of the registration and publication process). It also makes replications of previous findings more attractive to researchers because publication doesn’t depend on the results, which we know has lead to the neglect of replications (Nosek, Spies, & Motyl, 2012).

Could preregistration increase the likelihood that researchers focus solely on their preregistered analyses and ignore other potential findings? Maybe, but this worry is very abstract. This is contrasted with the very real (and very damaging) problem of questionable research practices (QRPs) which we know plague the literature (John, Loewenstein, & Prelec; 2012) and have a negative impact (Simmons, Nelson, & Simonsohn; 2011). Preregistration can help limit these QRPs.

Is preregistration the panacea for psychology’s replication crisis? No, but then it never claimed to be. It’s one of the (many) tools to help improve psychology.

References:

Bowman, S.; Chambers, D.C.; & Nosek, B.A. (2014). FAQ 5: Scientific Creativity and Exploration.  [OSF open-ended registration] Available at: https://osf.io/8mpji/wiki/FAQ%205:%20Scientific%20Creativity%20and%20Exploration/ [Accessed on 19/05/2016].

Chambers, D.C. (2015). Cortex’s Registered Reports: How Cortex’s Registered Reports initiative is making reform a reality. Available at: https://www.elsevier.com/editors-update/story/peer-review/cortexs-registered-reports [Accessed on 16/05/2016]

Chambers, D.C.; Feredoes, E.; Muthukumaraswamy, S.D.; & Etchells, P.J. (2014). Instead of “playing the game” it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1 (1), 4-17.

de Groot AD. (2014) The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. Acta Psychologica (Amst), 148, 188-194.

Ioannidis, J.P.A. (2005) Why Most Published Research Findings Are False. PLoS Med 2: e124.

John, L.K.; Loewenstein, G.; & Prelec, D. (2012). Measuring the Prevalence of Questionable
Research Practices With Incentives for Truth Telling. Psychological Science, 23 (5), 524–532.

Kerr, N.L. (1998). HARKing: Hypothesising After the Results are Known. Personality and Social Psychology Review, 2 (3), 196-217.

Nosek, B.A.; J.R. Spies; & Motyl, M. (2012). Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science,  7 (6), 615-631.

PsychBrief (2016). Keep Calm and Preregister. [Online]. Available at: http://www.keepcalm-o-matic.co.uk/p/keep-calm-and-preregister/ [Accessed on 23/05/2016]

Slater, M. (2016). Pre-Registration of Analysis of Experiments is Dangerous for Science [Online]. Available at: http://presence-thoughts.blogspot.co.uk/2016/04/pre-registration-of-analysis-of.html [Accessed on 15/05/2016].

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). Falsepositive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.