On social media, there are often discussions about why psychologists leave academia. Some argue that the new culture of criticism (where overly harsh[note]In their view[/note] criticism is leveled against those who make errors and accusations of malfeasance are rife) make academics, especially younger ones, change profession. They give examples of high profile cases where researchers have made errors or merely used previously accepted standards of practice and the obloquy they’ve received when their results don’t hold up. Others provide counter examples of former colleagues who grew frustrated at their inability to replicate supposedly rock-solid findings or who had a crisis in confidence about the validity of vast swathes of the literature. But one thing is lacking from this discussion.

Data.

Are people more likely to leave academia because they feel they’ve been bullied out by overly zealous critics? Or are they forced out by their despair over their weakening belief in the truth of their field? The tennis match of anecdotes between the two camps hasn’t answered many questions. So I sent out a questionnaire asking for demographic information and the reasons for leaving/considering leaving academia from those who had chosen to leave or were debating whether to do so. I shared it on social media and various forums dedicated to people who had left academia, specifying it was for psychologists. I also encouraged those who saw it to share it with colleagues or friends who had left. 559 started the questionnaire, 451 gave their demographic details, and 403 of those 451 answered question 9 (the key question where participants rated the relative importance of reasons for leaving academia). 2 participants were excluded from the entire data set because they gave joke answers[note]Their results are still in the “Full non-transformed data.xlsx” file but the means and percentages reported there will be slightly different from the ones in this blog post as this is using data with their answers excluded.[/note].The full data set can be found on the OSF, along with pictures of the questionnaire, and a full account of the analyses I performed (in the ReadME file[note]I’m sure the code is ugly but it works[/note]). I strongly recommend you download and read at least the ReadME file as you will be able to see the precise results of the tests I carry out[note]You don’t need to be able to read the programming language R in order to see the results[/note]. You can look at the survey online as it is still open, though results are not being collected (you will have to answer the questions to progress through the survey). The data is completely open so feel free to play around with it if you’d like. If you notice any errors please let me know and I will correct them.

Demographic information

I’ve summarised the data from the study below. For the more interesting questions, I’ve included all the data (often in a table for readability) and for the less interesting ones I’ve provided some of the more common answers. The table below shows the ages of the participants.

For where the respondents lived, unsurprisingly, the vast majority were from either the United Kingdom (36.74%) or the United States (32.07%). 258 (57.46%) were female and 181 (40.31%) were male, with 11 (2.45%) giving another answer. Most respondents were in the field of psychological science (43.43%), with the next most common occupation being working in industry (10.24%). Almost two thirds were currently in academia (61.47% said “Yes” versus 27.85% who said “No”, with 8.02% working as practicing psychologists). Almost half of the sample were PhD or Doctorate students (49.67%) at the time of completing the questionnaire or when they left academia. 17.37% were postdoctoral researchers and 13.59% were lecturers. As to whether they had left or considered leaving, the table below summarises the results.

Ratings for how relevant the different reasons for leaving academia are

Below is a table for question 9 “What were your reasons for leaving/considering leaving academia? Please rate how relevant they were/are”.

The lack of rewards/incentives, the large amount of effort needed for funding, and a desire for a better work-life balance had the highest weighted means of the 13 questions. Rob Chavez, John Sakaluk, and Richard Morey have made excellent graphs summarising the table above. Rob has transformed the frequency for each rating into a percentage of the total respondents for each question and turned it into a bar graph. A large-scale version can be found here. John has created pirate plots to show the individual data points (the blue circles), the relative number of people who gave different scores (the orange bean), and the mean (the black dots) with 95% confidence intervals (the lines above and below). He has a detailed blog post showing how to make these kind of plots here.

Morey created a frequency plot for the frequency of the different responses for each sub-question of question 9 and calculated the correlation between each sub-question. You can view the images here.

The code for all of the graphs are freely available at the same OSF page as the rest of the code.

Inferential statistics[note]I conducted a lot more tests than are reported here, these are a summary of the most relevant ones[/note]

One of the first areas I wanted to explore was whether there were sex differences in reasons for leaving academia. I used a chi-square test between the variable sex and the four variables related to the key area of interest: uncertainty about how much of the field is true, culture of criticism, being harshly criticised, and inability to replicate. The null hypothesis is that the participants sex is independent to the ratings of relevance they gave. There was a significant difference between the sexes in their rating of how relevant being harshly criticised was to them leaving and for being unable to replicate others findings. Women said being harshly criticised was a more relevant reason for them leaving than men did (p=0.028[note]After correcting for multiple comparisons[/note]), whereas men said failing to replicate others work was a more relevant reason for them than women (p=0.031[note]After correcting for multiple comparisons[/note]). There was no significant difference between the sexes for how much of the field is true and there being a criticism of culture.

To determine whether participants rated the variables of interest as equally relevant for them leaving, I analysed the variables (after ranking them) using a Mann-Whitney-Wilcoxon Test. The null hypothesis is that the participants ranked the reasons as equally relevant. There was a significant difference between there being a culture of criticism and failing to replicate others findings, with culture of criticism ranked as more relevant (p=0.00099[note]After correcting for multiple comparisons[/note]). There was no significant difference between doubts about the truth of the field and culture of criticism (p=0.42[note]After correcting for multiple comparisons[/note]). There was a significant difference between failing to replicate and being harshly criticised, but in the opposite direction to the earlier test: failing to replicate was ranked as a more relevant reason for leaving academia than being harshly criticised (p<8.8e-16[note]After correcting for multiple comparisons[/note]). Again there was no significant difference for doubts about the truth of the field, this time tested against being criticised (p=0.2113). This suggests that a culture of criticism was a greater reason for leaving academia than people failing to replicate other people’s results, which in turn was ranked as more relevant than being harshly criticised.

When comparing the variables of interest against the more traditional reasons for leaving academia (e.g. a desire for a better work-life balance, the inordinate amount of effort required to gain funding, or there being a lack of incentives), there were no significant differences after controlling for multiple testing. Being harshly criticised and the lack of incentives was almost significant (p=0.0567) so it may be worth exploring this question again in the future. But we cannot reject the null hypothesis that the reasons are ranked equally as relevant for the rest.

Conclusion

Looking over the results, we can see there is some evidence that men and women give different rankings for how relevant being harshly criticised and failing to replicate others work is. However, this would need to be replicated due to the weak nature of the evidence (Wagenmakers & Gronau, 2017). There is evidence that the culture of criticism that is believed to be taking over psychology is ranked as a more relevant reason than failing to replicate findings by other researchers. But the results also suggest being unable to replicate a finding is more relevant than being personally criticised. How this links into the questions that inspired this survey is unclear. Perhaps researchers are put off by what they interpret as a hostile environment but are more likely to leave the field because of a failure to replicate than because they themselves have been criticised. This survey by no means definitively answers the question why psychologists leave academia. Hopefully it can be a starting point for a more rigorous, preregistered study which doesn’t miss out a key reason (lack of jobs). It would also be interesting to see if there were any differences between participants at a difference stage in their career.

Note

[Special thanks to Dana Linnell Wanzer, Farid Anvari, Daniel E Bradford, Iva Čukić, and Julia Rohrer for providing highly informative feedback on the survey prior to its release, and to Jim Tyson for his invaluable help with analysing the data. Also thanks to Richard Morey for answering a statistical question.]

One of the forums I posted the questionnaire on had an enlightening discussion about reasons for leaving academia. You can read it here if you’re interested.

References

Wagenmakers, E-J. & Gronau, Q. Redefine Statistical Significance II: Caught in a Bad Romance? Available at: https://www.bayesianspectacles.org/redefine-statistical-significance-part-ii-caught-in-a-bad-romance/ [accessed on: 01/11/2017]