My blog feed for psychological methods blogs was the subject of a recent paper [zotpressInText item=”{5421944:VIVBL9IA}”][note]Open access version available here and code and data available here.[/note]. The paper was a quantitative analysis of the content of these blogs as of April 11th 2017. The primary purpose was to look at: who runs these blogs, what topics do they cover, who do they write about, and for what purpose? This paper came about because of strong comments made by Susan Fiske (the senior author) regarding highly critical writings on some of these blogs [zotpressInText item=”{5421944:3TDMI7V9}”]. Given how much time has been spent debating this piece, I don’t want to retread old ground. Instead, I will briefly look at the paper itself and provide some context about the researchers.

Who, what, where, why?

Most of the authors included in the feed at the time of data collection were male (38 out of 52 authors, with 2 anonymous), mid to late career (36 out of 54), and white [zotpressInText item=”{5421944:G5PDAVBV}”]. In addition, most of the blogs posted about once a month. There was, however, one notable exception: Andrew Gelman’s blog, which averaged 11.1 posts a week. The most common topics discussed across all blogs were: ‘statistics’, ‘replication’, and ‘science communication’ in general.

The question of greatest interest to some readers, however, centred around whether women were more likely to receive harsh criticism from these blogs. After searching through all the blogs and their comment sections for relevant names[note]Names of salient researchers were put forward by self-selected scientists.[/note], they did not find evidence for this conclusion. Daryl Bem, author of the notorious paper purporting to show evidence of extrasensory perception, was the most talked about researcher. 2nd was Diederik Stapel, famed faker of data. Brian Wansink, who has had a string of high profile retractions after data sleuthing revealed very suspicious data, came in 3rd. The first female researcher to appear on the list was Amy Cuddy, lead researcher behind the power posing phenomenon, who came in fifth.

Cuddy was at the epicentre of a heated debate about how people criticised research. Some argued that made unfair judgements about her character and technical ability. Others reasoned that her sub-par methodology was being rightly pointed out. Regardless of where you stood on the debate, this was a widely discussed issue. Given this, and the impact of the availability heuristic [zotpressInText item=”{5421944:T6TWS3HK}”], many thought she would have been mentioned the most by bloggers. This formed part of the broader debate around overly critical comments directed at women. The purpose of this paper, therefore, was to shed light on this issue. Given the results, the authors concluded there wasn’t evidence for a greater representation of women in critical blog posts[note]As the lead author points out, more analyses should be conducted to see whether other metrics, like publication rates, affects the rate of discussion in blogs (G. Nicolas, personal communication, 2019).[/note]

Was it worth it?

Simply put, I think this was a worthwhile question to ask. This is especially true as I believe blogs have an important role to play in improving our collective understanding. The paper is also an excellent example of open science: it is open access, the code and data are available, and it didn’t use p-values as it was exploratory. The authors are also very clear about the limitations of the study. For example, the methods used couldn’t give much information about the severity of the criticism. Some sentiment analysis is provided in the supplementary material but since this cannot interpret sarcasm or other subtleties of language it is regarded as preliminary. This frankness is commendable as there is a well known pressure to aggrandise findings [zotpressInText item=”{5421944:34W4X7PH}”]. Given the limitations, this paper is not incontrovertible proof. But it helps provide a more complete picture of how bloggers write about researchers.

My discussion with Susan Fiske

As an addendum, I think some may be interested in the conversation I had with Susan Fiske after she gave her presentation on the topic in February of 2018[note]See here for my write-up of the whole event and here for the video of her presentation.[/note]. Thus, I feel I can give some more context and their views on the topic. Given her comments, many would presume her to be anti-blog and anti-open science.

This, however, is not the case. She is in favour of methodology blogs and is pro open science. Just because she is critical of how some people behave doesn’t mean she is against the whole enterprise. Regarding this issue, she and I both agreed that the bad actors are few and far between. She also argued, and I agree wholeheartedly, that asking these types of questions ensures potential bias or abuse doesn’t go unnoticed. Only by asking ourselves questions and looking for answers do we gain a greater understanding of how we interact. In relation to the results, she had a prior that women were more likely to receive harsh sentiments from blog posts[note]Lead author Nicolas had a weak prior in the opposite direction.[/note]. However, the data did not show this.

The reason why I present this information is I think it’s valuable to get a better understanding of her views, rather than using only 1 post to judge her. Going by overly simplistic tribal divides, we should be on opposing sides of the debate about blogs and open science. But we agreed far more than we disagreed. By getting to know her beliefs better, I found out I had more in common with someone supposedly on the opposite side of the debate than I thought.

References

[zotpressInTextBib style=”apa” sort=”ASC”]