3117: Replication Crisis

Replication Crisis

|< < Prev #3117 (July 18, 2025) Next > >|

Title text: Maybe encouraging the publication of null results isn't enough--maybe we need a journal devoted to publishing results the study authors find personally annoying.

|< < Prev Random Explanation Next > >|

Explanation

This is one of 57 incomplete explanations: This page was created recently. Don't remove this notice too soon. If you can fix this issue, edit the page!

Transcript

This is one of 30 incomplete transcripts: Don't remove this notice too soon. If you can fix this issue, edit the page!

[Megan, Ponytail, and Cueball are standing at a lectern. Ponytail is talking into the microphone.]

Ponytail: In the early 2010s, researchers found that many major scientific results couldn't be reproduced.

[Ponytail turns slightly to look around the room.]

Ponytail: Over a decade into the replication crisis, we wanted to see if today's studies have become more robust.

Ponytail: Unfortunately, our replication analysis has found exactly the same problems that those 2010s researchers did.

[This panel shows a newspaper, with title "NEWS" surrounded by flourishes. There is a photo of the three researches at the lectern taking up about half the front page. The headline reads: "REPLICATION CRISIS SOLVED".]

  Add comment        Create topic (use sparingly)       Refresh 

Discussion

I believe the current explanation is a bit missing the point. It's supposed to mean that the authors shown in the comic failed to reproduce the result of the papers claiming that there are replication crisis, and therefore the original claim that there is a replication crisis going on is unfounded (since the papers claiming it cannot be replicated), and comically the headline in the last panel takes this to the next level by saying that this means there was no replication crisis to begin with. Justhalf (talk) 00:57, 19 July 2025 (UTC)

I favor the current explanation's interpretation. "Today's studies", I think, refers to 2025 primary research papers across fields of science, and the team finds issues with their reproducibility similar to those found with 2015 primary research papers. I argue that the headline appropriate for "falsifying the replication crisis" would be REPLICATION CRISIS DEBUNKED, not CRISIS SOLVED; the latter tacitly accepts the finding of a replication crisis. I argue further that the demons responsible for the replication crisis are legion, and include the sheer mass and rapid worldwide growth of 'the literature', the 'publish or perish' demands of employers and funders especially given the inadequate money and time granted by funders (before the currently unfolding catastrophe), the rules of (usually volunteer) print-journal editors desperate to save money and space, the collapse under multiple pressures of peer review, the devolution of most actual work to the least paid and least experienced, the disastrous consequences of replacing integrity with propaganda ("don't be such a scientist"), yada. Issues that won't be addressed by publication of null results (oh goody, yet another predatory for-profit journal opportunity!) or annoying results, even if that idea does stimulate a wry chuckle on first reading. Once upon a time, there was a Journal of Irreproducible Results. "So what happened to it?" "That's what they all are now." 2605:59C8:160:DB08:C1B3:77CD:F0E3:3391 02:59, 19 July 2025 (UTC)

I agree with the interpretation of Justhalf since it is simpler and more direct to the punchline than what is written in the explanation. Here's the joke: 1. in the 2010's a study showed there were too many research results that could not be replicated. 2. in 2025 another study looking at current research results, also found that many could not be replicated. 3. This second study confirms the first study thereby replicating their results, 4. the newspaper then announces "replication crisis solved" basing that on the fact that the second paper replicated the first one. Of course the newspaper got it wrong, because simply replicating one study, doesn't solve the problem that all the other studies were never replicated. That's the joke. It's very simple. please don't over think it and make it more complicated Rtanenbaum (talk) 20:06, 19 July 2025 (UTC)

This also makes fun of the tendency of newspapers to sensationalize the result of a single study (in this case, the replication of the earlier study on non-reproducible results). This is why we often see things flip-flopping -- one time they'll say that wine is bad for you, a year later they'll say that it's good for you, and so on. Barmar (talk) 01:47, 20 July 2025 (UTC)

I always prefer a self-referential explanation :-) BTW, we DO have a category Self-reference, does it encompass both in-universe (like here) and ex-universe (the comic itself is self-ref)?

This explanation is very convoluted and (IMHO), mostly wrong (and partially OK). I do not agree with Justhalt's interpretation, but I DO agree with Rtanenbaum's one (who says that they agree with Justhalt's! What!?). Here's my take in slow motion:

1. In 2010s, researchers reported some issues in science.

2. One decade later, the team in the panel wanted to see if these issues have improved over time.

3. The team in the panel replicated the exact same issues as the 2010s teams. So the issues haven't improved at all, but...

4. ... the team in the panel did replicate the 2010s' teams work exactly.

Now: because the "issues" are that a lot of studies couldn't be replicated, an absurd paradox arises: by replicating a study succesfully, the panel team seems to have solved the "replication crisis" (because it looks like new teams like the panel team use robust methods for their science), which is ridiculous because they actually confirmed that the crisis is ongoing. --181.234.89.171 20:43, 21 July 2025 (UTC)
        Add comment