Editing 2755: Effect Size

A meta-analysis, true to its name, is a statistical analysis of statistical analyses, usually those attempting to answer a single question. Meta-analyses are intended to account for possible individual error within each study, summarizing the general results of all of its studies in order to potentially draw a useful conclusion. For a meta-analysis to be possible, there must be some measured variable in common across the included studies.

Here, the meta-analysis consists of a graph of {{w|effect sizes}} for what is allegedly every single study ever conducted. Accordingly, even page 53,589 of the meta-analysis is only about 1/4 of the total graph, as the scroll bar on the right is only about 1/4 of the way down; this makes the total included in the meta-analysis approximately 210,000 pages, or around 2.3 million studies. Below the graph is an estimate of the "average effect" across all of these variables, the effect normally being the relationship being analyzed by the studies within a meta-analysis, though here it seems again to be just a conglomerate of all known effects, along with a (likely) 95% {{w|confidence interval}} for the findings of the meta-analysis. It's absurd to analyze all studies this way, as the variables that all of those studies measure are wildly different and it makes no sense whatsoever to average (or otherwise analyze) them together. In addition, 2.3 million scientific studies is much too small a number; a [https://www.stm-assoc.org/about-stm/ recent estimate] is that about 3 million papers are published ''each year'', and while not all of them would have a numerical hypothesis test, many others would have several such tests.

Statistical studies are produced by generating hypotheses and then testing those hypotheses. A meta-analysis of all studies would therefore include both studies where the original hypothesis turns out to be false, as well as studies where the original hypothesis is confirmed. Hypotheses that fail to be confirmed by studies are often discarded; however, these studies would still be included in this meta-analysis.

In the caption, Randall delivers the bad news: that the meta-analysis of "all of science" has finally been performed, and as it turns out, the results are not significant. {{w|Statistical significance}} is the degree  to which the results of a sample or study are likely due to a correlation, as opposed to chance or {{w|sampling variation}} alone. Apparently, across the entirety of human science in the study of our universe, the study has found a lack of significance, or of a relationship between all the variables measured by all the studies ever.

The joke lies in the absurdity of the claim that "all of science" can be analyzed at all. Science is not a singular term that can be subcategorized in such a manner, but is rather hundreds of different fields of study, many of which have little or no overlap. Doing a meta-analysis of geology and philosophy, for example, would be patently ridiculous, so the 53,589 (or 210,000) page study is comical in its very existence, much less conclusion. In addition the comic conflates two meanings of "significant": the statistical meaning, and the more everyday meaning of importance or noteworthiness.

 

In the title text, Randall reports that {{w|subgroup analysis}} is ongoing. The joke here is that since all scientific studies are subsets of the overall meta-analysis, every field of scientific endeavor can be separately assessed by constraining the subgroup to include only studies in that field. Hence the subgroup analysis could be considered to include analyses of every individual area or question that scientists have made subject to statistical studies. Again, analyzing any subgroup would lump together studies that measured very different things and hence would still be meaningless.

 

XKCD has previously considered the topic of subgroup analyses around the important issue of [[882|jelly beans]].  Subgroup analyses may be used as {{w|data dredging}} or [[1478|p-hacking]] in order to identify anything that is "significant" and thus publishable.

:[A forest plot is shown. In the tab on the top right, there is a label "Page 53,589". On the right side of the plot, there is a vertical scrollbar where the scroll box is about one quarter from the top. A horizontal axis centered on 0 is shown at the bottom and -1 and +1 on either side are labeled. In the middle of the plot, there is a dashed vertical line. On both sides of the vertical line in separate rows, there are black boxes of different sizes with horizontal bars of varying lengths on the sides of the boxes. Below the plot, slightly to the right of the vertical line, there is a black diamond wider than it is tall, labeled "0.17 (-0.14, 0.52)".]