2755: Effect Size

|< < Prev Comic #2755 (March 27, 2023) Next > >|

Effect Size

Title text: Subgroup analysis is ongoing.

Explanation

This explanation may be incomplete or incorrect: Created by a Meta-Meta-Meta Analysis - Please change this comment when editing this page. Do NOT delete this tag too soon. If you can address this issue, please edit the page! Thanks.

This comic outlines a meta-analysis, or more aptly THE meta-analysis, as its inclusion criteria are simply all studies.

A meta-analysis, true to its name, is a statistical analysis of statistical analyses, usually those attempting to answer a single question. Meta-analyses are intended to account for possible individual error within each study, summarizing the general results of all of its studies in order to potentially draw a useful conclusion.

Here, the meta-analysis consists of a graph of effect sizes for what is allegedly every single study ever conducted. Accordingly, even page 53,589 of the meta-analysis is only about a quarter of the total graph (approx. 210,000 pages) Below is an estimate of the average effect (the effect normally being the relationship being analyzed by the studies within a meta-analysis, though here it seems again to just be a conglomerate of all known effects), along with a (likely) 95% confidence interval for the findings of the meta-analysis.

Statistical studies are produced by generating hypotheses then testing those hypotheses. A meta-analysis of all studies would therefore include both studies where the original hypothesis turns out to be false, as well as studies where the original hypothesis is confirmed. Hypotheses that fail to be confirmed by studies are often discarded, however these studies would still be included in THE meta-analysis. The meta-analysis, therefore, could be considered to be confirming our intuition that a lot of hypotheses that scientists come up with are false.

In the caption, Randall delivers to us the bad news: that the meta-analysis of "all of science" has finally been performed, and as it turns out, the results are not significant. Statistical significance is the degree to which the results of a sample or study are likely due to a correlation, as opposed to chance or sampling variation alone. Apparently, across the entirety of human science in the study of our universe, the study has found a lack of significance, or a relationship between any two variables ever.

The joke lies in the absurdity of the claim, that "all of science" can be analyzed at all. Science is not a singular term that can be subcategorized in such a manner, but hundreds of different fields of study, many of which have little or no overlap. Doing a meta-analysis of geology and philosophy, for example, would be patently ridiculous, so the 53,589 page study is comical in its very existence, much less conclusion.

In the title text, Randall reports that subgroup analysis is ongoing. The joke here is that since all scientific studies are subsets of the overall meta-analysis, every scientific endeavor can be separately assessed by correctly constraining the subgroup to include only relevant studies. Hence the subgroup analysis could be considered to include analyses of every individual area or question that scientists have made subject to statistical studies.

Transcript

This transcript is incomplete. Please help editing it! Thanks.

Meta-analysis

Inclusion criteria: All studies

[A forest plot is shown. In the tab on the top right, there is a label "Page 53,589". On the right side of the plot, there is a vertical scrollbar where the bar is less than one quarter from the top. A horizontal axis centered on 0 is shown at the bottom and -1 and +1 on either side are labeled. In the middle of the plot, there is a dashed vertical line. On both sides of the vertical line in separate rows, there are black boxes of different sizes with horizontal bars of varying lengths on the sides of the boxes. Below the plot, slightly to the right of the vertical line, there is a black diamond wider than it is tall, labeled "0.17 (-0.14, 0.52)".]

[Caption below the panel:]

Bad news: They finally did a meta-analysis of all science, and it turns out it's not significant.

 add a comment! ⋅  add a topic (use sparingly)! ⋅  refresh comments!

Discussion

Wow, it looks like I'm first!162.158.146.40 16:40, 27 March 2023 (UTC)

Wasn't something like this actually done?

Robert Sapolsky mentions an obscure paper that actually did something like this. They did a meta-analysis of the average reported error throughout various disciplines in order of the physical size of the objects being studied (e.g., from cells to organs to etc.), and found no correlation between them. The conclusion was that this was evidence that philosophical reductionism was flawed. Fephisto (talk) 22:45, 27 March 2023 (UTC)

Did you manage to find it? 172.70.57.203 08:49, 28 March 2023 (UTC)

Here is the talk. He talks about the paper around 1:26:00. The figure is 1:26:50. Fephisto (talk) 13:18, 29 March 2023 (UTC)

Maybe LINK Titled "Reductionism and Variability in Data: A Meta-Analysis" Sapolsky, R.; Balt S.; Perspectives in Biology and Medicine 39(2), 1996Tier666 (talk) 16:21, 29 March 2023 (UTC)

But does the meta-analysis include itself? Technically, it too is part of Science... Artinum 172.70.91.151 13:06, 28 March 2023 (UTC)

It's SCIENCE all the way Down! Kev (talk) 18:39, 28 March 2023 (UTC)

I know this is facetious, but to answer seriously, the meta-analysis is a break down of specific areas of science, and meta-analyses was not one of the categories that was analyzed. Fephisto (talk) 14:42, 15 August 2023 (UTC)

scroll box location is ~25.5% down track: scroll box is 10px high, scrollbar is 290px high, 54px above box, 226px below = center of scrollbox is 59/231 = 25.541..% = ~209,815 pages of total studies. Adjusted to 210,000 to account for rounding errors. (Plus the scroll box might not even move a pixel for a number of pages).162.158.146.41

Wait, if the scrollbar is 290px high, then shouldn't the position be 59/290 = 20.345%? It looks a lot more like 1/5th down than 1/4th down to my eyes. --Orion205 (talk) 17:16, 29 March 2023 (UTC)

The assumption here is that the scroll bar corresponds to the page numbers. However, that is not normally the case, it's more common to have a scroll bar per page, meaning we are here 20% into page nr 53589... -- Pbb (talk) 16:56, 7 April 2023 (UTC)

Did anyone notice the asterisk next to one of the graph elements? There's got to be a lot of those... Not all scientific studies (I would say very few) can be boiled down to a single numerical output.162.158.146.41

Unless I misunderstand this, there's also an aspect of this that's due to sign - because some studies of some outcomes expect negative results, and some expect positive, mixing even results that are overall statistically significant may cause the effects to cancel out. Mattwigway (talk) 15:32, 28 March 2023 (UTC)

I think that could be squaredTier666 (talk) 17:03, 29 March 2023 (UTC)

meta-analyses are also referenced in 1477: Meta-Analysis 172.71.26.104 16:18, 28 March 2023 (UTC)Bumpf

1477 Is Star Wars? Kev (talk) 18:39, 28 March 2023 (UTC)

sorry, I meant 1447: Meta-Analysis :) 172.71.166.248 13:04, 29 March 2023 (UTC)Bumpf

Would this meta-analysis of all science satisfy Life Goal #28 (assuming it's rejected, as it probably should be)? Barmar (talk) 15:29, 29 March 2023 (UTC)

172.70.251.39 07:01, 30 March 2023 (UTC) SCIENCE IS HIGHLY SIGNIFICANT

If we (i) postulate that the picture of page 53,589 of the meta-analysis of all science is a representative sample, and if we (ii) postulate that the model of the meta-analysis is just simple random sampling, without stratification (and I think that is a reasonable guess, since if you really have data of ALL science or want to make an assumption about ALL science based on a sample, then Simple random sampling is okay since weighting of different scientific disciplines is proportional to the number of studies in your sample, SRS guarantees getting an unbiased estimate ...), and if we (iii) postulate that the study-specific variance is independent from the single-study means, we can approximately calculate the correct confidence interval.

Let's do it: The authors say that the weighted least square estimator of the population mean is 0.17. The picture shows 11 studies. I eye-balled the effects being (-0.125; 0.5; 0.375; 0.75; -0.375; 3.75; 0.125; 1.25; 0; 0.55; -0,2) and calculated the "between study standard deviation" (using Excel ) being 1.146 and the mean of that sub-sample being 0.6. (Remark: We can ignore the within study variation, since the dominating source of variation is "between studies" and the within error is enclosed in "between study stddev" due to error propagation). Of course, data analysis can be done with a mixed model with clustered data, but doing an analysis with the study means will give a very good approximation.

Now, first step is to calculate the confidence interval of the mean effect size based on the studies we see. We have 11 studies, 10 degrees of freedom. Assuming a t-distribution the (unweighted) 95% confidence interval of the studies in the picture is 0.6 +/- [2.228*1.146/sqrt(11)] = 0.6 +/- 0.77 = [-0.17, 1.37]

The C.I. includes zero but also includes the full meta study mean of 0.17. So, we have no evidence against our hypothesis that page 53,589 which we see on the website is representative for the full meta analysis. So, we can go on

The 95% confidence interval for ALL studies assuming a number of around 250,000 studies would be 0.17 +/- [1.96*1.146/sqrt(250000)] = 0.17 +/- 0.00572 = [0.16428, 0.17572].

The 99.9% confidence interval for ALL studies assuming a number of around 250,000 studies would be 0.17 +/- [3.3*1.146/sqrt(250000)] = 0.17 +/- 0.00756 = [0.16244, 0.17756].

meaning, on average SCIENCE IS HIGHLY SIGNIFICANT (p<0.001)

162.158.86.191 10:31, 30 March 2023 (UTC) I re-viewed the graph and read the comments on the web page. They say the underlying number of papers is 2,3 million. My fault was that I havent multiplied the number of pages with number of studies per page. So, the confidence interval will become even more narrow

The 95% confidence interval for ALL studies assuming a number of around 2,100,000 studies would be 0.17 +/- [1.96*1.146/sqrt(2100000)] = 0.17 +/- 0.00155 = [0.16845, 0.17155].

The 99.9% confidence interval for ALL studies assuming a number of around 250,000 studies would be 0.17 +/- [3.3*1.146/sqrt(2100000)] = 0.17 +/- 0.00261 = [0.16739, 0.17261].

162.158.86.191 10:31, 30 March 2023 (UTC) Interesting is, that the population mean is 0.17 and not 0.000. When averaging the effects of so many studies, all different in topic and investigated treatments and strata, one would expect that the global mean of all effects is zero. But it is 0.17. Clear indication of publication bias. There is higher probability for a positive effect to be published in a paper.