Editing 2726: Methodology Trial

{{comic
| number    = 2726
| date      = January 18, 2023
| title     = Methodology Trial
| image     = methodology_trial_2x.png
| imagesize = 339x459px
| noexpand  = true
| titletext = If you think THAT'S unethical, you should see the stuff we approved via our Placebo IRB.
}}

==Explanation==

When testing the efficacy of a potential medical treatment, researchers compare subjects who received the treatment against subjects who received a {{w|placebo}}. Usually each subject does not know whether they received the treatment or placebo, and neither do the practitioners, until the end of the trial. This distinguishes the actual effects of the treatment from the effects of simply participating in a study. People who receive a placebo (or an ineffective treatment) often believe their treatment is working due to such causes as paying more attention to one's health or expecting to feel better. This misattribution of effect to a non-treatment is called the "placebo effect".

In this comic a team of researchers appears to have studied some medical treatment, using a placebo controlled test. They present their findings in which a particular subset of participants (out of at least four distinct groups) shows an apparently significant result. The graph shows that three groupings have results whose error-bars indicate that they might easily have zero (or neutral) true effects, if not negative ones. But, even at the lowest extent of the accepted uncertainty, the fourth stands out as definitively having some degree of positive effect (of whatever kind this particular graph is plotting). 

However, it is revealed that the 'treatment' they were given was also a placebo. Their own study was the subject of a placebo controlled test conducted on their methodology. They were the placebo group, while a different team presumably used the exact same methodology to study the real treatment. Thus, all of this team's findings were due to the placebo effect, or else the trial size and scope allowed a purely statistical 'blip' to occur, instead of there being any real merit to the "treatment". This indicates that their methodology shouldn't be used for any real world applications. This may be a subtle dig at the recent {{w|aducanumab}} Alzheimer's drug trial controversy, where post-hoc reanalysis of one subgroup of patients revealed a surprising result when the overall trial had otherwise failed.

The particular flaw in the methodology appears to be dividing too few subjects into too many sub-groups, allowing a chance cluster of anomalous results to overly influence an apparent result. The researcher did find significance in one sub-group, even though in reality there was no signal, just noise, since it was all placebo groups. This references the same p-hacking problem as [[882: Significant]]. Only in this case the researcher themself is the subject of the real trial.

If the non-placebo study had the exact same size and design (as it should have, in such a meta-study), it would cast doubt upon whether any similar-looking findings in London were significant. Especially if they also found that the same subgroup were again exhibiting the sole significant effect, which might reveal an inbuilt flaw in the procedure. On the other hand, it could just further show how likely any particular grouping was to falsely show a result; if all groups had apparently benefited, the chances are that most of them were correct, whether or not [[2268: Further Research is Needed]].

Treatments ''can'' be more effective on specific subgroups of the population; for example, an anti-cancer drug might only work against specific mutations that cause cancer. But any such result needs to have appropriate statistical significance and new subjects from that subgroup should be tested to ensure the result is repeated.

The title text points out how the experiment has almost certainly violated some set of ethical standards, because one researcher offers what he believes to be genuine treatment to a large number of participants only for a third party (the offscreen speaker) to replace all his medicine with placebos, ultimately deceiving the patients. The title text implies that it was approved by a genuine Institutional Review Board (IRB), the group which decides whether a proposed experiment is ethical to perform. However they also have a "placebo IRB", presumably made up of people who have no qualifications to make such judgements well, or perhaps not made up of people at all, but simply a mechanism for generating random decisions.

However, such a methodology trial using all placebos wouldn't necessarily be unethical.  In addition to using a placebo, most studies are "double blind" meaning neither the patients nor the doctors/nurses treating them know who is getting the placebo and who is not; only the researchers conducting the study know.  This is so doctors/nurses cannot inadvertently let the patients know who is getting real medicine (by acting with remorse around patients they know are not being treated, or being more cheerful with patients they know who are).  It is considered perfectly ethical for doctors to give patients what they believe is medicine but is not (the placebo).  This is because without the double blind procedure it may not be possible to identify real medicines from ones that have no effect, and the impact of preventing real medicine from being used by millions is greater than the deceit towards the small number receiving a placebo in the experiment.  By extension it could be ethical to have the researcher conduct a trial with two placebos without knowing it.  For instance if the London team and the team in the comic were finding beneficial effects in new drugs that other researchers found had no effect (or finding other drugs didn't work when others had evidence they did) then it may be worth investigating if their shared methodology has the flaw demonstrated in the article.  That way regulatory agencies could exclude their flawed data when they make decisions on what drugs to approve, while the two teams could shift to a better methodology and return to contributing to medical science.

==Transcript==

:[Cueball stands in front of a poster holding a pointer. The poster shows a scatter plot with four points and error bars, with one data point labeled "Subgroup" is marked with an asterisk and is placed somewhat higher up than the other three points.]
:Cueball: However, we see clear evidence that the treatment is more effective than the placebo for some subgroups.
:Off-panel voice: Thank you.
:Off-panel voice: However, we can now reveal that the '''''London''''' team was studying the real treatment. Both groups in your study got a placebo.
:Cueball: Aw, '''''maaan...'''''
:[Caption below panel:]
:Researchers hate it when you do placebo controlled trials of their methodology.

:[Cueball is presenting some data to people off-panel. He has a pointer raised to indicate a graph showing four subgroups, the rightmost of which is visibly raised in comparison to the others, its lower interval rising above the x-axis.]
:Cueball: However, we see clear evidence that the treatment is more effective than the placebo for some subgroups.
:Voice from off-panel: Thank you
:Voice from off-panel: However, we can now reveal that the ''London'' team was studying the real treatment. Both groups in your study got a placebo.
:Cueball: Aw, ''maaan...''

:[Caption below the panel]:
Researchers hate it when you do placebo controlled trials of their methodology.

{{comic discussion}}
[[Category:Comics featuring Cueball]]
[[Category:Scientific research]]