How to evaluate a review study
From
Follow the steps below to evaluate a review study.
Contents |
Step 1: Examine the component studies
The ostensible purpose of a review study is to make some sort of sense out of the body of studies that preceded it. You may need to get copies of the component studies to do this. A review study may contain summaries of its component studies, but those summaries may be incomplete and/or wrong. A [review study] author may accept at face value the conclusions drawn by the authors of the component studies. But if these conclusions are wrong or unwarranted, then the review study author’s analysis will be similarly afflicted. Each component study should be examined as you would individual clinical trial, cohort and case-control studies.
Step 2: Have any studies been excluded?
Don’t assume that a review study contains the entire, relevant body of scientific literature. Not only may a biased review study author exclude studies that don’t fit his predetermined conclusion, but keep in mind the phenomenon of publication bias, which tends to favor the publication of “positive studies” over “negative” studies. While not much can be done about publication bias, the former problem can be checked with the data base. Sometimes authors will inadvertently hint that studies have been excluded by, for example, mentioning cut-off dates for study publication, English-language only studies or other exclusion criteria. If you locate excluded studies, what results do they report? If the results of excluded studies are of a contradictory or debunk-atory nature, then you may have just learned why they were excluded. Any review study that excludes contradictory studies should be viewed with prejudice.
Step 3: Not all studies are created equal
Once you have a good sense of the component studies (and what studies, if any, have been excluded), next consider how the review study author has evaluated the individual studies in his own analysis. Keep in mind that epidemiology studies are inherently different from one another. Cohort studies have a better design than case-control studies. Every population is different, demographically and size-wise. Data may be collected, measured or estimated in different ways that impact its reliability. Accordingly, some studies are better-conducted and, hence, more credible, than others. Consider also how the review study authors interprets/handles contradictory studies. Do positive results inexplicably carry more weight with the review author than negative results? Does the author seem to go out of his way to rationalize/apologize for shortcomings in studies with positive results while treating negative results dismissively? Affirmative responses to these questions may indicate author bias.
Step 4: Is there one good negative study?
It only takes one exception to disprove a rule. Similarly, one solidly-conducted study with negative results can invalidate a raft of dubiously-conducted studies touting positive results. Has the review study acknowledged this fact?
Step 5: Independent replication
A hallmark of science is that studies should be capable of being replicated independently by other scientists. Moreover, such independent replication should be attempted and the results published. Beware of a body or line of scientific literature in a review study that has largely been produced by one researcher or a small group of researchers. Lack of independent replication should be a red flag.
Step 6: Independent review
Neither a researcher nor his colleagues should review the researcher’s own work. As obvious as that point may be, researchers and reviewers violate it all the time.
Return to the step-by-step guide for debunking.
Return to the DEBUNKosaurus™ Main Page.
