How to evaluate an ecologic study
From
Follow the steps below to evaluate a ecologic study.
Step 1: What is the relative risk result?
The first step in evaluating a ecologic study is considering the magnitude of the relative risk (RR) statistic, the key result of the study. Remember that a relative risk statistic has nothing to do with risk (e.g., the likelihood that exposure will cause disease. The magnitude of the relative risk statistic is merely indication of the strength of the strength of association.
Where can you find out what the relative risk is? The RRs calculated by the researchers will be located in the results section of the study abstract and, usually, in a table in the full study.
If you are reading or hearing about the study from the news media, they will likely report the RR as follows: The exposure increased the risk of disease by X%, where X is some number greater than 0. Although this is an utterly incorrect way to talk about RR (see above paragraph), it is unfortunately common usage by the media and junk scientists. To determine what the study's RR is from such a report, simply divide X by 100 and then add the result to 1.0. So, for example, if there was 75% more disease among those exposed, then the RR would be 1.0 plus 0.75 = 1.75.
Step 2: Evaluating the relative risk: Magnitude
Since, ecologic studies typically test whether the occurrence of disease is associated with exposure, news touting a positive ecologic study will generally feature an RR greater than 1.0 (or, in incorrect shorthand, an increase in risk greater than 0%).
But just because an RR is greater than 1.0, that does not necessarily mean that the disease is actually associated with exposure. We need to evaluate some characteristics of the RR. The first characteristic is its magnitude or size, also referred to as the strength of association.
As a rule of thumb, ecologic studies with RRs less than 2.00 (i.e., a 100% increase in risk in junk science-ese) should be viewed with extreme suspicion. The rationale for this judgment is that the low RR is a weak statistical association.
Based on strength of association guidelines, more confidence is to be had in RRs that are 2.00 or greater. The greater the RR is, then the more confidence that may be had in the existence of a statistical association between exposure and disease.
Step 3: Evaluating the relative risk: Statistical significance (Part 1 of 2)
It is not enough that a ecologic study RR is 2.0 or greater, i.e., that the disease is highly correlated with exposure. The RR must also be statistically significant. That is, we want to have some confidence that the RR result is not a fluke or happened simply by chance.
There are two standard tests of statistical significance that should be reported for each RR. You will most likely have to obtain a copy of the actual study to verify statistical significance.
The first test of statistical significance concerns the p-value of the RR. The p-value will usually be found in the same table in the study as the RR. The p-value indicates how much confidence can be had that the RR is different from the no-effect level of 1.0.
By convention:
- If the p-value is 0.05 or less, then the RR is statistically significant. This does not necessarily mean that the RR is not junk science, it just means that the RR has passed the p-value test. Go to #Step 4: Evaluating the relative risk: Statistical significance (Part 2 of 2)
- If the p-value is greater than 0.05, then the RR is not statistically significant at the 95% confidence level. If the RR is not statistically significant, then it is viewed as a meaningless result and as having been debunked.
If a study does not present the p-values of its RRs, interpret this to mean that the researchers were too embarrassed to publish them because they would expose the RRs as not statistically significant, i.e., meaningless.
Step 4: Evaluating the relative risk: Statistical significance (Part 2 of 2)
The second test of statistical significance for an RR involves its confidence interval.
The confidence interval for an RR is typically found in the same place as the RR and its p-value. It is often indicated with the abbreviation C.I. or CI. The confidence interval is a range of values, typically indicated with parentheses) within which the true RR lies, according to a certain degree of confidence. The standard degree of confidence is 95%, designated as 95% C.I. For example, if the RR=2.50, and its confidence interval ranges from 2.0 to 3.0, then this may be designated as RR=2.50 (95%CI 2.0, 3.0).
The two numbers in the confidence interval are referred to as its lower bound and upper bound. In the above example, 2.0 is the lower bound and 3.0 is the upper bound.
In order for the RR of a ecologic study to pass the second test of statistical significance, the lower bound of the confidence interval must be greater than 1.0. The reason for this is that if the confidence interval includes 1.0, then we cannot be sure with 95% confidence that the true value of the RR is greater than 1.0. In other words we cannot safely exclude the possibility that the true RR isn't 1.0, the no-effect level. For example, if you see a RR and CI presented as RR=2.60 (95% CI 0.95, 5.25), the RR is not statistically significant because its lower bound (0.95) is less than 1.0 and includes the no-effect level as well as a range (from 0.095+ to 1.00) indicating that the exposure may not be statistically associated with the occurrence of disease.
So in the case of a ecologic study, if the lower bound of the confidence interval touches or crosses the 1.0 no effect level, then the RR is not statistically significant, i.e., it is meaningless and debunked.
Confidence intervals are be subject to game-playing by junk scientists. The width of the confidence interval may be adjusted by altering the confidence level. For example, a 90% confidence interval is narrower than a 95% confidence interval, while a 99% confidence interval is wider than a 95% confidence interval. This makes sense if you think about it. You will have less confidence in a narrower range and more confidence in a wide range. So here's the game.
Keeping in mind that the 1.0 or the no-effect level is the third rail of an epidemiology study and means instant death for an RR which confidence interval touches or crosses it, there is incentive to narrow the confidence interval so that the upper bound stays below 1.0. This may be accomplished by choosing to use a lower confidence level than the standard 95% level. This lower confidence level is typically 90%. So if you see a 90% confidence interval, this is a red flag that the researcher is trying to conceal the fact that his CI touches or crosses the no-effect level.
Step 5: Confidence interval width
Another tell-tale sign of a dubious relative risk is a wide confidence interval. The CI indicates the range of values within which the true RR likely falls. A “too-wide” confidence interval indicates that the underlying data are unusually disparate, including too many outliers and oddball data points. A “tight” confidence interval indicates more uniformity and less variance among the data.
How wide is too wide? As a rule of thumb, if a CI is wider than the magnitude of the RR, then you’re looking at goofy data. For example, assume you see a RR=20 (95% CI 5, 167). The width of the confidence interval (162) is many times the size of the size of the RR. A nice tight confidence interval would be, for example, something like RR=20 (95% CI 18, 22), where the width of the CI is only a fraction of the size of the RR.
Consider the width of the confidence interval as an indication of the “margin of error” concept used in polling. When the polling results between, say, two candidates are within the margin of error, the candidates are considered to be in a statistical dead heat. If the CI is wider than the RR is large then the RR is statistically dead.
Step 6: Non-empirical considerations
If a ecologic study is debunkable, then there's a good chance that you will have already debunked it by following the procedure described in Steps 1 to 4. If that empirical process didn't do it and there is something that still seems fishy about the study, here are a few qualitative issues to explore with a ecologic study:
- Ecologic fallacy. Regardless of the above analysis, the ecologic fallacy is the Acheilles heel of an ecologic study. Since data on either individual study subject exposures or health characteristics will not have been collected, researchers will have substituted average man or presumed exposure characteristics to study subjects.
- Biological plausibility. This is an essential element of a cause-and-effect relationship. Researchers should be able to provide some sort a physiological explanation for the results, i.e., why their treatment worked/didn't work. But be aware of the limitations of biological plausibility. Marginal statistical results cannot be buffed into science even by the best physiological explanation, unless that explanation can also explain the marginal results.
- Dose-response relationship. Where study subjects are exposed to different levels of a potential substance or condition, greater exposures ought to be associated with greater RRs — and the RRs should be statistically significant to boot. Some researchers will try to establish a dose-response relationship through trend analysis. You may see researchers claims a statistically significant trend (i.e., a p-value for trend of 0.05 or less), although one or more of the RRs making up that trend are nonsignificant. Remember that nonsignificant RRs are essentially non-results and so cannot be claimed be data points in a dose-response relationship. A corollary to the greater exposure/greater RR rule is that similar exposures should produce similar results. While this may seem like an obvious point, consider the following example. Let’s say a study is examining whether consumption of processed meats is associated with a particular health effect. The results for hot dogs should be similar to the results reported for bologna, since hot dogs are just rolled up bologna. Consideration of the dose-response relationship is an extension of the biological plausibility analysis.
- Other studies. A hallmark of science is replication of results. Similar experiments should produce similar results. You should check to see if there are other epidemiologic studies similar in purpose, scope and (ideally) design to the one of interest. Compare the results. Are they consistent? If not, this should be a red flag — something is problematic, either the study of interest or the other study or studies. If it turns out to be a case of one study against many studies, don't necessarily assume that the one study is necessarily wrong. It could very well be that the many studies are wrong for a variety of reasons — say, for example, they've all associated with a single researcher or a single organization with reasons to cook the books.
- Publication bias. It may be difficult to locate and compare prior ecologic studies because of the phenomenon of publication bias — i.e., the tendency of journal to publish positive results and to ignore the rest. Past the tendency of the journals, researchers who don't like their negative results may not submit them for publication.
- Fraud. Scientists are people, people can commit fraud, so scientists can commit fraud. It's sad, but true — fraud can never be discounted as an explanation for study results.
Return top the Step-by-step guide for debunking.
Return to the DEBUNKOsaurustrade; Main Page.
