Why Medical Science Is Never An Exact . . . Science
In 2002, the National Institutes of Health stopped a Women’s Health Initiative (WHI) study because it found that post-menopausal women taking hormones were at higher risk of a suffering from a heart attack. In 2004, another WHI study was halted because it indicated women taking estrogen had an increased risk of suffering from a stroke.
Today, the Wall Street Journal (subscription required) and other publications reported the results of a JAMA study that represents a giant flip-flop by government researchers. New data indicates that hormone replacement therapy is not as dangerous to the heart as earlier research indicated. What?!
Yes, now the Journal is reporting that “women who use hormones in the first five years after the arrival of menopause are not at increased heart risk.” Here’s the catch. While the study meets the normal test of statistical significance (p = .05), it does not satisfy a newer, more conservative standard set after the data was reviewed by JAMA.
According to the Journal: “At issue is something called the P value, an index that measures how strong the evidence really is. Traditionally a P value of 0.05 is acceptable, showing that there is only a 1-in-20 chance that the result isn’t real. That is the standard used in most WHI analyses. The analysis easily met this test of significance, coming in with a P value of 0.02. But once the paper started being reviewed, a decision was made to further lower the standard for statistical significance — to 0.01. ‘Both the internal reviews and the journal reviews drew attention to the need to be cautious and recommended this,’ Dr. Rossouw said. A more conservative approach was needed, he says, because the data had already been subjected to a number of statistical analyses, a process that statisticians agree increases the likelihood of a false positive.”
In this case, researchers were concerned that women could be at increased risk for coronary heart disease when the data indicated they weren’t (under the old standard). Under the new statistical test, they can now conclude that women taking hormones five years after menopause may still be at risk. However, scientists usually set the significance level at the beginning of a study. Changing it mid-stream is very, very unusual and increases the possibility of statistical error.
The ongoing debate about the WHI study illustrates why science is almost never an exact . . . science. Results of clinical trials should be interpreted and communicated with caution. However, because research is always ongoing, we can never be sure if we have the full story.
