Skip to main content
Article
Pisces did not have increased heart failure: Data-driven comparisons of binary proportions between levels of a categorical variable can result in incorrect statistical significance level
Journal of Clinical Epidemiology (2008)
  • Peter C Austin, Institute for Clinical Evaluative Sciences
Abstract

Objective We examined the impact on statistical inference when a χ2 test is used to compare the proportion of successes in the level of a categorical variable that has the highest observed proportion of successes with the proportion of successes in all other levels of the categorical variable combined.

Study Design and Setting Monte Carlo simulations and a case study examining the association between astrological sign and hospitalization for heart failure.

Results A standard χ2 test results in an inflation of the type I error rate, with the type I error rate increasing as the number of levels of the categorical variable increases. Using a standard χ2 test, the hospitalization rate for Pisces was statistically significantly different from that of the other 11 astrological signs combined (P=0.026). After accounting for the fact that the selection of Pisces was based on it having the highest observed proportion of heart failure hospitalizations, subjects born under the sign of Pisces no longer had a significantly higher rate of heart failure hospitalization compared to the other residents of Ontario (P=0.152).

Conclusions Post hoc comparisons of the proportions of successes across different levels of a categorical variable can result in incorrect inferences.

Keywords
  • Chi-squared test,
  • Maximal proportion,
  • Type I error rate,
  • Contingency table,
  • Astrology,
  • Significance testing
Publication Date
2008
Citation Information
Peter C Austin. "Pisces did not have increased heart failure: Data-driven comparisons of binary proportions between levels of a categorical variable can result in incorrect statistical significance level" Journal of Clinical Epidemiology Vol. 61 (2008)
Available at: http://works.bepress.com/peter_austin/26/