Statistical Mistakes That Get Papers Rejected (A Reviewer's Checklist)

What it looks like: You tested 20 different outcomes and reported the three that were statistically significant, without mentioning the other 17. Or you compared six treatment groups to each other (15 pairwise comparisons) and reported the p-values from individual t-tests.

A specific example: A study compares gene expression across four patient groups. Instead of an ANOVA, the authors run six pairwise t-tests and report two comparisons with p < 0.05. They don't apply Bonferroni correction. With six comparisons at α = 0.05, you'd expect about one "significant" result by chance alone.

Why reviewers flag it: Because it inflates false positives. If you run enough tests, something will be significant by accident. Every stats-literate reviewer knows this.

How to fix it: Use an appropriate correction (Bonferroni, Benjamini-Hochberg, Tukey's HSD). Report all outcomes in a supplementary table, not just the hits. Pre-register your primary outcomes if possible. Twenty independent tests at alpha = 0.05 yields a 64% chance of at least one false positive.

What it looks like: n=3 per group for an experiment that would need n=15 to detect a meaningful effect. No power calculation. No justification for the sample size.

A specific example: A drug reduces tumor volume by 20% compared to control. Sounds promising. But with n=4 mice per group, the confidence intervals span from -10% to +50%. The effect could easily be zero. A power analysis would show they'd need at least 12 animals per group to reliably detect a 20% difference.

Why reviewers flag it: Because underpowered studies produce unreliable results. A "significant" finding from a tiny sample is likely a false positive or a grossly overestimated effect size.

How to fix it: Do a power analysis before the experiment and report it in your methods. G*Power (free) or R's pwr package make this straightforward. If you couldn't achieve the ideal sample size, say so. Reviewers respect honesty about constraints far more than silence.

What it looks like: Running a t-test or ANOVA on data that's clearly skewed, bimodal, or has massive outliers. No test for normality mentioned.

A specific example: Comparing hospital length-of-stay between two patient groups using a t-test. Length-of-stay data is almost never normally distributed. The median is 4 days, but the mean is 11. A t-test on these data compares means that don't represent either group well.

Why reviewers flag it: Parametric tests assume normal distribution. When that assumption is violated, the p-value can be meaningless.

How to fix it: Test for normality (Shapiro-Wilk for n < 50, Q-Q plots for visual assessment). If your data isn't normal, use non-parametric alternatives: Mann-Whitney U instead of a t-test, Kruskal-Wallis instead of ANOVA. Or log-transform skewed data. Report which test you used and why.

What it looks like: The results show a correlation. The discussion says "causes" or "drives" or "leads to."

A specific example: "Higher coffee consumption was associated with lower mortality (r = -0.32, p < 0.01). These findings demonstrate that coffee consumption reduces mortality risk." No. People who drink more coffee might also exercise more or have higher income. The study measured both variables and found they moved together. That's it.

How to fix it: Use precise language. "Associated with," not "caused." "Correlated with," not "driven by." If you believe the relationship is causal, make the argument explicitly using evidence from the study design.

What it looks like: The overall analysis showed no effect. But when the authors split by age, gender, and disease severity, they found a significant effect in women over 65 with severe disease. The paper focuses on this subgroup as if it were the planned analysis.

A specific example: A clinical trial shows no difference in the primary endpoint (p=0.34). The authors test the drug in 12 pre-specified and 8 post-hoc subgroups. They find a significant benefit in patients with baseline CRP > 10 (p=0.03). The abstract leads with this subgroup finding.

How to fix it: Pre-specify your subgroups and register them. If you find an interesting subgroup effect post-hoc, label it clearly: "In an exploratory analysis..." Don't bury the negative primary result.

"Treatment improved survival (p=0.03)." Improved by how much? One day? One year? You can't tell.

A specific example: A cardiovascular trial reports that a new antiplatelet agent "reduced major adverse cardiac events (p=0.04)." But the absolute risk reduction was 0.3% (3.2% vs 3.5%). The number needed to treat is 333. Statistically significant, but clinically marginal.

Why reviewers flag it: A p-value tells you whether an effect is likely to be non-zero, not how large or precise it is. The ASA's 2016 statement on p-values makes this point explicitly: "Scientific conclusions should not be based only on whether a p-value passes a specific threshold."

How to fix it: Report effect sizes with 95% confidence intervals for every comparison. Include absolute numbers, not just relative measures. CONSORT requires both absolute and relative effect measures for clinical trials, and journals like PNAS now require effect sizes for all statistical comparisons.

What it looks like: Bar graphs with error bars for continuous data with 5-20 data points per group.

A specific example: Two treatment groups, n=8 each: the bar graph shows nearly identical means with overlapping error bars. But the individual data reveals Group B has a bimodal distribution, four complete responders and four non-responders. The bar graph hides the most interesting finding.

Why reviewers flag it: Weissgerber et al. (2015) in PLOS Biology showed that dramatically different distributions can produce identical bar graphs. Their paper has been cited over 3,000 times, and many journals updated their figure policies.

How to fix it: Use dot plots, box plots, or violin plots. Show individual data points. JBC, PLOS, and eLife now require individual data points for small n.

What it looks like: A study follows patients from diagnosis to outcome, but only includes patients who survived long enough to be enrolled. The sickest patients died before they could be included.

A specific example: A study examining five-year outcomes after chemotherapy requires participants to have completed at least two cycles. Patients who died after cycle one aren't in the data. The reported 70% five-year survival rate is inflated because the highest-risk patients were excluded by design.

Why reviewers flag it: This creates an artificially favorable picture of outcomes.

How to fix it: Use intention-to-treat analysis where appropriate. Draw a CONSORT flow diagram showing exactly how many patients were excluded at each stage and why. If your enrollment criteria exclude high-risk patients, run a sensitivity analysis showing how results change if they're included.

What it looks like: A neuroscience paper reports n=200 neurons. But those came from 4 mice (50 per animal). The paper runs statistics as if each neuron is independent, but neurons within the same mouse are correlated.

A specific example: The effective sample size isn't 200, it's closer to 4. The intraclass correlation within a single animal might be 0.3 or higher, meaning the 200 "independent" observations carry the statistical weight of roughly 15-20 truly independent measurements. The paper's p-value of 0.001 recalculated at the animal level might be 0.15.

Why reviewers flag it: This is one of the most common errors in biology. The eLife review identified it as particularly widespread in neuroscience.

How to fix it: Analyze at the level of the independent unit, or use mixed-effects models with a random effect for animal/subject/cluster. Report both the number of independent units and total observations. ARRIVE 2.0 now requires explicit reporting of the experimental unit for each analysis.

What it looks like: "Treatment increased expression 10-fold (p < 0.01)." Impressive, until you learn baseline expression was 0.001 units.

A specific example: A 15-fold increase from 0.2 pg/mL to 3.0 pg/mL sounds dramatic. But when the normal physiological range is 50-200 pg/mL, you're still in the noise floor. Conversely, a 1.3-fold increase in serum creatinine (from 1.0 to 1.3 mg/dL) is clinically meaningful, it indicates acute kidney injury under KDIGO criteria.

How to fix it: Always report absolute numbers alongside fold-changes. Include reference ranges when available. If your fold-change is large but the absolute values are near the assay's detection limit, acknowledge that limitation.

Statistical errors vary by discipline. Here's what reviewers flag most often in each area:

In our pre-submission review work, the most damaging statistics problems are often the ones authors think are too small to matter. A paper may have respectable analyses overall, but one subgroup result is oversold, one figure legend hides the error-bar definition, or one animal study treats measurements from the same subject as independent. After that, reviewers start distrusting the rest of the paper.

According to ICMJE recommendations, authors should describe statistical methods in enough detail that a knowledgeable reader can verify the reported results. ARRIVE 2.0 also requires authors to state the experimental unit explicitly. Those requirements line up with what we see in rejections: papers do not only fail because the statistics are wrong, they fail because the manuscript does not make the statistical logic auditable.