Pre-Submission Review for Data Science Papers

Data science reviewers often ask:

• can the analysis be reproduced from code, data, and documentation?
• are data provenance, permissions, and preprocessing clear?
• is there leakage between training, validation, test, or external data?
• are benchmarks and baselines current and fair?
• do evaluation metrics match the real problem?
• are statistical comparisons and uncertainty handled correctly?
• are ablations, sensitivity analyses, or robustness checks present where needed?
• does the paper fit a data science, statistics, ML, domain, or methods journal?

If those points are weak, a good-looking result table can become a rejection trigger.

In our pre-submission review work, data science manuscripts most often fail because authors treat the result table as the manuscript instead of treating the workflow as the evidence.

Code-visible but not runnable: a repository exists, but dependencies, environment, data paths, seeds, or instructions are incomplete.

Data leakage: preprocessing or feature selection accidentally sees information from the test set or future period.

Benchmark convenience: the paper compares against weak, old, or poorly tuned baselines.

Metric mismatch: the headline metric does not match the real decision problem or class balance.

Domain-free claim: the model performs well numerically, but the claim ignores domain constraints.

A useful review should find the first reproducibility or evaluation objection.

Journal of Data Science author instructions say manuscripts must be typeset in LaTeX and that code files need to be submitted, with authors responsible for ensuring results are reproducible. The page also says a reproducibility-checking team will check when review concludes.

INFORMS Journal on Data Science describes replication as a fundamental scientific principle and says its data and code disclosure policy is intended to assure availability of materials needed to replicate published research.

Those public signals are clear: for data science, reproducibility is not decoration. It is part of the submission package.

Send the manuscript, target journal, code repository, README, environment file, dependency versions, data access plan, preprocessing scripts, train-validation-test split description, random seeds, benchmark list, metric definitions, ablation tables, sensitivity checks, and any domain constraints.

If data cannot be public, send the access explanation and synthetic or minimal reproducible materials. If code cannot be public, explain restrictions and what reviewers can inspect.

A useful data science pre-submission review should include:

• reproducibility verdict
• code and data packaging critique
• leakage and split-design review
• benchmark and baseline audit
• metric and statistical comparison check
• claim and domain-validity note
• journal-lane recommendation
• submit, revise, retarget, or diagnose deeper call

"Share the code" is not enough. The useful version is "the repository needs a locked environment, preprocessing command, seed note, and script that reproduces Table 2."

Before submission, authors often need to:

• add an environment file or container
• make a README reproduce the main tables
• document data provenance and preprocessing
• fix train-test leakage
• add current baselines
• report uncertainty or repeated-run variance
• add ablations for the claimed contribution
• narrow claims to the tested setting
• retarget from an ML venue to a domain journal or from a domain journal to a data-science methods journal

These fixes often decide whether reviewers trust the result.

A predictive modeling paper needs split discipline, leakage control, calibration, and external validation where possible. A statistical methods paper needs assumptions, proofs or simulations, and comparison to established methods. A workflow paper needs runnable code, data access, and documentation. A benchmark paper needs dataset quality, task definition, baseline fairness, and metric justification. An applied data-science paper needs domain validity, not only technical performance.

The AI manuscript review can flag whether the blocking risk is reproducibility, leakage, benchmark weakness, or journal fit.

Use this page when the manuscript's risk is the data pipeline, reproducibility, benchmark design, statistical comparison, or applied data workflow. Use the AI page when the main contribution is a model architecture, training method, AI system, or machine-learning novelty claim.

Many papers use machine learning but are still data science papers. The difference is what reviewers will attack first.

Do not submit a data science paper if the main table cannot be reproduced from the stated code, data, and instructions. Reviewers do not need production-grade software, but they do need enough structure to understand what was run and how the result was produced.

Also pause if the paper relies on a hidden preprocessing step, a private split, a manually tuned baseline, or a metric chosen after seeing the result. Those issues make a strong manuscript look fragile. The safest pre-submission move is to write a reviewer-facing reproducibility path: where the data came from, what code runs, what command produces the main result, and what cannot be shared. If that path is hard to write, the manuscript is probably not ready.

For applied papers, add one more check: can a domain reviewer explain why the benchmark result matters outside the dataset? If not, the paper may need a narrower claim or a different target journal.