Pre-Submission Review for Data Science Papers

Data science reviewers often ask:

• can the analysis be reproduced from code, data, and documentation?

• are data provenance, permissions, and preprocessing clear?

• is there leakage between training, validation, test, or external data?

• are benchmarks and baselines current and fair?

• do evaluation metrics match the real problem?

• are statistical comparisons and uncertainty handled correctly?

• are ablations, sensitivity analyses, or robustness checks present where needed?

• does the paper fit a data science, statistics, ML, domain, or methods journal?

If those points are weak, a good-looking result table can become a rejection trigger.

In our pre-submission review work on data science manuscripts, the recurring issue is that authors treat the result table as the manuscript instead of treating the workflow as the evidence. Reviewers usually want to know what data entered the pipeline, what code transformed it, which choices could change the result, and whether the comparison would still look fair under a skeptical rerun.

The specific failure pattern is a reproducibility gap hidden behind a strong benchmark; in practice this means the code, data, evaluation split, and abstract claim need to tell the same story.

Code-visible but not runnable: a repository exists, but dependencies, environment, data paths, seeds, or instructions are incomplete.

Data leakage: preprocessing or feature selection accidentally sees information from the test set or future period.

Benchmark convenience: the paper compares against weak, old, or poorly tuned baselines.

Metric mismatch: the headline metric does not match the real decision problem or class balance.

Domain-free claim: the model performs well numerically, but the claim ignores domain constraints.

Notebook-to-paper drift: figures in the manuscript no longer match the repository because a notebook cell, random seed, data extract, or preprocessing branch changed after drafting.

Private-data dead end: the manuscript says data cannot be shared but does not provide a reviewer-facing access explanation, synthetic example, code path, or minimal reproducibility package.

Applied-journal mismatch: the paper has a data-science method but is aimed at a domain journal whose reviewers need domain interpretation first, or the reverse: the paper sells a domain result to a methods audience without enough algorithmic contribution.

We see data science papers improve fastest when the review ties each objection to a manuscript component: the abstract's claim, the methods section's workflow, the repository README, the environment file, the data availability statement, the benchmark table, the figures, and the cover letter's journal-fit argument. That is the difference between a generic reproducibility warning and a submission decision about whether the paper belongs in Journal of Data Science, INFORMS Journal on Data Science, a statistics venue, an ML venue, or a domain journal.

A useful review should find the first reproducibility or evaluation objection.

Journal of Data Science author instructions say manuscripts must be typeset in LaTeX and that code files need to be submitted, with authors responsible for ensuring results are reproducible. The page also says a reproducibility-checking team will check when review concludes.

INFORMS Journal on Data Science describes replication as a fundamental scientific principle and says its data and code disclosure policy is intended to assure availability of materials needed to replicate published research.

Method note: official reproducibility policies define the expected materials, but they do not test whether the submitted package is usable. Manusights interpretation focuses on the reviewer action: can a competent reader inspect the code, data, methods, figures, and claims without reconstructing the lab's private workflow?

Those public signals are clear: for data science, reproducibility is not decoration. It is part of the submission package.

Send the manuscript, target journal, code repository, README, environment file, dependency versions, data access plan, preprocessing scripts, train-validation-test split description, random seeds, benchmark list, metric definitions, ablation tables, sensitivity checks, and any domain constraints.

If data cannot be public, send the access explanation and synthetic or minimal reproducible materials. If code cannot be public, explain restrictions and what reviewers can inspect.

A useful data science pre-submission review should include:

• reproducibility verdict

• code and data packaging critique

• leakage and split-design review

• benchmark and baseline audit

• metric and statistical comparison check

• claim and domain-validity note

• journal-lane recommendation

• submit, revise, retarget, or diagnose deeper call

"Share the code" is not enough. The useful version is "the repository needs a locked environment, preprocessing command, seed note, and script that reproduces Table 2."

Before submission, authors often need to:

• add an environment file or container

• make a README reproduce the main tables

• document data provenance and preprocessing

• fix train-test leakage

• add current baselines

• report uncertainty or repeated-run variance

• add ablations for the claimed contribution

• narrow claims to the tested setting

• retarget from an ML venue to a domain journal or from a domain journal to a data-science methods journal

These fixes often decide whether reviewers trust the result.

A predictive modeling paper needs split discipline, leakage control, calibration, and external validation where possible. A statistical methods paper needs assumptions, proofs or simulations, and comparison to established methods. A workflow paper needs runnable code, data access, and documentation. A benchmark paper needs dataset quality, task definition, baseline fairness, and metric justification. An applied data-science paper needs domain validity, not only technical performance.

The AI manuscript review can flag whether the blocking risk is reproducibility, leakage, benchmark weakness, or journal fit.

If the data-science manuscript is really an IoT, communications, or transportation-systems paper, check the venue-specific route before polishing the analysis: IEEE IoT Journal submission guide, IEEE JSAC call for papers and submission guide, or IEEE T-ITS submission guide.

Use this guide when the manuscript's risk is the data pipeline, reproducibility, benchmark design, statistical comparison, or applied data workflow. Use an AI review when the main contribution is a model architecture, training method, AI system, or machine-learning novelty claim.

Many papers use machine learning but are still data science papers. The difference is what reviewers will attack first.

Do not submit a data science paper if the main table cannot be reproduced from the stated code, data, and instructions. Reviewers do not need production-grade software, but they do need enough structure to understand what was run and how the result was produced.

Also pause if the paper relies on a hidden preprocessing step, a private split, a manually tuned baseline, or a metric chosen after seeing the result. Those issues make a strong manuscript look fragile. The safest pre-submission move is to write a reviewer-facing reproducibility path: where the data came from, what code runs, what command produces the main result, and what cannot be shared. If that path is hard to write, the manuscript is probably not ready.

For applied papers, add one more check: can a domain reviewer explain why the benchmark result matters outside the dataset? If not, the paper may need a narrower claim or a different target journal.

Ready to submit if:

• code, data, and workflow are auditable enough for review

• splits and preprocessing avoid leakage

• baselines and metrics are fair

• claims match the benchmark evidence

• target journal matches the contribution

Pause first if:

• the repository does not reproduce core results

• data access is vague

• one metric hides poor performance elsewhere

• the paper sells a domain claim from a narrow benchmark

Pros: a field-specific review can find data leakage, weak baselines, underreported preprocessing, unclear dataset licensing, missing code citations, and claims that depend on a benchmark the target journal will not trust.

Cons: it is less useful if the repository is not yet runnable, the evaluation question is still changing, or the manuscript needs a new experiment rather than a submission-readiness diagnosis.

Data-science source details make the same point in different ways. Data Science Journal caps research papers and reviews at 8,000 words and asks authors to cite datasets or source code formally, including associated DOIs where relevant. Nature Computational Science lists a gold open-access APC of $12850 / £9390 / €10850, which makes journal fit and funder compliance a real pre-submission concern.

IOS Press launch materials identify named editorial leadership, but authors should verify the current editors on the journal site before quoting any name in submission materials. These are concrete constraints, but the review job remains the same: prove that the data, code, evaluation, and claim are aligned.