Pre-Submission Review for Information Retrieval Papers

IR reviewers often ask:

• what is the retrieval task?
• are queries, documents, users, sessions, or recommendations defined clearly?
• are relevance judgments valid, consistent, and appropriate for the task?
• do metrics match the use case?
• are baselines current, tuned, and fair?
• is there leakage between train, validation, test, judgments, prompts, or collection construction?
• are artifacts, code, data, indexes, and runs available enough for reproduction?
• does the paper fit SIGIR, TOIS, ICTIR, CHIIR, RecSys, CIKM, WSDM, or an applied venue?

The paper has to make the evaluation credible before the result can matter.

In our pre-submission review work, IR papers most often fail when the experimental design makes the ranking improvement hard to believe.

Task blur: search, recommendation, question answering, conversational retrieval, and RAG are mixed without a precise evaluation target.

Metric mismatch: nDCG, MAP, MRR, recall, precision, success, satisfaction, or latency is used without matching user intent or system goal.

Baseline weakness: the comparison set omits a strong lexical, neural, hybrid, tuned, or simple baseline.

Judgment fragility: relevance labels, pooling, annotation, assessor agreement, or gold-standard construction is underexplained.

Leakage risk: training data, query logs, prompt examples, document collections, or candidate sets leak test information.

A useful review should identify the first retrieval-evaluation objection.

ACM SIGIR artifact badging emphasizes reproducibility, replicability, and artifact sharing as part of IR research culture. ACM reproducibility guidance describes artifact review and badging as a way to prepare and review research artifacts. Recent IR reproducibility work on recommender systems reports problems such as data split errors, train-test leakage, artifact-paper inconsistency, and weak baselines.

Those signals make IR readiness more than model novelty. Evaluation design and artifact consistency are central.

Send the manuscript, target venue, code repository or archive, dataset or collection description, query and document construction details, relevance judgment protocol, run files, baseline settings, evaluation scripts, metric definitions, train-validation-test split logic, prompt or RAG construction if relevant, and prior reviews if available.

If the paper uses proprietary logs or private collections, include the reproducibility compromise and what can be shared.

A useful IR pre-submission review should include:

• retrieval-contribution verdict
• task and collection critique
• relevance-judgment and metric review
• baseline and leakage-risk check
• artifact and reproducibility readiness note
• user-context and venue-fit recommendation
• submit, revise, retarget, or diagnose deeper call

The review should not only say "add baselines." It should identify the baseline, metric, or judgment problem that will decide reviewer trust.

Before submission, authors often need to:

• define the retrieval task more narrowly
• justify metrics against the user or system goal
• add lexical, neural, hybrid, or simple baselines
• document relevance judgments and pooling
• check for leakage in splits, prompts, or collection construction
• package code, run files, and evaluation scripts
• narrow claims from general search improvement to a tested retrieval setting
• retarget from SIGIR to CHIIR, ICTIR, RecSys, CIKM, WSDM, TOIS, or a domain venue

These fixes make the paper easier to trust and reproduce.

A ranking paper needs strong baselines, metric justification, and leakage control. A recommender paper needs split discipline, candidate set clarity, user or item cold-start context, and comparison to simple baselines. A RAG paper needs retrieval-grounding evidence separate from generation quality. A conversational search paper needs session, interaction, and user intent clarity. A collection paper needs construction, annotation, licensing, and reuse value. An evaluation paper needs metric validity and failure-mode analysis.

The AI manuscript review can flag whether the blocking risk is task definition, metrics, baselines, leakage, artifacts, or venue fit.

Use this page when the manuscript's submission risk depends on search, ranking, relevance judgments, retrieval metrics, collections, recommender evaluation, RAG retrieval evidence, or IR venue fit. Use ML review when the main claim is a general learning method, architecture, optimization, or benchmark outside retrieval-specific evaluation.

That distinction keeps the page focused on the IR buyer's actual problem.

Do not submit an IR paper if the evaluation task cannot be stated in one sentence. If reviewers cannot tell what kind of retrieval success the paper optimizes, they will disagree about whether the metrics, baselines, and collection are appropriate.

Also pause if the strongest result depends on a baseline that may be undertuned. IR reviewers are used to seeing new methods beat weak comparisons. A simple, well-tuned lexical or hybrid baseline can be more damaging to a paper than a complex competing model.

For RAG or conversational search papers, pause again if generation quality is masking retrieval weakness. The manuscript should separate retrieval relevance from downstream answer fluency.

For recommender papers, pause if the candidate set and negative-sampling logic are not explicit. A model can look strong when the evaluation excludes realistic alternatives or samples negatives too easily. Reviewers need to know whether the task reflects the choice set users or systems actually face, not just a convenient offline benchmark.

If the paper uses click logs, make the bias story explicit. Position bias, popularity bias, bot traffic, and changing product surfaces can turn a clean-looking signal into a misleading relevance proxy.