Question before method
Define the quantity, contrast, relationship, prediction, or decision before selecting an analysis.
Sources and scope · Guide reviewed September 22, 2026
- Coverage
- Scientific target, design, units, outcomes, data states, methods, assumptions, uncertainty, sensitivity, provenance, and claim control
- Source basis
- ASA, ICMJE, ICH, CONSORT, STROBE, ARRIVE, NIH, and EQUATOR primary guidance
Prepared by the Manusights editorial team. Publisher and institutional instructions take precedence.
Analysis provenance desk
Make every result
traceable.
The defensible unit of statistical work is not a test name. It is a visible chain from the scientific question to the design, data, method, uncertainty, and claim.
Question
Design
Data
Method
Estimate
Claim
10 primary sources7 decision gates8 release testsEdition 2026.09.22-analysis-1
Five-page analysis provenance record
Statistical analysis decision record
Record the target, unit structure, outcomes, data states, analysis warrant, uncertainty, sensitivity checks, deviations, result locators, and bounded claim. Licensed CC BY 4.0 for the original Manusights framework.
- Edition
- 2026.09.22-analysis-1
- Format
- License
- CC BY 4.0
- Evidence reviewed
- September 22, 2026
Method boundary
This is not a statistical test picker
The same data type can require different analyses when the target, design, unit structure, timing, sampling, or intended claim changes. A normality label and a group count are not an analysis plan. The record makes those hidden decisions inspectable before software produces an answer.
Design before rows
Preserve experimental, sampling, observation, and analysis units plus clustering, pairing, and repetition.
Estimate before threshold
Report magnitude, direction, uncertainty, assumptions, and robustness before turning output into a claim.
Interpretation boundary: The ASA states that a p-value does not measure the probability that a hypothesis is true, the probability that chance alone produced the data, effect size, or importance. A conclusion should not rest only on whether a value crosses a cutoff. Source A
Inferential chain
A result is only as strong as its weakest handoff
Each stage must carry a specific payload forward. If that payload disappears, the next stage inherits an undocumented assumption.
Link 1
Question Target
Must carry: Population or system, outcome, comparison or exposure, time point, and the quantity or decision of interest.
Break signal: The stated aim can be paired with many incompatible analyses.
Link 2
Target Design
Must carry: The contrast or quantity the design must identify, estimate, predict, or describe.
Break signal: The design cannot support the intended inference or its scope is left unstated.
Link 3
Design Data
Must carry: Units, allocation or sampling, dependence, timing, measurement, eligibility, and planned analysis populations.
Break signal: The dataset structure hides clusters, pairing, repeated observations, exclusions, or changed definitions.
Link 4
Data Method
Must carry: Variable definitions, coding, missingness, censoring, transformations, groupings, and provenance of analysis-ready values.
Break signal: The method operates on an undocumented data state or silently chooses a denominator.
Link 5
Method Estimate
Must carry: Model, assumptions, covariates, interactions, dependence handling, software, validation, and uncertainty method.
Break signal: An estimate cannot be reproduced or interpreted without hidden defaults.
Link 6
Estimate Claim
Must carry: Magnitude, direction, uncertainty, analysis population, robustness checks, multiplicity context, and limitations.
Break signal: A p-value or model output is translated directly into a scientific conclusion.
ICH E9(R1) formalizes this alignment for clinical trials by connecting the clinical question, estimand, design, conduct, analysis, sensitivity analysis, and interpretation. The chain here generalizes the provenance idea without claiming that the clinical-trial framework governs every biomedical design. Source F
Original Manusights framework
Seven gates turn output into a defensible record
Each gate asks a question, identifies what must be recorded, defines evidence of completion, and names the failure it prevents.
- 1
Question and target
What quantity, contrast, relationship, prediction, or decision is the analysis meant to address?
Record
State the scientific question, population or system, outcome, comparison or exposure, time point, target quantity, and the claim the study could support if the evidence is adequate.
Evidence of completion
The target is specific enough that a method can be judged against it, and the manuscript does not substitute a convenient test for an unstated scientific question.
Failure signal
The analysis begins with a software menu, a preferred p-value, or a vague aim such as finding differences.
- 2
Design and units
What was assigned, sampled, exposed, measured, clustered, paired, or repeated?
Record
Identify the sampling unit, experimental unit, observation unit, unit of analysis, dependence structure, allocation or sampling process, comparator, timing, and design features that must enter the analysis.
Evidence of completion
The analysis unit and independence assumptions follow the design rather than the number of rows in the data file.
Failure signal
Repeated measurements, technical replicates, clusters, matched observations, or blocked factors are treated as independent without a design-based rationale.
- 3
Outcomes and variables
What exactly is measured, derived, transformed, grouped, or selected for analysis?
Record
Define each outcome and predictor, measurement method, units, participant- or specimen-level analysis metric, aggregation, time point, coding, derivation, transformation, threshold, and primary or secondary status.
Evidence of completion
A reader can trace every reported result to a prespecified or explicitly labeled variable definition.
Failure signal
The outcome name stays constant while the scale, time point, denominator, transformation, or grouping changes.
- 4
Data states and populations
Which observations enter each analysis, and what happened to all others?
Record
Preserve inclusion and exclusion rules, analysis populations, attrition, missingness, censoring, protocol deviations, outlier handling, data corrections, and the counts entering every outcome analysis.
Evidence of completion
The path from collected or eligible units to analyzed units is visible, and each exclusion or data-state change has a timing, rule, and rationale.
Failure signal
The final denominator appears only in a figure or software output and cannot be reconciled with the study flow.
- 5
Method and assumptions
Why does the selected model or method answer this target for this design and data structure?
Record
Name the estimand or target quantity where applicable, model or method, covariates, interactions, dependence handling, assumptions, diagnostics, validation, software and version, and the rationale for each decision.
Evidence of completion
A knowledgeable reader with access to the data could understand the analysis and judge its appropriateness without guessing hidden defaults.
Failure signal
A test name is offered as the entire rationale, or an assumption is declared satisfied because one diagnostic crossed a threshold.
- 6
Uncertainty and robustness
Which analytical choices or departures could materially change the estimate or interpretation?
Record
Specify effect estimates, uncertainty intervals or other uncertainty measures, multiplicity decisions, missing-data assumptions, sensitivity analyses, model checks, alternative specifications, and the boundary between prespecified and exploratory work.
Evidence of completion
The main conclusion is shown beside the assumptions and alternative analyses that test its stability, not reduced to a binary threshold.
Failure signal
Only the most favorable analysis is visible, or statistical significance is treated as effect size, importance, truth, or the probability that chance produced the data.
- 7
Result and claim
What does the analysis support, and what remains outside the study's inferential reach?
Record
Link each central claim to the analysis population, estimate, uncertainty, model, sensitivity result, relevant limitation, and the exact manuscript, table, figure, code, and data locations.
Evidence of completion
The wording of the claim matches the design and evidence, all primary and secondary outcomes are accounted for, and exploratory findings are labeled.
Failure signal
The conclusion widens from association to causation, sample to population, surrogate to clinical outcome, or exploratory result to confirmation without a defensible bridge.
Analysis contract
Write the decisions that software cannot infer
ICMJE asks for enough statistical detail that a knowledgeable reader with the data can judge appropriateness and verify results. A software command is not enough; the record also needs the scientific and design warrant. Source B Source J
Analysis identity
Analysis ID, protocol or plan version, outcome, target, population, time point, and whether the analysis is primary, secondary, sensitivity, supplementary, or exploratory.
Design payload
Units, dependence, allocation or sampling, comparator, measurement schedule, stratification, blocking, clustering, pairing, and repeated-measure structure.
Data-state contract
Source data version, inclusion and exclusion rules, missing and censored values, transformations, coding, derived variables, outliers, and the analysis denominator.
Method warrant
Model or method, rationale, covariates, interactions, assumptions, diagnostics, validation, multiplicity handling, software, package and version.
Robustness plan
Alternative assumptions, sensitivity or supplementary analyses, missing-data scenarios, influence checks, and the result that would change the claim.
Reporting payload
Estimate, uncertainty, exact analysis count, effect direction and magnitude, model and sensitivity result, table or figure locator, code and data locator, and bounded claim.
Worked control record
One fictional example, bounded by design
This example demonstrates traceability, not the selection of a model. It deliberately stops at the information a statistical collaborator would need to assess the plan.
- Question
- In adults enrolled in a defined clinic cohort, what is the adjusted difference in the prespecified 12-week outcome between the two observed exposure groups?
- Units
- Participant is the sampling and analysis unit; repeated measurements occur within participant; clinic is a recorded clustering factor.
- Outcome
- Prespecified continuous score at week 12, with scale, direction, derivation, time window, and baseline value defined before analysis.
- Data state
- Eligible cohort, observed analysis set, missing outcome counts by group, exclusion reasons, and date-stamped data version are reconciled.
- Method
- Adjusted model and covariance structure are justified against the design; assumptions, software version, and diagnostics are preserved.
- Claim
- Report the adjusted estimate and uncertainty as an association in this cohort, with missing-data and residual-confounding limits.
Why the boundary matters: The design is observational, so this record does not promote an adjusted association to a causal effect. STROBE asks authors to describe confounding control, missing data, participant flow, estimates, precision, sensitivity analyses, and limitations, but reporting those items does not itself identify a causal effect. Source G
Failure patterns
Five shortcuts that sever the record
Test-first planning
Failure: The team asks which test to run before fixing the question, target, design, units, and data structure.
Repair: Write the inferential target and design payload first. Select or review a method only after the required structure is explicit.
Row-count independence
Failure: Rows, wells, images, visits, specimens, or measurements are treated as independent observations because they are separate records.
Repair: Identify the experimental, sampling, observation, and analysis units and show how dependence is handled.
Silent data-state change
Failure: Exclusions, missing values, transformations, regrouping, outcome substitutions, or denominator changes appear after results are visible without a preserved decision trail.
Repair: Version the data-state contract, label deviations, preserve the reason and timing, and show their effect through an appropriate sensitivity or supplementary analysis.
Threshold-only conclusion
Failure: A result is called real, absent, important, or unimportant solely because a p-value falls on one side of a cutoff.
Repair: Report magnitude, direction, uncertainty, design and model context, limitations, and the scientific consequence. Treat the threshold as one declared decision rule, not the claim.
Exploration presented as confirmation
Failure: A subgroup, time point, outcome, model, or data-driven choice selected after inspection is reported as though it were the prespecified primary analysis.
Repair: Distinguish prespecified, changed, supplementary, sensitivity, and exploratory analyses and preserve the multiplicity and selection context.
Scope-specific routing
Use the reporting guideline that matches the study
The shared record captures provenance. It does not erase design-specific reporting obligations.
Randomized trial
Use current CONSORT guidance for trial outcomes, analysis populations, multiplicity, missing data, additional analyses, and participant flow.
Observational epidemiology
Use the applicable STROBE checklist for cohort, case-control, or cross-sectional reporting.
In vivo animal research
Use ARRIVE 2.0 to report experimental units, design, sample-size reasoning, exclusions, randomization, blinding, outcomes, and statistical methods.
Biomedical manuscript
Use current journal instructions with ICMJE and relevant EQUATOR guidance; do not treat a general checklist as journal approval.
Release audit
Eight independent tests before the claim leaves the team
A fully documented model can still answer the wrong target. A precise estimate can still support an overbroad claim. Run every test separately.
Target test
Pass: The primary scientific question and target quantity, contrast, prediction, or decision are explicit.
Repair: Rewrite the question so the analysis can be judged against a defined target rather than a general topic.
Unit test
Pass: Sampling, experimental, observation, and analysis units plus dependence are named and consistent with the model.
Repair: Reconstruct the unit hierarchy and resolve clusters, pairing, repeated measures, technical replication, or blocking before interpretation.
Outcome test
Pass: Variables, metrics, aggregation, time points, coding, transformations, and primary or secondary status are traceable.
Repair: Create one versioned definition for each outcome and reconcile methods, analysis, figures, tables, and claims.
Population test
Pass: Every reported analysis count reconciles with eligibility, attrition, exclusion, missingness, censoring, and protocol deviations.
Repair: Add an analysis-population record and explain each difference between collected, eligible, assigned, observed, and analyzed units.
Method test
Pass: The method is warranted by the question, design, data structure, and assumptions, with software and versions recorded.
Repair: Replace the test label with the full method rationale and obtain qualified statistical review when the judgment exceeds team expertise.
Uncertainty test
Pass: Magnitude, direction, uncertainty, multiplicity, missing-data assumptions, and material robustness checks accompany the result.
Repair: Add the missing uncertainty or sensitivity context and remove binary threshold language that outruns the evidence.
Provenance test
Pass: Prespecified, changed, supplementary, sensitivity, and exploratory analyses are distinguished with versions and reasons.
Repair: Create a deviation ledger and relabel analyses honestly instead of reconstructing a false prespecification story.
Claim test
Pass: Each central claim is linked to an analysis, estimate, uncertainty, robustness result, limitation, and stable result locator.
Repair: Narrow the claim to the design and evidence or supply the missing analysis record before publication.
Evidence and maintenance
Primary-source register and design boundaries
The web guide, printable record, and source ledger share one edition. Every source is shown with the job it supports and the boundary that prevents overgeneralization.
- AASA Statement on Statistical Significance and P-Values
American Statistical Association / Six principles on p-values and statistical significance. Supports: P-values describe incompatibility with a specified model; they do not give the probability that a hypothesis is true or that chance alone produced the data, and they do not measure effect size or importance. Decisions should not rest only on a threshold, and transparent reporting is required.
Boundary: The statement corrects common interpretations of p-values. It does not select a design, model, uncertainty measure, or decision threshold for a specific study.
- BPreparing a Manuscript for Submission to a Medical Journal
International Committee of Medical Journal Editors / Methods, Statistics, Results, and Discussion. Supports: Methods detailed enough for verification, effect estimates with uncertainty, avoidance of sole reliance on p-values, software and version disclosure, separation of prespecified and exploratory analyses, complete outcome reporting, and claims bounded by the evidence.
Boundary: ICMJE recommendations address medical-journal reporting. The exact journal instructions and study-design guideline still apply.
- CCONSORT 2025 Item 14: Outcomes
CONSORT-SPIRIT Group / Outcome definition components. Supports: Outcome definitions that distinguish the measurement variable, participant-level analysis metric, aggregation method, and analysis time point.
Boundary: CONSORT 2025 is for randomized trials. Its outcome-definition structure is useful beyond trials, but trial-specific requirements must not be generalized automatically.
- DCONSORT 2025 Item 21a: Outcome Comparison
CONSORT-SPIRIT Group / Primary and secondary outcome analysis methods. Supports: Analysis methods detailed enough for verification, statistical analysis plan provenance, model specification, covariates, assumptions, and transparent handling or non-handling of multiplicity.
Boundary: This is randomized-trial reporting guidance, not a universal method-selection rule.
- ECONSORT 2025 Item 21c: Missing Data
CONSORT-SPIRIT Group / Planning, analysis, reporting, and sensitivity guidance for missing data. Supports: Recording missing-data amounts and reasons, assumptions, methods, justifications, analysis populations, and sensitivity analyses rather than treating missingness as a hidden software detail.
Boundary: The examples concern randomized trials. Missing-data methods and assumptions remain study-specific and need qualified statistical judgment.
- FICH E9(R1) Addendum on Estimands and Sensitivity Analysis in Clinical Trials
International Council for Harmonisation / Sections A.2 through A.6. Supports: Aligning the clinical question, estimand, design, conduct, main estimation, sensitivity analysis, interpretation, and documentation, including explicit treatment of intercurrent events.
Boundary: ICH E9(R1) is regulatory clinical-trial guidance. Its alignment principle does not make its estimand framework mandatory for every biomedical design.
- GSTROBE Checklists
STROBE Initiative / Cohort, case-control, and cross-sectional reporting checklists. Supports: Design-specific reporting of study size, quantitative variables, confounding control, subgroup and interaction methods, missing data, loss to follow-up, sensitivity analyses, participant flow, estimates, precision, and limitations.
Boundary: STROBE is reporting guidance for observational epidemiology. It does not certify study quality or prescribe one analysis.
- HARRIVE Guidelines 2.0
NC3Rs / Essential 10 and Recommended Set. Supports: Transparent animal-study reporting, including experimental units, study design, sample-size reasoning, inclusion and exclusion, randomization, blinding, outcome measures, statistical methods, data access, and protocol information.
Boundary: ARRIVE applies to in vivo animal research. Experimental-unit and design records must be adapted rather than copied into human, in vitro, or computational studies.
- IRigor and Reproducibility in NIH Applications: Resource Chart
National Institutes of Health / Scientific rigor and biological variables. Supports: Explicit connections among experimental design, methodology, analysis, interpretation, reporting, relevant biological variables, and unbiased results.
Boundary: This chart is general NIH application guidance. Applicants must follow the current notice of funding opportunity and application instructions.
- JSAMPL Guidelines
EQUATOR Network / Guiding principles and basic statistical reporting. Supports: Enough method detail for verification, effect estimates and uncertainty, multiple-comparison disclosure, treatment of outliers, test direction, model validation, and software reporting.
Boundary: SAMPL is an older reporting aid for common biomedical analyses and was not developed through a formal consensus process. Use current design-specific guidance and statistical expertise where they conflict or go further.
Scope: An educational provenance record for biomedical research teams connecting a scientific question to design, data, analysis, uncertainty, and a bounded claim. It is not statistical advice, a power calculation, a sample-size determination, study-design approval, causal identification, journal compliance, or a substitute for a statistician or biostatistician. Recheck all primary sources after a material revision and no later than 90 days after this review. At the time of use, verify the study-specific protocol, statistical analysis plan, reporting guideline, journal instructions, software version, data state, and any regulatory, funder, or institutional requirements.
Common questions
Use the record without turning it into a rulebook
Does this record choose the right statistical test for my study?
No. A test name cannot be selected responsibly from the number of groups and a normality label alone. The question, target quantity, design, units, dependence, data state, assumptions, and intended claim all matter. Use the record to expose those decisions, then obtain study-specific statistical review when needed.
Does a p-value below 0.05 prove that a finding is real or important?
No. The ASA states that a p-value does not give the probability that a hypothesis is true, does not show that chance alone produced the data, and does not measure effect size or importance. Interpret it with the model, design, estimate, uncertainty, multiplicity, robustness checks, and scientific context.
Should every study use a power calculation?
No single sample-size procedure is appropriate for every design or objective. Record how the study size was determined, the target and inputs when a formal calculation was used, and the rationale when another approach was required. ARRIVE, for example, notes animal-study settings in which an a priori power calculation is not appropriate.
Can a reporting checklist validate the analysis?
No. CONSORT, STROBE, ARRIVE, ICMJE, and SAMPL can reveal what should be reported for their scopes. They do not certify that the design identifies the target, the model is correct, the data are unbiased, or the conclusion is warranted. Reporting completeness and statistical validity are separate judgments.
Cite this resource
Citation formats
Free to use and cite, with attribution. See the reuse policy for what you may republish.
- APA
- Manusights. (2026). Statistical analysis decision record. Edition 2026.09.22-analysis-1. https://manusights.com/resources/statistical-resources-guide
- MLA
- Manusights. "Statistical Analysis Decision Record." Manusights, 22 Sept. 2026, https://manusights.com/resources/statistical-resources-guide.
- Vancouver
- Manusights. Statistical analysis decision record [Internet]. 2026 Sep 22. Available from: https://manusights.com/resources/statistical-resources-guide