What review changes, and what the paper says

Drafted v0.3 · October 27, 2026 · round 1, 58 open-review papers · paper codebook 1.5.0 · model-coded and model-labelled, no human check yet

• Question. When referees ask for changes and authors make them, does the published paper show that the work changed in review?
• Answer, 58 papers with open review, 33 of them in The Cryosphere. No. None of 580 changes made during review is marked as a revision in the published paper. 97% are in the paper, written as if planned; 3% are not in the paper at all.
• For geoscientists. The checks that referees ask for (sensitivity tests, comparisons with independent data or methods, reruns with other parameters) are the kind of change most often promised in a reply and not found in the paper.
• For NLP researchers. In these 58 papers the published text contains the changes made in review but never marks them as such. A model or agent that learns research practice from papers like these sees the outcome of review written as the original plan.
• Scope. Codings with quotes, derived tables and figure provenance are public in the data release, data/roadmap-release/. Paper texts, abstracts and review texts are not redistributed. Per-paper records of the papers in human coding are held back until that coding is done. Per-paper bars in panel c of the results figure are anonymous and unordered by journal.

Corpus

• Source. Round 1 of the roadmap corpus: 525 eligible 2022 papers, 374 coded from full text (see workflows and outcomes and topics and journals). The 38 Copernicus papers of round 2 have not been through this analysis.
• Copernicus journals. 64 of the 374 are in journals published by Copernicus (DOI prefix 10.5194). All but two are journals of the European Geosciences Union (EGU); European Journal of Mineralogy and Scientific Drilling are published by Copernicus for other societies and have no open discussion, which is why 3 of the 64 have none. 61 have an open discussion; 58 have a review sequence that passed the format check.
• Open discussion. In EGU journals the submitted manuscript is posted as a citable discussion preprint. Referee comments, author replies, community comments and editor comments are posted beside it, and the revised paper is published separately and links back to it (Pöschl 2012). The record works like OpenReview for geoscience. Across the 61: 134 referee comments, 147 author comments, 6 community comments and 4 editor comments.
• Journals. The Cryosphere (ice and snow) 33; Natural Hazards and Earth System Sciences (NHESS, hazards), Geoscientific Model Development (GMD, model code), Hydrology and Earth System Sciences (HESS, hydrology) and Biogeosciences (ecosystems and biogeochemistry) 4 each; Atmospheric Chemistry and Physics (ACP) 3; Earth Surface Dynamics (ESurf, landforms) and Earth System Science Data (ESSD, data sets) 2 each; Geoscientific Instrumentation (GI) and Solid Earth 1 each. The result mostly describes one journal.

Method

• Coding. Claude Opus 5.5 (claude-opus-5-5, Anthropic’s asynchronous batch mode, effort high, no temperature set) read the published paper (PDF and publisher XML text) and the open discussion, and wrote the review process as a sequence of steps on the graph of inquiry. A node is the kind of move (15 types, for example V vary or replicate); a mode of inquiry is the kind of epistemic work the step serves (13 types, for example estimation or reconstruction from traces); both are defined on workflows and outcomes. Each step has a node, a mode of inquiry, an action, a location in the review record, a quote of at most 25 words, a move (forced by review, declined, return after failure) and a trigger (referee, editor, commenter, author). $26.76 for 58 papers.
• What the coder read. The discussion comments (referee, author, community and editor) with the supplements attached to them. The tracked-changes manuscripts, author-response PDFs and manuscript versions were downloaded for all 58 papers but not given to the coder, and neither was the article’s own supplement. Because the coder also read the final paper, it may have phrased steps from the final text, which would favour “as if planned” over “not in the paper”.
• Labelling. Two model labelers, separate runs of a Claude model inside the Claude Code tool (model id not recorded), each put every step in one of four classes without seeing the other’s labels. A third run settled disagreements. Coder, labelers and adjudicator are all Claude models, so their errors may be correlated.

Label	Rule
Marked as a revision	In the paper, and the paper ties it to review. A generic thank-you to referees does not count.
As if planned	In the paper, with no sign it came from review.
Not in the paper	A change described in review that the published paper does not contain.
No change	A declined request or a reply that only argues.

• What the labelers searched. The publisher XML text of the final paper. Its converter keeps body text and captions; only 12 of the 58 texts include an acknowledgements section, and none includes the supplement. A change moved to the supplement would be labelled “not in the paper”.
• Checks. Every “in the paper” label carries a quote from the published text, matched by script against that text. Papers, not steps, are the unit for intervals.

Results

Review steps and their visibility in the published paper (a), share of changes absent by graph node (b), and changes per paper (c). Round 1, 58 open-review papers.

• Papers marking any review change: 0 of 58. One-sided 95% upper bound: 5.2% of papers.
• Independent check by script. In the published PDFs, 53 of 58 papers mention reviewers or referees, every time in a thank-you or in the journal’s review statement; none uses a phrase that ties content to review (“as suggested by the reviewer”, “in the revised manuscript”, and similar). The labelers’ XML texts contain no mention at all, because most back matter is missing from them. The zero therefore holds on the full PDF text, for the phrases searched.
• Not in the paper: 20 of 580 changes, 3.4% (95% interval 1.7% to 5.6%, bootstrap over papers); 13 of 58 papers have at least one. The article supplements were not checked, so some of these may sit there.
• What goes missing: V changes (sensitivity tests, comparisons with independent data or methods, controls, reruns) are the most common request and the most often missing: 13 of 102 not in the paper, against 7 of 478 for all other nodes. The 13 absences fall in 10 papers. With papers as units (absences shuffled within each paper, Benjamini-Hochberg over 14 nodes), V exceeds the other nodes by 11 percentage points (interval 4 to 20, q = 0.003); no other node differs. Exploratory: V was singled out after looking.
• Who asked: 729 of 746 steps were triggered by referees, 15 by community commenters, 2 by the authors, none coded as triggered by the editor. All 20 absences follow referee requests.
• Refusals that still changed the paper: 60 of 220 declined requests left a trace in the published text, usually a caveat or a justification.
• Labeler agreement: 720 of 746 steps (96.5%) before adjudication; Cohen’s kappa 0.91 (agreement corrected for chance, see Measures).
• Quote existence: 560 of 560 quotes are found verbatim in the published text. This shows the quotes exist, not that each shows the change its label claims; no entailment check (does the quote support the label?) has been run. 44 of the 560 quotes exceed the 25-word limit the labelers were given (7.9%), an instruction-following failure.
• October 21 loop pilot, 9 papers with review files (Nature peer-review files, EGU discussions, one preprint revision; a different pilot from the 10-paper labelling pilot of this experiment): 4 of 126 loops readable as loops, 70 as if planned, 52 absent. Same direction. The larger absent share probably reflects construction: the pilot counted analyses discussed in review files that never entered any version.

Reading the result

• For geoscientists. A convention, not misconduct. EGU journals put the whole discussion on the record, on a citable discussion preprint that the final paper links to (Pöschl 2012); the history exists, just not in the paper. Unless the journal publishes review files, readers of closed-review journals have no such record; some closed-review journals do publish them after acceptance (Nature Communications, for example).
• For NLP researchers. A loop designed from methods sections sees no review loop. V is among the steps the coded agent loops rarely run: 10% of 31 agent systems against 98% of round-1 papers, with Q (question) also at 10% and I (infer) at 5%, H (hypothesize) at 6% and P (protect) at 0%. Refinement (R) goes the other way: agent loops run it more often than papers (68% against 55%). The 31 systems were hand-picked for a comparison talk (14 science agents, 8 general agent patterns, 3 multi-agent, 3 coding, 2 deep-research systems, 1 workflow pattern), coded from their papers and documentation under a separate loop codebook (0.1.2) by two model coders, a Claude Opus run and a Claude Sonnet run. The comparison is between descriptions, not executed runs.
• The 31 agent systems. ReAct; Reflexion; Self-Refine; Plan-and-Solve; Tree of Thoughts; LATS; Voyager; BabyAGI; AutoGen, CAMEL and ChatDev (one entry); MetaGPT; Magentic-One; the evaluator-optimizer pattern; coding harnesses (Claude Code, OpenAI SDK); SWE-agent; OpenHands; deep research (OpenAI, Gemini); STORM; Agent Laboratory; Kosmos; Biomni; Virtual Lab; data-to-paper; AIDE; DiscoveryWorld Hypothesizer; Robot Scientist (Adam); Coscientist; Co-Scientist; Robin; The AI Scientist; FunSearch and AlphaEvolve (one entry); Bayesian adaptive design.

Node use in 374 coded round-1 papers (round-2 papers as open circles) against 31 AI agent systems (a), distinct nodes per workflow (b), and common researcher transitions that no agent loop runs (c).

Measures

• Review step. One change or refusal recorded in the open discussion, coded as one node on the graph of inquiry.
• Change to the work. A review step with any label other than “no change”; the denominator for all shares.
• Absent share. Changes labelled “not in the paper” divided by changes; interval is the 2.5 and 97.5 percentiles of 5000 bootstrap resamples of papers.
• Upper bound on marking. With 0 of n papers marking a change, the one-sided 95% upper bound on the rate is about 3/n, the rule of three (Hanley and Lippman-Hand 1983).
• Per-node share interval (panel b). Wilson 95% interval (Wilson 1927), steps treated as independent, not adjusted for the 14 comparisons. The paper-level test above replaces it for V.
• Labeler agreement. Share of steps where the two labelers gave the same label before adjudication.
• Cohen’s kappa. Agreement between the two labelers corrected for the agreement expected by chance from each labeler’s label frequencies; 1 is perfect, 0 is chance.
• Quote existence. Share of “in the paper” labels whose quote is found verbatim in the normalized published text.
• Agent node share. Share of 31 AI agent systems whose loop (the cycle of steps the system repeats) runs a node, mean of the two model coders (loop codebook 0.1.2).
• BH q. Benjamini-Hochberg adjusted p (Benjamini and Hochberg 1995); defined on workflows and outcomes.

Limits

• Coding, labelling and adjudication by Claude models; agreement between runs of one model family shows stability, not correctness, and errors shared by the runs do not show. No human check and no model from another family yet.
• The coder read the final paper as well as the review record, which can only raise “as if planned”. An ablation that codes the review sequence from the review record alone has not been run.
• The labelers searched the publisher XML text, which lacks most acknowledgements and every supplement. The PDF string check covers acknowledgements for the phrases searched; supplements remain unchecked.
• No recall test: the labelers have not been shown papers with known revision markings, so their ability to find one is assumed, not measured.
• Possible contamination: the papers (2022) and their public discussions predate the model’s knowledge cutoff (June 2026), so the model may recall them.
• 58 papers, one year, 33 from The Cryosphere, all open discussion. Closed-review journals may differ in either direction.
• Strict rule for “marked”: generic acknowledgements not counted.
• Three papers dropped at the format check: two for quotes over 25 words, one for a schema error.

Reproduce

• Code (this repository): tools/code_papers.py, tools/roadmap/g15_open_review.py, tools/roadmap/g16_visibility.py, tools/roadmap/workflows/visibility.js, codebook agents/coder/ 1.5.0; the string check, trigger counts and paper-level node test in tools/roadmap/g19_report_numbers.py; figures tools/roadmap/figures/fig06_agents.py, fig07_review.py.
• Data and codings: data/roadmap-release/, data/roadmap/derived/review/ (open-review index, visibility labels and results) and analysis/codings/api/calc1-review/. The review texts themselves are not included.

Benjamini, Yoav, and Yosef Hochberg. 1995. “Controlling the false discovery rate: A practical and powerful approach to multiple testing.” Journal of the Royal Statistical Society: Series B (Methodological) 57 (1): 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.

Hanley, James A., and Abby Lippman-Hand. 1983. “If nothing goes wrong, is everything all right? Interpreting zero numerators.” JAMA 249 (13): 1743. https://doi.org/10.1001/jama.1983.03330370053031.

Pöschl, Ulrich. 2012. “Multi-stage open peer review: Scientific evaluation integrating the strengths of traditional peer review with the virtues of transparency and self-regulation.” Frontiers in Computational Neuroscience 6: 33. https://doi.org/10.3389/fncom.2012.00033.

Wilson, Edwin B. 1927. “Probable inference, the law of succession, and statistical inference.” Journal of the American Statistical Association 22 (158): 209–12. https://doi.org/10.1080/01621459.1927.10502953.