From graph to decision tree

Adopted controller v0.5.1 · adopted by the convenor on October 21, 2026 · room discussion pending · graph v0.4.2 is the vocabulary · source agents/controller/decision-tree.json

1. What the eight walks say about v0.4

  • Nodes held, edges did not. Every coded step (91 of 91) landed on a v0.4 node, but only 24 of the 83 transitions between steps follow an edge the graph draws (walks).
  • The loops are between modes, for coder 1. At the node level the papers return to a node already visited 11 times. At the mode level they return to a mode already used 27 times, in all eight papers; Xu et al. return eight times. Counts are from the coded steps in private/analysis/success-set/build/steps.py, consecutive repeats merged.
  • The blind coders disagree on where the loops sit. Two blind model coders (Opus 5.5 and Sonnet 5.5, paper codebook 1.3.0, October 25, 2026) cut finer steps: 169 and 108 against 91. They find 83 and 33 node returns against 78 and 38 mode returns. The between-modes reading is not yet coder-independent (final-paper sequences in private/analysis/reliability/codings/papers/, consecutive repeats merged).
  • Hypothesis assessment is a hub. The most frequent two-way hand-offs:
Pair of modes Hand-offs, both directions Papers
Hypothesis assessment ↔︎ theory 6 4
Hypothesis assessment ↔︎ simulation 5 3
Simulation ↔︎ robustness 5 2
Hypothesis assessment ↔︎ robustness 4 4
Hypothesis assessment ↔︎ estimation 4 4
Hypothesis assessment ↔︎ observation 4 3
Methods ↔︎ observation 4 2
  • Hypothesis assessment is one side of five of the seven most frequent pairs, for coder 1. For both blind coders the most frequent pair is estimation ↔︎ observation (17 and 13 hand-offs, in 6 of 8 papers each).
  • Shape, for coder 1. Every paper enters through a capability or record mode (methods 2, instruments 2, observation 3, estimation 1). The middle is a loop around hypothesis assessment. Four of eight leave through synthesis.
  • Shape, for the blind coders. They open 5 (Opus 5.5) and 6 (Sonnet 5.5) of 8 papers with a question step; after it, a capability or record mode comes first in 7 and 5 of 8. The last step is in synthesis for 2 and 0 of 8; hypothesis assessment is their most common last mode (3 of 8 each). Codebook 1.3.0 has no rule for the mode of a K step, so exit modes are soft.
  • What this cannot show. The order is the order of the paper’s account; the table and the walks are one reader’s coding (coder 1); supplements were not read for four papers. Section 6 tests that limit on nine papers with process documents.

2. Cleland’s schema is the record

  • The schema (Cleland 2001, 987–88): a hypothesis \(H\), with auxiliary assumptions \(A\) and conditions \(C\), implies an expected observation \(E\). A mismatch refutes the conjunction, not \(H\) alone (Duhem 1954).
  • A project’s record holds each term, the observations \(O\) that \(E\) is compared with, and the status of the claim. Every mode is defined by what it reads from the record and what it writes. the figure on the modes page draws the same table.
Term What it holds Modes that write it Nodes
\(Q\) The target: a question, an anomaly, a goal Question-driven inquiry; exploration; observation Q, E
\(O\) Observations with their provenance Observation; instruments; reconstruction (independent traces); cross-study synthesis (pooled results) N, O, E
\(O^*\) A latent quantity inferred from \(O\) through a forward model Estimation I
\(A\) Auxiliaries: instrument theory, forward model, processing, background laws Instruments; methods; theory; revised only at P N, I, P
\(H\) Rival explanations, with how each was generated Theory; hypothesis assessment; regularities (a relation proposed as a law) H, R
\(C\) Conditions, set by the investigator or found in nature Hypothesis assessment (set); reconstruction and regularities (found) D, F
\(E\) The expectation from each rival Theory (derivation); simulation (computation) S
Verdict Comparison, robustness, decision, status Hypothesis assessment; replication and robustness C, V, X, K
  • What the schema adds to the modes. It names what a mode may not do. Estimation writes \(O^*\) and cannot write \(H\); a resolved parameter is not a mechanism. Simulation writes \(E\) and cannot write \(O\). Robustness varies \(A\) and \(C\); when it revises one instead, that is P and is logged.
  • What the modes add to the schema. Cleland’s schema has no term for the capability to observe or for whether a phenomenon exists. Instruments, methods, and exploration produce those, and most of the coded advances began there (modes, 13 of 18 episodes).

3. Three levels

  1. The record: the schema terms above, with provenance.
  2. The modes: each runs an inner loop over graph nodes until its stopping rule holds, then hands back what it wrote.
  3. The controller: the decision tree in section 4, which reads the record and calls the next mode.
Mode Inner loop (nodes) Stops when Hands back
Observation O, E, O The record repeats across sessions or sensors; coverage limits stated \(O\)
Exploration E, O, E, varying conditions and representations A phenomenon repeats and survives a check of the chain (Bogen and Woodward 1988; Steinle 1997) \(Q\)
Instruments N, O, V Response known; an independent sensor agrees \(A\) and new \(O\)
Methods N, I, V on synthetic data Recovers synthetics; failure cases known \(A\)
Estimation I, C, I Misfit at noise level; resolution and uncertainty stated (Backus and Gilbert 1968; Tarantola 2005) \(O^*\)
Theory H, S A valid derivation; a distinct expectation per rival \(H\), \(E\)
Simulation S, C, S Converged; verified against a reference; forecasts frozen before the outcome (Charney et al. 1950) \(E\)
Regularities C, K, C out of sample Holds out of sample; range stated \(H\) as a relation
Hypothesis assessment H, S, D, F, O, C, X or K, R One rival left, or abstention naming the deciding observation (Platt 1964; Chamberlin 1965) Verdict
Cross-study synthesis O over many studies, C, K The pooled estimate is stable to removing any one study; dependence stated (Gurevitch et al. 2018) \(O\) pooled
Reconstruction from traces F found, O over independent traces, C Independent traces converge on one account (Cleland 2002) Verdict
Question-driven inquiry Q, then back to the start The reframed question names what would answer it \(Q\)
Robustness V over \(A\) and \(C\) Survives variation; or scope revised (Wimsatt 2012; Lloyd 2010) Status
  • The stopping rules are the “evidence owed” sections of the mode pages, shortened. Each is a hypothesis for the room.

4. The decision tree

Start: a question, an anomaly, data, a capability, or a goal
1
Is there a phenomenon?
It repeats, and it is not an artifact of the measurement chain.
no: the chain is suspect
Instruments · methods N
characterize the chain↺ back to 1
no: not yet stable
Exploration E, O
vary conditions until it repeats↺ back to 1
yes
2
Is the target observed directly?
Or is it a latent quantity behind the data?
no
Estimation I
invert through a forward model; state resolution→ then 3
yes
3
What is the aim?
A mechanism, a relation, a forecast, or a pooled estimate.
a relation
Regularities C, K
fit across repetitions; test out of sample→ to 7
a forecast
Simulation S, C
freeze the forecast; compare when the outcome arrives→ to 7
a pooled estimate
Cross-study synthesis O, C, K
state the inclusion rule; model dependence between studies→ to 7
a mechanism
4
Two or more rivals?
Incompatible explanations, with "artifact" kept as one.
no
Theory · hypothesis H
generate rivals; record how each was generated↺ back to 4
yes
5
Does each rival predict something different?
An expectation for each rival, from H + A + C.
no
Theory · simulation S
derive or compute each expectation↺ back to 5
still identical
A new observable N
a capability that separates the rivals↺ back to 1
yes
6
Can you set the conditions?
Experiment, or conditions found in nature.
yes
Hypothesis assessment D, F, O, C
design the test, set the conditions, observe, compare→ to 7
no: nature repeats it
Regularities F, O, C
nature's repetition→ to 7
no: one event
Reconstruction from traces F, O, C
independent traces of the same event; rival histories→ to 7
evidence in hand
7
Does the result survive variation?
Replication and robustness (V): vary the auxiliaries A and the conditions C; independent data, methods, models.
no: an artifact?
↺ back to 1
no: an auxiliary is wrong
Protect P
revise A instead of H; logged; must predict something new (cap: open)↺ back to 5
yes
8
Verdict
Keep, exclude, contradict: each is a conclusion.
survivors remain
Refine R
split the survivors into subhypotheses↺ back to 4
no rival fits
Surprise E
an outcome no rival predicted: explore it↺ back to 1
no conclusion
Question-driven inquiry Q
reframe the question; record why the first framing could not be answered↺ back to start
still none after reframing
Failure, recorded
name the observation that would decide
a conclusion: draft the paper
9
Does the claim survive review?
Co-authors, referees, editors, readers.
a new control asked for
Robustness V
run it; it usually enters as a planned-looking figure↺ back to 7
a new rival named
Hypothesis assessment H
test it, or state why it is excluded↺ back to 4
the claim is too broad
Refine the claim R
narrow it to what the evidence carries→ to Keep
the request is declined
Argue and log
keep the design; record the reason→ to Keep
accepted
Conclusion: keep, exclude, or contradict the claim, with its status, and publish the record, the path including its loops, and the evidence the mode owes.
Figure 1: The research decision tree, v0.5.1 (adopted by the convenor; room discussion pending). Questions about the record run down the left; each answer that sends the work elsewhere calls a mode (right, with the graph nodes it runs) and returns to the question marked ↺. Every route that produces evidence reaches question 7, robustness, before the verdict. A project fails only when no conclusion is reached after the question has been reframed. Source: agents/controller/decision-tree.json, v0.5.1.
Question Reads from the record If the answer sends work elsewhere Source of the rule
1. Is there a phenomenon? \(O\) and the chain in \(A\) Instruments or methods if the chain is suspect; exploration if the record does not yet repeat Bogen and Woodward (1988); Steinle (1997); Hacking (1983)
2. Is the target observed directly? \(O\) Estimation, which needs a forward model in \(A\) Backus and Gilbert (1968); Tarantola (2005); Bokulich (2020)
3. What is the aim? \(Q\) A relation goes to regularities; a forecast to simulation; a pooled estimate to cross-study synthesis; a mechanism to question 4 Charney et al. (1950); Cleland (2002), p. 485; Gurevitch et al. (2018)
4. Two or more rivals? \(H\) Theory and hypothesis generation, artifact kept as a rival Chamberlin (1965); Platt (1964)
5. Distinct predictions? \(E\) for each \(H\) Derivation or simulation; if still identical, a new observable, hence a new capability Platt (1964); Mayo (1996)
6. Can you set the conditions? \(C\) Experiment if yes; if not, regularities when nature repeats, reconstruction from traces when it does not Cleland (2001); Cleland (2002); Turner (2007)
7. Does it survive variation? Verdict, \(A\), \(C\) Back to question 1 if it may be an artifact; protection of an auxiliary, logged, and acceptable only if the revision predicts something new Mill (2011); Wimsatt (2012); Duhem (1954); Lakatos (1970) (the novel-prediction test; any numeric cap is the seminar’s)
8. Verdict Verdict Refine survivors and return to question 4; a surprise returns to exploration; no conclusion sends the question to question-driven inquiry for reframing, then back to the start; still none is a failure, recorded Klahr and Simon (1999), p. 537; Cleland (2002), pp. 481–482; O’Malley et al. (2010)
9. Does the claim survive review? The draft and its record A new control returns to 7; a new rival to 4; an over-broad claim is narrowed; a declined request is argued and logged The nine process documents in section 6
  • Success and failure (convenor, October 21, 2026). A project succeeds when it reaches a conclusion: a claim kept with its status, a rival excluded, or a claim contradicted. A contradicted claim is a successful result. A project fails when no conclusion is reached after the question has been reframed.
  • Stopping. A project ends at a conclusion or at a recorded failure. A kept claim carries a status the mode page defines: confirmed, best explanation among stated rivals, regularity within a range, estimate with resolution, capability with its characterization.
  • A paper is the record plus the path. The tree gives a paper’s outline: the question, the gates passed, the loops taken, and the claim’s status.
  • Reframing is question-driven inquiry (O’Malley et al. 2010; Elliott 2012), called after question 8 finds no conclusion, not before the tree starts.
  • What the tree leaves out: the choice of project, and resource limits, which decide in practice when a loop stops.

5. The eight papers on the tree

  • Mode sequences from the coded steps, consecutive repeats merged; “replication” in one Shapiro step is read as robustness.
Paper Mode sequence Where it loops Reading on the tree
Shapiro et al. (2005) Methods, robustness, observation, estimation, robustness, observation, methods Robustness, observation, methods each twice Question 1 by method and robustness, question 2 by estimation, then Keep a capability; never reaches question 4
Sun and Davis (2026) Instruments, observation, hypothesis, robustness, estimation, theory, simulation, robustness, theory, hypothesis, estimation, synthesis Theory, robustness, hypothesis, estimation Question 1 loops on drift versus signal; question 2 by estimation; questions 4 to 7 with forward models; ends in synthesis
Normandeau et al. (2025) Observation, hypothesis, estimation, theory, hypothesis, synthesis Hypothesis Coder 1 enters at the change seen between surveys (O); question 5 by theory; a protective move at question 7 (the ice lasts several years). Both blind coders enter at a question (Q); Opus 5.5 codes the multi-year-ice move as P, Sonnet 5.5 as R
Vine and Matthews (1963) Observation, estimation, hypothesis, simulation, hypothesis, theory, synthesis, simulation, synthesis, hypothesis Hypothesis with simulation, synthesis Questions 4 and 5 loop twice through block-model simulations
Rodell et al. (2009) Estimation, instruments, observation, estimation, hypothesis, simulation, hypothesis, robustness, estimation, synthesis Estimation three times; hypothesis Question 2 dominates; rivals (rainfall, soil water) excluded at questions 5 to 7
Lam et al. (2023) Methods, robustness, simulation, hypothesis, robustness, simulation, robustness, simulation Simulation and robustness, three times The forecast branch of question 3: verify, vary, verify
Gräff et al. (2025) Instruments, observation, exploration, estimation, theory, synthesis, theory Theory Question 1 by a new instrument and exploration; the melt-amplifier claim stops at theory without question 7
Xu et al. (2025) Observation, methods, observation, methods, robustness, hypothesis, observation, theory, hypothesis, observation, theory, hypothesis, theory, synthesis Observation and methods twice; hypothesis and theory three times Question 1 loops on back-projection calibration; questions 4 and 5 loop without an exclusion
  • Reading. The tree places all eight, and their loops are its loops: six papers trade between hypothesis assessment and theory or simulation (questions 4 and 5); three return among capability and record modes before any hypothesis (question 1: Shapiro, Rodell, Xu). The test that matters is whether unsuccessful papers walk it differently (section 6).

6. What process documents show

Count, nine papers Loops
Loops in the process documents 126 (61 real, 65 textual)
Returns in the final papers’ own sequences 7, all repeated C or V steps
Loops readable as loops in the final paper 4 (3%)
Content kept but written as if planned 70 (56%)
No trace in the final paper 52 (41%)
Requests declined and argued 30
Node the loop returns to Loops Real
V, a new control or robustness check 29 22
R, a claim narrowed 24 1
H, a new rival named 17 2
D, a redesigned test 11 5
C, a new comparison 10 6
  • What the final paper hides is order and failure, more than work. A reviewer-forced control usually survives as an Extended Data figure that reads as planned. Rivals named in review are rarely tested: 15 of 17 are argued away or become caveats. Narrowed claims leave no trace.
  • Earlier attempts leak only by accident. At least eight pre-submission loops surface in rebuttals and nowhere else, for example a first liquid-core hypothesis for Mars (Bi et al. 2025) and a deformation inversion rerun once pressure data arrived (Isken et al. 2025).
  • The literature agrees. Between protocol and paper, 62% of randomized trials changed, added, or dropped a primary outcome (Chan et al. 2004). 96% of standard psychology reports supported their first hypothesis against 44% of registered reports (Scheel et al. 2021). Preprint text changes little before publication, so preprints do not recover loops either (Klein et al. 2019).
  • For the tree. The loops are question 9 (review) and question 7 (robustness), the two places the published sequence flattens. Codings should be labelled “final-paper sequence”; where a review file exists, a second “review sequence” is coded, with the trigger (author, referee, editor) and the kind (real, textual) on each return.
  • Source: private/analysis/success-set/loops/; one coder per paper; papers chosen because their reviews were substantive, which raises the counts.

7. Loops in automated science

  • The automated systems loop too, but almost always inside one mode, holding \(H\) and \(A\) fixed and changing the conditions.
System Loop, in node letters Returns when Level
Strong inference (Platt 1964) H, D, F, O, C, X, R, back to H a rival is excluded inside hypothesis assessment
Robot Scientist (King et al. 2004, 2009) H, S, D, F, O, C, X, back to H (Sonnet 5.5 adds K; Opus 5.5 codes the 2009 loop with I and K) each cycle; a falsification inside hypothesis assessment; the model \(A\) fixed
Bayesian adaptive design (Rainforth et al. 2024) S, D, F, O, I, back to S (Opus 5.5) or to D (Sonnet 5.5) every observation, until the budget is spent inside one mode; \(H\) and \(A\) fixed
Coscientist (Boiko et al. 2023) optimization loop: D, F, O, C, R, back to D (Sonnet 5.5: H, F, O, C, back to H; its planner loop D, F, O, R); neither codes S a software error; the yield inside one mode
Co-Scientist (Gottweis et al. 2026) H, D, S, C, X, R, back to H, with no O (Opus 5.5 adds P and K, in another order) tournament results inside hypothesis generation, without new data; compares against the literature
Robin (Ghareeb et al. 2026) Q, F, H, D, humans run F and O, I, C, K data return from the laboratory across modes, with humans at F and O
The AI Scientist (Lu et al. 2026) D, F, O, C, R per step; H to K across runs an error trace or a critique inner and outer
FunSearch (Romera-Paredes et al. 2024) H, R, S, C, K or X, back to H every evaluation inside one mode
  • Sources. The Robot Scientist (recoded from the 2004 and 2009 full texts), adaptive design, Coscientist, and Co-Scientist rows are from the two blind loop codings of October 25, 2026 (private/analysis/reliability/codings/agents/, loop codebook 0.1.0); the other rows are from the literature report private/analysis/modes-literature/07-loops.md.
  • What they rarely do by themselves. Only Co-Scientist revises an auxiliary (P), and only for one of two blind coders (Opus 5.5: deep verification keeps the core hypothesis when an error is not fundamental). None of the five coded systems here answers question 1 (is there a phenomenon); of all 31 coded systems, SWE-agent does for both coders, by reproducing the bug first. None is described revising the set of rivals after a surprise. Adaptive design “only capture[s] uncertainty within the model” (Rainforth et al. 2024, preprint p. 9).
  • Practice in laboratories: scientists faced with inconsistent data first protect the method, then repeat, and only then revise the hypothesis (Fugelsang et al. 2004). That is the path C, P, V, then H, which the tree’s question 7 allows.
  • For agents. A mode agent can run its inner loop as these systems do. The controller, which these systems lack, is what the tree adds: it decides when to leave a mode, when to doubt the phenomenon, and when to protect or exclude.

8. What would show the tree is wrong

  • Same route, different outcomes. If the matched controls and the low-cited Nature and Science papers pass the same questions in the same order as the successes, the tree describes papers, not success.
  • Missing questions. If peer-review files, preprint versions, or lab records show loops returning to a point the tree lacks, the tree needs that question. Reframing was added this way, at question 8, on October 21, 2026.
  • No gain for agents. If agents that follow the tree do no better on held-out cases than agents that choose modes freely, the tree adds nothing for agents.
  • Unobservable gates. If two coders agree on which question a step answers less often than κ = 0.6, the questions are not observable in papers.
  • Limits of the present evidence, from the walks and section 6:
    • Order is the paper’s narrative, not the work’s; 3% of recorded loops are visible in the final paper.
    • The walks are one coder’s, coding against the graph it tests. Two blind model coders agree on node use at alpha 0.63, below the 0.67 gate (private/analysis/reliability/RESULTS.md); no person has coded yet.
    • The walks’ coding (codebook v0) had no field for a failed attempt, so a return and a next step look alike. Codebook 1.0.0 added a move on every step.
    • Supplements unread for four papers.
    • Mode labels inconsistent in one case.
    • Eight successes as walks; five matched controls coded since; no failures.

9. Decisions and change log

Date Decision By
October 21, 2026 Graph v0.4.2 is the vocabulary; this tree is the controller Convenor
October 21, 2026 Reframing the question comes after a failure to conclude Convenor
October 21, 2026 Failure means no conclusion; a contradicted claim is a success Convenor
October 21, 2026 Synthesis split; question-driven inquiry added as a mode Convenor
Open A numeric cap on protective revisions Undecided
Version Date Change
0.5.0 October 21, 2026 First tree: eight questions over the record; review added as question 9 after the process-document test
0.5.1 October 21, 2026 Adopted as controller; reframing after no conclusion; outcomes defined; synthesis split across questions 3 and 6b; protection cap left open
  • The tree’s source is agents/controller/decision-tree.json; tools/build_tree.py writes the figure, and the validator fails if the page and the file differ. Every change gets a version and a row here.

10. For the room

  1. Is question 3 (the aim) asked before question 4, or does a mechanism question come first and the aim follow?
  2. Does the seminar want a numeric cap on protective revisions, beyond Lakatos’s novel-prediction test?
  3. Which stopping rule in section 3 is wrong for your own last paper?