Assessing scientific advance: novelty, correctness, and impact
Meeting 12 of 13
Drafted v0.4 · not yet held · record discussion
Central question
How do we assess a warranted gain in knowledge, understanding, or capability, using novelty, correctness, and impact without reducing advance to one proxy?
Anchor and companion
- Everyone reads the selected anchor sections and the companion abstract/overview. The rotating reader presents the companion in depth.
- Pairing: Empirical indicator / methodological reanalysis. Full records and access notes: bibliography.
Anchor
Wu, L., Wang, D., & Evans, J. A. (2019). Large teams develop and small teams disrupt science and technology. Nature, 566, 378–382. https://doi.org/10.1038/s41586-019-0941-9
Read Wu’s main argument, definition of disruption, and discussion.
Companion
Petersen, A. M., Arroyave, F., & Pammolli, F. (2025). The disruption index suffers from citation inflation: Re-analysis of temporal CD trend and relationship with team size reveal discrepancies. Journal of Informetrics, 19(1), 101605. https://doi.org/10.1016/j.joi.2024.101605
Read Petersen’s main critique and conclusions. Dellsén is optional philosophical framing, not a third required paper.
Why these readings belong together
Wu distinguishes developing and disruptive citation patterns and reports associations with team size. Petersen challenges how citation inflation and reference-list properties affect such inferences. Their disagreement concerns measurement as well as interpretation; neither supplies an uncontested scalar measure of scientific progress. (Wu et al. 2019; Petersen et al. 2025)
Prepare and discuss
Read the selected anchor sections and the companion abstract or overview. Bring one source passage or artifact relevant to the case; the rotating reader presents the companion in depth.
- Can a calibration, correction, archive, or replication be a major advance without an original theory?
- Is a “big leap” an intrinsic property or a retrospective judgment about consequences?
- How do entry, authority, infrastructure, standards, and funding shape a field?
Case exercise: scientific gain and epistemic mode
Assess a celebrated contribution and a modest enabling/corrective one before revealing citations. Record meaningful gains, novelty, warrant, and then realized impact. Calibrate judgments without ranking contribution roles as importance levels.
- Profiles to examine: robustness, synthesis.
- Record source kind and unknown chronology; distinguish documented practice, philosophical argument, association, and our proposed agent rule.
- Ask what was gained, what warrants it, which action helped, and what a comparable agent would need to demonstrate.
The agent
- After the meeting, say what these papers change in the unit skill, agents/skills/unit-of-inquiry/SKILL.md: a rule at a node, a new composition of units, or a planted flaw. Meeting 1’s page shows the form.
- Modes exercised here, whose “agent actions” sections the rule would enter: robustness, synthesis.
- Written from the texts after the papers are read in full, as for meeting 1; nothing here yet.
Optional extensions
- Sinatra et al. (2016): Quantifying the evolution of individual scientific impact. Science (2016). The highest-impact paper falls at a random position in a career once productivity is accounted for; a constant per-scientist factor Q does the rest. What “optimize for advance” can and cannot mean at the career level.
- Leibel and Bornmann (2024): What do we know about the disruption index in scientometrics?. Scientometrics (2024). The review of the index’s variants, its time-sensitive and data-induced biases, and its validity; read it with Petersen before trusting any CD number, including the ones the seminar computes for its own papers on the metrics page.
- Dellsén (2016): Scientific progress: Knowledge versus understanding. An understanding-based account of progress to compare with knowledge and capability gains.
- Azoulay et al. (2019): Does Science Advance One Funeral at a Time?. Publisher offers complimentary full-text PDF.
- Wang et al. (2017): Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Publisher/DOI page; full text may require institutional access.
- Park et al. (2023): Papers and patents are becoming less disruptive over time. Nature (2023). The claim Petersen’s critique responds to, at the scale of the whole literature. Read the two together with Wu.
- Chu and Evans (2021): Slowed canonical progress in large fields of science. Proceedings of the National Academy of Sciences (2021). Larger fields entrench their canonical papers and make it harder for a new idea to gain attention. Bears on whether an agent that adds papers adds progress.
- Bloom et al. (2020): Are Ideas Getting Harder to Find?. American Economic Review (2020). Research productivity has fallen across sectors as research effort has grown. The economists’ statement of the problem the seminar’s agents are supposed to help with.
- Azoulay et al. (2011): Incentives and creativity: evidence from the academic life sciences. An observational funding-program comparison with identifying assumptions, not an isolated causal test of tolerance for failure.
- Bird (2007): What Is Scientific Progress?. The knowledge-accumulation account of scientific progress, compared with Dellsén’s account of understanding.
Record after the meeting
Sign and date the notes; preserve disagreements and what changed your assessment. Keep confidential examples in private notes.