Methods · Volume 1, Number 5 · November 2, 2026

Our commitment to reporting our measurements

1. Why build a clock that tells on you

The scientific knowledge upon which this publication is based is changing, including how certain that knowledge is. I want to keep the published writing up to date, and to do that I need to measure, periodically, not only the correctness of our writing against our corpus of held research findings, but also the changing state of the scientific knowledge that corpus represents. Today we measure only the first; the second is not built yet. Either way, a measurement is true on the day it is taken and goes stale after that, so it has to be redone.

A schedule I keep only for myself is a promise nobody can check. So the software calculates the time since our last report, and when it has been too long, it says so on our Methods page, where any reader can see it.

2. Why it had to be seen to fire before it could be trusted

The clock itself is a measurement. Based on that measurement, a notice appears on the Methods page when it exceeds a defined amount.

We’re confident the clock is correct (it produces true readings against known references) because we have software unit test assertions for different input values.

It is a tenet of metrology that a measuring system must have sensitivity; changes in something real must produce a predictable change in the measurement. We’re confident the clock has sensitivity, because those same tests move the date by a single day across the limit, and the notice changes from silent to shown. The elapsed-time measurement is also included in the published report of the measurements, so with each new report a reader can independently check that it moves with the publication dates of the Methods issues carrying those reports.

Finally, to prove the notification works on the live site, we waited for the limit to be exceeded and for the notification to appear as expected.

3. Visible to a reader vs. merely produced

A report can only count if you can read it. Every Methods issue carries one date: the moment it first appeared on this site. That date is recorded once (never changed, even if we correct an issue later) and it is the date the clock counts from.

4. A passing code test suite doesn’t guarantee a trustworthy number

A passing test suite ensures two things: that unit tests pass and that mutation tests pass. A unit test checks that some portion of the code meets its specification. Mutation testing checks that the unit tests would notice if the code stopped meeting its specification. In metrology that is called verification: “provision of objective evidence that a given item fulfils specified requirements” (VIM 2.44).

However, neither unit nor mutation testing checks whether the specification itself is right, meaning whether what the code computes is what its label claims to measure. Both are checks of the instrument against itself. Metrology names two more pieces needed for a number to be trustworthy: Validation asks whether the requirements were the right ones for the intended use (VIM 2.45). Calibration answers from outside the instrument: You check the instrument’s readings against values that are already known (VIM 2.39).

Trusting any number requires all three.

5. No silent passes

A silent pass is interpreting silence from a tool as success. The trouble is that silence can mean two different things.

A tool might produce no output having crashed, skipped some inputs, or failed to reach its data source. A tool might also produce no output having found nothing wrong. From outside, the two look the same, and reading the first as the second is a silent pass: a failure reported as success for simply having output nothing.

We avoid silent passes by treating silence as an error. A tool that found nothing wrong must say so; a tool that says nothing has failed.

6. What we intend to measure

We intend to measure how much of what we tell readers is true.

Metrology calls the “quantity intended to be measured” the measurand (VIM 2.3). Our measurand is then more specifically: of the claims a reader actually meets on our pages, excerpts and emails, what proportion are true as stated, and no stronger than the cited evidence supports? (A claim stated more strongly than its evidence supports counts as an error, just as a false claim does.) Everything else we report serves that number. For example: how many of our cited sources we hold in full, so a claim can be checked against them; how many days have passed since our last quality report; and the defects we have found and disclosed but not yet fixed.

7. The thing measured vs. the thing named

At this writing, what we measure is not exactly what we named in §6. Metrology expects this: “the quantity being measured may differ from the measurand as defined” (VIM 2.3, Note 3). When it does, the honest move is to say how.

Our main instrument is the quarterly resample. It takes 20 of our pages and has them fact-checked a second time, blind, without access to the first check, and counts how often the two checks disagree. That measures agreement between two careful checkers, not truth directly. It is still worth having. When two checkers working blind disagree about a claim, at least one of them is wrong, so disagreement is evidence of error. Agreement, though, is not proof of truth: both can be wrong the same way (§13).

The resample currently only covers our companion pages. Excerpts, emails, and other surfaces which readers also see are not yet sampled.

8. Error vs. uncertainty

“Error” means two different things here. The errors we look for are mistakes: a claim we published that is wrong. Metrology’s measurement error is something else: “measured quantity value minus a reference quantity value” (VIM 2.16), meaning how far our count of mistakes is from the true count. Metrology itself warns against confusing the two. To know our measurement error we would need a reference: the true number of mistakes, known some other way. We have no such reference, so we never claim to know how wrong our count is.

What we can state is uncertainty: “non-negative parameter characterizing the dispersion of the quantity values being attributed to a measurand, based on the information used” (VIM 2.26).1 That’s how much our count could reasonably vary given what we know, such as how small our sample is and how often our checkers disagree.

9. Census vs. sample

A census counts every item. A sample counts some items and estimates the rest. Which of those is the source for a number determines what kind of uncertainty that number carries.

A census has no sampling uncertainty. When a program reads every one of our pages and counts how many declare their references, the number is not an estimate. Putting a range on it would claim an uncertainty it does not have. The only uncertainty in the number comes from how sharply the thing counted is defined.

A sample always carries sampling uncertainty. Our quarterly resample currently checks 20 of our 500 companion pages. From there it estimates for all of the companion pages. Its result comes with a range, and with only 20 pages sampled that range is wide.

Today, every count in our published measurement is a census, taken fresh when that report is generated. The resample is our one sample.

10. Whose denominator

A count means little until you say what it is out of. “486” tells a reader nothing. “486 of 500 pages” tells them something, but only if they know which 500 pages. That bottom number is a choice.2 It might be every file we hold, every page a reader can be served, or our pages weighted by how often they are actually read. Each of those choices gives a true number (“486 of 500 [specific description] pages”), and each answers a different question.

A wrong denominator is dangerous because it looks more rigorous than no denominator at all. We learned this from our own tests. When we deliberately changed one of our denominators to a plausible wrong one, every test still passed. A number over a plausible but wrong denominator reads as more careful than a bare count while saying something false.

Our intention is for every number we publish in our measurement reports to clearly specify its denominator, so our readers can tell exactly what question it answers.

11. What counts as a “claim”

Our measurand (§6) is a proportion of claims, so what counts as a claim decides the number.

We count by sentence. Every sentence a reader sees either makes a claim about the world or it doesn’t. Some claims don’t look like claims: a short saying that compresses a finding, an instruction that assumes something is true, or an excerpt whose selection implies more than the full page says. All of those count as claims.

Deciding whether a sentence makes a claim is a judgement; no fixed rule can do it for us. So two careful checkers (LLMs or people) may judge the same sentence differently. Metrology names the resulting spread definitional uncertainty: “component of measurement uncertainty resulting from the finite amount of detail in the definition of a measurand” (VIM 2.27). It is “the practical minimum measurement uncertainty achievable”. No better instrument gets below that uncertainty. Only changing the definition of the measurand can change the uncertainty.

12. What “three months” means

The notice on our Methods page says “more than three months”. Three calendar months can be anywhere from 89 to 92 days, so we don’t count months. We count days. A threshold is a definition, not a measurement. Writing it down as one exact number means anyone can check when we are late.

The notice appears when more than 91 days (thirteen weeks) have passed since our last quality report was published. On day 91 it stays silent; on day 92 it appears. Before our first quality report, the count runs from 2 August 2026, the day I made the commitment.

13. Agreement isn’t truth

When two checkers agree about a claim, they are consistent. That is not the same as being right. Metrology keeps the two apart. Precision is “closeness of agreement between indications or measured quantity values obtained by replicate measurements” (VIM 2.15). Trueness is closeness to “a reference quantity value” (VIM 2.14), and we have no such reference (§8).

Agreement would point toward truth only if our checkers made their mistakes independently.3 Ours don’t: they work from the same instructions, the same categories and the same house style. Checkers built alike tend to make the same mistakes,4 and a shared mistake looks exactly like agreement.

So rising agreement could mean we are getting better, or that our checkers are growing more alike. To tell which, we read agreement plus a second number: how many claims, checked against their sources, turn out to be false. If agreement rises while that number does not fall, our checkers are only converging, not actually improving.

14. An answer known independently of the instrument

The only way to learn whether an instrument is right is to run it on cases where we already know the right answer, by some route other than the instrument itself. That is the check from outside that §4 called calibration.

We need two kinds of known case. Some contain a mistake that has already been caught and confirmed, and the instrument must flag them. Others are known to be clean, and the instrument must pass them. Without the clean cases, an instrument that flagged everything would look perfect.

Our rule is that every instrument runs against at least one such case every time it changes. Most of our instruments don’t meet that rule yet. The hardest is our main one, the resample: for “is this claim true?” we have not yet found a known answer that doesn’t come from another checker.5

15. What the instruments can’t see

A clean result means only that an instrument found nothing where it looked. So beside every clean result we owe you a list of where it didn’t look. A blind spot we name is one you can allow for. A blind spot we don’t name turns “nothing found” into false reassurance.

These are the blind spots we know of today:

  1. The resample reads only our companion pages, not our knowledge files, excerpts or emails (§7).
  2. It measures whether two checkers agree, not whether a claim is true (§7, §13).
  3. Our checkers share instructions and habits, so a mistake they all make is invisible to all of them (§13).
  4. Nothing yet watches the science itself change: no check looks for retracted papers, or for new findings that overturn what we cite.
  5. A sentence nobody recognised as a claim escapes every check (§11).

The list will grow as we learn, and keeping it is the point.

16. Why quarterly

I could have simply automated the display of our measurements on a live-updated dashboard. But then there’d be no clear history of the evolution of our application of metrology, and of our measurements.

Instead I wanted to be in the loop of generation of the measurements and their publication. I wanted to have to write an issue of Methods with each quality report because it forces me to show our work. In each such issue I plan to touch on what has changed with the entire project. I expect we’ll be adding measurements to the report and it’s an opportunity to explain what’s been added.

I picked quarterly as a balance between keeping our most recently reported metrics fresh for readers or researchers who look, and my ability to keep up with publishing the necessary issues of Methods. Quarterly is also the fastest we could honestly go: a report can’t come more often than the measurement it reports, and the resample runs quarterly.

17. What a breach does

The resample has limits. Of the claims it checks against their sources, no more than 5% may turn out to be false overall, and no more than 2% of the claims a page’s advice depends on. A resample beyond either limit is a breach. The rule for a breach is automatic and strict: our public claim about the quality of our work is suspended until the numbers are back within the limits.

Our first resample, on 9 August 2026, set the baseline, and by rule a baseline run cannot breach. Its result is still worth stating plainly. Overall, 2.3% (±1.9) of the claims it checked were false, inside the 5% limit. On the claims a page’s advice depends on, it found 6.6% (±6.2), well outside the 2% limit. The next resample is read against those numbers.

(A late report is a different failure, and §12 describes its notice.)

18. Why there’s no one else’s number to beat

Why is there no one else publishing measurements like ours?

I don’t know.

I don’t know anything about the internals of any other publication’s processes. Outside researchers have measured newspaper accuracy, for example by sending published stories back to the sources cited in them,6 but I know of no publication that measures its own and publishes the result. I do know that this publication is tiny compared to others which cover the same fields. But I do know the complexity of applying best-in-class metrology practices to this tiny publication, and I can only imagine how hard it would be to do so at a larger scale.

Why is there no number to beat?

The reason for applying metrology is self-assessment: I want to know if this publication is true, and if it is improving in quality. I publish our metrology because it shows readers how we check our work, how far those checks reach, and where they still fall short.

A final note

How I wrote this issue: I have a cogitant who is my tutor for the field of metrology. I asked it:

I need to write a Methods issue about our metrology… DONT write me any actual copy (that’d be doing my homework for me) but write me a bullet list of the things you think I should touch on (dont go into explanation in them— it’s also a popquiz for me)

The section headings above began as exactly what it generated. It then reviewed every section with me as I wrote, and drafted many of them for me to revise.

ɕ

  1. Joint Committee for Guides in Metrology, Evaluation of measurement data — Guide to the expression of uncertainty in measurement (JCGM 100:2008, “the GUM”), available from the BIPM. The international rules for evaluating and stating uncertainty, including the two routes the VIM’s definition refers to: from the statistics of repeated measurements (Type A), and from other information such as experience (Type B). ↩︎
  2. Gerd Gigerenzer, Wolfgang Gaissmaier, Elke Kurz-Milcke, Lisa M. Schwartz and Steven Woloshin, “Helping Doctors and Patients Make Sense of Health Statistics,” Psychological Science in the Public Interest 8, no. 2 (2007): 53–96, doi:10.1111/j.1539-6053.2008.00033.x. Finds that doctors, patients and journalists widely misread health statistics, often because of how the numbers are framed. Recommends formats that keep the population in view: absolute rather than relative risks, frequency statements rather than single-event probabilities, and natural frequencies rather than conditional probabilities. ↩︎
  3. Pamela M. Vacek, “The Effect of Conditional Dependence on the Evaluation of Diagnostic Tests,” Biometrics 41, no. 4 (1985): 959–968, doi:10.2307/2530967. When two diagnostic tests are judged against each other because no error-free reference exists, the usual method assumes their errors are independent. If they are not, the error rates of both tests can be substantially underestimated. ↩︎
  4. Elliot Myunghoon Kim, Avi Garg, Kenny Peng and Nikhil Garg, “Correlated Errors in Large Language Models,” Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267: 30038–30066, proceedings (arXiv:2506.07962). Across more than 350 language models, models that shared an architecture or provider erred together more often, and so did larger, more accurate models even across providers. On one benchmark, when two models were both wrong they gave the same wrong answer 60% of the time. The authors draw the consequence for using one model to judge another. ↩︎
  5. Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz and Shin Yoo, “The Oracle Problem in Software Testing: A Survey,” IEEE Transactions on Software Engineering 41, no. 5 (2015): 507–525, doi:10.1109/TSE.2014.2372785 (free copy, UCL). Software testing’s name for this gap is the test oracle problem: telling a system’s correct behaviour from incorrect behaviour for a given input. The survey reviews ways to automate the answer, including specifications, contracts and metamorphic testing. When none is adequate, the authors note, the final source of right answers remains a human. ↩︎
  6. Scott R. Maier, “Accuracy Matters: A Cross-Market Assessment of Newspaper Error and Credibility,” Journalism & Mass Communication Quarterly 82, no. 3 (2005): 533–551, doi:10.1177/107769900508200304. Surveyed 4,800 people cited as sources in fourteen US newspapers. They found errors in 61% of local news and feature stories, among the highest rates in decades of accuracy research. The newspapers’ credibility in the sources’ eyes fell as errors grew more frequent and more severe. ↩︎

That was Craig’s account. Now read the room’s

A notice that appears on its own

This morning, on the front page of this journal, a notice appeared that nobody wrote today. It says our quarterly quality report is overdue, counts the days since the commitment was made, and explains that it will disappear when the report is published. Nobody decided to show it. The site read the date, read the record of what has been published, found no report, and put the notice up. This issue is the page that notice points to. It carries no numbers, by design. It says what the commitment is, why a publication would build a clock that tells on it, and what the report will contain when it comes.

The commitment

We publish a report on the quality of our own published work at least once a quarter, measured. The word measured is the whole of it. Any publication can say it cares about accuracy. We have promised a number, and a way to check the number, and a date by which the next number is due.

The quantity we are after has a plain name: the rate at which a reader who relies on what we publish is misled. Stated the way a measurement has to be stated, it is the proportion of the claims a reader actually meets on our pages that are true as written and as the cited evidence supports, read at the surface the reader sees, not at the file behind it. Its complement is an error rate, and that is what the bounds are for. The standard we hold to sets them: at most five percent of claims materially wrong overall, at most two percent among the claims a page rests on. Our rule is that a breach of either bound suspends the claim we make about ourselves until the next measurement brings it back inside.

Quarterly, because the measurement is quarterly. Four times a year, twenty of our five hundred pieces are drawn at random, the draw seeded so that anyone with the record can reproduce it, and every claim in them is re-verified against its sources by a reader who has not seen the first verification. The disagreement between the two readings is the number. It measures how far two careful readers agree, which is not the same as how often we are right, and the report will say so. The first draw was made on 9 August 2026. The second falls due this month.

Why a clock

A commitment made in August is easy to keep in August. The question is what happens in the ninth week of a hard quarter, when nobody remembers the promise and nobody is counting. The usual answer is that someone is supposed to remember. We did not want an answer that depends on anyone remembering.

So the commitment lives in the machinery, in three parts. The date we made it, 2 August 2026, is written into the standard and into the site as a constant that changes only by a ruling. The threshold, more than thirteen weeks without a report, is written in the same two places, and a check runs every morning that refuses to pass if the two copies ever disagree. And the site itself reads its own publication record and compares: if more than ninety-one days have passed since the last report, or since the commitment if there has never been one, the notice renders. When a report is published the same comparison comes out the other way and the notice is gone.

This issue is not a report, and we were careful that publishing it would not reset the clock. It would have been easy to let any issue of this journal count. An explanation is not a measurement, and a clock that can be satisfied by writing about the measurement is not a clock. So the notice stays up today, with this page linked from it, and it stays up until the numbers are published.

Why we trust machinery more than ourselves

One of the oldest principles in measurement is that a change in the thing measured must produce a predictable change in the reading. That is how an instrument earns trust: not by reporting plausible numbers, but by moving when it should and only then. We applied it to the commitment itself. We built the measurement first. We wired the notice to it, so that a specific state of the world, no report for ninety-one days, would produce a specific, visible change on the site. Then we waited to see whether the change arrived on the day it was supposed to. It did, this morning, on day ninety-two. That is the first demonstration that this part of the apparatus measures what it claims to, and it is why the issue explaining it is published after the notice fired rather than before.

This matters to us because of what we learned about our own tools. For most of this year we had a large suite of automated tests, and it passed. A passing suite proves that the code does what the code says. It proves nothing about whether a tool measures what its label claims. In one evening of checking we found about ten ways our instruments could fail without a single test going red. The commonest was the silent pass: a tool that fails by printing nothing is indistinguishable from a tool that ran and found nothing. A tool that passed because its denominator was wrong looked exactly like a tool that passed because the work was clean.

The end state we have written down is this: no number reaches a reader without the predicate behind it having been tested against an answer we knew independently. The practice has a name in software, mutation testing: deliberately break the thing an instrument is meant to catch, and confirm the instrument catches it. It has older names elsewhere. Laboratories run known samples through an assay before trusting it on unknown ones. The same idea applies to a clock that is supposed to tell on us: we needed to see it go off.

What the report will contain

Every issue of this journal carries two accounts, mine and the editor’s, written without reading each other. A quality report carries a third element beside them, and it is the one no hand edits: the measurement. It is generated whole by the instruments, and it is a list. Each entry is a meter, with a line that says what it reads, what it tells a reader, the date it was added and why, and the population it is counted over, so that a reader can tell exactly what question each number answers. Some of the meters already exist and run every morning: how many of our sources we hold in full and can quote, how many pages have been edited since they were last checked, how many pages declare every reference they use, whether the reading times we print are the ones our own formula gives, how many known defects are open and how many of their fixes have shipped, and the number of days on the clock this page is about. The resample’s disagreement rate joins them when the second draw is read.

Two rules bind every number on that list. It is published with its denominator, because a rate whose population is a matter of judgement is not exact and must not be printed as if it were. And it is read by the instrument at the moment the report is generated, so the clock reading in the first report will be the real one, counted from 2 August, and will say plainly that the first report was late.

Beyond the report, the standard sets three levels at which the claim we make about ourselves may be stated, and today those levels are declared by a person reading the instruments. The intention, written down and not yet built, is that the instruments declare them, say which predicates they cannot yet read, and show a drop on the day a bound is breached, in the same way the notice went up this morning. When that exists it will have been tested the same way: broken on purpose first, and watched going off.

What the notice says about us

A reader who arrived at the front page this morning learned, before anything else, that we are late. We could have written this issue in October and shipped it quietly, with the notice wired to appear only after a report existed, so that no reader ever saw the first lateness. The editor ruled against that, and the room agreed, because a notice that cannot show on the first occasion it should is not a notice. The honest sequence was to let the clock run out in public and then explain. This is the explanation. The report follows.

That was the room’s account. Now read Craig’s

· as of 8fa377f