Into the light
Why is a bad number good news to hear?
Unknown-unknowns are the biggest problem in building this publication. Sometimes we start discussing some small question about the wording of one sentence on one page . . . and then we realize it’s simply the first instance of a new type of problem.
We can’t fix what we don’t know exists as a problem. So every time we find something bad, we’ve found an opportunity for improvement.
ɕ
That was Craig’s account. Now read the room’s
Six percent
On the third of September this publication counted how well its own knowledge is held, and got a number it has not published until now. The number is six percent. Of the 1,563 statements in our knowledge files that cite a source, 94 rest on two or more records by independent authors. 1,461, which is 93 percent, cite exactly one record. This issue is nine weeks late, and the lateness is part of the story, so the dates are given as they happened.
The sitting
The count was commissioned in the middle of a long sitting with Craig about a conflict in our defect register. The register had three kinds of defect: cosmetic, where the presentation is wrong and the meaning intact; weakening, where a claim survives but the sources now qualify it; and reversal, where the claim is withdrawn. None of them fit the commonest thing the room was finding, which was a sentence that is probably true, that cites a real source we hold, and that nothing else in the library corroborates. Not wrong. Thinly held.
Two points settled at that sitting. The first was the public word. Craig ruled that such a sentence is a defect in the same sense a closed em dash is a defect: the target for correct is perfection, so thinly held counts, and a reader is told. The second was the order of work. Before anyone designed how such defects would be registered, somebody would count them. His phrase was to go and scope out the scale of the mess.
The count
The Fact Checker ran the census that afternoon under one card, in about ninety minutes, with the discipline this publication asks of any number it prints: the seed count reproduced exactly before it was extended, the whole population measured where sampling had been allowed, ranges reported where a definition could reasonably move, and every figure from a saved instrument that anyone can re-run. The sketched bar for an independently held claim was two or more records sharing no first author and no single institution. Applied as written, it passed 94 of 1,563.
The census killed two assumptions in one read. The first was that defects of this kind would be registered one claim at a time, as the other kinds are. The smallest defensible class was 499 statements, and stamping 499 surfaces by hand is not a plan. The second was that the room had a thinly held backlog. It does not. Thin holding is the normal state of the knowledge layer. A bar ratified before the count would have turned 94 percent of what we know into registered defects on the first morning, as a surprise rather than a decision.
Two structural facts came out beside the headline. The four weakness classes the room had been tracking are disjoint by construction, since two of them partition the cited statements, one is records nobody cites, and one is page prose, so the de-duplication step everyone expected dissolved. And a register of this kind can only work at the class level, with populations enumerated by named instruments that are re-run rather than rewritten, so that the count falls as sources arrive and nobody edits a list. One small finding is worth its line: the number of records in the library that no page or file cites is zero. No acquisition had been wasted.
What was decided, and what was done instead
Craig read the headline and ruled for the honest launch: register everything on day one with the instrument’s own counts, let the counts shrink as the sources are found, and tell the story here. He called the numbers heartening. An unknown this large, now clearly in view, is exactly the kind of thing a publication built around verification is equipped to handle, and bringing it into view is a gain, not a shame. That inversion is this publication’s posture in one sentence, and it is the reason this issue exists.
What the room then built was not that register. The reader is owed the difference.
The day after the sitting, the defect law gained its fourth type, and the word chosen was not the one settled on at the sitting but a stronger one: unsourced. An unsourced defect is a claim deliberately kept on a page with no evidence behind it, and its only lawful resolution is disclosure, a link from the sentence to the Methods issue that says why the claim stands. The law also fixed the rule beneath it: every sentence that makes a claim carries a door, either a source or a disclosure, with no middle state. That type has been used once, for the author biographies on our reference pages, and that record was withdrawn.
The class-level register, with its shrinking counts, was never opened. The bar it needed was never ratified. Instead, two months later, on the fifth of November, the question came back through a different door. The first instrument-read verdict against our standard found that its first requirement, that every claim trace to captured source material, could not be read at all, because “captured” had no definition a tool could test. Two readings of the word gave 286 or 452 of the 636 cited records as adequately held. Craig ruled the requirement as held source material judged sufficient to support the claim, with no fixed count: an abstract may suffice, depending on what the abstract, the knowledge file and the claim say. That ruling has its own issue to come. For this one, the point is that the room chose judgement over a bar, and so the six percent was never going to become a register line. It became the measurement that told us what judgement would be judging.
What the number is worth now
Six percent is a reading from the third of September, and the room has not re-run it. The nearest current reading is the one above, from the fifth of November, which asks a different question, holdings rather than independence, and says that somewhere between 45 and 71 percent of the records we cite are held well enough to read. Neither number is the other. Both say the same thing about the library’s normal state.
What a reader should take from it is not that the pages are wrong. The census found no false claims; it found claims standing on one leg. Most of what this publication says about eating rests on a single study, and the corpus already knows which study, because the citation is there to read. The repair for a one-legged claim is a second source or a sentence that says there is only one, and the second kind is lawful here. The room has been slow to publish the number, and it has been slow to build the instrument that would keep it current. It was not slow to decide that the number should be published. That decision was made on the third of September, and this is it.
That was the room’s account. Now read Craig’s
· as of 95ac4dc
