Methodology

How All True Fact scores a claim, what is arithmetic and what is still a model's judgment, how the dials were set, and what the backtest does and does not show. This page is the source for the site's methodology page and the methodology episode. Written 2026-09-17 (Phase 4); every number here names the dataset it came from.

The one-paragraph version

A claim is split into checkable sub-claims. For each sub-claim the engine searches the web for evidence on both sides, fetches the documents, and asks a language model a series of narrow questions about each one: how good is this source (asked without telling the model which side the document helps), how directly does it bear on the sub-claim, which way does it point and how hard, and what does its content rest on (an observation, or an estimate or assertion). Documents that rest on the same underlying source are grouped, and only the first in a group counts in full. Each document's push is added up in log-odds, the sub-claims are combined through the claim's AND/OR structure, and the result is a probability with a label. The same rubric is run with five different models on the same evidence, and the divergence between them is published next to the consensus. The model is also asked cold, with no evidence, what it thinks; that number is published too, so the reader can see what the structure changed.

What is arithmetic and what is judgment

Judgment (a model's opinion, compartmentalised): for every document, four atomic calls. quality_v1 rates the source on a 0 to 1 scale blind to the sub-claim and to which side the document helps. directness_v1 rates relevance to the exact sub-claim. direction_v2 returns direction (+1, 0, −1), strength (0 to 1) and basis (finding, mixed or assertion), and is run a second time under the negated sub-claim on a 20 percent sample as a consistency check. roots_v1 names what the document rests on (dataset, study, author group, funder, primary source), extracted once per pool by one designated model so every scoring model sees the same cluster structure; roots_canon_v2 merges spellings of the same root. Decomposition of the claim into sub-claims is hand-written for video claims and model-written (decompose_v1) otherwise, and is always reviewed. Query writing (queries_v1, queries_side_v1) is a model call too.

Arithmetic (a pure function of the cached judgments and one parameter file):

weight_i       = quality_i ^ quality_exp × directness_i ^ directness_exp × basis_i
independence_i = independence_discount ^ (rank of i within its cluster − 1)
push_i         = direction_i × strength_i × scale × weight_i × independence_i
shrink         = 1 / (1 + contested_pull × mixed_mass / directional_mass)
L_leaf         = logit(prior) + shrink × Σ push_i
p_leaf         = sigmoid(L_leaf)
p_and          = Π p_children       p_or = 1 − Π (1 − p_children)

Mass (Σ weight × independence) below mass_min makes a leaf "insufficient evidence", which is a published verdict, not a failure. A leaf whose weaker side has directional mass below side_mass_min is flagged one-sided and, since Phase 4, gets a bounded second retrieval pass for that side. The arithmetic re-runs in milliseconds; changing a dial never calls a model. Every dataset records the rubric version, the prompt versions, the parameter snapshot and the params_version.

Consensus is the median leaf log-odds across the five models, re-aggregated through the tree. Divergence is reported per leaf (spread and mean pairwise disagreement) and per item (the documents the models scored most differently).

What this does not do. It does not remove the model's judgment. Source quality, directness, direction, basis and roots are all the model's calls, and a lean in any of them can run systematically across many small calls. What the structure does is make every call visible, keep the blind call blind, run the same calls on five models so a lean in one shows against the others, and keep the aggregation out of the model's hands. Bias is compartmentalised and auditable, not eliminated.

The dials

All in atf/params.py. Internally every dial is open; the public endpoint exposes one, strictness, which maps onto independence_discount, mass_min and quality_exp; videos and published datasets always use the locked defaults.

Why these values, in one line each: the 2026-09-16 cheap configuration (pool 25, polarity 20 percent, public profile) was set by Curtis on cost after the first frontier run priced at $19 a claim; basis_weights and contested_pull came out of the Phase 3 gate on the hand-run pair, where model-based estimates and "this is contested" reviews were being counted as if they were observations; the rest are the Phase 2 starting values. None of the scoring dials has yet been moved by the backtest (next section), and the log says so.

The backtest

The set. 51 resolved claims in data/backtest/claims/, 28 true and 23 false, each with the source's original wording, the declarative paraphrase used in every prompt, the resolution, the date the fact became knowable, and the press or primary record it was checked against on 2026-09-17. 49 are resolved prediction-market questions (Manifold; Metaculus's API is now login-only), chosen for a real-world, unambiguous resolution and at least 40 traders; 2 are record claims (the Iraq WMD fixture and the ICC warrant). Candidates whose resolution turned on a disputed definition (Russia's net territorial change in July 2026, which the trackers disagreed on; whether a July 2026 remark was a campaign announcement) were left out. Political-judgment ratings are not in the set: they are judgments, not ground truth.

Post-cutoff, per model. Each pinned model's stated training cutoff is recorded in data/models.yaml with the page it was read from; where the vendor states none (DeepSeek V4.1 Flash), the model's release date is used as a hard upper bound and the report says so. A claim is post-cutoff for a model when its resolution date is after that bound. Resolution dates run from 2004 to 16 September 2026, so the flag varies by model: 28 claims are post-cutoff for every model with a stated cutoff, another 12 are post-cutoff for some models and not others, and 11 are pre-cutoff for all of them.

Two measurements, stated honestly.

1. Rubric versus raw, all claims. Every model is also asked cold, with no evidence, for its probability. The delta between the rubric's number and the raw number is real whether or not the model knows the answer. The headline is deviation direction: on each claim, did the rubric move the model toward the recorded resolution, away from it, or not at all (under 0.1 log-odds)?
2. Brier score and log loss, pre- and post-cutoff separately. Brier is the mean squared error of the probability against the 0/1 outcome (0.25 is a coin flip at 0.5, 0 is perfect). Reported for rubric and raw, per model and for the consensus, with calibration tables.

The leakage caveat, in full. On a pre-cutoff claim the model was trained on text that states the answer, so its raw number is recall, not a forecast, and its judgments of the evidence were made knowing the answer. On a post-cutoff claim the model cold does not know, but the rubric's retrieval step reads today's web, where the outcome of a resolved question is normally reported directly, so the rubric side is not blind either. The post-cutoff subset therefore measures whether retrieval plus rubric reaches the recorded answer from the documents, and the pre-cutoff subset measures what the structure changes about a model that already knows. Neither is a forecasting score, and this page does not call either one that. Paraphrasing the claims reduces verbatim recognition; it does not remove knowledge.

Tuning. atf backtest tune re-scores the cached judgments under a grid of the scoring dials (1,944 points), hill-climbs from the best point, prints how sensitive the objective is to each dial, and runs 5-fold cross-validation: tune on four fifths of the claims, score the held-out fifth under the tuned dials and under the defaults. A dial moves only if the held-out score improves, because with ~50 claims and eight dials an in-sample improvement is easy to find and means little. Tuning never calls a model; it is the same arithmetic on the same judgments.

Results

Filled from data/backtest/report.md and data/backtest/tuning.md as the set is run; both are regenerated by make backtest-eval and make backtest-tune with zero model calls.

Status (2026-09-18): 10 of 51 claims run

The OpenRouter balance allowed a first slice of the set, not the set: three claims on 2026-09-17 (two live on five models, plus the Phase 3 run of the Iraq WMD fixture registered without re-running) and seven more on 2026-09-18 on the four cheapest models, Grok 4.3 dropped for cost at Curtis's direction. One of the seven (Tim Cook stepping down) was cut short by the spend cap and is marked incomplete. Live spend for the ten: $4.42. Nine are post-cutoff for every model with a stated cutoff; one (the fixture) is pre-cutoff for all. Five true, five false.

Deviation direction, consensus per claim: toward the record on 8, flat on 1 (the AfD claim, which every model already put at 0.00 to 0.02 cold), away on 0; the incomplete claim has no consensus baseline. Consensus Brier 0.000 under the rubric against 0.49 cold on the post-cutoff nine.

Per model, pooled over claims:

The three "away" moves on post-cutoff claims are all of the same kind: a model that cold already put a false claim at 0.01 or 0.02 and the rubric put it at 0.03 or 0.05, a move of a few tenths of a log-odds unit in the wrong direction on a claim both numbers get right. The flats are models already at 0.00 cold. Every one of the nine post-cutoff claims lands on the correct side of 0.5 for every model that scored it. The "What moved it" tables name what did it: the Fed's own statement, the Hungarian and Swedish election results, Apple's and Stripe's newsroom posts, Starmer's resignation coverage, congress.gov for the bill that did not pass. The pre-cutoff row is the leakage caveat in numbers: cold Brier 0.28 against 0.47, because the models remember Iraq.

What ten claims do and do not show. They show that retrieval plus rubric reaches the recorded answer on recent, well-reported, settled questions, and that the movers are the primary record rather than commentary. They do not show calibration on contested claims, which is the product's actual job, and they cannot tune a dial: data/backtest/tuning.md is a smoke of the procedure and no value in it is to be used. The remaining 41 claims cost about 45 cents each on four models.

What is still judgment, listed

• Which documents the search finds (two backends, both fallible, one ending on 2027-01-01); the pool is reported as a source of uncertainty (leave-one-backend-out spread) and the second pass exists because thin retrieval on one side was the first thing the hand-run pair showed.
• Source quality, directness, direction, strength and basis: a model's call on each document, blind where it can be.
• Roots: one model's opinion of what a document rests on decides the clustering, and clustering decides how much a pile of similar articles is worth. This is the hardest and most consequential judgment in the pipeline and the one most likely to be wrong.
• Decomposition: which sub-claims a claim is made of, and which parts are values and excluded.
• The dials themselves: numbers chosen by people, recorded with the reasons, and open to be moved by the backtest with the evidence written down.

Every one of these is recorded per call, per dataset, with the model string and prompt version that produced it, so a reader who disagrees can point at the exact call.

Rendered from docs/METHODOLOGY.md in the engine repository. The page and the file are one text.