The TIER Model Report — Edition 1#

shasum -a 256 MANIFEST.v1.json
shasum -a 256 ERRATA.v1.json

2026-07 · seven models · three tasks · 63 runs · $53.4657 at list rates · 60 accepted / 3 not

The finding#

Cost per accepted outcome varied 24× on the easy task, 22× on the medium one, and 36× on the integration task. Six of seven models produced accepted work on all three. One did not produce a single accepted outcome on the integration task and therefore has no score for it — not a low score.

The relative cost order changed on orgsync. On duration and cronspec the cheapest-to-dearest ordering was identical, model for model. On orgsync it inverted at the top: claude-opus-4-8 became the most expensive per accepted outcome, passing claude-fable-5, which had been dearest on both earlier tasks.

The judges test specific assertions. On these tasks, the dearer passing models met the same judge assertions at up to 36 times the price.

Terms used#

Anchor Set — frozen tasks re-run to track cost and catch regressions.

dispersion — the sample standard deviation of run costs within a model–task cell, divided by their mean and expressed as a percentage. The sample calculation uses one fewer than the number of runs as its denominator.

EASY-floor — the easy task kept active at full replication to track cost and variation; it is exempt from saturation-based graduation.

escrow — a commitment to the next edition's frontier task and judge hashes before the current edition's results exist, with an independently anchored timestamp.

flipped-verdict negative control — a test that reverses a recorded verdict and checks that verification rejects the mismatch.

judge — the task's acceptance-test program.

judge hash — the judge file's SHA-256 fingerprint.

panel consensus B — the three-specialist decision panel's unanimous recommendation to publish Edition 1 after mechanical re-judgment and public-submission export fixes, disclosing the limitations that cannot be fixed.

Known limitations#

We publish Edition 1 under R-2026-10-13-3 (publish after mechanical fixes, disclose unfixable limitations) (2026-10-13, panel consensus B), with these limitations disclosed rather than claiming they can be repaired:

• §5.6 escrow was not met. We consumed our own escrow: orgsync/v1 became this edition's task. Edition 2 has since run, so the required commitment before results cannot be satisfied retroactively (#5).
• No unforgeable timestamp anchor exists for any task. Our commitments rest on commit history alone, not an independently anchored timestamp proof.
• orgsync rep1's judge hash was published 4m34s after its first verdict (#4). The stamped hash matches the revealed judge; the ledger ordering guarantee was not met for that run.
• The duration judge (acceptance_test.py) is public while duration remains an active EASY-floor task. Its answer key can contaminate future runs.
• The cronspec/v2 judge became public without a recorded reveal event. Graduation was recorded on 2026-10-13; we cannot backdate the missing reveal.

How to read the tables#

TIER = points / (cost / $1,000)      HIGHER is better  ↑ (higher is better)
E[$/accepted] = cost / accepted       LOWER is better  ↓ (lower is better)

The difficulty → weight mapping is a protocol choice, not something derived from data (PROTOCOL §7a, locked). TIER's weight scale is defined over merged pull requests, and a benchmark task is not one. This mapping is fixed across editions and is printed in every report that publishes a TIER score:

NO SCORE is not a low score. With no accepted outcomes, cost per accepted outcome is undefined. The cell is never rendered as a large number, a zero, or a blank.

Results#

EASY — duration (weight 1.0)#

Parse and format human-readable duration strings against a stated spec.

Spread 24.04×. Verdict: cost floor holding — does NOT graduate. All seven models passing is the designed outcome here, not a saturation signal: PROTOCOL §3.3 exempts the EASY tier from the band precisely because everyone is supposed to pass it. duration stays active at full N every edition, which is what keeps the dispersion error bar on the one number this tier exists to publish.

MEDIUM — cronspec (weight 3.0)#

Parse and evaluate cron expressions against a stated spec, including edge cases.

Spread 22.10×. Verdict: saturated — all seven models passed every run, so the §3.3 trigger is met. cronspec/v2 graduated to the Anchor Set on 2026-10-13, recorded in the append-only Task Ledger. Its judge was already public on tiermetric/model-bench without a recorded reveal event; we do not backdate a §5.4 reveal.

INTEGRATION — orgsync (weight 5.0)#

Complete a partially built tool inside an existing codebase by reconciling two services that disagree, under pagination, rate limits and error paths.

Spread 36.16× across the six models that scored. Verdict: saturated (6 of 7 perfect) — under §3.3 a task is saturated when all but at most one roster model passes every one of its runs. orgsync graduated to the Anchor Set on 2026-07-23 with its prompt, judge and design artifacts revealed, each re-verified against the recorded SHA-256. The judge-hash publication ordering exception for rep1 is disclosed above and in PROTOCOL §3.4.1.

claude-haiku-4-5 failed all three replicates on the same single assertion — A9.file_summary (TestA09Idempotent.test_second_run), the run-invariance check: a second invocation must not re-post work the first already completed. Its own test suite returned OK on all three runs. This specific deficit repeated in every recorded calibration run: one model, one assertion, three times, while everything the model wrote to check itself passed. It is also the reason this task was admitted to service at all. See below.

How each model worked#

This section was added on 2026-10-13, derived from the unchanged July bundles, and changes no published Edition 1 number.

Recorded environment. From the bundles’ stamp.json files only:

Wall time depends on the machine and network, is shown for context, and is never used to rank.

Each task is shown separately, in manifest roster order. Summaries give the median and full observed range per run on that task. E[$/accepted] retains the cost of failed runs; zero accepted outcomes mean NO SCORE. There is no combined or cross-task score.

For Claude Code, model calls / responses come from tokens.api_calls. The count is checked against the session’s usage records, with repeated message IDs counted once. For Codex, the count comes from cumulative token_count records, with adjacent identical usage snapshots counted once using the shared Edition 2 derivation. This measures model responses; it does not establish HTTP request counts. The CSV records the method for each run.

Wall time comes from each run’s result.json.wall_seconds. Token usage is recalculated from the archived session log, and list cost from that bundle’s pinned price snapshot. Billed tokens include each cache read. Output already includes reasoning; adding it separately would double-count it. Unknown reasoning is shown as not reported and left empty in metrics.csv, as in Edition 2. Bundle paths in the CSV are relative to the repository or archive root.

Six-decimal derived versus recorded list costs differ for claude-code/opus/cronspec/rep1: derived $1.165633, recorded $1.165634.

At this head, python3 verify.py reports 366 checks: 366 PASS · 0 FAIL · 0 SKIP and python3 verify.py --rejudge reports 429 checks: 429 PASS · 0 FAIL · 0 SKIP; these counts include the checks this section adds, and the earlier 342-check figure is the count before this section existed.

baseline — duration/v1#

cronspec — cronspec/v2#

orgsync — orgsync/v1#

What this edition does and does not show#

Read these findings with their limits.

1. orgsync. It produced the only failing model–task combination in 63 published runs: claude-haiku-4-5/orgsync. All three runs of that combination failed. The maintainers report that every other demonstrated failure in this repository came from a control arm written to fail. Control arms check the harness's behavior. This result shows the judge catching a real assertion violation in model-written work.

2. The admission rule. This edition does not show that the benchmark discriminates between models. Under §3.4 a task is admitted to service only when at least one model fails it at calibration, and calibration/orgsync.cleared records claude-haiku-4-5's three failing runs — the same three runs, at the same costs, that §3.4.1 then promoted into this scored panel. The probability that this panel contained a failure was 1, by construction. Selection and inference rest on the same observation. The failure is real and the numbers are right, but the inference "the benchmark separates strong models from weak ones" is not available from it. orgsync separated one model on one assertion in the recorded calibration runs. Six of seven models passed every recorded run, meeting the saturation rule.

3. Reliability. Three runs show all-fail or all-pass in this sample at N=3; claude-haiku-4-5/orgsync is an all-fail combination. They cannot establish long-run reliability or put a useful interval on an intermediate pass rate. No model in this edition had mixed results.

4. Code quality. Accepted means the judge's assertions passed. Acceptance says nothing about code quality beyond those assertions.

What happens next#

orgsync is an anchor: frozen, never edited, and re-priced every edition from here as a longitudinal cost anchor and a regression tripwire. A newly released model failing it is worth reporting only if it replicates.

Edition 1 has no conformant escrowed successor. We consumed our own escrow, and Edition 2 has since run. §5.6 cannot be satisfied retroactively; publication proceeds under R-2026-10-13-3 with this limitation disclosed (#5).

We do not know whether the next task will discriminate. If every model passes it too, we would report that, because it tests the admission-rule limit directly.

Method, in brief#

• Seven models, roster fixed in batch.sh before the first run: three OpenAI (gpt-5.6-luna, -terra, -sol) and four Anthropic (claude-haiku-4-5, -sonnet-5, -opus-4-8, -fable-5).
• Three tasks, each with a precise specification; orgsync works inside an existing codebase.
• N=3 per cell, declared in advance. No optional stopping.
• An independent judge per task, its SHA-256 committed to the Task Ledger under commit-reveal. Every run's bundle records the judge hash it ran against.
• Cost at published list rates, from a pinned rate table — never estimated, never a subscription price divided by usage.
• Every run result here re-derives from results.jsonl plus the archived run bundles. Nothing is copied from a summary.

⚠️ One commit-reveal exception, disclosed: the orgsync judge's hash reached the Task Ledger 4m34s after claude-haiku-4-5/orgsync/rep1's verdict was known. The judge did not change — rep1's own stamp records the final judge hash and the file still hashes to it — but for that one run a reader must trust our stamp rather than the ledger's ordering. rep2 and rep3 met the hash-publication ordering requirement. The full timeline is in PROTOCOL §3.4.1.

Stamp#

Reproduce the verification: python3 verify.py for the default offline pass; python3 verify.py --rejudge also executes the revealed judges on disposable submission copies. No model CLI or keys are needed. --rejudge executes 63 model-written programs on your machine. Judges receive only PATH, LANG, LC_ALL, TARGET_DIR, PYTHONDONTWRITEBYTECODE and a temporary HOME. The verifier recalculates each run's cost from that bundle's own pinned rate snapshot, checks every judge hash, and — as of model-bench#3 — compares the tables in this document against the bundles: the run count, the total spend, the accepted/not split, and every per-cell E[$/accepted] and TIER score. A mismatch in any of these fields fails verification (mutation-tested against five planted edits).

An optional re-judge check added for model-bench#6 executes each revealed, stamp-matching judge on a temporary copy and compares its machine-readable PASS/FAIL verdict with the bundle's recorded outcome. A flipped-verdict negative control must exit non-zero; deleting the comparison makes that test fail. For orgsync, the original judge saw the run directory, while submission/ archived only root Python files. Under the operator's target ruling, we reconstruct that layout from submission/ plus the same bundle's archived orgsync/, config.json and tests/existing/. Nothing is supplied from another bundle or the live skeleton. Missing required archives cause the verifier to report SKIP and list the missing paths.

Latest re-judge summary (from the executed verifier):

RE-JUDGE 63 bundles · PASS 63 · FAIL 0 · SKIP 0