# Methodology

Paths such as `data/`, `tasks/` and `templates/` below refer to the private benchmark repository. The public results repository has `data.json`, `pricing.json` and this file at its root; task contracts, briefs and hidden suites stay private.

## What one result is

One result is one **entry**: one model configuration (`model@effort`) writing a task's deliverable from one lead's brief, in one run. An entry is graded five ways:

| Measure | Source | Notes |
|---|---|---|
| **Score** | Council | Mean of the seats' 0–10 scores. Seats review anonymously and calibrate across all entries of the run. |
| **Hidden** | `tasks/<t>/bin/hidden` | Categories passed out of the total. Pass/fail per category, never partial credit. |
| **Speed** | `tasks/<t>/bin/perf` | Speed-up over the task's reference implementation. Above 1 is faster. Speed is not scored when the perf workload's output was wrong. |
| **Cost** | Harness logs | Tokens priced at list price, standard tier (`data/models.json`). |
| **Time** | Harness logs | Wall clock from the contestant's first to last logged event. |
| **Slop** | Council findings | Defined below. |

The score is the headline. The others explain it and add the dimensions a team actually buys on: price, latency and how much clean-up the code needs.

## Council

Full protocol: `docs/COUNCIL.md`.

- **Judges** have stable ids (`data/judges.json`) and can join or retire. Each review session is fresh: at most 10 entries, re-lettered and shuffled, plus anchor entries from earlier runs of the same task.
- **Basis.** Judges judge against the task's `SPEC.md` and a fixed rubric (`templates/council-rubric.md`), never the lead's brief, so scores compare across leads. The 2026-10-31 run predates this: it was judged against its brief (`basis: "brief"`).
- **Score** is the mean over the run's **panel**: the judges who scored every live entry. The **cross-vendor score** uses only judges whose vendor differs from the entry's. Both appear in tooltips and in the findings drawer.
- **Calibration.** Anchors give each judge an offset relative to earlier panels (`tools/bench council calibrate <task>`). Offsets are reported, not applied.
- **Findings** each have a severity (critical, high, medium, low or nit) and exactly one category from `tools/benchlib/council.py`:

  | Group | Categories |
  |---|---|
  | Defects | `correctness`, `robustness`, `performance`, `abi` |
  | Under-delivery | `tests`, `docs` |
  | Slop | `misleading-docs`, `dead-code`, `redundancy`, `verbosity` |
  | Other | `style` |

- **False positives.** The orchestrator verifies surprising findings. A finding shown to be wrong stays in the data with `status: "false_positive"` and a note, and counts in no metric. Score revisions are logged in `council.revisions`.
- **Legacy reviews.** The 2026-10-31 run predates explicit categories. Its findings were classified by keyword rules plus documented manual overrides (`tools/legacy/import_2026_10_31_range_coder.py`).

## Slop score

For each seat, sum the severity weights of its valid slop findings (`misleading-docs`, `dead-code`, `redundancy`, `verbosity`). Average over all seats; seats with no slop findings count as 0.

| Severity | Weight |
|---|---|
| critical | 4 |
| high | 3 |
| medium | 2 |
| low | 1 |
| nit | 0.5 |

Lower is better. 0 means no seat reported any slop. For example, two seats each flag one low-severity false comment (1 + 1) and a third flags a nit dead branch (0.5); over four seats that is 2.5 / 4 = 0.63.

Why council-based: slop is a judgement about whether text and code say true, necessary things. Static metrics cannot see a wrong comment. Implementation lines and comment share are shown beside the score (tooltips and the findings drawer) as context, not mixed into it.

The sibling **defect points** use the same weights over every valid finding of any category, averaged per seat. They appear in tooltips as "all findings".

## Cost

- **Tokens.** Every agent run records its tokens from its harness's own logs: contestants, leads writing briefs, council judges and kit builders (`tools/benchlib/usage.py`). The tokens are split into uncached input, cache reads, cache writes and output; reasoning is a reported subset of output.
- **Prices.** `data/pricing.json` holds every model's full price card from the models.dev catalog: input, cache read, cache write (5-minute and 1-hour for Anthropic), output, service-tier multipliers, long-context bands and peak-hour multipliers.
- **Cost = list price × tokens, cache included.**
  - **Formula:** uncached input × input price + cache reads × cache-read price + cache writes × cache-write price + output × output price, at the standard tier (`tools/benchlib/pricing.py`).
  - **Claude cache writes:** Claude Code writes 1-hour cache entries, priced at $8 per million tokens for Opus 5.5.
  - **Visibility:** the per-component split is stored as `cost_breakdown` and shown in the site's "Tokens and cost" table.
- **Fast mode.** Codex records `service_tier: priority` when it runs fast. Those runs also carry `cost_usd_as_run`, twice the list price, which is what the API bills at that tier.
- **Recorded by the harness.** OpenCode stores what it billed. That figure is kept as `cost_recorded_usd` and shown beside the list price.
  - It matches list price for DeepSeek.
  - It is $0 on subscription plans (GLM on the Z.ai coding plan; MiMo on OpenCode Go).
  - It is *below* list for Qwen ($0.29 against $0.38) and Grok ($2.90 against $5.09), because OpenCode Go bills cache reads below the catalog price.
- **Comparisons always use list price.**
- **History.** Until 2026-11-01, Claude cache writes were priced at the 5-minute rate ($5) and Qwen and Grok used OpenCode's recorded figure. The pilot round was re-priced with the rules above, which raised its Claude entries by about 14–16% and Grok to list price.

## Time and fast mode

Wall time is measured, not estimated. Entries that ran on a faster service tier are flagged (`harness.fast_mode`, ⚡ on the site). The time chart adds an estimated standard-tier extension at 2× the measured time. That 2× is an assumption, not a measurement; replace it by running the same configuration on both tiers.

## Speed

- **Measurement.** Each task's `bin/perf` runs a fixed workload on a fixed host: the Ryzen 9 9950X for everything except NEON, which runs on the Apple M1. It reports median timings for the candidate and the reference, with `speedup = reference_time / candidate_time`.
- **Aggregation.** A task with several metrics (encode/decode, or one metric per function) is summarised by their geometric mean. The overall view takes the geometric mean across tasks.
- **Expert baselines.** Where a task has expert baselines (dav1d, wpd), the speed chart marks them. A `*` means the baseline does not meet the contestant contract (`task.json` `perf.baseline_notes`), so it is context, not a target.
- **Proxy timings.** If the Mac is unreachable, the NEON task falls back to qemu instruction counts, flagged `proxy: true`. These are not comparable with real timings.

## Final score

The headline score combines the three things the benchmark measures:

> final = council average × (hidden-suite categories passed ÷ total) − 0.5 × slop

- **Council average** (0–10): the panel's mean score.
- **Hidden-suite share:** a hard, judge-independent check. An entry that fails 1 of 12 categories keeps 11/12 of its council score.
- **Slop:** severity-weighted slop findings per judge; each point costs half a point. Judges already see some slop, so the half weight keeps the double count small.

Per configuration, the final score is averaged over runs, and in the overall view over tasks. The council score stays on the site next to it. The top picks still rank on the council score, because only entries that pass (nearly) every hidden category qualify and slop is already in their defect points. (Owner decision, 2026-11-03; chosen over an additive mix and a harsher squared hidden share, which reorder the council ranking more.)

## Top overall picks

The site headlines up to six picks, not one, because the trade-off between quality and price rarely has a single answer.

> value = score − 0.2 · log2(cost in USD) − 0.5 · defect points

- **Price.** Each doubling of cost costs 0.2 points. Cheap matters, but a small saving can't buy back a quality difference.
- **Liability.** Defect points are the council's severity-weighted findings, averaged per judge (critical 4, high 3, medium 2, low 1, nit 0.5). They cover every category, slop included. Each point costs 0.5. Every finding needs another agent run to fix, so an entry with findings costs more by the time it can be merged.
- **Eligibility.** Only entries that pass at least 98% of hidden-suite categories, summed over their runs (and in the overall view over tasks), qualify. A single run of one task has 11–16 categories, so it must pass all of them; across many runs, a configuration that misses one category check out of a few hundred still qualifies, and the pick shows its hidden count. (Owner decision, 2026-11-04: with every lead combined, almost no configuration was perfect over about 30 runs. Until then the rule was every category in every run.) The ranking covers every effort level. Under *Best per model* or *Best per lab* each model or lab appears once, at its best-value effort level; under *All effort levels* a model can appear more than once. In the overall view, cost and defect points are means per task.

In round 1 (all five tasks, every lead, every effort level) the top three picks are:
1. Opus 5.5 xhigh: council 9.01, $5.74 per task, 1.12 defect points.
2. Opus 5.5 high: council 8.77, $2.75, 1.30 defect points.
3. GPT‑6.1 Sol max: council 8.43, $0.44, 1.72 defect points (hidden 333/335).

Opus 5.5 xhigh's quality outweighs its price here, but under the Opus 5.5 lead alone (*Best per model*) GPT‑6.1 Sol xhigh ranks first: council 8.80 at $0.40.

History: until 2026-11-01 the pick was a single entry, scored as score − 0.5·log2(cost). That favoured Terra max, because it charged price heavily and ignored findings.

## Comparing leads

Under *All leads*, the "Leads compared" card gives each lead one number per task: the **median** final score over every configuration that ran from its briefs, at every effort level, whatever the view. Every lead runs the same roster, so the weak configurations don't favour any lead; but one breakdown near 0 moves a 23-entry mean by about 0.3 points and can reorder leads, while the median barely moves. Weak configurations still count: a brief that lifts them lifts the median. The plain mean is shown on hover and a strip draws every configuration, so breakdowns stay visible. *Overall* averages each configuration over the tasks every shown lead has scores for, then takes the median. (Owner decision, 2026-11-04, chosen over a trimmed mean, a mean without low outliers, and the plain mean.)

## Repeat runs

**Round 1 ran each configuration once per lead and task** (replicate 1): 23 configurations × 5 leads × 5 tasks. With one run per cell, treat differences of a few tenths within a single cell as ties. Comparisons that pool cells (a configuration over five tasks, or a lead over 115 entries) are steadier, and the Findings section says how strong each one is.

The site also supports repeat runs, called **replicates**: separate runs with the same lead, task and brief, numbered `replicate` 1, 2, 3 in `data/runs/<run>.json`, each reviewed by the full council like any run. When a selection has more than one:

- **Combined (default).** Every number is the mean over the replicates: score, cost, time, slop and defect points. The matrix shows the run-to-run standard deviation under the mean (`±0.22`); hovering lists each run. The Pareto chart draws the score range vertically and the cost range horizontally; the time and slop charts draw the range over the bar.
- **Run 1, 2, 3.** The **Runs** switcher shows a single replicate on its own. It appears only when the selection has more than one.
- **Hidden suite.** Combined counts add up over runs (`33/33` = 11 categories × 3 runs).
- **Reading differences.** Two configurations whose ranges overlap are not clearly different.

A batch that adds configurations to an existing run shares that run's replicate number (`run new --replicate N`); a configuration appears at most once per replicate.

## Code metrics

Measured by tools, not judges, so they are exactly repeatable (`tools/bench run quality <run>`, `tools/benchlib/quality.py`). Only the implementation counts: test files, and in Rust the `#[cfg(test)]` module, are excluded.

- **Lint.** Warnings from the task's own linter (for example `cargo clippy`), from `bin/check`.
- **Duplicated.** Share of significant lines (not blank, not comment-only, not a lone bracket) inside a block of six or more consecutive significant lines that appears more than once, after whitespace normalisation. Some repetition is inherent: an encoder and decoder with the same accessors repeat them. Read it as a measurement, not a verdict; the council's redundancy findings are the judgement.
- **Complexity.** Cyclomatic complexity per function from [lizard](https://github.com/terryyin/lizard): mean, maximum, and functions above 10. Lizard cannot parse Elixir or assembly, so those tasks show no complexity.

## Verified defects

`data/defects/<task>.json` lists the defects confirmed by building and running the code (or by a failing hidden-suite category), one entry per distinct defect, with the entries that have it and the judges' findings that report it. It gives the council a ground truth to be measured against:

- **Judge recall.** In the Council judges table, "Verified defects found" is the share of listed defects, in entries the judge reviewed, that it reported.
- **Trials.** Candidate judges and effort levels are compared on how many listed defects they find and how many false positives they raise (see `docs/COUNCIL.md`).

A defect enters the list only with a verification method (`probe`, `hidden-suite` or `inspection` for documentation and test defects). Unverified findings stay as findings.

## Council judges

Per selection, the site reports how each judge scores, how the judges relate to each other, and what the reviews cost:

- **Offset.** A judge's score minus the other judges' mean on the same entries. Positive = more generous. It is a tendency, not a quality: the 2026-10-31 Opus judge was about 0.7 points stricter.
- **r.** Correlation of the judge's scores with the other judges' mean: does it rank entries the same way?
- **ICC.** Intraclass correlation (two-way random effects, absolute agreement) over entries scored by every judge. ICC(2,1) is the agreement between single judges; ICC(2,k) is the reliability of the panel mean, which is the published score.
- **Mean and median score.** The judge's raw 0–10 scores over the entries it scored in the selection.
- **Review cost and time.** The list price of the judge's council sessions on the selected runs (from its harness's logs, like every other cost; Muse Wonder Full at the Muse Wonder 1.0 API rate), cost per entry scored (calibration anchors excluded from the count), and wall-clock time per session.

## Aggregation on the site

- **Task view.** One row per configuration. Repeat runs are averaged (see Repeat runs); each judge column is that judge's mean over the runs.
- **Overall view.** For each configuration, the mean over the tasks it completed of the per-task score, slop, cost and time. Speed uses the geometric mean. The `n/N tasks` tag shows coverage. Configurations that skipped hard tasks are not penalised for it, so read coverage before comparing.
- **Best per model.** Keeps, per model, the effort level with the highest score; ties go to the cheaper one. "All effort levels" shows every configuration.
- **Leads.** The lead selector scopes everything to one lead's runs. "All leads" makes the score columns per-lead averages, so the same configuration can be compared across leads' briefs.
- **Did not finish.** DNF entries carry cost and time but no score. They are excluded from score averages and shown hatched in the time chart.

## Colours

Vendor colours are fixed per vendor and never depend on rank (`data/models.json`):

| Vendor | Colour |
|---|---|
| Anthropic | terracotta orange |
| OpenAI | teal green |
| DeepSeek | cyan blue |
| xAI | black, or light grey in dark mode |
| Meta | Facebook blue |
| Z.ai | ochre in light mode, yellow in dark mode |
| Moonshot | muted purple |
| Google | red |
| others | neutral grey |

Eight brand hues cannot all pass colour-blindness separation checks, so identity never relies on colour alone. Every mark has a text label, the Pareto chart gives each vendor its own marker shape, score tables show each lab's mark next to the model name, and every chart has a table view.

## Known biases and caveats

- **Run-to-run variance.** Round 1 has one run per configuration per lead and task, so a single cell carries run-to-run noise; treat differences of a few tenths there as ties. Where replicates exist, use the shown ranges.
- **Judges share families with contestants.** Anonymisation helps, but self-preference is possible. Check the per-seat columns: the 2026-10-31 run's top three were identical without the Opus seat.
- **Opus 5.5 built the kits.** The benchmark's operating agent, an Opus 5.5 xhigh session, chose the tasks, wrote every task contract (SPEC), wrote the range-coder hidden suite and had its own builder subagents write the others. Its own briefs from that work (the 2026-10-31 pilot) were written knowing the hidden categories, so the pilot is excluded from the results. The round-1 Opus 5.5 xhigh lead is a separate session, briefed with the same short prompt and the same inputs as every other lead (`task.json`, the task contract, `scaffold/` and `reference/`), with no access to the hidden suites. It is still the same model that wrote the contracts, which could favour its briefs; the Findings note this next to the Opus-lead result.
- **Single-file scope.** The benchmark measures careful implementation of a specified component, not design of large systems or long agentic sessions.
- **Isolation in the first run.** In the first run, contestants worked next to the harness directory, under a rule not to read outside their own directory. From the standard layout onward, contestant directories live outside the repository (`~/.cache/code-bench/work/<run>/`).