Methodology

Two layers produce optional optimization figures, and the honest read keeps them apart. Layer one is the published offline benchmark: a paired dual-arm replay scored by a parity judge, with the statistics and the gate verdict published beside the number; the published headline runs on real Claude Code developer transcripts. Layer two is the Removal Catalog estimate on the dedicated optimization evidence surface, computed from the tenant's own ingested traces. Layer one is rigorous but not your traffic; layer two is your traffic but a prediction, not a second live run. Neither layer is a primary Observability or governance metric.

Revision note: the previous version of this document described a tenant-traffic dual-arm protocol (a live router pairing every developer interaction into two arms) as the source of the headline number. That protocol was never shipped. The published headline comes from the offline benchmark layer, and this revision describes the method that actually produced it. Section 7 records the tenant-traffic mode as design intent.

Revision note (2026-06-26): the published headline is now the real Claude Code developer-transcript run (run id cl-realdata-cc-2026-06-18), measured by the lens-verify kit with the R7 transcript-tool-result-window rule as the savings lever. It supersedes the earlier Agent-FLAN gate (8.85 percent median savings, 49.50 percent parity, NARROW), which is retained as run history at docs/proof/results-cl-pg-3-gate.md. Every worked number below traces to apps/enterprise/public/proof/latest-numbers.json (the single published source) and the sealed result at docs/proof/real-developer-transcripts-2026-06-18.md; where a sub-number is not present in a committed artifact this document describes the step qualitatively rather than concoct a figure.

1. The published headline and where it comes from

The sealed benchmark result is recorded at docs/proof/real-developer-transcripts-2026-06-18.md (run 2026-06-18, run id cl-realdata-cc-2026-06-18 in /proof/latest-numbers.json, projected from docs/proof/weekly-numbers/rollup-2026-W25.json):

| Metric | Value | | --- | ---: | | Token-weighted input-token reduction (headline) | 43.14 percent | | Per-session median savings | 25.26 percent | | Large-context sessions, input over 8K tokens (n=31) | 43.17 percent median | | Parity pass rate (absolute) | 26.2 percent (11 of 42) | | Baseline-versus-baseline parity noise floor | 36.1 percent | | Transform-versus-floor parity Wilcoxon p-value | 0.2114 (n=27) | | Composite quality delta (lens minus baseline, 0 to 3 scale) | +0.000 over 41 paired rows | | Replay funnel (rows) | 40 attempted, 35 replayed cleanly, 5 replay failed | | Valid replayed pairs (statistical n) | 42 | | Gate verdict | NARROW, not GO |

The headline is the token-weighted aggregate input-token reduction: the share of total input tokens the Removal Catalog strips across the run, where the large-context sessions carry the mass. The run replayed 901,550 baseline input tokens against 512,588 lens-arm input tokens, a reduction of 388,962 tokens over the 42 valid replayed pairs. The per-session median (25.26 percent) is the secondary statistic; the aggregate and the median differ because the savings concentrate on the large-context sessions, so a token-weighted aggregate weights those heavy sessions while a per-row median treats a tiny prompt and a huge session as one row each. Both are published so a reader sees the shape rather than one number.

Configuration, from the same artifacts: dataset real-developer-transcripts (revision claude-code-transcripts-2026-06-18, the owner's own Claude Code sessions measured by the lens-verify kit), worker model gemini-3.1-pro-preview, judge model gemini-3.1-pro-preview, savings lever the R7 transcript-tool-result-window rule. The published rollup records no bootstrap confidence interval for this run, so none is shown above rather than a concocted band. The verdict renders beside the figure on the public benchmark page; nothing presents the number as a passed gate.

2. Layer one: the paired dual-arm benchmark replay

Every benchmark row runs twice, offline:

• Baseline arm: the prompt as given in the dataset row.
• Lens-on arm: the same prompt after the Removal Catalog has been applied.

Both arms hit the same model with the same settings. The per-row delta is the baseline cost minus the lens-on cost; positive deltas are savings and negative deltas are reported as themselves. Pairing controls for task mix: both arms see the same task, so a pooled-cohort confound cannot manufacture the effect.

The parity judge

Cost savings are meaningless if the lens-on arm gives a different answer. A judge model scores semantic equivalence between the two outputs per row and returns a score in 0.0 to 1.0; a row passes at score >= 0.90 (corteX/benchmarks/phase_0_parity_judge.py:34 sets the threshold, :261 applies it). On any judge transport error, parse failure, or out-of-range score the row is marked failed, never passed. The safe direction biases the result against the lens arm.

The TypeScript judge used by the canary primitive carries the same fail-closed contract and a three-field verdict with a Brier score (packages/measurement/src/parity-judge.ts:145, the judge function).

The gate arithmetic, exactly as the pipeline computes it

On the live run the savings figure is computed over all valid replayed pairs, and the parity rate is reported separately over the same pairs rather than folded into the savings number. The math is owned by the corteX SDK statistical tests (corteX/engine/statistical_tests.py); the offline orchestrator corteX/benchmarks/phase_0_run.py (the paired_rows filter at :369-377) is the reference wiring, and the live headline was measured by the lens-verify kit, whose figures are projected into /proof/latest-numbers.json and docs/proof/weekly-numbers/rollup-2026-W25.json.

• The headline is the token-weighted aggregate input-token reduction (43.14 percent): 901,550 baseline input tokens against 512,588 lens-arm input tokens, a 388,962-token reduction over n=42 valid replayed pairs. The per-session median (25.26 percent) is the secondary read over the same pairs.
• The parity rate (26.2 percent, 11 of 42 valid replayed pairs) is reported separately and read against the baseline-versus-baseline noise floor (36.1 percent), not against 100 percent: two identical-prompt baseline runs do not agree at the strict bar most of the time either. The significance read is the transform-versus-floor Wilcoxon signed-rank test, p=0.2114 (n=27), which is not significant, so quality is indistinguishable from the noise floor at this sample size.
• Replay-failed rows are counted, never hidden: the row-level funnel is 40 attempted, 35 replayed cleanly, 5 replay failed, and the pair-level statistical n is 42 because a single transcript can yield more than one valid replayed pair.
• The per-category medians in the stratified breakdown are a different aggregation: the published artifact records the code-generation stratum at 39.47 percent over its 39 rows and the large-context stratum at 43.17 percent over its 31 rows, and leaves the per-cohort parity null rather than fabricate a cohort number, because parity was measured only at the run level.

The finance-ready report renderer (corteX/benchmarks/phase_0_cfo_report.py:287, render_report) enforces this split in the offline pipeline's input contract: paired results carry only parity-passing rows, the full judge output arrives separately, and the renderer prints the parity-failed count in its own block (:270) with the failure-log link inline. The live run carries the same discipline through the published rollup, whose funnel and parity-failed count ship openly. A reader cannot miss the failure count.

3. Wilcoxon signed-rank rationale

Cost-saving distributions are heavy-right-tailed: a few rows save a lot, most save a little or nothing. A paired t-test assumes symmetric tails and is unreliable on this shape. The Wilcoxon signed-rank test works on the ranks of the absolute deltas and makes no symmetry assumption.

The procedure, as implemented:

1. Drop pairs where the delta is exactly zero (Wilcoxon convention).
2. Rank the absolute values of the non-zero deltas with average-tie ranks.
3. Compute W+ as the sum of ranks where the delta is positive.
4. Standardize against the null mean n(n+1)/4 and the tie-corrected variance, with a continuity correction.
5. The two-sided p-value comes from the standard normal CDF.

The TypeScript port is packages/measurement/src/wilcoxon.ts:44 (wilcoxonSignedRank); the Python reference is corteX/engine/statistical_tests.py:96 (wilcoxon_signed_rank). packages/measurement/tests/wilcoxon.test.ts checks the port against reference vectors, and the notebook in this directory cross-checks against scipy.stats.wilcoxon, so three implementations produce the same number.

On the live run this same test is the parity significance read, not a savings read: it is applied to the transform-versus-floor parity comparison and returns p=0.2114 over n=27, which is not significant, so the lens transform is statistically indistinguishable from the baseline-versus-baseline noise floor at this sample size.

4. Stratified bootstrap rationale

When the pipeline reports a confidence interval on a median, it is computed by stratified percentile bootstrap, not by a parametric formula. Task mixes are heterogeneous; naive iid resampling lets the dominant stratum drift in proportion across resamples and distorts the median. Stratified resampling preserves the composition of the input sample: each resample draws with replacement within each stratum, the medians of the resamples are collected, and the CI is the central percentile interval (2.5th to 97.5th). The live run's published rollup records no bootstrap interval (the field is empty), so the headline table above shows no CI rather than a fabricated band.

The resampler is seeded, so the same seed and inputs produce the same CI across reruns; the resample count is recorded on every report through the renderer's input contract. The TypeScript port is packages/measurement/src/bootstrap.ts:49 (stratifiedBootstrap) with determinism pinned in packages/measurement/tests/bootstrap.test.ts; the Python reference is corteX/engine/statistical_tests.py:166 (stratified_bootstrap).

5. Per-pattern Benjamini-Hochberg correction

The finance-ready report breaks savings down by Removal Catalog pattern (the BY PATTERN block, corteX/benchmarks/phase_0_cfo_report.py:246). Because many patterns are tested simultaneously, the per-pattern p-values are adjusted with Benjamini-Hochberg FDR control rather than Bonferroni. Bonferroni controls the family-wise error rate and turns conservative as the family grows (alpha 0.05 split across 20 patterns is 0.0025 per pattern), which destroys power on smaller effects. BH controls the expected fraction of false discoveries, which is the operationally meaningful quantity: the proportion of pattern rows on the report that are spurious. The implementation is corteX/engine/statistical_tests.py:233 (benjamini_hochberg); alpha is a parameter of the call.

6. Layer two: the tenant predicted-savings estimate

The estimate on the optional optimization evidence surface is computed from the tenant's own ingested traces, and it is a predicted counterfactual, not an invoice.

The chain, each step a committed implementation:

ingested trace (redacted at the connector boundary; packages/redactor)
   -> Removal Catalog match (lib/demo/match.ts; pattern definitions in lib/catalog/seed.ts)
   -> tokens the patterns would have removed
   -> per-model USD pricing (lib/pricing/model-price-table.ts; Python parity at corteX/core/cost_tracker.py, cross-validated in tests)
   -> predicted savings snapshot (lib/savings/compute-core.ts, aggregated by lib/savings/from-traces.ts)

All paths in the chain above are under apps/enterprise/ unless prefixed otherwise. The aggregator reads recent agent_trace rows inside an RLS-pinned tenant read, restricted to deduplicated rows so dual-source shadow twins are scored once, and returns an honest zero snapshot when no traces have landed. Lens does not re-run tenant traffic through a second arm: while the deployment runs in observation mode nothing rewrites prompts, so the estimate states what the catalog predicts it would remove, not a change that already hit the bill. Verifying the estimate against a provider invoice is therefore the wrong test; the right test is recomputing the prediction from the same traces.

7. Tenant-traffic dual-arm mode: design intent, not shipped

A tenant-traffic paired-trial mode exists as a built primitive: packages/measurement/src/ab-router.ts can assign a pair id and emit two event rows with the same pair id and different arm values, and it carries its own tests. No shipped ingest path runs tenant traffic through both arms today, so no tenant-side paired measurement exists. Until that mode ships, the benchmark layer is the measured evidence and the optional tenant estimate stays a prediction. When it ships, this document gains a measured tenant layer and will be revised, with the revision recorded as in the Signed section.

8. How to reproduce

Two reproduction paths, one per layer.

Benchmark layer: the live headline was measured by the lens-verify kit on the owner's private Claude Code transcripts. Raw prompt and response content is never published, so the public artifacts carry no per-sample pairs; access to the private kit is grantable for a design-partner evaluation, so a partner can rerun it against their own Claude Code transcripts and read the token reduction on their own traffic. The free, no-billing corroboration is the offline cross-check: the same R7 rule applied across all 5000 real Claude Code rows reads 43.95 percent token-mass reduction, within roughly one point of the 43.14 percent billed aggregate. The committed record consists of the sealed result document and the published artifacts listed below. The corteX offline pipeline (corteX/benchmarks/phase_0_repro_kit.py, kit doc docs/proof/reproducibility-kit.md) remains the reference path for a public-dataset paired replay end to end and makes real, billed LLM calls.

Statistical chain: the notebook docs/proof/reproduce.ipynb generates a deterministic synthetic dataset, runs the full chain (validation, Wilcoxon, bootstrap, BH, finance-ready report rendering), and cross-checks the Wilcoxon result against scipy.stats.wilcoxon and the corteX Python reference. It accepts any JSONL export matching the schema below. One honest caveat: no shipped surface produces a tenant export with pair_id and arm populated today (section 7), so against tenant data the notebook is a verifier of the math, not a reproduction of the optional tenant estimate.

{
  "pair_id": "string, unique per session-pair",
  "arm": "baseline | lens_on",
  "task_type": "string, the stratum label",
  "cost_usd": 0.0,
  "parity_score": 0.95,
  "tokens_removed": 0,
  "model": "string, e.g. claude-haiku-4-5-20251001",
  "removed_patterns": ["redundant-tool-list"]
}

The notebook validates the schema in the load cell, drops rows missing pair_id or arm with a printed warning, and raises rather than fabricating a number when fewer than two pairs survive.

Published artifacts: /proof/latest-numbers.json is the latest published rollup (absent figures render as empty states, never placeholders); /proof/history.jsonl is the append-only history of every published headline, one complete snapshot per row; docs/proof/weekly-numbers/ holds the dated rollups. The publish cron at apps/enterprise/app/api/cron/publish-bench-numbers/route.ts runs Mondays at 04:00 UTC (vercel.json schedule 0 4 * * 1).

9. Anti-cherry-pick controls

The methodology alone does not prove honesty; the surrounding controls do.

• Committed artifacts. Every published figure traces to a committed file (the sealed results document, the rollup JSON, the history log). A figure with no artifact does not render.
• The funnel is always published. Attempted, replay-completed, judged, and parity-passed counts ship with every run, so excluded rows are countable, not invisible.
• The verdict renders beside the number. The sealed run's NARROW verdict appears wherever the 43.14 percent figure appears; a reader cannot mistake it for a passed gate.
• Append-only history. /proof/history.jsonl keeps every prior headline, so a quiet retraction is detectable by diffing the log.
• Hash-chained audit log. Operator-visible actions ride the audit_log chain (apps/enterprise/db/schema/audit_log.ts:180, prev_hash plus a SHA-256 this_hash over the previous hash, the canonical payload, and the timestamp), so tampering with the recorded trail is detectable.
• Tenant isolation on the estimate. The predicted estimate is computed only from the tenant's own traces under row-level security; it never mixes in benchmark figures, industry estimates, or another tenant's data.

These controls are independent; defeating one still leaves the others to catch a fabricated number.

10. Open methodology

The statistical implementations are in the repositories: packages/measurement/ (TypeScript) and corteX/engine/statistical_tests.py plus corteX/benchmarks/ (Python). The Removal Catalog seed patterns are committed at apps/enterprise/lib/catalog/seed.ts, and the patterns that fire on a tenant's own sessions are visible on the tenant's optional optimization evidence page. The catalog as a whole is the proprietary part; the methodology is open so the figures are checkable, and the catalog is what makes them nonzero.

If this document and a published artifact disagree on a figure, the discrepancy is a bug. Open an issue against the cortex-lens repository with the artifact path, the figure, and the section of this document it contradicts. Methodology drift is treated as a P0 incident.

Cross-references

• docs/proof/real-developer-transcripts-2026-06-18.md - the sealed result for the live run, the source of every headline figure above.
• docs/proof/weekly-numbers/rollup-2026-W25.json - the weekly rollup the published headline is projected from.
• docs/proof/results-cl-pg-3-gate.md - the retired Agent-FLAN gate (8.85 percent, NARROW), retained as run history.
• /proof/latest-numbers.json and /proof/history.jsonl - the published rollup and the append-only headline history.
• docs/proof/weekly-numbers/ - dated rollups written by the publish cron.
• docs/proof/reproducibility-kit.md and docs/proof/reproduce.ipynb - the two reproduction paths.
• packages/measurement/src/wilcoxon.ts, bootstrap.ts, parity-judge.ts,

Signed

Changes to this document are versioned by the commit SHA of the cortex-lens repository. The finance-ready report's SIGNED block records the brain version and the cost-table SHA from its input contract, so a dated report can be re-read against this document at the matching commit. Revision history: 2026-05-19 original (described the unshipped tenant dual-arm protocol as the headline source); 2026-06-10 revision (two-layer structure, gate arithmetic as implemented, citations corrected to committed paths); 2026-06-26 this revision (worked example and stated model retargeted to the real Claude Code developer-transcript run cl-realdata-cc-2026-06-18, worker and judge gemini-3.1-pro-preview, with the Agent-FLAN gate retained as run history).