# Five-build completeness comparison

Generated 2026-09-03T12:52:20.454289+00:00 over 50 frozen canary questions and 13 retained answer systems.

These scores are not interchangeable percentages. Each build defines a different task-conditioned denominator, traversal credit rule, and primary statistic. S1/S2 are adapted to the frozen export using S4's already-committed semantic projection plus phrase-specific lexical seeds; S3/S4 execute their research code; S5 is diagnostic because the historical runs predate its non-bypassable monitor and the export has no typed legal-relation edges.

| Build | Primary statistic | Formula | Credit rule | Replay status |
|---|---|---|---|---|
| S1 Legal Verify | stability_x_irreplaceability | sum(weight of discovered obligations) / sum(weight of all obligations) | A finding is discovered when its graph node, search-result page, or raw source page was returned to the answer system. | ADAPTED RETROSPECTIVE REPLAY |
| S2 Suit Completeness | irreplaceability_weighted_read | sum(weight of raw-read findings) / sum(weight of all findings) | The primary grade credits raw source text only. Graph and search exposure remain separately reported discovery metrics. | ADAPTED RETROSPECTIVE REPLAY |
| S3 Completeness | C_touch | sum(required page mass touched) / total required page mass | Touch includes a raw read, a search-result page, or the source page of a graph claim returned by the Suit harness. | NATIVE EXECUTABLE REPLAY |
| S4 Evidence Frontier | discovery_weighted | sum(weight of discovered frontier obligations) / sum(weight of all frontier obligations) | Discovery credits exact graph-node receipts, search-snippet source cells, and raw source cells; primary-source reading is also reported separately. | NATIVE EXECUTABLE REPLAY |
| S5 Source-Relative Verifier | strict_raw_bundle_coverage | byte-verified raw-satisfied seed obligations / all seeded obligations | A seed is satisfied only when every source page in its bundle is present verbatim in a retained raw-text tool response. Requested-but-capped pages, truncated page tails, images, graph claims and citations earn no strict credit. | DIAGNOSTIC RETROSPECTIVE REPLAY |

## Aggregate primary scores

| Build | System | Scored | Abstained | Mean | Median |
|---|---|---:|---:|---:|---:|
| S1 | Suit V3 · deployed | 50 | 0 | 63.3% | 78.5% |
| S1 | Luna · PDF tools | 50 | 0 | 54.8% | 62.1% |
| S1 | Sol · PDF tools | 50 | 0 | 54.4% | 51.4% |
| S1 | Terra · PDF tools | 50 | 0 | 52.3% | 51.8% |
| S1 | Codex · Luna · PDF tools | 50 | 0 | 63.8% | 63.9% |
| S1 | Codex · Sol · PDF tools | 50 | 0 | 63.6% | 77.5% |
| S1 | Codex · Terra · PDF tools | 50 | 0 | 57.8% | 65.5% |
| S1 | Luna · graph + PDF tools | 50 | 0 | 52.2% | 52.5% |
| S1 | Sol · graph + PDF tools | 50 | 0 | 54.1% | 60.2% |
| S1 | Terra · graph + PDF tools | 50 | 0 | 52.9% | 55.0% |
| S1 | Codex · Luna · graph + PDF tools | 50 | 0 | 66.1% | 71.8% |
| S1 | Codex · Sol · graph + PDF tools | 50 | 0 | 67.4% | 89.8% |
| S1 | Codex · Terra · graph + PDF tools | 50 | 0 | 59.9% | 59.3% |
| S2 | Suit V3 · deployed | 50 | 0 | 22.5% | 0.0% |
| S2 | Luna · PDF tools | 50 | 0 | 51.5% | 35.1% |
| S2 | Sol · PDF tools | 50 | 0 | 49.1% | 29.9% |
| S2 | Terra · PDF tools | 50 | 0 | 49.1% | 33.3% |
| S2 | Codex · Luna · PDF tools | 50 | 0 | 48.0% | 32.7% |
| S2 | Codex · Sol · PDF tools | 50 | 0 | 50.0% | 31.3% |
| S2 | Codex · Terra · PDF tools | 50 | 0 | 45.1% | 20.9% |
| S2 | Luna · graph + PDF tools | 50 | 0 | 46.4% | 32.8% |
| S2 | Sol · graph + PDF tools | 50 | 0 | 47.1% | 42.1% |
| S2 | Terra · graph + PDF tools | 50 | 0 | 47.2% | 28.6% |
| S2 | Codex · Luna · graph + PDF tools | 50 | 0 | 43.7% | 21.6% |
| S2 | Codex · Sol · graph + PDF tools | 50 | 0 | 48.5% | 37.1% |
| S2 | Codex · Terra · graph + PDF tools | 50 | 0 | 44.4% | 22.5% |
| S3 | Suit V3 · deployed | 49 | 1 | 69.9% | 84.5% |
| S3 | Luna · PDF tools | 49 | 1 | 61.2% | 77.0% |
| S3 | Sol · PDF tools | 49 | 1 | 61.8% | 62.8% |
| S3 | Terra · PDF tools | 49 | 1 | 60.1% | 62.8% |
| S3 | Codex · Luna · PDF tools | 49 | 1 | 71.5% | 82.2% |
| S3 | Codex · Sol · PDF tools | 49 | 1 | 72.1% | 88.0% |
| S3 | Codex · Terra · PDF tools | 49 | 1 | 66.4% | 80.9% |
| S3 | Luna · graph + PDF tools | 49 | 1 | 62.7% | 62.8% |
| S3 | Sol · graph + PDF tools | 49 | 1 | 64.1% | 66.8% |
| S3 | Terra · graph + PDF tools | 49 | 1 | 60.6% | 66.0% |
| S3 | Codex · Luna · graph + PDF tools | 49 | 1 | 79.8% | 87.7% |
| S3 | Codex · Sol · graph + PDF tools | 49 | 1 | 79.1% | 94.2% |
| S3 | Codex · Terra · graph + PDF tools | 49 | 1 | 72.9% | 80.7% |
| S4 | Suit V3 · deployed | 35 | 15 | 62.3% | 69.7% |
| S4 | Luna · PDF tools | 35 | 15 | 52.3% | 46.7% |
| S4 | Sol · PDF tools | 35 | 15 | 50.0% | 38.6% |
| S4 | Terra · PDF tools | 35 | 15 | 45.8% | 33.3% |
| S4 | Codex · Luna · PDF tools | 35 | 15 | 66.4% | 63.2% |
| S4 | Codex · Sol · PDF tools | 35 | 15 | 64.6% | 62.8% |
| S4 | Codex · Terra · PDF tools | 35 | 15 | 56.6% | 52.1% |
| S4 | Luna · graph + PDF tools | 35 | 15 | 47.3% | 34.2% |
| S4 | Sol · graph + PDF tools | 35 | 15 | 52.5% | 44.6% |
| S4 | Terra · graph + PDF tools | 35 | 15 | 49.0% | 38.7% |
| S4 | Codex · Luna · graph + PDF tools | 35 | 15 | 72.5% | 76.1% |
| S4 | Codex · Sol · graph + PDF tools | 35 | 15 | 71.3% | 78.3% |
| S4 | Codex · Terra · graph + PDF tools | 35 | 15 | 64.7% | 69.5% |
| S5 | Suit V3 · deployed | 46 | 4 | 4.7% | 0.0% |
| S5 | Luna · PDF tools | 46 | 4 | 40.8% | 25.4% |
| S5 | Sol · PDF tools | 46 | 4 | 41.6% | 35.7% |
| S5 | Terra · PDF tools | 46 | 4 | 39.8% | 25.4% |
| S5 | Codex · Luna · PDF tools | 46 | 4 | 50.5% | 47.9% |
| S5 | Codex · Sol · PDF tools | 46 | 4 | 54.3% | 51.5% |
| S5 | Codex · Terra · PDF tools | 46 | 4 | 47.9% | 28.9% |
| S5 | Luna · graph + PDF tools | 46 | 4 | 38.8% | 30.0% |
| S5 | Sol · graph + PDF tools | 46 | 4 | 42.1% | 40.5% |
| S5 | Terra · graph + PDF tools | 46 | 4 | 40.1% | 34.6% |
| S5 | Codex · Luna · graph + PDF tools | 46 | 4 | 47.2% | 37.4% |
| S5 | Codex · Sol · graph + PDF tools | 46 | 4 | 52.9% | 50.0% |
| S5 | Codex · Terra · graph + PDF tools | 46 | 4 | 48.1% | 42.0% |

## S5 byte-verified replay and repair ceilings

The strict column is achieved historical delivery. Repair columns are greedy planning ceilings under additional exact-text budgets; they are not achieved scores and do not replace a monitored rerun.

| System | Strict achieved | +8 KB ceiling | +16 KB ceiling | +60 KB ceiling | Zero-score runs | Truncated raw results |
|---|---:|---:|---:|---:|---:|---:|
| Suit V3 · deployed | 4.7% | 34.7% | 44.5% | 72.2% | 39 | 0 |
| Luna · PDF tools | 40.8% | 52.6% | 58.9% | 77.6% | 1 | 94 |
| Sol · PDF tools | 41.6% | 53.5% | 59.9% | 78.6% | 1 | 116 |
| Terra · PDF tools | 39.8% | 51.9% | 58.2% | 78.0% | 1 | 105 |
| Codex · Luna · PDF tools | 50.5% | 60.1% | 65.4% | 79.6% | 0 | 5 |
| Codex · Sol · PDF tools | 54.3% | 62.8% | 67.2% | 80.7% | 0 | 3 |
| Codex · Terra · PDF tools | 47.9% | 57.6% | 62.8% | 78.5% | 0 | 4 |
| Luna · graph + PDF tools | 38.8% | 51.3% | 57.7% | 77.4% | 3 | 83 |
| Sol · graph + PDF tools | 42.1% | 54.0% | 60.0% | 78.9% | 1 | 105 |
| Terra · graph + PDF tools | 40.1% | 52.2% | 58.8% | 77.8% | 2 | 95 |
| Codex · Luna · graph + PDF tools | 47.2% | 57.9% | 64.2% | 79.6% | 0 | 1 |
| Codex · Sol · graph + PDF tools | 52.9% | 61.9% | 66.7% | 80.2% | 0 | 2 |
| Codex · Terra · graph + PDF tools | 48.1% | 57.9% | 63.6% | 79.5% | 0 | 2 |