Retain RRF.

How much graded evidence do we capture?

At depth 20, the authenticated postmortem diagnostic finds 316 / 12,984 known-relevant documents (2.43%) and 2,492 / 84,560 graded-gain units (2.95%). This diagnostic is bound to the pinned qrels and RRF ranking but does not alter the sealed evaluation depths. At depth 1,000, RRF captures 3,953 / 12,984 known-relevant documents (30.45%) and 28,815 / 84,560 graded-gain units (34.08%). The full accepted union raises those ceilings to 38.66% binary recall and 42.38% graded recall.

Known-relevant binary recall gives every judgment at grade 2 or higher equal weight. Graded gain uses 2**grade - 1, so higher-grade evidence contributes more. These are separate questions and neither is precision over unjudged documents.

Judgment-pool dependent. The denominators cover known judgments only. Unjudged candidates remain unknown, not nonrelevant, and the full-union values are a ceiling within this accepted candidate pool rather than corpus-wide recall.

Why Topics 31 and 300 fail at the cutoff

The pooled gain hides two decisive regressions. Open the native disclosures for the incoming/outgoing label mix, source-family and facet-depth attribution, and the bounded replay. No document text or document identifiers are exposed.

What this result means—and what it does not

This is a retrospective full-development stress test over all 22 development topics using already known judgments. It measures retrieval of known-relevant documents and is useful for diagnosing headroom and regressions. It is not evidence of generalization to new topics, unseen judgments, or production traffic. The downstream RAG answer generation is out of scope; no claim is made about answer accuracy, faithfulness, citation quality, or user utility.

The v1/v2 ranking/evaluation rejected label is intentional: only portable rankings_v3 and evaluation_v3 are admissible here. Their ranking bytes and scientific conclusion are unchanged from corrected v2; v3 removes checkout-local paths from sealed identity and enforces the judged-rate promotion guard.

Exact preregistered selection ladder

The baseline appears first for orientation; alternatives follow the sealed ladder exactly. Aggregate improvements are insufficient when a zero-loss guard fails.

Recall and retained facet evidence by depth

Recall rises as the cutoff expands, and DUAL variants retain more facet-only evidence. The aggregate curves explain the attraction of the alternatives; the per-topic guard below explains the decision.

Judged-rate caveat: RRF judged rate falls from 97.8% at 100 to 28.2% at 1,000. Unjudged documents are not negatives, so deep precision and yield are conservative and judgment-pool dependent.

Known-relevant yield falls with facet depth

The first 50 results of each deduplicated facet stream carry the highest pooled known-relevant yield. Later buckets still add evidence but at lower density, supporting depth discipline rather than blanket expansion.

Per-topic deltas expose the promotion blockers

Every topic is shown. Count deltas are alternative minus RRF; negative values are regressions. The final column gives primary DUAL deltas at depths 250 / 500 / 1,500.

Representative diagnostics

Method and evidence boundary

The portable ranking freeze has 22 topics, 6 complete arms, 270,864 ranking rows, and 206,030 audit rows. The evaluation independently binds that freeze before opening the pinned qrels. The paired exact sign-flip test for primary DUAL estimates +3.394 percentage points mean topic recall (95% bootstrap CI +2.118 to +5.052; raw p=0.000002623; Holm-adjusted p=0.000006676). Statistical significance cannot override the preregistered topic-loss rule.

Costs and execution accounting

Original-query requests: 0. Retrieval failures/retries: 0/0. Shared-score cache reuses: 68,079. Completed score windows: 848,183. Peak device/host memory: 489,715,200 / 12,323,328,000 bytes. Ranking and evaluation added zero retrieval, inference, model-load, hosted, paid, or network calls.

Limitations, decision, and next step

• Retrospective known-judgment evidence can overstate certainty and does not establish held-out generalization.
• Judgment incompleteness grows with depth; unjudged candidates may contain useful evidence.
• The experiment evaluates retrieval and ranking only; answer generation remains untested.
• Topic 31 and Topic 300 losses are small in pooled terms but decisive under the frozen guard.

Recommendation: retain RRF, run a bounded recovery lane for Topics 31/300-like failures, and start the RAG lane from a deeper pool with Source-diverse evidence selection. Do not pass the protected top 20 through unchanged and call it facet-aware RAG.

Authenticated provenance

Ranking v3 root: 6f4c35899f90c1d60324caf24bf8834d3482c5e8b9e785f8316ab0eec55fc305
Evaluation v3 root: e49ce3f7f0cfeedf1863670ae4cfe5781d40163449a172a5822d8eb11eeffe84
Pinned qrels SHA-256: 42bf933ae06eb22213312b22e3f2bc39f3dcc2d54e87ebcd8125e9528ddfcc37
Planning root: bc1351cca8aa05dd0979a342c8dd5f72395f2b7a9207ae20668aba05f1ab85a2
Retrieval root: f2191c295f600d0243f4dcd2b1dc23a05b9fa8a7d9a4d6258983a0d3c9433aeb
Scoring root: 4f67107770b1cd35598adc75beeeafe205ba984f201cedf4b505c4d20515e990

This sanitized report contains aggregate metrics and hashes only: no credentials, raw qrels, raw documents, document identifiers, request logs, or local filesystem paths.

Back to top