From one frozen source to five accepted TREC RAG 2026 runs

The final Retrieval and RAG submission architecture, told as a guided story

Final scope. This guide documents the five files accepted by Evalbase: three Retrieval runs and two Retrieval-Augmented Generation runs. It explains the shared frozen source that produced them, the point where the two task branches separate, and the validation that protects each organizer artifact. It contains only aggregate, checked-in provenance—no test narrative, passage, prompt, provider response, or generated answer.

Open the project artifact hub or go directly to the submission ledger.

The central correction to the earlier architecture story is simple: The Retrieval TSV is not Generation input. One authenticated source retrieval run was frozen. Cache-first replay over that source produced the three Retrieval submissions, while a separate sealed selected-evidence handoff from the same source fed the two RAG strategies.

A small glossary

Narrative

The complete official information need. It stays intact as the original retrieval lane and as the answer contract.

Source lane

One authenticated pool from the frozen source artifact: either the original narrative lane or one facet:<subnarrative>:text lane.

Candidate core

The inclusive union of documents admitted by robust per-lane score gates. It is narrative-specific and shared by all three Retrieval orderings.

Selected evidence

Exact passages retained by source retrieval. Selected passages are factual authority. Canonical claim hints are advisory routing aids.

Sealed handoff

generation_handoff_manifest.json plus its authenticated topic payloads. It is the only Retrieval-side state that either RAG strategy may consume.

Organizer artifact

One accepted r_output_trec_rag_2026.tsv or rag_output_trec_rag_2026.jsonl file.

How to read every image

• Teal names source and submission Retrieval; indigo names RAG Generation.
• Amber names advisory model work or a bounded control decision.
• Solid arrows carry data or evidence. Dashed arrows carry advisory, fallback, or validation control.
• Folded documents are durable artifacts; cylinders are stores; diamonds are validation gates.
• Counts distinguish documents, chunks, passages, selected evidence, and advisory hints rather than treating them as one unit.

Frozen source, two submission branches

Takeaway: One source retrieval artifact is the provenance fork. Its two children are sibling submissions, not a sequential Retrieval-TSV-to-RAG pipe.

Figure 1: An official narrative enters bounded source retrieval and produces the authenticated facet-deepseek-b40-v3 source. One branch creates a robust variable-depth candidate core and three Retrieval TSVs. A separate sealed selected-evidence handoff feeds single-pass and multi-stage RAG, producing two JSONLs. All five files pass local validation and are accepted by Evalbase.

Open the reusable whole-system SVG

The frozen facet-deepseek-b40-v3 source preserves the outputs of planning, BM25 retrieval, neural passage scoring, evidence selection, and canonical hint generation. The Retrieval submission branch replays only authenticated lanes and cached scores to decide a narrative-specific candidate set and three rankings. The RAG branch authenticates a purpose-built selected-evidence handoff and never opens the final Retrieval runs.

The split matters because the tasks answer different questions. Retrieval asks which ClimbMix documents should be ranked as useful evidence. RAG asks for a cited answer after the system has performed its own retrieval. Shared provenance does not make one organizer file the other’s runtime input.

Text equivalent: The official narrative flows into bounded source retrieval and then the frozen authenticated source. From there, one branch produces three variable-depth Retrieval TSVs and the other passes a sealed evidence handoff to two RAG strategies. A final validation gate precedes five Evalbase-accepted files.

Implementation notes and sources

• The final artifact map and exact hashes live in the submission ledger.
• The Retrieval branch is documented in its bundle architecture.
• The RAG branch is documented in its bundle README.
• The source artifact identifier is provenance, not a public link; its private contents remain outside this report.

Source retrieval: bounded widening, fixed evidence

Takeaway: The source system widens once, applies the same retrieval boundary to every query, and freezes selected passages before either submission branch.

Figure 2: The untouched narrative and bounded DeepSeek subnarratives create query lanes. Every lane searches the climbmix-400b BM25 index through Pyserini, chunks returned documents, scores passages with the Mixedbread cross-encoder, selects exact evidence, and creates advisory hints before the source artifact is frozen.

Open the reusable source-retrieval SVG

The untouched narrative always supplies one lane. One strict deepseek/deepseek-v4-flash-20260423 call may add zero to eight focused subnarratives, each with one to three BM25 queries. That gives at most 25 searches including the original lane, without an open-ended search–replan loop.

Each query can retrieve at most 1,000 documents from the Pyserini REST climbmix-400b index. Documents change units into overlapping chunks using 3500 characters with 350-character overlap. The pinned mixedbread-ai/mxbai-rerank-base-v2 cross-encoder scores query–passage pairs, and source retrieval retains at most 100 passages per focused query before local evidence selection. A one-shot Nuggetizer stage may organize advisory claim hints, but it cannot replace the exact selected passages.

Text equivalent: The original narrative and bounded planned queries cross the same Pyserini BM25 and Mixedbread scoring boundary. Documents become chunks, then ranked passages, then selected evidence. Exact passages and advisory hints are sealed together in the authenticated frozen source.

Implementation notes and sources

• The source configuration is configs/rag26_competition_retrieval_v2.yaml.
• Planning is implemented in facet_extraction.py; source orchestration is in competition_retrieval.py.
• Passage scoring uses the pinned model revision recorded in the final Retrieval bundle metadata.
• The 1,000-document and 100-passage values are per-query internal ceilings, not final submission depth.

Planning stays one-shot

Takeaway: DeepSeek may widen the search, but the original narrative remains unconditional and the planner cannot initiate a second research round.

Figure 3: One DeepSeek V4 Flash request receives the complete narrative. The original lane remains. Zero to eight subnarratives may each propose one to three BM25 queries. Schema or semantic failure falls back to the original lane, and no retrieved result returns to the planner.

Open the reusable bounded-planning SVG

Planning decomposes a long information need into bounded, search-oriented subnarratives. It is useful for breadth, but it is not factual authority. The system owns the executable limits, preserves the original narrative, validates the structured plan, and uses an original-only fallback when the plan is not usable.

For the final Retrieval replay, each authenticated facet:<subnarrative>:text lane is one pooled source. Individual planned query strings are not retrospectively treated as separate pools because the frozen artifact does not preserve distinct result lanes for them. This keeps the replay faithful to available provenance.

Text equivalent: A single planner call may emit zero to eight subnarratives with 1–3 BM25 queries each. The original narrative lane bypasses the planner gate and always survives. The planner has no edge from search results back into itself.

Implementation notes and sources

• The bounded planning and fallback contract is enforced by facet_extraction.py and competition_retrieval.py.
• The final source and pooled-lane interpretation are recorded in the Retrieval architecture.
• No qrels, gold nuggets, or RAGDoll results participate in planning or replay.

Retrieval submission: robust variable depth

Takeaway: Final Retrieval depth is evidence-driven. Each source lane admits outliers, their union becomes the shared candidate core, and no fixed cutoff is padded or imposed.

Figure 4: Each authenticated source lane computes its median aggregate score and median absolute deviation. Documents above the robust lane threshold are admitted. The original and all subnarrative admissions form an inclusive union. An empty union would retain the best original-lane document. The observed final depth ranges from one to 121 documents.

Open the reusable candidate-core SVG

For source lane (u), let (A_u(d)) be a document’s finite aggregate score, (m_u) the lane median, and (mathrm{MAD}_u) the median absolute deviation. The robust admission rule is:

MAD > 0: A(d) >= median + 2.5 × 1.4826 × MAD
MAD = 0: A(d) > median

The final core is the inclusive union of the original-lane admissions and all subnarrative-lane admissions. If that union were empty, the strongest original-lane document would survive with deterministic retrieval-rank and UTF-8 document-ID tie breaks. No topic needed this fallback.

Across 119 narratives, the final core contains 4,246 topic–document pairs. Its variable depth spans 1–121 documents, with 61 distinct values of (k). Every submitted Retrieval run uses the same core for a given narrative.

Text equivalent: Every authenticated lane applies a robust median/MAD gate. Admitted documents are unioned across the original and subnarrative lanes. The result is one narrative-specific candidate core shared by all three runs, with observed depth from one to 121 and no padding.

Implementation notes and sources

• The exact admission formula, fallback, and observed counts are preserved in ARCHITECTURE.md.
• retrieval_candidate_core.py implements robust candidate admission.
• Final per-topic cutoff receipts are authenticated in retrieval-baseline-runs-manifest.json.

Three orderings, one candidate set

Takeaway: Cache-first targeted scoring changes order, not eligibility. The three Retrieval runs are an interpretable comparison over identical documents.

Figure 5: The shared candidate core is chunked and scored against the untouched narrative and every subnarrative. Cached Mixedbread scores complete one targeted matrix. Percentiles feed three orderings: narrative plus subnarrative, evidence breadth, and narrative only. All three emit the same narrative-specific document sets.

Open the reusable targeted-scoring SVG

Every candidate document is scored against the untouched narrative and every subnarrative. Passage scores are overlap-suppressed and aggregated with weights 0.55, 0.25, 0.13, and 0.07 over the best four retained passages, then converted to tied percentiles within each narrative and semantic unit. Final replay used 167,597 cached query–passage scores, zero cache misses, and zero model batches.

The three accepted orderings are:

1. Narrative + subnarrative — r26-narr-facet-v1. A 50/50 blend of narrative percentile and a facet score made from the best two subnarrative percentiles (0.7 best + 0.3 second-best).
2. Evidence breadth — r26-facet-breadth-v1. Lexicographic ordering by supported subnarrative count, strong passage count, combo score, source rank, and document ID.
3. Narrative only — r26-narrative-v1. Narrative percentile followed by source rank and document ID.

Each output has 4,246 rows. Their document set is identical within every narrative; only ranking changes.

Text equivalent: The candidate core becomes one targeted document-by-semantic-unit matrix. Cache-first Mixedbread scoring and percentile normalization feed three ranking heads. All heads emit the same per-narrative documents as separate standard Retrieval TSVs.

Implementation notes and sources

• retrieval_baseline_runs.py implements the shared matrix and three ordering kernels.
• The Retrieval bundle README records the accepted files and validation counts.
• metadata.json records exact hashes, model identity, cache accounting, and run descriptions.

The handoff is the RAG trust boundary

Takeaway: RAG authenticates selected evidence and allowed citations. It cannot browse sibling Retrieval artifacts or evaluation material.

Figure 6: The sealed generation handoff contains the exact narrative, 691 evidence groups across 119 topics, selected passages, advisory hints, allowed ClimbMix citation IDs, source receipts, and hashes. Single-pass and multi-stage generation authenticate it. The Retrieval TSV, full-text ZIP, qrels, gold nuggets, and RAGDoll scores are barred.

Open the reusable handoff-contract SVG

The RAG bundle uses a handoff with 119 topics and 691 evidence groups. Each topic binds the exact narrative, selected passages, advisory claim hints, allowed citation domain, and source receipts through canonical serialization and hashes. Loading fails closed on malformed, contradictory, non-canonical, or unauthenticated state.

Generation may open only this handoff. It does not read any accepted Retrieval TSV, the source full-text ZIP, qrels, gold nuggets, or RAGDoll scores. The first two are sibling Retrieval artifacts; the last three are evaluation or gold material. Canonical claim hints help organize coverage, but selected passages remain the factual authority.

Text equivalent: A folded handoff document names narrative, evidence groups, passages, advisory hints, citation IDs, receipts, and hashes. It crosses an authentication gate into both RAG strategies. Five barred artifacts have no edge into Generation.

Implementation notes and sources

• generation_handoff.py defines canonical serialization, payload hashes, context hashes, and validation.
• The RAG bundle records handoff manifest SHA-256 31dc1b3578741339101b8199c0ea027e2d8405e828532aa58f86bf74c0542a0d.
• The private handoff itself is intentionally not copied into this report or the submission bundle.

Single-pass RAG

Takeaway: Single-pass Sol receives the complete authenticated selected- evidence context, writes one structured candidate, and gets at most one semantic retry after local validation.

Figure 7: The authenticated handoff becomes one deterministic selected-evidence prompt for GPT-5.6 Sol. Sol uses medium reasoning, strict structured output, and a 12,000-token ceiling. Local validation checks exact narrative metadata, the allowed citation domain, reference uniqueness, exact-hint citations, and the 1,024-word ceiling before publishing rag26-ss1.

Open the reusable single-pass SVG

The single-pass strategy uses openai/gpt-5.6-sol with medium reasoning, strict structured output, no explicit temperature, a 12,000-token ceiling, and four-topic concurrency. It renders a deterministic projection of the selected-evidence handoff and asks Sol for organizer-shaped answer objects with raw allowed ClimbMix document IDs.

Local validation checks the exact topic and narrative, required metadata, answer structure, unique references, allowed citation domain, exact-hint citation rules, and the 1,024-word ceiling. Two of 119 topics used the configured second semantic attempt. The final accepted rag26-ss1 file used 121 Sol calls, cost $52.14302500 as reported by the provider, and contains answers ranging from 203 to 749 words with 10 to 40 references.

Text equivalent: The sealed handoff is projected into a strict Sol request. The result crosses a local validation diamond; one bounded semantic retry may return to the request. Valid document IDs are converted deterministically into organizer citation indexes before the accepted single-pass JSONL is written.

Implementation notes and sources

• competition_rag.py implements request rendering, transport retry, semantic validation, resume identity, and atomic publication.
• configs/rag26_competition_rag_gpt_sol_v2.yaml preserves the checked-in model and request shape.
• Aggregate production counts and cost are in the RAG bundle metadata.json.

Multi-stage RAG

Takeaway: Multi-stage Generation spends bounded Luna work on coverage and edit safety around a Sol draft; every topic retains a locally valid fallback.

Figure 8: Luna creates a narrative blueprint and audits each authenticated evidence group. Sol writes a structured draft, then may propose a bounded splice revision. Luna screens proposed operations for support, atomicity, materiality, redundancy, and replacement safety. Local code applies only accepted operations; malformed or rejected work preserves the validated draft. Final validation publishes rag26-ms1-final.

Open the reusable multi-stage SVG

The final multi-stage strategy uses openai/gpt-5.6-luna for a compact narrative blueprint, one audit per authenticated evidence group, and an optional decisions-only operation screen. openai/gpt-5.6-sol writes the draft and may propose a bounded splice revision. The normal ceiling is two Sol reservations—draft and revision—with a third allowed only for deterministic validation repair.

Local code owns every boundary. The blueprint must anchor obligations to the narrative and authenticated evidence. Revision operations are indexed against the immutable draft and must fit hard insertion/replacement/citation/word budgets. The Luna screen evaluates support, atomicity, materiality, redundancy, and replacement safety. Only accepted operations are applied; malformed, ambiguous, or unsafe work preserves the already validated draft.

The accepted rag26-ms1-final run used 894 Luna calls and 264 Sol calls—1,158 provider calls total—at a provider-reported cost of $50.423054730. Six planner responses required deterministic deadline normalization and strict revalidation; there were no final failures and no operation-screen crash fallbacks. Answers range from 531 to 1,011 words with 14 to 136 references.

Text equivalent: Luna planning and group audits create a bounded coverage view. Sol writes a validated draft and proposes at most one bounded revision. An optional Luna gate screens each operation; deterministic local assembly either applies the safe subset or falls back to the draft. Final validation precedes the accepted multi-stage JSONL.

Implementation notes and sources

• competition_rag_multistage.py provides full-run identity, bounded concurrency, resume, failure recording, final-row reload, and all-or-nothing publication.
• narrative_blueprint.py, bounded_splice.py, and operation_screen.py define the bounded stage contracts.
• The final run statistics and warnings are preserved in the RAG bundle metadata.json.

Five accepted organizer files

Takeaway: Every accepted file is tracked at its exact submitted bytes, with one ledger connecting run identity, validation result, priority, and SHA-256.

Figure 9: Three Retrieval TSV files and two RAG JSONL files pass task-specific validation. Retrieval checks cover six fields, exact topics, dense ranks, finite non-increasing scores, unique ClimbMix document IDs, and stable run IDs. RAG uses organizer AutoJudge for schema, topics, narratives, citations, and word limits. SHA-256 comparison protects the exact accepted bytes.

Open the reusable accepted-submissions SVG

Retrieval · priority 1

r26-narr-facet-v1
Narrative + subnarrative
4,246 rows · depth 1–121
Accepted by Evalbase

Retrieval · priority 2

r26-facet-breadth-v1
Evidence breadth
4,246 rows · depth 1–121
Accepted by Evalbase

Retrieval · priority 3

r26-narrative-v1
Narrative only
4,246 rows · depth 1–121
Accepted by Evalbase

RAG · priority 1

rag26-ms1-final
Multi-stage Luna + Sol
119 valid reports
Accepted by Evalbase

RAG · priority 2

rag26-ss1
Single-pass Sol
119 valid reports
Accepted by Evalbase

The submission ledger links every exact file, its full SHA-256, and the corresponding archived portal response. It records the Evalbase run-page identifiers and the portal timestamps that were actually observed, without inventing finer precision.

Fresh local preflight produced three Retrieval PASS results—each 4,246 rows, 119 topics, depth 1–121—and two RAG PASS results with 119/119 organizer-valid reports, no extras, and no duplicates. File hashes matched the ledger and both bundle metadata records before this report was finalized.

Text equivalent: Three Retrieval documents and two RAG documents converge on task-specific validation and an exact-hash gate. All five leave the gate with accepted status. The Retrieval and RAG files remain visually separated as sibling branches.

Implementation notes and sources

• Retrieval combo TSV
• Retrieval breadth TSV
• Retrieval narrative-only TSV
• Multi-stage RAG JSONL
• Single-pass RAG JSONL
• Submission validation skill

Final analysis reports

Takeaway: The final analysis set keeps the existing Retrieval quality view separate from two independent, private RAGDoll citation-support reports.

The project has one tracked 119-topic Retrieval quality analysis and two separate 119-topic RAGDoll citation-support reports, one for each accepted RAG run. RAGDoll measures citation support, not official TREC correctness. qrel/gold metrics are unavailable, and this evaluation did not influence accepted priority: the accepted files and their recorded priorities remain unchanged.

Retrieval Quality Analysis

Retrieval Quality Analysis
Existing tracked 119-topic Retrieval coverage analysis.

RAG Analysis: rag26-ss1

RAG Analysis: rag26-ss1
Separate 119-topic RAGDoll citation-support report for accepted single-pass RAG. It is not an official TREC correctness score.

RAG Analysis: rag26-ms1-final

RAG Analysis: rag26-ms1-final
Separate 119-topic RAGDoll citation-support report for accepted multi-stage RAG. It is not an official TREC correctness score.

Sources, validation, and scope

This report is a reader-facing synthesis of the accepted bundle documentation, machine-readable metadata, reusable implementation, checked-in configs, and pinned official contracts linked above. The canonical authored source is reports/2026-competition-architecture.qmd; the standalone HTML is generated with Quarto 1.9.38. Reusable diagrams and their source map live in reports/2026-competition-architecture/.

What is verified here: architecture boundaries, aggregate source and submission counts, model/call identities recorded in checked-in metadata, organizer-format validation, exact accepted-file hashes, and artifact ownership. What is not claimed: official effectiveness scores, causal superiority among the five runs, or fresh validation of the absent private handoff. Acceptance establishes valid delivery, not comparative quality.

The project is complete. Historical specifications and experiment reports remain valuable provenance, but they are records—not an implied next-work list.