TREC RAG 2026 Briefing, rebuilt as a field guide.

This guide explains the current TREC RAG task, the ClimbMix collection, organizer-provided development examples, and a practical path for building a cited retrieval-augmented answer system.

See sample docs and answers Jump to the 2026 pipeline Open the 2025 technique library

Read the narrativeIdentify the facets the answer must cover, not just the keywords to search.

Retrieve ClimbMix evidenceUse BM25 as the control run, then add coverage-oriented queries and reranking.

Write with citationsDraft human-readable prose where factual claims point back to selected document references.

Check support and coverageDo not confuse a well-cited answer with a complete answer.

Start Here: How to Use This Briefing

Use this page as the current-track companion to the deeper 2025 writeup report. It is organized by what a new participant needs to understand first.

10-minute path

Understand the task shape

Read RAG in plain English, ClimbMix, sample documents, and the answer JSON. That is enough to explain the competition to someone new.

builder path

Turn it into a system

Read the 2025 lesson bridge, pipeline, implementation kit, and failure diagnosis. Then build a measured baseline ladder.

audit path

Check claims and evidence

Use the sample qrels, organizer nuggets, source caveats, and evaluation section to keep relevance, coverage, and citation support separate.

RAG In Plain English

A TREC RAG system is a reading assistant with receipts: it finds evidence first, writes second, and makes important claims traceable.

retrieval

Find candidate documents

Search ClimbMix for document IDs that might help answer the narrative.

evidence

Select what matters

Keep the passages or evidence cards that cover distinct facets, not only the highest lexical matches.

generation

Write readable prose

The final RAG output should be a proper answer, not a dump of chunks.

verification

Check the receipts

Every substantive factual answer claim should be supported by the cited references.

Glossary

Narrative

A long user request with multiple aspects. In this task, the narrative is not just search text; it is the specification for evidence coverage and final answer content.

What Is ClimbMix?

ClimbMix is the 2026 evidence collection: a broad curated web/pretraining-style collection rather than a narrow domain corpus.

official unit

Submit document IDs

The official retrieval unit is a ClimbMix row/document with a ClimbMix docid such as shard_05975_34428.

local chunks

Chunk internally when useful

You can chunk documents locally for indexing, reranking, evidence cards, or generation, but keep traceability back to official document IDs.

BM25 baseline

Use BM25 as the control

The Pyserini baseline gives a plain lexical starting point. The narrative then guides decomposition, reranking, evidence selection, and generation.

Important mental model: ClimbMix documents can be long and varied. Treat documents as source containers, then use local passage-level work to find the evidence inside them.

Collection And Dev Set At A Glance

The main corpus is huge; the development files are small practice artifacts layered over that same corpus. Keep those two ideas separate.

ClimbMix corpus

553,240,576 documents

Hugging Face lists the default ClimbMix train split as 553M text rows, about 600 GB of Parquet data.

RAG25 dev

22 narrative topics

These are long, multi-facet TREC RAG 2025 narratives reused as 2026 development queries.

Dev qrels pool

26,341 judgments

The current RAG25 UMBRELA qrels judge pooled ClimbMix candidate documents; they are not a separate dev corpus.

ResearchRubrics

30 deep-research prompts

These prompts come with rubrics for answer diagnosis, not with ClimbMix relevance qrels.

Artifact	Count	What It Means	Source In This Workspace
ClimbMix source corpus	553,240,576 documents / rows	The full evidence collection, exposed through the hosted `climbmix-400b` Pyserini index or a compatible custom index.	Hugging Face `karpathy/climbmix-400b-shuffle`
RAG25 development topics	22 topic narratives	Practice TREC-style narratives with many subtopics each.	`topics/rag25-topics-dev.tsv`
RAG25 development qrels	26,341 judged topic-document rows over 26,151 unique ClimbMix docids	Projected relevance judgments over pooled candidate documents for the 22 RAG25 dev topics.	`rag25-dev-umbrela-qrels/*.qrels`
RAG25 answer nuggets	1,255 nuggets across 22 topics	Atomic coverage targets for diagnosing whether an answer addresses the important facets.	`rag25-dev-nuggets/rag25-dev-nuggets.jsonl`
ResearchRubrics development prompts	30 prompts with 755 rubric criteria	Open-ended deep-research prompts with rubric-based answer quality diagnostics.	`researchrubrics-dev-rubrics/*.jsonl`

RAG25 topics

Mostly public, social, and technical issues

Environment and energy: e-waste, nuclear energy, deforestation, climate change.
Health and safety: vaccines, cancer, chemicals, alcohol and drug use.
Society and policy: banks, abortion, immigration, housing, race, euthanasia.
History and geopolitics: the Holocaust, Korean War, Africa and development.
Technology and media: technology in society, violent games, social media and mental health.

ResearchRubrics domains

Deep-research prompts are broader

The 30 ResearchRubrics prompts cover AI and ML, technical documentation, historical analysis, hypotheticals and philosophy, STEM, business planning, creative writing, current events, and consumer research.

tailoring note

Optimize for coverage, not one domain

The corpus is broad web/pretraining data, and the topics are broad narratives. A good system should decompose needs, retrieve across facets, and then assemble evidence without overfitting to a single subject area.

Set	ID	Example Topic	What It Stresses
RAG25	`31`	Environmental and health impacts of e-waste, recycling benefits, innovations, and practical waste-handling steps.	Multi-facet public-policy synthesis with current practices and risks.
RAG25	`58`	Nuclear energy pros, cons, safety, accident risks, climate impact, fusion comparison, and provider examples.	Evidence coverage across technical, historical, and policy subtopics.
RAG25	`213`	Korean War origins, ending, US involvement, Cold War strategy, political effects, major errors, and presidential views.	Historical causality, chronology, and competing explanations.
ResearchRubrics	`683a58c9a7e7fe4e7695848b`	Explain an NLP research project using de-identified Electronic Health Records to extract Asian subgroups and study autoimmune disease prevalence.	Teaching, research planning, health-data caveats, and method explanation.
ResearchRubrics	`6847465956a0f6376a60535d`	Design a framework for regulating autonomous AI agents that interact with the open internet.	Legal synthesis, containment mechanisms, interpretability, and risk analysis.
ResearchRubrics	`6847465956a0f6376a605391`	Advise a plant-based meat startup on entering Southeast Asian markets such as Singapore, Thailand, and Indonesia.	Market research, regional comparison, regulatory issues, and execution recommendations.

Dev-set document count nuance: the released development data does not define a separate mini corpus. It defines topic files, answer diagnostics, rubrics, and projected qrels over candidate ClimbMix documents.

Sample Documents And Organizer Answers

A concrete dev example makes the task easier to hold: one narrative, judged ClimbMix documents, organizer-provided answer nuggets, and an illustrative response object.

RAG25 dev qid 58: The user wants nuclear energy's pros, cons, safety, accident risks such as Chernobyl, uses, climate impact, comparison with fusion, and examples such as Bison Energy and Peninsula Clean Energy.

Docid	Qrel	Size	What The Document Looks Like
`shard_05975_34428`	4	4,356 words	Article-style web document about nuclear energy as alternative energy, with discussion of climate, waste, safety, and accidents.
`shard_04044_42652`	4	552 words	Short explainer about fission reactions, reactor types, electricity generation, and tradeoffs.
`shard_05087_1995`	4	5,378 words	Long discussion-style web document touching uranium supply, nuclear reliability, alternatives, and policy arguments.

organizer nuggets

Answer supervision is atomic

The dev data provides nuggets, not a polished prose answer. The qid 58 file has 39 nuggets. Examples:

Vital: nuclear energy produces very low greenhouse gas emissions.
Vital: major accidents include Chernobyl, Fukushima, and Three Mile Island.
Vital: nuclear waste is toxic and difficult to manage long term.
Vital: nuclear energy is used mainly for electricity generation.
Okay: fusion energy is experimental.

document record

What a retrieved ClimbMix document actually looks like

A ClimbMix document is not a pre-cut answer passage. The hosted Pyserini API returns a document record with metadata plus one long doc text field. For this short qid 58 example, the full doc field is 552 words; the report shows a complete short excerpt plus a paraphrased outline rather than embedding the entire raw web-corpus text.

{
  "api": "v1",
  "index": "climbmix-400b",
  "docid": "shard_04044_42652",
  "doc_excerpt_20_words": "Fission reactions occur when heavy atomic nuclei, such as uranium-235, are bombarded by neutrons, causing them to become highly unstable."
}

Official ID: cite or submit the parent docid, here shard_04044_42652.
Stored text: the doc field is the full source text returned for that ClimbMix document, not a generated summary.
Document outline: this record explains nuclear fission basics, reactor designs, electricity generation, safety controls, waste issues, and the tradeoffs of nuclear power as a low-carbon energy source.
Local passage work: a system can chunk this text internally, but outputs should still trace back to the official ClimbMix docid.
Fetch pattern: GET /v1/climbmix-400b/doc/shard_04044_42652 retrieves the complete record through the Pyserini REST service.

submitted shape

A RAG answer is prose plus citation indices

The citations values point into the zero-indexed references array.

{
  "metadata": {
    "team_id": "example-team",
    "run_id": "sample-q58-rag-v1",
    "type": "automatic",
    "narrative_id": "58",
    "title": "Nuclear energy benefits, risks, and comparisons",
    "prompt": "Answer using only selected ClimbMix evidence."
  },
  "references": [
    "shard_05975_34428",
    "shard_04044_42652",
    "shard_05087_1995"
  ],
  "answer": [
    {
      "text": "Nuclear energy is mainly used to generate electricity, commonly through fission reactors that produce heat for power generation.",
      "citations": [1]
    },
    {
      "text": "Its climate advantage is low direct greenhouse-gas emissions compared with fossil-fuel generation, but the answer also needs to discuss waste, safety controls, and accident risks.",
      "citations": [0, 1]
    }
  ]
}

second topic

Korean War example

Qid 213 asks about origins, ending, US involvement, Cold War strategy, US politics, major errors, and presidential views. It has 50 organizer nuggets.

Vital: North Korea invaded South Korea on June 25, 1950.
Vital: the Truman Doctrine aimed to contain communism.
Vital: the US underestimated Chinese intervention risk.

how to use samples

Evaluate by facets

Use qrels to inspect retrieved evidence, nuggets to inspect coverage, and answer JSON to inspect whether each factual sentence cites actual support.

The 2025 Lesson Library

The 2025 interactive report is the deeper technique map. This briefing uses it as a source of build patterns, not as a single leaderboard story.

coverage ledger

Long narratives hide many needs

Decompose the narrative into coverage slots and track which slots have strong evidence.

hybrid retrieval

Keep several search signals alive

BM25, dense retrieval, learned sparse retrieval, HyDE, and RRF can contribute different useful evidence.

evidence cards

Compress before writing

Turn passages into source-linked facts or cards before asking the model for polished prose.

dual evaluation

Citation support is not coverage

A cautious answer can be well supported and still miss vital nuggets, so evaluate both lenses.

Where to go deeper: open the 2025 interactive writeup for team dossiers, leaderboards, architecture figures, and a reusable playbook.

The 2026 Build Pipeline

Click a stage to see how I would turn the briefing into a concrete system. This is the shortest practical path from baseline to cited answers.

Stage 1

Baseline with title-first BM25

Run the title query directly against ClimbMix BM25. This creates a cheap, debuggable control run before adding decomposition, dense retrieval, reranking, or generation complexity.

Log: query text, docid, rank, score, and whether each top document covers a real narrative facet.

Implementation Kit

Keep these small data structures and experiments visible. They make it easier to debug whether retrieval, evidence selection, or generation caused a failure.

schema

NarrativeNeed

{
  "qid": "58",
  "need": "major nuclear accidents and impacts",
  "importance": "vital",
  "status": "covered|weak|missing"
}

schema

EvidenceCard

{
  "docid": "shard_05975_34428",
  "need": "climate impact",
  "claim": "Nuclear power has low direct greenhouse gas emissions.",
  "support_span": "short local excerpt or passage pointer"
}

schema

ClaimCitation

{
  "sentence": "Nuclear energy is mainly used for electricity generation.",
  "citations": [1],
  "support_label": "supported"
}

Experiment	What It Tests	Keep It If	Failure Risk
BM25 baseline	Exact-term evidence	It finds useful docids for entity-heavy topics.	Vocabulary mismatch.
Facet searches	Coverage of multi-part narratives	They add distinct relevant docs.	Query drift.
Hybrid retrieval	Complementary lexical and semantic signals	It adds unique useful evidence.	Noisy fusion.
Evidence cards	Cleaner generation input	They preserve rare but vital facts.	Compression loss.
Citation-first generation	Claim support	It improves support without collapsing coverage.	Safe but incomplete answers.

Evaluation: Coverage Plus Support

Think of grading as two linked questions: did the system find evidence that covers the need, and did the final answer make only claims that evidence can support?

retrieval

Facet-aware relevance

A document can be valuable if it answers several parts of a long narrative, even if it is not the best match for one exact phrase.

generation

Readable answer quality

RAG output should be human-readable prose with citations, not only a list of cited chunks.

support

Every citation must prove the claim

Related text is not enough. A citation should support the specific sentence it is attached to.

Citation support is not coverage: a short answer can cite every sentence correctly and still miss vital nuggets. A long answer can cover many nuggets and still contain unsupported citations. Track both.

Failure Diagnosis

When a run looks bad, name the failure before adding another model. Most issues fall into a small set of repeatable patterns.

missing facet

Answer is fluent but incomplete

Likely cause: direct query over-weighted the easiest part of the narrative.
Fix: add a coverage ledger and retrieve for weak facets.

query drift

Decomposition retrieves generic pages

Likely cause: subqueries dropped the topic anchor.
Fix: keep entity and topic terms from the original narrative in each facet query.

citation decoration

Citations are topical but not proving

Likely cause: citations were attached after drafting.
Fix: generate claim-source pairs and reject unsupported claims before final prose.

Symptom	Likely Cause	Debug Signal	Repair
High retrieval score, weak answer	Evidence was unorganized	Generator prompt has many duplicate passages	Use evidence cards or nuggets before generation.
High support, low coverage	Answer is too cautious	Many vital nuggets absent	Force coverage check before final answer.
Long answer, weak citations	Unsupported synthesis	Sentences cite docs that only mention the topic	Run a sentence-level support judge.
Facet search hurts	Subqueries are vague	Retrieved docs answer the wrong domain	Anchor each facet with original entities.

Appendix

Exact formats, local source paths, and caveats live here so the main report stays readable.

2026 submission formats

Retrieval submissions use the standard six-column TREC row:

topic_id Q0 docid rank score run_id

RAG submissions use JSONL with metadata, references, and answer. Citation indices in answer objects point into the zero-indexed references array.

Development data mental model

Local development data includes 22 RAG25 narrative topics, 30 ResearchRubrics prompts, organizer nuggets, ResearchRubrics rubrics, and projected qrels. Treat them as practice material for understanding topic shape, citation behavior, and evaluator expectations.

Development qrels are useful for experiments and error analysis, but they cover pooled candidate documents rather than the full 553,240,576-document ClimbMix corpus. A missing judgment does not prove that every unjudged ClimbMix document is irrelevant.

Sources

trec-rag-data/trec-rag-2026/development-data/topics/rag25-topics-dev.tsv for sample topic narratives.
trec-rag-data/trec-rag-2026/development-data/topics/research-rubrics-topics-dev.tsv for ResearchRubrics development prompts.
trec-rag-data/trec-rag-2026/development-data/rag25-dev-nuggets/rag25-dev-nuggets.jsonl for organizer-provided answer nuggets.
trec-rag-data/trec-rag-2026/development-data/rag25-dev-umbrela-qrels/rag25-climbmix-umbrela-codex-gpt5.5-medium-reasoning-v1.qrels for sample ClimbMix relevance judgments.
trec-rag-data/trec-rag-2026/development-data/rag25-dev-umbrela-qrels/README.md for qrels creation notes and the 26,341-judgment count.
trec-rag-data/trec-rag-2026/development-data/researchrubrics-dev-rubrics/research-rubrics-dev-rubrics.jsonl for ResearchRubrics domains and rubric criteria.
Hugging Face ClimbMix dataset page for corpus row count and file size.
Official TREC RAG site
Official 2025 TREC RAG page
TREC RAG overview on arXiv
TREC 2025 RAG proceedings browser
Local 2025 interactive writeup report