Reports Home

TREC RAG 2025 · source-backed architecture synthesis

The Most Promising TREC RAG 2025 Architecture

Evidence synthesis

Short answer

The most promising architecture pattern from TREC RAG 2025 was the path from a long narrative to explicit coverage units, broad complementary retrieval, unit-level evidence selection, evidence-backed claims, and a final synthesis with separate coverage and support checks.

Turn the narrative into explicit coverage units; retrieve broadly with complementary methods; rerank/select evidence per unit; draft only evidence-backed claims; then synthesize and verify the final answer.

The strongest evidence is split. UTokyo-HitU supplied the best verified retrieval front end: query expansion + HyDE, BM25 + SPLADE + two dense retrievers, RRF, then sliding-window LLM reranking. Its 4method_merge run led the official manually assessed retrieval table at nDCG@30 0.6934. Official overview, PDF p. 10, Table 5; UTokyo paper, pp. 1–4, §§2–3

NC State LAS supplied the strongest documented full-RAG result in the fully manual nugget table: its single-agent run scored 0.37 strict-vital and 0.65 sub-narrative coverage, second overall behind an RMIT run for which this repository has no team paper. The richer LAS selector-agent workflow scored lower. Official overview, PDF p. 11, Table 7; LAS paper, pp. 5–13, §§3–4

MITLL supplied the clearest reproducible end-to-end blueprint: narrative → minimal spanning subquestions → SPLADE retrieval per subquestion → pointwise reranking → RRF → set-level evidence selection (SETR) → GPT-5 cited answer. MITLL paper, pp. 1–4, §§1–3 CFDA supplied the clearest claim-controlled generator: subquery-specific evidence pools → one short supported answer per subquery → integrated paragraph with traceability → sentence-level support verification. CFDA paper, pp. 2–5, §§3.1–3.3

The path from narrative to final RAG

Seven-stage recommended RAG architecture: narrative contract, coverage plan, hybrid retrieval, rerank and set selection, evidence ledger, facet answers, and final coverage and support checks.
Recommended composite architecture distilled from the TREC RAG 2025 systems. The original narrative remains the global contract throughout.
Open full-size architecture figure
  1. Long narrative → coverage plan. Create 5–10 non-redundant facets or subquestions, optionally marking likely vital facets, while carrying the original narrative with each facet.
  2. Retrieve broadly for each facet. Combine BM25 or keyword expansion, learned sparse retrieval (SPLADE), dense retrieval, and HyDE-enhanced dense retrieval.
  3. Fuse, rerank, and select a set. Use late fusion (RRF), reranking, then diversity or set selection rather than a single global top-k.
  4. Maintain an evidence ledger. For every facet, record claims / nuggets + source IDs, supporting segment IDs, support span, and confidence.
  5. Draft one short cited sub-answer per facet. A claim enters prose only from evidence assigned to its facet.
  6. Synthesize under the original narrative. Integrate facet answers without treating them as an excuse to lose coherence or readability.
  7. Run two separate checks. Coverage asks which facets or vital nuggets are missing; support asks whether every substantive sentence has proving evidence.

Definitions and the organizer/runtime boundary

Subquery
A search instruction derived from the narrative.
Nugget
An atomic fact or information unit that should be covered.
Claim
A sentence-level assertion the system actually proposes to put in the answer.
Citation
Acceptable only when its segment supports the claim, not merely when it is topically related.

Teams sometimes used the word “nugget” for a generated subquery or facet, so the terminology is not perfectly consistent across papers. CFDA explicitly calls its decomposed subqueries “nuggets,” while LAS uses nuggets for atomic content distilled from initially retrieved documents. CFDA, PDF pp. 2–3, §3.1; LAS, PDF p. 8, §4.1.2

How the leading systems actually did it

Cross-team roles in the recommended composite architecture
RoleTeamVerified evidenceWhat to reuse
Simple controllerNC State LASSingle-agent full-RAG run: 0.37 strict-vital / 0.65 sub-narrative coverage; selector-agent scored lower.Simple controller, explicit stopping reason, and caution about added roles.
Retrieval front endUTokyo-HitU4method_merge led manual retrieval at nDCG@30 0.6934.Query expansion, mixed HyDE vector, four-way RRF, sliding-window reranking.
Evidence blueprintMITLLMinimal spanning subquestions, parent narrative appended, SPLADE, pointwise reranking, RRF, SETR.Facet coverage contract and set-level evidence selection.
Claim-controlled generatorCFDASubquery evidence pools, short supported answers, traceable integration, sentence support checks.Evidence-first subanswers and final support verification.
Nugget-to-claim interfaceGenAIusQuery-conditioned atomic nuggets retain passage provenance, and each answer sentence cites nugget IDs.Use an auditable evidence ledger between retrieval and final claims.
AlternativesTokyo University of Science; WaterlooClarkeRetrieval-level fusion and answer portfolios offer complementary diversity strategies.Use diversity carefully while preserving atomic evidence provenance.
1. NC State LAS: simplest agent won the best documented manual full-RAG result

Verified flow

  1. An agent could decompose or rewrite the narrative and search BM25 and SPLADE tools.
  2. It gathered retrieved context and wrote an answer with citations.
  3. miniblame assigned citations by sentence-level semantic similarity.
  4. The submitted single-agent run beat LAS’s planner/researcher/writer/reviewer selector system.

LAS also tested a separate retrieval-only feedback loop: retrieve with the original query, use GPT-4o to distill non-overlapping “vital” and “okay” nuggets from top documents, turn vital nuggets into new searches, and combine results. Its 2025 ablation found that simply concatenating decomposed subquestions into one SPLADE query was the “best bang for the buck”; the second nugget-driven search round improved top-rank precision but not recall enough to justify its cost. LAS, PDF pp. 8, 11–13, §§4.1.2, 4.2.3–4.3

What the scores establish

The official fully manual table places LAS-agentic-RAG-agent at 0.37 strict-vital / 0.65 sub-narrative coverage and LAS-agentic-RAG-selector at 0.30 / 0.60. This is unusually valuable evidence that added agent roles did not automatically improve the final answer. Official overview, PDF p. 11, Table 7

Caveat

The paper describes the agent’s available tools and general behavior more clearly than its exact stopping policy and intermediate data contract. It is a strong outcome, but a less inspectable implementation template than MITLL or CFDA.

2. UTokyo-HitU: the strongest retrieval front end

Verified flow

  1. GPT-4.1 generated keyword expansion for sparse retrieval and a roughly 150-word hypothetical answer for HyDE.
  2. Sparse search used keyword-expanded BM25 and SPLADE.
  3. Dense search used BGE-small and Qwen3 embeddings. Their “HyDE Vector Mix” retained both the original-query vector and hypothetical-answer vector rather than replacing the query.
  4. The four top-1,000 lists were fused with RRF.
  5. GPT-4.1-mini reranked up to 200 candidates with three sliding-window passes and saw title, URL, and segment text.
  6. GPT-4.1/Ragnarok generated from the original query and top 20 segments, then segmented sentences and extracted citations.

UTokyo, PDF pp. 1–4, §§2.1–2.8

What the scores establish

The official manual retrieval table ranks 4method_merge first at nDCG@30 0.6934, nDCG@100 0.6134, recall@100 0.2331. UTokyo’s paper reports recall@100 0.257; the official overview’s table is used here when values differ. Official overview, PDF p. 10, Table 5

Caveat

UTokyo did not explicitly decompose the long narrative into coverage slots, and its final generator was comparatively direct. Its two-method RAG run scored better than its four-method RAG run on the team paper's official vital/coverage table (0.56 / 0.84 versus 0.53 / 0.79), and the authors report that more complex MMR, filtering, and multi-step generation did not beat direct Ragnarok generation. Its retrieval stack is the best front end, not proof that more stages always make the best final answer. UTokyo, PDF p. 4, Table 8 and discussion

3. MITLL: best inspectable narrative-to-answer blueprint

Verified flow

  1. Gemma 3 27B decomposed the narrative into a minimal spanning set of fully independent subquestions—about eight on average.
  2. MITLL concatenated the parent narrative to each subquery because a subquery can lose meaning outside its original context.
  3. Each expanded subquery went through SPLADEv3 retrieval and a Qwen3 or Gemma pointwise reranker.
  4. RRF fused the per-subquery lists.
  5. SETR selected a cohesive passage set intended to contain all information needed, rather than treating passage relevance as wholly independent.
  6. GPT-5 generated the final answer, with 1–3 citations required per sentence.

MITLL, PDF pp. 1–4, §§1–3

What the ablation establishes

MITLL’s team-reported official table shows the full pipeline at 0.47 strict-vital / 0.79 sub-coverage, while removing decomposition scored 0.50 / 0.77 and removing the reranker scored 0.48 / 0.77. Thus decomposition and reranking did not dominate on every metric; their clearest gain was coverage, while simpler variants slightly improved strict-vital. MITLL, PDF p. 4, Tables 2–3

This is why the architecture is promising, not proven universally superior.

4. CFDA: turn facets into verified claims before final prose

Verified flow

  1. An LLM decomposed the original query into focused “nugget” subqueries, directly or with pseudo-relevance feedback.
  2. BM25 and MiniLM retrieved broad lexical and semantic pools; RRF combined them.
  3. Qwen3 embeddings and then MiniLM/ColBERT variants refined the list.
  4. For each subquery, a cross-encoder selected a small evidence pool.
  5. An LLM generated one concise answer of under roughly 35 words, with every factual claim constrained to that subquery’s evidence.
  6. Another LLM integrated indexed sub-answers into a coherent final paragraph while preserving sentence-to-subanswer traceability.
  7. For each final sentence, the union of supporting passages from its source subanswers was checked by a support-evaluation LLM.

CFDA, PDF pp. 2–5, §§3.1–3.3 and Figure 2

Why it matters

This is the cleanest answer to “where do claims come from?” Claims are not extracted blindly from the narrative. Facets retrieve evidence; evidence produces small supported sub-answers; final sentences inherit their supporting evidence; then support is checked.

Caveat

CFDA’s final official scores were competitive rather than leading, so this is an architectural lesson, not a winner claim.

5. TUS and WaterlooClarke: two useful alternatives

TUS (Tokyo University of Science) decomposed the narrative into viewpoints and compared retrieval-level fusion: diversify or consolidate documents across decomposed queries, then generate once; and answer-level fusion: generate a partial answer for each decomposed query, then edit or integrate them.

Its uema2lab_B4 retrieval-level-fusion run was third among systems with local writeups in the fully manual table (0.33 / 0.55), while the paper warns that subanswer fusion can introduce redundancy and lose readability. TUS, PDF pp. 2, 9–15, §§2–3; official overview, PDF p. 11, Table 7

WaterlooClarke generated a portfolio of answers: a nuggetizer pipeline that converted a cited answer into atomic nuggets and retained vital ones; GARE, which drafted an answer, extracted atomic claims, retrieved evidence for each, then validated or rewrote each claim; an explicit multi-step retrieval/generation plan; and a combined run pooling passages from all strategies before nuggetization.

combined and auto_selected placed just behind TUS in the fully manual table. This supports portfolio diversity, but not a claim that answer-first factual reconstruction is safer than evidence-first generation. WaterlooClarke, PDF pp. 1–3, §§2–7; official overview, PDF p. 11, Table 7

6. GenAIus: the cleanest nugget-to-claim interface

GenAIus worked in the fixed-retrieval AG task rather than end-to-end retrieval, but its intermediate representation is especially instructive:

  1. Take the supplied top 20 passages.
  2. Extract query-conditioned atomic nuggets from each passage while retaining passage provenance.
  3. Either generate directly from all nugget IDs or cluster nuggets by subtopic first.
  4. Require each answer sentence to cite nugget IDs.

Direct nugget generation modestly beat clustered generation in the team's results; the authors argue that clustering can lose atomic granularity. This is the clearest primary-source example of turning retrieved evidence into an auditable fact ledger before final claims. GenAIus, PDF pp. 1–3, 6–8, §§2.1 and 4

Facts, inference, and unknowns

Verified

  • UTokyo led the official manually assessed retrieval table.
  • LAS’s single-agent run was the best full-RAG run with a team paper in this repository under the fully manual nugget table.
  • MITLL, CFDA, and TUS explicitly decomposed narratives and retained subquery-specific evidence or answers.
  • LAS’s more complex selector-agent submission did not beat its single agent.
  • Official evaluations disagree by assessment setting; rankings should not be flattened into one “winner.”

Inference

  • The most promising practical design is a facet-led, evidence-first, claim-verified pipeline, not an unconstrained autonomous agent.
  • UTokyo’s retriever and CFDA’s generator are complementary modules worth combining.
  • A coverage ledger is a better intermediate contract than a loose list of subqueries because it can drive retrieval, evidence selection, generation, and final auditing.

Unknown

  • No 2025 submission tested the exact composite recommended here.
  • The repository has no paper for overall manual winner Kun-Third (RMIT-IR), so its architecture cannot be verified locally.
  • The official overview and some team papers report slightly different numbers or evaluation variants; the official overview tables take precedence for cross-team comparisons.

Primary sources

All provenance links below are repository-relative local PDFs from the TREC RAG 2025 writeups.