TREC RAG 2025, explained for a first-time reader.

Architecture

CFDA built a two-sided system: a multi-stage retrieval stack and a hierarchical AG stack. Retrieval starts with BM25 and lightweight embedding retrieval, combines candidates with RRF, applies Qwen reranking, and can add ColBERTv1/MiniLM reranking. Generation decomposes the narrative into subqueries, generates concise cited answers for each subquery, indexes those partial answers, then synthesizes a final sentence-cited response.

What they tried

They evaluated Q2E query expansion, Q2E with pseudo-relevance feedback, BM25-only retrieval, embedding-only retrieval, RRF fusion, Qwen reranking, ColBERTv1 reranking, MiniLM reranking, and multi-reranker fusion. On the AG side, they compared their compositional pipeline against a single-pass baseline that retrieves, concatenates, and asks the LLM to answer once.

Models and tools

BM25, lightweight embeddings, Qwen reranking, ColBERTv1, MiniLM, RRF, GPT-4.1-Mini, Llama-3.1-8B, AutoNuggetizer-style grading, Union Nuggets Coverage, and sentence-support metrics.

Results and observations

RRF across BM25 and Stage 2 improved retrieval to nDCG@10 0.5432 and Recall@5000 0.7734. ColBERTv1 alone reached nDCG@10 0.5928, while fusing ColBERT, MiniLM, and earlier stages produced the best short-cutoff nDCG@3 0.6307 and nDCG@5 0.6167. Their AG pipeline improved UNC@4 from 0.6981 to 0.9457 with GPT-4.1-Mini and from 0.4819 to 0.9352 with Llama-3.1-8B, with sentence support remaining broadly comparable to baseline.

Caveats

Their UNC and sentence-support analysis uses third-party/autograder style tooling rather than only final official TREC human judgments. The design also trades simplicity for many moving parts, so deployment would need careful stage-by-stage monitoring.

What we can reuse

• Reusable move: Turn one hard narrative into a set of evidence-backed subanswers, then synthesize the final cited answer from those subanswers.
• Plain English: Do not ask the model to solve the whole messy question in one breath. Break it into smaller questions, collect receipts for each, then write the final answer.
• Try first: Generate 5 to 8 subquestions, retrieve with BM25 plus embeddings for each one, fuse with RRF, and create one cited mini-answer per subquestion before final synthesis.
• Measure: Compare nugget coverage and sentence support before and after decomposition; CFDA's useful gain was better coverage without a clear support collapse.
• Watch for: This adds many moving parts, so log every stage. If a subanswer is weak, the final synthesis may sound authoritative while inheriting that weakness.
• Related teams: uogTr, TUS, MITLL, and NC State LAS also treat decomposition as a coverage tool rather than just prompt decoration.

Architecture

NIT Agartala built a straightforward comparative pipeline across all subtasks. Retrieval had three variants: BM25 lexical retrieval, dense DPR-style retrieval with FAISS, and a hybrid sparse+dense approach followed by cross-encoder reranking. Generation used retrieved evidence segments in structured prompts and post-processed citations into the required JSONL format. Their RJ run used a BAAI/bge-reranker-large2 style model to score query-segment pairs.

What they tried

They submitted BM25, dense, and hybrid retrieval runs; AG and full RAG runs with instruction-tuned answer generation; and an automated relevance-judgment run. The point of the paper is largely comparative: how far a clean sparse baseline gets, how much dense retrieval adds, and whether hybrid reranking improves both.

Models and tools

BM25 via Anserini, DPR-style dense embeddings, FAISS indexing, BAAI/bge-reranker-large2, Falcon-7B-Instruct for generation, cross-encoder reranking, and TREC-format post-processing.

Results and observations

BM25 scored nDCG@30 0.397 and Recall@100 0.112. Dense retrieval improved to nDCG@30 0.486 and Recall@100 0.133. The hybrid run kept nDCG@30 at 0.486 but improved nDCG@100 to 0.407 and Recall@100 to 0.158. Their full RAG result was strict vital score 0.19, sub coverage 0.36, weighted precision 0.472, and weighted recall 0.472. Their AG run scored strict vital 0.18, sub coverage 0.35, weighted precision/recall 0.481.

Caveats

The generation results show the ceiling of a relatively simple evidence-to-answer stack: retrieval improves, but answer coverage remains limited when the retrieved evidence does not cover all query aspects. Their automatic RJ agreement was also low, with kappa around 0.06, showing the difficulty of automated relevance labels.

What we can reuse

• Reusable move: Use a baseline ladder: BM25, dense retrieval, hybrid fusion, reranked hybrid, then full RAG.
• Plain English: Before adding clever agents or complicated prompts, build the boring versions and know exactly how much each upgrade buys you.
• Try first: Keep four runnable configs: BM25-only, dense-only, BM25+dense, and BM25+dense+cross-encoder reranker. Use the same generator for all four.
• Measure: Track nDCG@30, Recall@100, strict vital score, sub coverage, and weighted precision/recall for the same topic set.
• Watch for: Better retrieval alone did not make their generated answer complete; answer coverage still needed better evidence selection and prompt structure.
• Related teams: MITLL is the closest ablation mindset; UTokyo-HitU and CFDA show stronger hybrid retrieval variants after the baseline ladder is understood.

Architecture

uogTr compared two generation workflows on top of the same retrieval setup. Run 1 explicitly decomposes the narrative into subqueries, retrieves/generates for each, and merges the partial answers. Run 2 uses Search-R1, an agentic RAG model that determines additional information needs and searches iteratively. Retrieval used PyTerrier-style sparse/dense components and MonoT5-style reranking.

What they tried

The key experiment was explicit planning versus agentic search. Run 1, genSubQ_merge, represents a planned decomposition-and-generation strategy. Run 2, e5_monot5_searchR1, relies more on a trained agent deciding what to retrieve next.

Models and tools

PyTerrier, PyTerrier-RAG, E5 dense retrieval, sparse retrieval, MonoT5, Search-R1, Llama-family generation, explicit prompts for decomposing description-style queries into subqueries.

Results and observations

Run 1 beat Run 2. With automatic nuggets, genSubQ_merge reached strict vital score 0.3071 and sub-narrative coverage 0.5590, while e5_monot5_searchR1 reached 0.1510 and 0.3562. With post-edited nuggets, Run 1 scored 0.2465/0.5178 versus Run 2's 0.1383/0.3596. The team observed that Run 1 issued 9.71 searches per query on average, while Run 2 issued only 3.82, likely missing more aspects.

Caveats

Both runs were below the track median. Explicit decomposition improved coverage but can introduce less relevant subqueries and less tightly integrated final answers. Agentic search is flexible but needs stronger control over when to search and when to stop.

What we can reuse

• Reusable move: Prefer an explicit query plan when the user request has multiple facets, and compare any agent against that plan.
• Plain English: A self-directed search agent sounds powerful, but a simple checklist of subquestions may cover the user's need better.
• Try first: Generate subquestions, run retrieval for each, write a short answer per subquestion, then merge. Log how many searches each subquestion actually triggered.
• Measure: Count searches per query, covered subquestions, strict vital score, and sub-narrative coverage. uogTr's stronger run searched more and covered more.
• Watch for: Agentic retrieval can stop too early. If the agent issues few searches, it may look efficient while missing evidence.
• Related teams: GRILL Lab explores iterative search loops; NC State LAS shows that a transparent single-agent baseline can beat more complex agent teams.

Architecture

WING-II worked in the AG setting where retrieval was fixed. Their primary architecture selects 24 diverse evidence segments with a greedy submodular objective, compresses selected segments into short evidence cards, generates citation-first claims, then optionally applies a post-hoc refiner.

What they tried

They compared a primary submodular/evidence-card/citation-first system, a primary-plus-refiner variant, a lighter No-LLM concatenation-style baseline, and a No-LLM-plus-refiner variant. This directly tested whether upstream evidence organization or downstream rewriting mattered more.

Models and tools

Greedy submodular selection, pool-IDF n-gram coverage, BM25 relevance, host-level novelty, evidence-card compression, strict JSON citation-first generation, and a post-hoc rewriting/refinement stage.

Results and observations

The primary family scored strict vital 0.35 and sub coverage 0.51, versus the No-LLM baseline at 0.18 and 0.37. Adding a refiner to the primary system preserved coverage while improving weighted precision/recall from 0.578 to 0.690. Applying a refiner to the weaker baseline produced high weighted scores in the released package, but weaker strict vital and sub coverage, and the paper warns the metric topic subsets differ.

Caveats

Because retrieval was fixed, the results isolate answer generation and evidence selection rather than end-to-end retrieval. Weighted precision/recall were not reported on the same topic set for every run, so coverage metrics are the cleanest comparison.

What we can reuse

• Reusable move: Select a diverse evidence set before generation, then compress it into evidence cards.
• Plain English: Do not dump the top retrieved chunks into the prompt. First decide which chunks deserve to be the answer's receipts.
• Try first: Score candidates for relevance, n-gram coverage, source novelty, and redundancy. Pick a small set, then summarize each item into a source-linked card.
• Measure: Compare strict vital, sub coverage, and sentence support with and without submodular selection and evidence cards.
• Watch for: A refiner can make weak evidence sound better. Polishing should happen after evidence selection, not instead of it.
• Related teams: TUS uses keystone document selection, IDACCS uses extractive compression, and GenAIus/HLTCOE use nuggets as the evidence unit.

Architecture

GRILL Lab used a saturation-based retrieval architecture. A query is decomposed into subquestions; each subquestion goes through Retrieve -> Expand -> Refine. Stage 1 uses BM25, Stage 2 uses Query-by-Document expansion, and Stage 3 uses LLM gap analysis to formulate supplementary queries. Results are merged with RRF, with relevance filtering between stages.

What they tried

For RAG retrieval, they compared GPT-4.1 decomposition against a GPT-5-class decomposition model. For full RAG, they paired those retrieval pipelines with GPT-4.1 or GPT-5 generation. The same architecture was also explored in IKAT and simulation settings.

Models and tools

BM25, Query-by-Document expansion, LLM gap analysis, RRF, MonoT5 reranking, GPT-4.1 nano fine-tuned as a binary relevance filter, GPT-4.1 and GPT-5-class models for planning/generation.

Results and observations

The paper emphasizes architecture and limitations more than final official RAG scores. It argues that iterative saturation can improve topic coverage, but the final ranking budget and stage ordering can cap the benefit. The system's main claim is that LLMs should be active retrieval-loop components: rewriting, filtering, and identifying missing information.

Caveats

The hard reranking cap of 500 documents may discard useful candidates discovered by multi-round expansion. Iteration also has cost overhead, and the reordered gap-before-expansion variant was described as an architectural hypothesis still needing empirical validation.

What we can reuse

• Reusable move: Use retrieval loops: retrieve, expand from promising documents, ask what is missing, then retrieve again.
• Plain English: Search once, inspect what you found, then search again for the gaps instead of hoping the first query was enough.
• Try first: For each subquestion, retrieve with BM25, expand using query-by-document terms from useful hits, run LLM gap analysis, and fuse results with RRF.
• Measure: Track new relevant evidence per loop, uncovered facets removed per loop, and cost per useful document added.
• Watch for: Iteration can explode cost and noise. Add a relevance filter and stop when new searches stop adding coverage.
• Related teams: uogTr and NC State LAS expose the need for search accounting; UTokyo-HitU and CFDA show how late fusion can preserve multiple retrieval signals.

Architecture

IIUoT framed RAG as a passage-to-claim alignment problem. Their stack starts with BM25 over the corpus, reranks candidates with progressive sliding-window listwise reranking, optionally diversifies the selected context with MMR, and then uses a generator designed to emit references before finalizing claims. Invalid or unsupported citation references are discarded rather than patched afterward.

What they tried

Their central experiment was whether a strong listwise LLM reranker plus citation-constrained generation could produce better supported answers. They used a windowed reranking scheme because the reranker cannot inspect the full candidate list at once: a window of candidates is reranked, the top items are carried forward, and the process continues until the final evidence set is chosen.

Models and tools

BM25 with Anserini, RankZephyr/castorini rank_vicuna_7b_v1_fp16 through RankLLM, progressive sliding windows of about 50 candidates with top-20 carryover, MMR diversity, and a Llama-3.1 generation setup with dual adapters for answer text and evidence references.

Results and observations

The run scored low on coverage but high on support-style measures: strict vital score 0.20 against a median near 0.41, sub-narrative coverage 0.24 against a median near 0.67, but weighted precision 0.77 and weighted recall 0.76 against medians around 0.40 and 0.39. In plain English, the system was conservative: it supported what it said well, but missed many expected facts.

Caveats

Reranking and citation constraints can over-optimize for safe claims. If the selected evidence set is too narrow, the generator becomes precise about a partial answer. The paper also exposes a common TREC RAG tension: high support does not automatically mean high nugget coverage.

What we can reuse

• Reusable move: Make citation assignment part of answer generation, not a cleanup pass after the answer is already written.
• Plain English: The model should write each claim with its receipt attached. If it cannot name the receipt, the claim should not survive unchanged.
• Try first: Rerank broadly, optionally diversify context with MMR, then require the generator to emit sentence-level source references before accepting each claim.
• Measure: Track weighted precision/recall beside strict vital and sub coverage. IIUoT shows citation discipline can raise support while coverage stays low.
• Watch for: Citation constraints can create a safe but incomplete answer. Support metrics and coverage metrics must be read together.
• Related teams: WING-II and HLTCOE also bind generation to evidence; CFDA and WaterlooClarke use cited synthesis as part of broader pipelines.

Architecture

GenAIus built an AG and RJ system around nuggets. Given the fixed input passages, the pipeline asks GPT-4o to extract concise factual nuggets, then either feeds those nuggets directly into generation or clusters them by subtopic before synthesis. The same nugget inventory is reused for relevance-judgment signals such as nugget count, normalized nugget count, cluster count, unique cluster count, and citation count.

What they tried

They compared a nugget-generation run against a cluster-generation run. The direct variant preserves every atomic fact as its own unit; the clustered variant tries to organize facts into subtopics before writing, which can make the answer more coherent but may hide minority facts.

Models and tools

GPT-4o nugget extraction, subtopic clustering, cited synthesis prompts, AutoAssign and AutoNuggetizer evaluation views, and simple RJ scoring features based on nuggets, clusters, and citations.

Results and observations

With post-edited nuggets plus AutoAssign, both generation styles reached strict vital score 0.38, while direct nugget generation had slightly higher sub-narrative coverage at 0.66 versus 0.64. Under AutoNuggetizer, direct nuggets scored strict vital 0.41 and sub coverage 0.63, while clustering scored 0.40 and 0.62. Manual support evaluation favored direct nuggets too: weighted precision/recall 0.7780 versus 0.7483. Their citation retrieval nDCG@30 was 0.5373.

Caveats

The RJ signals were weak as judge replacements: agreement fractions were only about 0.26-0.28 and kappa values stayed low. Nugget counting is useful as a feature, but not enough to decide relevance by itself.

What we can reuse

• Reusable move: Keep atomic nuggets as a first-class intermediate representation before prose generation.
• Plain English: A nugget is one small useful fact. Make a fact inventory first, then use it to write and check the answer.
• Try first: Extract concise facts with source citations, save the raw list, deduplicate near-paraphrases, then cluster only after preserving the original facts.
• Measure: Compare direct-nugget generation against clustered generation on strict vital, sub coverage, and weighted support. GenAIus found direct nuggets slightly stronger.
• Watch for: Nugget counts alone are weak relevance judges. They help structure evidence, but they do not replace calibrated support labels.
• Related teams: HLTCOE generates candidate nuggets first, WaterlooClarke tests a nuggetizer strategy, and CFDA uses nugget coverage to inspect AG quality.

Architecture

Tokyo University of Science proposed Keystone-Docs RAG. The pipeline decomposes the narrative, extracts topic constraints to prevent query drift, expands those topics, retrieves with hybrid sparse+dense search, fuses ranked lists with RRF, assigns segments to the decomposed needs, and then generates from selected keystone documents rather than from a huge undifferentiated context dump.

What they tried

They submitted several variants: uema2lab_B4 uses retrieval-level fusion with a Parasol Score; uema2lab_base uses answer-level fusion with the top 10 passages per decomposed query; and uema2lab_rag_fewdoc uses only the top 5 to reduce context burden. The comparison directly asks whether the generator benefits more from more evidence or from less but cleaner evidence.

Models and tools

Keystone-Docs decomposition, topic extraction, topic expansion, BM25-style sparse retrieval, dense retrieval, RRF, Parasol Score selection, segment assignment, and Gemini 1.5 Pro for final generation.

Results and observations

In the official manual table, uema2lab_B4 reached strict vital 0.33 and sub coverage 0.55. uema2lab_base scored 0.27 and 0.53. The few-document variant dropped to 0.17 and 0.41. The takeaway is nuanced: reducing context helps only if the remaining documents still cover the narrative; underselecting evidence can collapse coverage.

Caveats

The paper is more design-forward than ablation-heavy and notes that future work will analyze evaluation results more deeply. The term "keystone" should be read as an evidence-selection hypothesis, not a solved guarantee that a few documents always suffice.

What we can reuse

• Reusable move: Treat document selection as a coverage problem, not just a top-k ranking problem.
• Plain English: A few great documents are useful only if together they cover the whole question. Fewer documents is not automatically better.
• Try first: Extract topic constraints, expand subtopics, assign retrieved segments to narrative needs, then choose keystone documents that jointly cover those needs.
• Measure: Compare full evidence versus few-document evidence on strict vital and sub coverage. TUS's few-document variant lost coverage when it underselected.
• Watch for: Topic expansion can drift. Keep constraints from the original narrative so expanded queries do not wander away from the user's question.
• Related teams: WING-II selects diverse evidence segments, IDACCS compresses selected evidence, and CFDA/uogTr decompose the narrative before selecting evidence.

Architecture

HLTCOE built Crucible, a nugget-first system that reverses the normal "retrieve then write" pattern. It generates candidate nuggets from retrieved documents, merges paraphrases, filters none/irrelevant/unreasonable answers, selects roughly 10-20 high-value nuggets, extracts supporting sentences, verifies citation and nugget coverage, chooses compact evidence, trims to length, and polishes the final report.

What they tried

The team explored different retrieval sources and filtering strategies, including BM25, PLAIDX, Qwen3, LSR, dense approaches, Llama-3.3-70B listwise reranking, and SVC-based nugget ranking. They also compared ablated versions to see what happens when nugget ideation, validation, or retrieval strength changes.

Models and tools

Crucible nugget ideation, paraphrase clustering, answer filtering, nugget ranking, supporting-sentence extraction, Llama-3.3-70B reranking, BM25, PLAIDX, Qwen3, LSR, dense retrieval, and verification checks for sentence support and nugget coverage.

Results and observations

The reported RAG run with answer retrieval reached sub-nugget coverage 0.541 and weighted precision 0.686, while an ablated run reached lower sub-nugget coverage 0.481 but higher weighted precision 0.798. In RAGTIME-style support checks, sentence support was high, around 0.881-0.973, while nugget coverage stayed around 0.267-0.348. They note that Milco/LSR-style retrieval helped nugget coverage but could reduce sentence support.

Caveats

The whole design depends on the quality of the generated nugget set. Verification can improve support, but if the system-generated nuggets do not match the official gold nuggets, the final response may be well supported yet miss judged content.

What we can reuse

• Reusable move: Reverse the normal workflow: propose nuggets first, verify them, then write the answer from the verified evidence.
• Plain English: Instead of asking "what should the final paragraph say?", first ask "what facts would a good answer need?"
• Try first: Generate candidate nuggets, merge paraphrases, filter unreasonable claims, rank nuggets, retrieve supporting sentences, and only then produce the final answer.
• Measure: Check sentence support and nugget coverage separately. HLTCOE shows those can move in different directions.
• Watch for: Nugget ideation can hallucinate desired facts. Every candidate nugget needs supporting evidence before it becomes part of the answer plan.
• Related teams: GenAIus uses direct nugget extraction, WaterlooClarke compares a nuggetizer strategy, and CFDA uses subanswers as a related intermediate layer.

Architecture

MIT Lincoln Laboratory built a clean multistage retrieval-to-generation pipeline. Gemma-3-27B decomposes the long narrative into subqueries, SPLADEv3 retrieves learned sparse candidates, Qwen3-Reranker-8B or a Gemma pointwise reranker reorders them, RRF fuses ranked lists, SETR selects a final passage set, and GPT-5 generates the answer.

What they tried

The paper is especially useful because it reports ablations. Run 1 used SPLADE only. Run 2 added Qwen reranking. Run 3 added Gemma decomposition plus SPLADE and RRF. Run 4 combined decomposition, SPLADE, Qwen reranking, and RRF. Run 5 swapped in Gemma reranking. This isolates how much each stage contributes before generation.

Models and tools

Gemma-3-27B query decomposition, SPLADEv3 retrieval, Qwen3-Reranker-8B, Gemma pointwise reranking, RRF, SETR set selection, and GPT-5 answer generation.

Results and observations

Retrieval was strong across variants: Run 1 scored nDCG@30 59.6, nDCG@100 53.9, and Recall@100 23.2 in percentage form; Run 4 rose to 62.5/57.0/24.6; Run 5 reached the best nDCG@30 at 64.4. Full RAG scores were also strong: Run 1 strict 0.44/sub 0.74, Run 2 0.50/0.77, Run 3 0.48/0.77, and Run 4 0.47/0.79, above the reported median of 0.35/0.69.

Caveats

The team observed that strong GPT-5 generation may compensate for modest retrieval differences, so retrieval gains did not map cleanly into RAG gains among the top variants. That makes end-to-end conclusions harder unless each stage is ablated.

What we can reuse

• Reusable move: Keep ablation runs as product infrastructure, not as a one-time paper exercise.
• Plain English: If you add SPLADE, decomposition, RRF, or a reranker, keep a switch that lets you turn it off and see what changed.
• Try first: Maintain fixed generation with retrieval variants: SPLADE-only, Qwen-reranked, decomposed, RRF-fused, and alternative-reranker runs.
• Measure: Track retrieval metrics and final RAG metrics side by side. MITLL's retrieval gains did not map perfectly into answer gains.
• Watch for: A strong generator can mask weak retrieval or smooth over retrieval differences, making the final answer look less diagnostic than it is.
• Related teams: NIT Agartala shows the simpler baseline ladder; UTokyo-HitU shows a deeper retrieval ablation style with HyDE and fusion choices.

Architecture

IDACCS used a hybrid summarization architecture. It retrieves with PLAID-X, reranks chunks with mxbai, builds extractive summaries with occams at roughly twice the target length, then asks GPT-4o or GPT-4.1 to convert that compressed evidence into nugget, hybrid, abstractive, or fluent outputs with attribution through blame/miniblame-style source assignment.

What they tried

They adapted earlier RAGTIME and NeuCLIR methods to the 400-word AG setting. The paper compares extractive summaries, abstractive rewriting, nugget-like intermediate representations, stratified retrieval over languages, and different GPT models for length-controlled generation.

Models and tools

PLAID-X retrieval, mxbai reranking, occams extractive summarization, GPT-4o, GPT-4.1, nugget and hybrid summary prompts, blame/miniblame attribution, and a cost-conscious non-agentic RAGTIME-inspired pipeline.

Results and observations

They found stratified retrieval by language had better nugget recall than a single multilingual top-4n pool. GPT-4.1 handled longer target lengths better than GPT-4o, which tended to undergenerate. Their best hybrid RAG summary stayed below 0.4 strict content but reached roughly 0.65 sub-narrative coverage; attribution precision/recall was about 75%. The paper estimates the hybrid approach around $0.01 per topic versus about $0.50 for a more agentic approach.

Caveats

The approach is cost-stable and attribution-friendly, but weaker on coverage than the strongest agentic systems. Extractive compression can also discard material before the generator ever sees it.

What we can reuse

• Reusable move: Compress evidence extractively before asking an LLM to write the final answer.
• Plain English: Shrink the evidence by copying important spans first, then let the model rewrite from that smaller evidence set.
• Try first: Retrieve with a strong lexical/neural stack, rerank, build an extractive summary around twice the target answer length, then rewrite with citations.
• Measure: Compare cost, latency, strict vital, weighted support, and citation accuracy before and after compression.
• Watch for: Compression is an information bottleneck. Keep source IDs and spans attached so fluent rewriting does not sever attribution.
• Related teams: WING-II uses evidence cards, GenAIus and HLTCOE use nuggets, and TUS explores choosing fewer keystone documents.

Architecture

NC State LAS explored agentic RAG. Retrieval experiments used query rewriting and nugget search over SPLADEv3. Generation used Microsoft Autogen agents and an MCP server exposing BM25, SPLADEv3, and T5/qdrant embedding search. The agent designs ranged from a single agent to selector teams, digraphs, swarms, and planner-executor roles such as planner, research assistant, writer, and reviewer.

What they tried

On retrieval, they compared concatenated subquestions and nuggets against separate-query fusion. On generation, they compared single-agent and multi-agent workflows, including selector, digraph, swarm, and planner-executor variants. This makes the report a useful cautionary test of whether multi-agent complexity actually pays for itself.

Models and tools

Microsoft Autogen, MCP search tools, BM25, SPLADEv3, T5 sentence embeddings in qdrant, Open-RAG-Eval, AutoNuggetizer, explicit query decomposition, and planner-executor style agent teams.

Results and observations

The best retrieval variant, LAS_con-que-con-nug, scored nDCG@30 0.6692, nDCG@100 0.4861, and Recall@100 0.1906. The simpler LAS_con-que had similar nDCG@100 0.4862 and higher Recall@100 0.2072, so the team saw the best compute tradeoff in the simpler query representation. Official RAG results put LAS-agentic-RAG-agent at strict 0.37/sub coverage 0.65 and selector at 0.30/0.60. The single agent unexpectedly beat multi-agent variants in key official scores.

Caveats

Agents add cost, latency, and new failure modes. The team notes that a second round with nuggets improved nDCG@30 but did not improve recall enough to justify the compute. Open-RAG-Eval also showed groundedness and citation quality can diverge from factuality.

What we can reuse

• Reusable move: Start with a transparent single-agent baseline before adding teams, swarms, or planner-executor graphs.
• Plain English: Make one agent use the tools well before deciding that more agents will help.
• Try first: Give one agent BM25, SPLADE, and embedding-search tools. Log every tool call, returned evidence, selected evidence, and final citation.
• Measure: Compare single-agent, planner-executor, and multi-agent variants on coverage, support, tool-call count, and evidence uniqueness.
• Watch for: Multi-agent structure is not automatically better. NC State LAS's single agent beat several richer agent teams in official scores.
• Related teams: uogTr shows planned decomposition can beat an agentic run; GRILL Lab shows where iterative search loops may still help.

Architecture

UTokyo-HitU built the clearest retrieval-first system in the collection. It preprocesses the query with HyDE and keyword expansion, searches with sparse BM25 and SPLADE, searches with dense BGE-small and Qwen3 embeddings, mixes original-query and hypothetical-answer embeddings through HyDE Vector Mix, fuses everything with RRF at k=60, and applies GPT-4.1-mini sliding-window LLM reranking before GPT-4.1/Ragnarok generation.

What they tried

They tested keyword expansion, HyDE-only dense search, weighted HyDE Vector Mix, four-method sparse+dense fusion, URL/context-aware LLM reranking, and different reranking window/pass settings. They also tried query paraphrasing and GPT-5-mini reranking, which did not help in their reported setting.

Models and tools

BM25, SPLADE, BGE-small, Qwen3 embeddings, HyDE, HyDE Vector Mix with alpha around 0.7, RRF k=60, GPT-4.1-mini sliding-window reranking, GPT-4.1 generation, and Ragnarok direct generation for answer writing.

Results and observations

Their four-method merge placed first in retrieval with nDCG@30 0.693, nDCG@100 0.613, and Recall@100 0.257. Keyword expansion improved BM25 nDCG@5 by 24.9%. HyDE Vector Mix improved dense retrieval, for example BGE-small nDCG@10 rose from 0.493 to 0.564. LLM reranking improved a BM25+BGE setup from nDCG@5 0.486 to 0.778. Their AG gpt41 run reached vital 0.54 and sub coverage 0.86; RAG variants reached vital 0.53/sub 0.79 and vital 0.56/sub 0.84.

Caveats

The system is compute-heavy and not every LLM trick helped. Query paraphrasing and GPT-5-mini reranking underperformed in their tests, and complex AG prompt tricks did not beat direct Ragnarok-style generation.

What we can reuse

• Reusable move: Use HyDE as one extra dense signal inside a hybrid retrieval portfolio, not as a replacement for the original query.
• Plain English: Let the model imagine what a good answer might look like, embed that imagined answer, and search with it alongside the real query.
• Try first: Run BM25, SPLADE, BGE embeddings, Qwen embeddings, original-query dense retrieval, and HyDE dense retrieval, then fuse with RRF before LLM reranking.
• Measure: Ablate keyword expansion, HyDE-only retrieval, HyDE Vector Mix, URL/context-aware reranking, and candidate depth.
• Watch for: A hypothetical answer can drift from the user's actual question. Keep original-query retrieval alive through fusion.
• Related teams: CFDA, MITLL, and NIT Agartala also preserve multiple retrieval signals; GRILL Lab uses expansion and gap analysis in an iterative form.

Architecture

DUTH focused only on relevance judgment. The system retrieves candidate query-passage pairs with BM25 plus ColBERT reranking, normalizes them into JSONL, uses small open-weight LLMs to pre-label relevance/justification, supports human review through a CLI, and exports validated TREC qrels. Their pipeline is about making judging reproducible and auditable, not generating answers.

What they tried

They tested Qwen2.5-3B-Instruct, StableLM 2-1.6B-Chat, optional LLaMA3-8B style components, deterministic decoding, lexical-overlap and BM25 confidence features, calibration thresholds, and hybrid confidence blending to reduce uncertain ties.

Models and tools

BM25, ColBERT reranking, Qwen2.5-3B-Instruct, StableLM 2-1.6B-Chat, optional LLaMA3-8B, PyTorch, Transformers, bitsandbytes, deterministic decoding, Jaccard lexical overlap, BM25 confidence, calibration, and a CLI human review workflow.

Results and observations

The hybrid Qwen run scored kappa 0.050 with agreement 0.250, disagreement 0.730, and tie 0.120. A calibrated Qwen variant scored kappa 0.040 with agreement 0.260. StableLM variants were weaker, including one run with kappa 0.000 and tie 0.420, and others around -0.010. Calibration reduced ties but did not solve the deeper ambiguity of justification-level labels.

Caveats

Low kappa is not just a model failure; relevance judgment for RAG asks whether a passage justifies a claim, which is much harder than topical matching. Raw agreement, disagreement, and tie rates need to be read beside kappa.

What we can reuse

• Reusable move: Separate topical relevance from justification support in judging and citation checks.
• Plain English: A passage can be about the same topic and still fail to prove the sentence you wrote.
• Try first: Use a small LLM and lexical features to pre-label obvious cases, then send uncertain or high-impact labels to human review.
• Measure: Report agreement, disagreement, ties, kappa, and calibration by confidence bucket rather than only raw accuracy.
• Watch for: DUTH's low kappa shows that raw model confidence is not enough. Relevance judging needs calibration and disagreement analysis.
• Related teams: GenAIus uses nugget and citation features for RJ; IIUoT, WING-II, and HLTCOE expose why sentence-level support matters.

Architecture

WaterlooClarke treated RAG as a portfolio of answer strategies. The shared front end uses BM25 followed by Mono-T5/Duo-T5 reranking and keeps the top 15 passages whose UMBRELA score is at least 2. Then GPT-4o-mini powers several pipelines: Nuggetizer, GARE, Auto Plan, Combined, and Auto Selected.

What they tried

Nuggetizer retrieves evidence, drafts a cited answer, validates citations, filters vital/okay nuggets, and combines vital nuggets. GARE generates an answer first, splits it into atomic claims, retrieves and checks evidence for each claim, and revises unsupported claims. Auto Plan asks the model to create and execute an ordered retrieval/answer plan. Combined pools unique passages from the other strategies. Auto Selected compares generated responses pairwise twice with order swapped to reduce position bias.

Models and tools

BM25, Mono-T5, Duo-T5, UMBRELA passage filtering, GPT-4o-mini, nugget filtering, claim extraction, evidence retrieval per claim, automatic planning, response pooling, and pairwise LLM judging with order swapping.

Results and observations

In the official overview, the combined run scored strict 0.32 and sub coverage 0.63. Auto Selected scored 0.31 and 0.63. Auto Plan scored 0.29 and 0.62. These were not the top scores, but the variants clustered closely, suggesting that several routes reached similar coverage with different failure modes.

Caveats

The paper mostly describes the system design and official outcomes rather than extensive internal ablations. LLM pairwise selection can still inherit judge preferences even with order swapping, so the selected answer is not guaranteed to be the truly best answer.

What we can reuse

• Reusable move: Run a portfolio of answer strategies when hard questions may need different evidence paths.
• Plain English: Make several candidate answers in different ways, then choose the best one using evidence-aware judging instead of betting on one prompt.
• Try first: Compare a nuggetizer answer, an answer-then-support answer, an automatic-plan answer, a pooled-evidence answer, and an auto-selected final answer.
• Measure: Evaluate each candidate on support, coverage, answer length, missing vital facts, and judge preference after order swapping.
• Watch for: LLM pairwise judges have position bias. WaterlooClarke swapped answer order and judged twice to reduce that failure mode.
• Related teams: HLTCOE and GenAIus supply nugget-based candidates; CFDA supplies decomposed subanswer candidates; NC State LAS supplies agentic candidates.