Music-CRS Blind-B: a candid retrospective

Executive Summary

• My final result was materially behind the four leading public-code entries. My composite was 0.3811, compared with 0.5759–0.5866 for the comparison set. The exact decomposition shows that ranking quality and the LLM judge account for nearly all of those gaps; catalog diversity was essentially neutral.
• The strongest evidence points to two broad issues: an unreliable in-sample development estimate, and weaker hidden-set ranking and response outcomes than the released leaders. The first issue explains why I had misplaced confidence in the system; it does not explain the Blind-B score itself. Reusing model-fitting data for evaluation is a measurement failure, not a direct hidden-set cause.
• The public systems differed in consistent, relevant ways. They more often used leakage-safe validation, behavioral or sequential retrieval, broader candidate unions or late fusion, and richer response grounding, sampling, selection, or critique. I treat those differences as plausible contributors—not session-level causal proof—because the hidden 80-session labels and counterfactual runs are unavailable.
• What I built still had real strengths. Structured state, broad multimodal retrieval, a traceable ranking handoff, and a reproducible offline bundle are worth preserving. The lesson is to pair that engineering discipline with honest model selection and equal attention to every scored component.

Chapter outline

1. Final result
2. Complete system lifecycle
3. Conversation to query
4. Data and model knowledge
5. Retrieval and candidate construction
6. Ranker feature families
7. Response generation
8. Our submitted system
9. Public-team case studies
10. Cross-team synthesis
11. Retrospective lessons
12. Acknowledgements
13. Caveats and evidence

How the challenge was scored

The challenge combined four signals. nDCG@20Source for 20Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv asks whether relevant tracks appear near the top of the 20Source for 20Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv-track recommendation list; higher-ranked hits count more. Catalog diversity measures how broadly the system uses the catalog across the submission. Lexical diversity measures variety in the generated responses. The LLM judge scores response quality from 1 to 5Source for 5Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv.

The official composite is 0.50 × nDCG@20 + 0.10 × catalog diversity + 0.10 × lexical diversity + 0.30 × (LLM judge − 1) / 4. Ranking therefore carries half the score, while the normalized judge contributes 30%Source for 30%Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv. This makes response quality a first-class objective rather than packaging around the recommender.

The final result exposed two large deficits

My entry, npatta01, finished at 0.381109682Source for 0.381109682Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv composite: 0.253684925Source for 0.253684925Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv nDCG@20Source for 20Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv, 0.031484353Source for 0.031484353Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv catalog diversity, 0.786187845Source for 0.786187845Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv lexical diversity, and 3.3000/5 LLM judge. Each comparison entry was roughly 0.195Source for 0.195Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv–0.206Source for 0.206Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv composite points ahead. The table keeps the official precision so the calculation can be audited rather than inferred from rounded cards.

The signed bars answer a narrower question than the leaderboard: which scored terms mathematically make up each leader’s advantage over my submission? Ranking and judge contributions dominate every panel. Catalog diversity is essentially neutral. Lexical diversity helps volart and swyoo, while it slightly favors my entry against niwatori and team2_s2; those negative lexical terms correctly point below zero instead of being hidden inside a positive stack.

The four component terms reconcile to the total gaps shown in the panel labels and exact table within 0.000000001Source for 0.000000001Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv, the last-decimal rounding residual in the reviewed published values. This is an arithmetic decomposition, not a causal decomposition: it tells me where the score difference appears, not which implementation choice created it.

Chart source: Official Music-CRS Blind-B final results.

A common lifecycle makes unlike systems comparable without making them identical

The five released systems expose different models, branches, and validation choices. The shared lifecycle below is a comparison grammar: it names the handoff boundaries that every case study must explain, not a claim that the implementations were the same.

Two scored boundaries deserve separate audits. Retrieval quality ends at the ranked, selected track IDs; response quality begins from those IDs and the evidence passed to the generator. Both nevertheless depend on the same upstream interpretation of the conversation: a mistaken intent, history, or constraint can degrade candidate construction and also give the response model the wrong explanation context.

The first design choice was how each system turned dialogue into search work

A conversational recommender does not send one universal text string everywhere. The important comparison is which dialogue window is read, what structure is extracted, whether an LLM rewrites or summarizes it, and how each downstream retriever receives a representation suited to its evidence type.

Question answered by this matrix: how did dialogue become one or more search representations? The conversation window is what each path could see. Interpretation is explicit state, entities, deterministic intent, or learned encoding derived from it. LLM query work separates model-authored rewriting or summarization from deterministic rendering. Query variants names the representations produced before candidate construction; history filtering and candidate constraints are reserved for the later retrieval matrix. Not documented is an evidence boundary, not proof that a capability was absent.

Audit detail: the disclosure immediately below preserves the exact prompt excerpts, complete commit-pinned inventory, and additional reviewed files behind this matrix.

Sources for the query audit: npatta01 state-extraction prompt; volart retrieval-rewrite prompt; swyoo session-summary prompt.

Data provenance and model knowledge entered at different stages

External records, generated artifacts, and latent model associations are not interchangeable. Keeping them separate shows what can be reproduced from files, what must be regenerated, what was checked against the catalog, and where an uncited musical claim can still enter the response.

The practical boundary is reproducibility, not whether an LLM appears anywhere. Cached states, summaries, and descriptions can be replayed when their artifacts and versions are preserved; latent associations in fresh prose cannot be independently audited without another record. swyoo documented the broadest external structured-data path and field-level guardrails, while the reviewed author and team2_s2 paths did not document an external music dataset. None of those differences alone establishes Blind-B causality.

Candidate construction determined what the ranker was allowed to rescue

Each system rendered multiple query forms for sparse, dense, collaborative, history, lookup, or transition sources. The important boundary is the union handed to the ranker: evidence outside that candidate set is unrecoverable regardless of downstream model quality.

Rankers differed most in the families of evidence they could combine

Raw feature counts are only useful after the underlying evidence families are named. The comparison therefore starts with the signals each row represents, then reports documented counts and validation lineage.

Response generation was a second system, not a formatting step

The ranked IDs fixed what was recommended; the response pipeline decided whether the explanation was grounded, specific, stable, and varied. The comparison separates drafting, checking or repair, selection or critique, rewriting, lexical control, and ID integrity so that a multi-pass editor is not confused with best-of-N selection.

Question answered by this matrix: after track selection, how many response candidates and stages existed, and what grounding, selection, repair, rewriting, or ID protection occurred? Drafting names candidate count when documented. Grounding separates catalog or structured records from latent model knowledge. Checking / repair is distinct from selection / critique: a path may revise one draft, choose among many, do both, or document neither. ID integrity asks whether prose operations can alter already-selected recommendations.

The pipelines buy different kinds of control. Sampling and selection can improve variety but do not automatically verify facts. Critics and second-pass editors can improve relevance or style but need explicit guards to preserve IDs and structured claims. Deterministic lexical stabilization protects a metric but should be read separately from the semantic quality of the core explanation. These are architectural comparisons, not proof that any single stage caused a hidden-set score.

Sources for the response walkthroughs: npatta01 response contract; volart generation; volart critic; volart refinement; volart hardening; volart lexical pass; niwatori seeded generation; niwatori diversity selector; swyoo PAS generator and validator; swyoo lexical stabilization; team2_s2 verified fact renderer; team2_s2 two-pass pipeline.

What my system actually built

I built a state-aware retrieve–rerank–respond pipeline whose durable strengths were explicit state, multiple candidate branches, useful CF/BPR centroids, multimodal and catalog evidence, a learned ranker, and response instructions that did not oversell a poor fit. The comparable walkthrough below follows the same query, data, candidate, feature, validation, and response boundaries used for the public teams.

Conversation-to-query path. DeepSeek read the multi-turn session memory and emitted fact-first V1 state: the current request; artist, album, track, and attribute facts; explicit exclusions; played-track feedback and references; temporal constraints; and lyrical theme when relevant. A resolver grounded names to catalog IDs. The compiler then rendered retriever-specific forms: weighted BM25 clauses; field-aligned lyric, metadata, attribute, sonic, and visual strings; anchor and user centroids; resolved-artist discography and era-popularity lookups; and a deterministic previous-turn/current-turn/last-track query for the b1_cos scorer.

Data and model knowledge. The submitted path used official catalog metadata and embeddings, Blind-B conversations and users, and public labeled conversations for learned retrieval and ranking. Generated inputs included cached DeepSeek states and committed LLM-authored artist known-for lines in the b1 document text. External structured music data was Not documented in the deployed path. The response model could add uncited associations from latent model knowledge; those associations had no independent record.

Retrieval and candidate construction. BM25, multimodal Qwen/CLAP/SigLIP approximate-nearest-neighbor branches, catalog lookups, and both deployed CF/BPR behavioral centroids—the default anchor centroid and explicit user centroid—formed the filtered candidate union. The compiler recorded a weighted fusion order for trace and fallback purposes, but LightGBM—not RRF—produced the submitted final ordering. Explicit track rejection could hard-drop, tag rejection could demote, new/different-artist evidence could gate a final artist veto, and era was normally soft. The union contained up to 500 hits from each traced branch; LightGBM scored that union; the trained Qwen3-Embedding-4B two-tower was present as b1_cos, not as its own candidate lane.

Ranker and feature families. The 142 documented features included per-branch ranks, raw scores, margins, ratios, presence, standardization, branch identity, and percentiles; dense and multimodal similarities; CF/BPR and catalog lookup evidence; popularity, era, tag, artist, album, duration, culture, and age fields; request, intent, routing, rejection, and history descriptors; and cross-source agreement and interactions. The count describes the deployed model, not its quality by itself.

Evaluation lineage. The held-out OOF table records 0.1970 nDCG@20 for production-like features and 0.2032 with b1_cos, using predictions for rows outside each fold’s fit. Later full-model dev replays were in-sample diagnostics under the documented fit/evaluate sequence; a separate selection lockbox was Not documented.

Response-generation subsystem. Ranking fixed the top-1 ID before one temperature-zero Qwen3-30B-A3B call. The prompt received latest state and XML-delimited title, artist, album, and tags, asked for one or two concise sentences, prioritized the latest request, required supported reasons, and handled explicit conflicts honestly. The metadata-echo guard implementation existed, but Blind-B set echo_retries to 0, so the submitted path made one generation call with no echo-triggered retry. An independent fact checker, multi-candidate selector, critic, lexical pass, and post-edit ID-integrity guard were Not documented.

Diagram sources: O1, O2, O3, O8, O9 — Blind-B configuration; O4 and O6 — bi-encoder; O5 — reranker reproduction; O7 — response contract.

What worked and is worth preserving

Acknowledged contribution. The submission turned heterogeneous dialogue into an inspectable state-and-query contract rather than one opaque string. Sparse fields, several multimodal dense views, two CF/BPR centroids, discography, and era-popularity lookups reduced dependence on one representation, while the 142-feature LightGBM ranker could combine branch evidence.

Traceability mattered. Extracted state, compiled queries, candidate-source scores, ranker stages, and the final recommendation handoff made it possible to see where an item entered and where it was lost. Commit-pinned models, caches, configuration, and a shared feature function reduced train/serve drift.

The response stance was restrained. The single-pass prompt prioritized the latest request, asked for supported reasons, and explicitly handled conflicts without claiming that a poor fit was perfect. These strengths did not guarantee hidden-set quality, but they remain technically useful and made the limits diagnosable.

The central mistake was measurement, not a direct Blind-B cause

In-sample evidence. The pinned diagnostic leaderboard records the two headline dev numbers I had been using—0.3844 and 0.4562 nDCG@20. They came from applying a full-data model to the same dev population that supplied its training features and labels. Those values described the fitted model; they did not estimate generalization.

OOF evidence. The held-out results were 0.1970–0.2032 nDCG@20, with each row predicted outside its fold’s fit. A separate model-selection lockbox was Not documented.

Blind-B nDCG@20 was 0.2537. Inferred, not proven: that result is more consistent with the OOF range than with the in-sample range. The hidden set was different, small, and potentially shifted, so proximity is not proof. The narrower conclusion is that I selected and discussed the system with more confidence than the validation design justified.

Best-supported ranking contributors

The candidate boundary was part of the ranker. Any relevant track outside the union of up to 500 hits from each traced branch was irrecoverable. niwatori documented full-union scoring, while team2_s2 documented routed and late-fused ranking over explicitly diverse sources. Those contrasts are plausible contributors, not hidden-session proof.

Behavioral/co-occurrence coverage remains Partial, not No. My deployed path had two useful CF/BPR centroid branches—anchor and user—plus user/last/centroid CF features, discography, and era-popularity lookups. It did not document a direct track co-occurrence lane or sequential-transition lane. Centroid similarity is related behavioral evidence but a different mechanism, which is exactly why the high-level matrix retains Partial.

The learned retriever was a feature, not a source. b1_cos joined the dense-similarity family inside LightGBM, but the Qwen3 two-tower candidate branch was not deployed. OOF evidence records a modest gain for that feature; no hidden-set counterfactual establishes the effect of serving its retrieval lane.

Feature families, not counts, define ranker evidence. The submitted ranker combined source ranks/scores/presence and transformations; dense, lyric, acoustic, image, and catalog similarities; CF/BPR centroids and lookup evidence; metadata/popularity/era fields; state, history, rejection, and routing descriptors; and agreement/interactions. The public systems document different behavioral, routing, and OOF lineages. Larger feature counts do not by themselves establish quality.

Best-supported response contributors

The pinned Blind-A response sweep records 4.7000/5 for the selected template; Blind-B received 3.30/5. That measured difference does not isolate a mechanism because sessions, ranked tracks, and judge cases changed together.

Actual response stages. The submitted path fixed one top-1 recommendation, rendered latest state and catalog metadata in XML, and made one temperature-zero Qwen call. The style contract prioritized the latest request, asked for supported reasons, required concise prose, and discouraged overselling explicit conflicts. The metadata-echo guard implementation existed, but Blind-B set echo_retries to 0, so this submission made one generation call with no echo-triggered retry.

Not documented: independent structured fact checking; multiple response candidates; a selector or independent critic; contradiction, theme, or citation repair; a post-generation lexical pass; and an independent ID-integrity guard. volart documented candidate selection plus critique and guarded edits, niwatori selected among ten seeded variants for corpus diversity, swyoo validated and repaired themes and citations, and team2_s2 grounded one draft in verified catalog facts before a second editorial pass.

Inferred, not proven: deeper grounding, selection, or repair are plausible response-quality contributors. I do not compare response lengths because the reviewed evidence does not trace both submitted response files to exact public artifacts, and length alone would not establish quality.

Four public systems show distinct, reusable implementation patterns

Each case uses the same ten visible roles: acknowledged contribution, verified final outcome, conversation-to-query path, data and model knowledge, retrieval and candidate construction, ranker and feature families, response-generation subsystem, difference from our submission, transferable implementation patterns, and limits or undocumented details. The official outcome provides context; architecture remains contributor-level evidence, not causal attribution.

volart — broad hybrid retrieval plus editorial response control

Acknowledged contribution. artvolgin/music-crs-recsys2026 documents a compact hybrid retriever, disjoint ranker validation, challenge-derived behavioral priors, and unusually explicit response selection and editing while preserving recommendation IDs.

Verified final outcome. volart led this public-code comparison at 0.586611845Source for 0.586611845Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv composite, with 0.396534887Source for 0.396534887Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv nDCG@20Source for 20Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv, 0.926832194Source for 0.926832194Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv lexical diversity, 0.031611820Source for 0.031611820Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv catalog diversity, and 4.9000/5 judge. Its lead over my entry was +0.205502163Source for +0.205502163Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv.

Conversation-to-query path. The rewriter saw the stated goal, latest user query, and available history, then GPT-4o-mini produced one cached concise retrieval rewrite. A separate cached JSON extraction kept positively requested artists, tracks, albums, and a validated era. The text rewrite fed the lexical and dense lanes; structured entities/era and played-track IDs fed their own sources.

Data and model knowledge. The path used official catalog, conversations, played history, public labels, challenge-train co-occurrence, gold-frequency, and MOVES-rate priors. The pinned submission config excludes LFM-2B/Last.fm grounding; no external music dataset is documented in the submitted path. Generated artifacts included query rewrites, entity records, per-track retrieval descriptions, response candidates, critiques, and edits. Gemini could add musical associations beyond catalog blocks, and the editorial critics did not independently verify every such claim against structured records.

Retrieval and candidate construction. Five lanes—OpenAI metadata dense, BM25, positive entity/era matches, dense LLM-generated descriptions, and challenge-train co-occurrence—were fused with RRF. Played tracks seeded co-occurrence and were excluded from results; entity/era matches received dedicated slots. The fused top 500 formed the LambdaMART boundary.

Ranker and feature families. The 69 documented features included retriever scores/ranks and lane membership; metadata cosine and mood/query overlap; artist/album/tag/era continuity; recency-weighted matches; direct co-occurrence sum/max; gold-frequency and MOVES priors; popularity/year/tags; demographics; turn/history/goal descriptors; and agreement or cross-history consistency. A deterministic 1,500-session holdout was excluded from the documented 5,000-session fit.

Response-generation subsystem. Gemini generated three candidates at temperatures 0.7, 1.0, and 1.2 from the same tracks. Low repeated-bigram overlap informed initial selection; an independent critic flagged weak replies; selective refinement, hardening, and a guarded lexical de-tic pass followed. These passes changed response text without touching track IDs. Structured musical fact verification remained Not documented.

Diagram sources: V1, V3, V4 — method overview; V2 — LambdaMART features; V5 — Blind-B configuration; V6 — response refinement; V7 — response hardening; V8 — lexical pass.

What differed from our submission. Both systems used hybrid retrieval, a learned ranker, and a separate response step. volart explicitly used RRF to fuse a top-500 pool before LambdaMART; our system unioned up to 500 hits from each traced branch, then LightGBM produced the submitted final ordering. volart additionally carried direct co-occurrence and challenge priors into ranking, used a disjoint validation subset, generated three response candidates, and applied independent critique plus guarded editing. The decomposition is consistent with that contrast: judge contributed +0.120000000, ranking +0.071424981, and lexical diversity +0.014064435 to its lead; it does not identify which mechanism mattered.

Transferable implementation patterns. Future systems can study four source-backed patterns here: keep rewritten text, structured entities, and played-track co-occurrence as distinct query inputs; expose source ranks/scores and lane presence alongside behavioral priors to the ranker; generate alternatives only when a documented selector or critic consumes them; and preserve selected recommendation IDs through selective editorial and corpus-level lexical passes.

Limits and undocumented details. Not documented: a general hard rejection or novelty policy in the reviewed retrieval files, structured verification of every musical claim, component-by-component Blind-B ablations, or the stage responsible for any particular hidden session. The disjoint split validates the reranker more cleanly than in-sample replay, but it does not certify generated facts or prove that any architecture difference produced the outcome.

niwatori — full candidate union, OOF learned retrieval, and response ensembles

Acknowledged contribution. ryowk/recsys2026-niwatori documents source-specific safe queries, a fourteen-source ordered union, OOF learned-retrieval artifacts, a behavior-rich LambdaRank model, and seeded response diversity selection.

Verified final outcome. niwatori scored 0.585920046Source for 0.585920046Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv composite, 0.493389952Source for 0.493389952Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv nDCG@20Source for 20Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv—the strongest ranking result in this comparison—0.773563770Source for 0.773563770Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv lexical diversity, 0.031186930Source for 0.031186930Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv catalog diversity, and 4.4500/5 judge. Its lead over my entry was +0.204810364Source for +0.204810364Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv.

Conversation-to-query path. niwatori used source-specific deterministic representations rather than a query-stage LLM: five-field BM25 and TF-IDF text, extracted genre/mood/descriptor tags, a supervised two-tower query, exact entity surfaces, artist/album history keys, co-occurrence keys, and a last-track Markov key. Blind-B-safe inputs explicitly omitted unavailable goal and thought fields. Windows varied by source, from current/recent text to complete played history and the most recent track.

Data and model knowledge. Official catalog, conversations, user/history fields, public labels, and challenge-derived behavior were joined by documented TalkPlayData roles: TalkPlayData-1 augmented mapped co-occurrence/transition counts and supported two-tower experiments; TalkPlayData-2 supplied only the Spotify-to-catalog mapping. Unmapped IDs were dropped and duplicate-overlap audits were documented. No LLM-generated retrieval rewrite, profile, document, or label was documented; ten response drafts were generated at inference, and any Qwen musical detail absent from the prompt was latent and not separately verified.

Retrieval and candidate construction. Fourteen lexical, dense, exact, history, co-occurrence, transition, and learned sources formed an ordered deduplicated union. History branches used artist/album continuity, co-occurrence aggregated played tracks, and Markov transitions used the last track. Already-played filtering and message-safe input constraints were explicit. The complete ordered union—not a globally truncated head—reached the ranker.

Ranker and feature families. The 176 documented features covered per-source presence/rank/transforms and public/TPD1 components; TF-IDF, intent-tag, Qwen dense, and two-tower similarity; track/album/artist co-occurrence and Markov probability; history consistency; ISRC year/country, duration, age-era, seasonal tags, popularity, tag-chain, and PPMI evidence; safe-query similarity; and cross-source agreement. Five-fold CV produced OOF learned-source artifacts for reranker rows and full-public artifacts for Blind-B.

Response-generation subsystem. Qwen3.6-27B generated ten variants under seeds 0–9 from the same profile, conversation, current message, and top-three tracks. A greedy ensemble selected whole records using corpus distinct-unigram and distinct-bigram scores. This was a lexical-diversity selector, not a factual critic; separate fact repair and cross-variant track-ID equality checks were Not documented.

Diagram sources: N1–N6 — submitted method; N7 — response ensemble.

What differed from our submission. niwatori used its learned retriever as a candidate source, kept direct co-occurrence and sequential-transition signals distinct from semantic similarity, generated ranker inputs out of fold, and scored the full ordered union. Its +0.119852514 ranking contribution is salient but not mechanistically identified. Lexical diversity was slightly below mine, contributing −0.001262408 to its total lead.

Transferable implementation patterns. Future systems can study four source-backed patterns here: render retriever-specific query representations instead of forcing one universal string; carry co-occurrence and Markov sequence evidence separately from dense or centroid similarity; generate learned source features OOF and use full-public artifacts only for serving; and treat response diversity as a corpus-level scored selection problem when choosing among documented candidates.

Limits and undocumented details. Not documented: a submitted hard rejection or novelty layer, independent response fact checking or repair, cross-variant recommendation-ID equality, per-source Blind-B recall, and hidden-set ablations of full-union scoring or response selection. Seeded generation also notes CUDA nondeterminism. The architecture is consistent with strong ranking evidence; it does not prove why any hidden session scored as it did.

swyoo — fold-safe QLoRA retrieval and evidence-checked responses

Acknowledged contribution. yoobros/music-crs-challenge documents three distinct query renderings, a fold-routed QLoRA retriever bag, guarded external metadata and generated documents, OOF ranker evidence, PAS response repair, and deterministic lexical stabilization.

Verified final outcome. swyoo scored 0.578429003Source for 0.578429003Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv composite, 0.382893624Source for 0.382893624Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv nDCG@20Source for 20Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv, 0.952048495Source for 0.952048495Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv lexical diversity—the highest in this comparison—0.030273417Source for 0.030273417Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv catalog diversity, and 4.8500/5 judge. Its lead over my entry was +0.197319321Source for +0.197319321Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv.

Conversation-to-query path. The submitted path kept three independently rendered representations. BM25 received content-filtered current text plus played-track metadata; QEmb received a document-aligned current/history string; the two-tower received recent listens, chat-derived context, current request, and an optional cached one-sentence summary of the evolving preference trajectory. The summary was injected only into the two-tower after turn one. All three documented retrievers removed session-seen tracks.

Data and model knowledge. Official catalog metadata/embeddings, train conversations/labels, demographics, and prior train-session interactions were enriched from LRCLIB and Genius lyrics plus MusicBrainz identifiers, ISRCs, labels, countries, dates, and tags. Two-tower documents directly formatted MusicBrainz tags, label, and country; the response crawl block separately exposed only caption, a short lyric hook, key, and tempo. Generated artifacts included session summaries, synthetic mood/theme/use-case track cards, PAS drafts, and repairs. Crawl guardrails checked completeness, text quality, title/artist/duration matches, and critical fields; generated and latent musical associations were not all independently fact-checked.

Retrieval and candidate construction. Music-only BM25, metadata-rich QEmb, and the five-adapter Qwen3-Embedding-8B two-tower bag produced source-native pools over enriched documents, with RRF over the fused candidate head and already-seen filtering. The reviewed sources establish those submitted pools but do Not document a full-union boundary comparable to niwatori’s.

Ranker and feature families. LightGBM received native rank and score for each logical pool, RRF rank/score, turn number, and goal category/specificity; documented tooling could also derive pool-hit count, min/mean/std/span ranks, best pool, and pairwise rank differences. Metadata and learned behavior entered through pool evidence rather than separate final raw columns. Each training row was routed to its held-out adapter for OOF learned-source features; Blind-B used the five-adapter bag. The exact submitted column count was Not documented.

Response-generation subsystem. PAS deterministically proposed evidence-typed intent groups; one GLM/DSPy prediction assigned tracks and selected structure; validation checked themes, citations, and legal titles against the recommendation pool, then retried or repaired failures. Packaging appended real tags in a deterministic low-overlap order without changing the ranked top-20 IDs. An alternative-draft count or best-of-N independent critic was Not documented.

Diagram sources: S1, S2, S3, S6 — method overview; S4 — OOF ranker training; S5 — PAS generator and validator.

What differed from our submission. swyoo used distinct query forms per retriever, made fold separation part of both learned retrieval and reranker feature construction, enriched retrieval documents with guarded external records, and added explicit response evidence validation plus deterministic lexical stabilization. Its ranking lead was +0.064604349, while judge (+0.116250000) and lexical diversity (+0.016586065) made up most of the remaining gap; that decomposition does not isolate a stage.

Transferable implementation patterns. Future systems can study four source-backed patterns here: align each retriever query with its document representation; route learned-source training rows through held-out adapters; guard external record matching before using enriched fields and limit response prompts to documented fields; and validate/repair legal titles and citations before a deterministic corpus-level lexical pass that preserves selected IDs.

Limits and undocumented details. Not documented: a separate direct co-occurrence or sequential-transition feature column in the final builder, the exact submitted ranker column count, an alternative-response candidate count, a best-of-N independent critic, per-session PAS repair records, or a Blind-B counterfactual without lexical stabilization. Public APIs, token/rate limits, caches, and generated fields also bound reproducibility. These differences support architectural hypotheses, not hidden-session causal claims.

team2_s2 / music-crs-team2 — diverse evidence, routed rankers, and grounded polishing

Acknowledged contribution. lopsandrea/music-crs-team2 documents retriever-specific conversation windows, collaborative and acoustic candidate sources, OOF learned lists, routed and late-fused rankers, covariate-shift weighting, verified catalog facts, and two-pass response editing.

Verified final outcome. The team2_s2 leaderboard entry from lopsandrea/music-crs-team2 scored 0.575916042Source for 0.575916042Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv composite, 0.445184942Source for 0.445184942Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv nDCG@20Source for 20Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv, 0.765483535Source for 0.765483535Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv lexical diversity, 0.030252172Source for 0.030252172Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv catalog diversity, and 4.6500/5 judge. Its lead over my entry was +0.194806360Source for +0.194806360Source: Official Music-CRS Blind-B final results CSVTable: https://nlp4musa.github.io/music-crs-challenge/static/results.csv.

Conversation-to-query path. Query windows were source-specific and deterministic. Conversation BM25 recency-weighted every prior user turn toward the current request; the live text retriever kept the last three user turns; item sources used the last one, last three, or all played tracks. Two BM25 strings, Qwen metadata-neighbor vectors, recent CF-BPR and CLAP anchors, an ALS session vector, a live supervised text query, and cached structured BGE-base/BGE-large lists formed separate representations. A shared query-stage LLM state was Not documented.

Data and model knowledge. The path used the official 47,071-track catalog, conversations, user data, precomputed Qwen/CF-BPR/CLAP embeddings, and public labels. External structured music data was Not documented. No persistent generated retrieval rewrite, profile, document, or label was documented; the response draft and refinement were generated at inference. Gemini could add uncited music knowledge beyond the catalog-fact string, and those associations lacked a separate record.

Retrieval and candidate construction. Lexical BM25, semantic Qwen, recent CF-BPR, ALS session affinity, full-catalog CLAP acoustic similarity, the live text bi-encoder, and structured BGE-base/BGE-large lists entered the source pool. All sources filtered already-played tracks; current lexical intent and source-specific history windows were enforced. The reviewed query code did Not document independent explicit rejection, era, popularity, or novelty rules.

Ranker and feature families. The 37 documented features covered RRF and six live-source inverse ranks/presence; text and structured retriever ranks/presence/cosine; Qwen and CLAP source evidence; CF-BPR/ALS membership, rank, and ALS session dot product; history recency, artist/album matches, query overlap, tag Jaccard, popularity, and pool distribution; and source agreement plus margin routing. Five-fold structured lists supplied ranker training; final retrains used tempered importance weights from a five-fold dev-versus-Blind-A shift model built from gold-free input descriptors.

Response-generation subsystem. Catalog-derived title, artist, album, year, duration, popularity, and tags grounded one Gemini 3.1 Pro Preview draft. Gemini 3.5 Flash then removed helpdesk phrasing, filler, and repetition; edits under the programmatic length floor fell back to the original draft. A best-of-N selector, independent structured fact checker, and post-edit recommendation-ID integrity check were Not documented.

Diagram sources: T1 — training method; T2 and T5 — system and response overview; T3 — ranker; T4 — transfer weighting.

What differed from our submission. team2_s2 used several learned retrievers as candidate sources, carried collaborative and acoustic evidence explicitly, routed between two LightGBM boosters by score margin, late-fused with CatBoost, reweighted blind-like training rows, and grounded a second editorial response pass in catalog facts. The lead split between ranking (+0.095750008) and judge (+0.101250000), while lexical and catalog diversity slightly favored my entry; the split is descriptive, not causal.

Transferable implementation patterns. Future systems can study four source-backed patterns here: let each retriever consume the conversation/history window it is designed for; produce OOF learned-retriever lists for ranker training and full lists for serving; combine specialized routed rankers only through documented score calibration and late fusion; and render verified track facts before an editorial pass with a deterministic fallback.

Limits and undocumented details. Not documented: external structured music data, a query-stage LLM rewrite/state, explicit rejection/era/popularity/novelty enforcement, a per-component Blind-B ablation, alternative-draft counts, an independent structured post-edit fact check, or a post-edit ID-integrity assertion. Live two-pass Gemini access and latent uncited associations remain reproducibility and factual-reliability boundaries. These differences do not establish session-level causality.

Cross-team synthesis: coverage mattered more than one magic model

The matrix uses only Yes, Partial, and Not documented. “Not documented” means the reviewed pinned sources do not establish the technique; it does not mean the team definitely omitted it. “Partial” marks a related capability that does not fully match the dimension as defined.

What our submitted path contained

Two CF/BPR behavioral centroid branches—anchor and user—plus discography and era-popularity lookups.

What the leaders documented

Direct track co-occurrence, artist/album history, sequential or Markov transitions, collaborative filtering, and train-frequency priors appeared in one or more released systems.

Why the matrix says Partial

Our centroids supplied related behavioral evidence, but centroid similarity is not a direct track co-occurrence mechanism and is not a sequential-transition lane.

Across the comparison, the most consistent pattern is coverage across several risk surfaces: behavioral evidence, learned retrieval, leakage-safe validation, candidate/ranker breadth, and response control. No single row proves the Blind-B outcome, but my system had more partial or undocumented coverage in several rows that align with the two dominant score terms.

What I would preserve, reconsider, and avoid

This is a retrospective judgment table, not an implementation program. Preserve means the choice produced durable technical value. Reconsider means the evidence exposes an important tradeoff or blind spot. Avoid means the reasoning pattern made the result harder to assess honestly.

Lessons I will carry into future ML competitions

• Separate model-selection data from model-fitting data. An in-sample score can describe a fitted system; it cannot tell me how well the system generalizes.
• Treat public-leaderboard evidence as sparse feedback, not a training oracle. Hidden sets are small and noisy, and repeated adaptation can overfit the visible signal.
• Keep simple behavioral signals beside powerful semantic models. Co-occurrence, transitions, history, and collaborative evidence encode continuity that text similarity may miss.
• Make the candidate boundary explicit. A reranker cannot rescue an item it never sees, so union construction and pruning are part of the learned-ranking design.
• Optimize every scored component. Retrieval, ranking, response grounding, response selection, and lexical variety all entered the composite; none was merely presentation.
• Use architecture comparisons as hypotheses, not verdicts. Released code can reveal better measurement and coverage patterns, but hidden labels are required for causal attribution.

Acknowledgements

This retrospective is possible because the teams released code and enough technical detail to inspect the boundaries, not merely the scores. I am grateful to:

• volart / artvolgin/music-crs-recsys2026 for documenting hybrid retrieval, disjoint validation, the LambdaMART ranker, and the multi-stage editorial response pipeline.
• niwatori / ryowk/recsys2026-niwatori for documenting the broad candidate union, history and transition features, the rich reranker, and ten-draft response selection.
• swyoo / yoobros/music-crs-challenge for documenting leakage-safe folds, learned retrieval, generated and profile evidence, PAS grounding, and lexical response stabilization.
• team2_s2 / lopsandrea/music-crs-team2 for documenting routed rankers, collaborative and acoustic evidence, covariate-shift weighting, verified track facts, and two-pass response generation.
• The Music-CRS challenge organizers for the task, data, metrics, and final results that made a shared technical comparison possible.

Caveats and assumptions

The Blind-B set contained only 80 hidden sessions, so a few difficult cases can move an aggregate metric. I do not have per-session relevance labels, judge rationales, or counterfactual submissions that change one component at a time. I therefore separate Verified arithmetic and documented architecture from Inferred contributors and Not documented details.

The comparison set is deliberately narrow: the four highest-scoring final entries among the public repositories supplied for review. Repository documentation varies in depth, and absence from a pinned source is not evidence of absence. Competitor feature counts, candidate counts, and pass counts are descriptive; larger numbers are not automatically better.

The in-sample dev range, OOF scores, Blind-A response sweep, and Blind-B outcome come from different evaluation contexts. Their contrast diagnoses confidence and transfer risk; it does not establish a single root cause.

Evidence notes

Evidence boundary. Official values and score contributions come from the final results CSV and the stated composite formula. Architecture claims come from public repositories pinned to the exact SHAs exposed in each diagram marker and in the report’s source metadata.

Report-shape note. The approved retrospective intentionally omits prescriptive work planning. The executive specification’s action-oriented closing role is replaced by retrospective principles because this artifact’s scope is explanation and transferable learning.

Chart map. The native quantitative chart is a zero-based five-entry composite overview with an adjacent exact leaderboard table. The gap decomposition uses the approved self-contained HTML fallback because the available native runtime does not preserve four-panel faceting and nine-decimal signed labels: four team panels share a −0.125 to +0.125 scale, center zero, label every signed component and total exactly, and include all 16 rows in an adjacent semantic table without pagination or movement formatting. The five architecture figures use the proven responsive two-rail HTML pattern with open text equivalents.