# Reproducing the LightGBM reranker The v10 state-ranker path makes ranking explicit: `state_ranker_v10_rrf_devset` serves the `candidate_fusion` stage, while the LGBM configs serve a LightGBM bundle selected by their `ranking.model_path` values. The current active bundle for devset, Blind-A, and Blind-B is `models/reranker_v12_goalfree`. `model_version` is the trace/stage label, not the source of truth for which booster is loaded. This doc lists what ships in the repo, what does not, and the two ways to reproduce: **FAST** (use the committed model) and **FULL** (retrain from fresh v10 traces). Do not treat v9/v0plus artifacts as current training inputs. ## What ships in the repo | Path | What | Size | |---|---|---| | `models/reranker_v12_goalfree/` | current active goal-free LightGBM bundle | 1.9 MB | | `models/reranker_v10/` | retained historical v10 LightGBM bundle | 1.9 MB | | `mcrs/qu_modules/lgbm_reranker.py` | online in-pipeline reranker (config-driven) | — | | `scripts/rerank/replay_lgbm.py` | offline replay runner for applying a model bundle to saved retrieval traces | — | | `run_pipeline.py` | staged experiment runner for retrieval → rerank replay → explanation → evaluation | — | | `scripts/rerank/features.py` | `compute_turn_features` — the single per-turn feature function called by **both** the offline builder and the online server (anti-drift) | — | | `scripts/rerank/build_features.py` | offline builder — loads the catalog/caches and writes the training parquet by calling `compute_turn_features` (same schema as serving) | — | | `scripts/rerank/build_constraint_features.py` | constraint sidecar (is_played, rejection flags) | — | | `scripts/rerank/build_label_weights.py` | train-time label-quality weights | — | | `scripts/rerank/train_lgbm.py` | LightGBM trainer — stages `build → fold ×5 → finalize → full_model` (`full_model` writes `model_full.txt`) | — | | `scripts/build_tag_embedding_index.py` | builds the tag-embedding index (below) | — | | `configs/state_ranker_v10_rrf_devset.yaml` | devset candidate-fusion/RRF baseline | — | | `configs/state_ranker_v10_lgbm_devset.yaml` | devset goal-free reranker | — | | `configs/state_ranker_v10_lgbm_blindset_A.yaml` | blind-A goal-free reranker + response gen | — | | `configs/state_ranker_v10_lgbm_blindset_B.yaml` | blind-B goal-free reranker + response gen | — | | `configs/pipelines/state_ranker_v10_lgbm_devset.yaml` | staged devset pipeline config for local iteration | — | The model bundle is referenced via `${oc.env:MCRS_MODEL_DIR,}`, where the default is `models/reranker_v12_goalfree` for all active LGBM configs. That resolves to a committed path locally and inside the Modal image (the repo is copied to `/app`; `models/` is not in the image ignore list), so no volume upload is needed for these bundles. Setting `MCRS_MODEL_DIR` overrides all four bundle files for a run. ## What does NOT ship (and how to get it) | Artifact | Role | Required? | How to obtain | |---|---|---|---| | `cache/lancedb/music_track_catalog` | 47k-track catalog + all vectors; needed by **all** retrieval | **Required** | Build per the base setup; on Modal it lives on the `music-crs-models` volume | | `cache/tag_embedding_index/qwen_0_6b.npz` (68 MB) | tag resolver (compiler **and** reranker); hard-loaded at reranker init | **Required** | `python scripts/build_tag_embedding_index.py --db-uri cache/lancedb --out cache/tag_embedding_index/qwen_0_6b.npz --min-track-count 5` (needs `DEEPINFRA_API_KEY`). On Modal: `music-crs-cache` volume | | `exp/ground_truth/devset.json` (1.8 MB) | eval labels | for eval | `python evaluator/make_ground_truth.py` (auto-generated on Modal) | | `exp/analysis/rerank/q06_memo.json` (~570 MB) | query-embedding cache | **Optional** | Serving live-fills via DeepInfra (`DEEPINFRA_API_KEY`); host on `music-crs-cache` to warm-start | | `exp/analysis/rerank/raw_msg_store/` (~470 MB) | message-embedding store | **Optional** | Same — `NpzEmbedStore` live-fills on miss; host to warm-start | | training intermediates (`rerank/v10/features/`, `X.npy` 12 GB, `rerank/v10/constraint_features.parquet`, `rerank/v10/label_weights.parquet`) | retrain only | retrain only | Regenerated by the FULL path; never committed | **Secrets:** `DEEPINFRA_API_KEY` (Qwen 0.6B embeddings), `OPENROUTER_API_KEY` (deepseek-v4-flash extractor + qwen3-30b responses), `VLLM_API_KEY` (qwen 8B), and an HF token (`hf auth login`). On Modal these come from the `ENV_SECRET`. ## FAST path — use the trained model Runs with the committed seed model; no retraining. ```bash uv pip install -e . # lightgbm is now a declared dependency hf auth login export DEEPINFRA_API_KEY=... OPENROUTER_API_KEY=... VLLM_API_KEY=... # Tag index (required, 68 MB) — build once if not already on the Modal volume: python scripts/build_tag_embedding_index.py \ --db-uri cache/lancedb \ --out cache/tag_embedding_index/qwen_0_6b.npz --min-track-count 5 # Devset (Modal, 50 shards by default): python run_experiment.py --backend modal --tid state_ranker_v10_lgbm_devset --batch_size 8 # → exp/scores/devset/state_ranker_v10_lgbm_devset.json # Blind-A submission: python run_experiment.py --backend modal --tid state_ranker_v10_lgbm_blindset_A \ --eval_dataset blindset_A --batch_size 8 bash prepare_submission.sh state_ranker_v10_lgbm_blindset_A # Blind-B submission path: python run_experiment.py --backend modal --tid state_ranker_v10_lgbm_blindset_B \ --eval_dataset blindset_B --batch_size 8 bash prepare_submission.sh state_ranker_v10_lgbm_blindset_B blindset_B ``` ### Staged FAST path — reuse retrieval traces For reranker iteration, run retrieval once and replay the committed model bundle locally over the saved trace. This keeps extraction/retrieval variability out of small reranker experiments. ```bash # Full staged local devset run python run_pipeline.py --config configs/pipelines/state_ranker_v10_lgbm_devset.yaml # Later: replay rerank/eval from the same retrieval trace python run_pipeline.py \ --config configs/pipelines/state_ranker_v10_lgbm_devset.yaml \ --from rerank \ --retrieval-run exp/pipeline/runs/ \ --run-id # Swap model bundles without rerunning retrieval python run_pipeline.py \ --config configs/pipelines/state_ranker_v10_lgbm_devset.yaml \ --only rerank \ --retrieval-run exp/pipeline/runs/ \ --model-ref models/ \ --run-id ``` `run_pipeline.py` writes per-run artifacts under `exp/pipeline/runs//`. The retrieval stage delegates to `run_experiment.py`; the rerank stage calls `scripts/rerank/replay_lgbm.py` and uses the same `features.compute_turn_features` function as training/serving. Rerank replay can shard local workers and, for fast ranking/eval loops, can skip rerank trace output with `rerank.write_trace: false`. Training remains in the FULL path below. On Modal the catalog (`music-crs-models`) and the tag index / warm caches (`music-crs-cache`) are read from the persistent volumes. For a brand-new Modal account, upload them once: ```bash python -m modal volume put music-crs-models cache/lancedb/ lancedb/ python -m modal volume put music-crs-cache cache/tag_embedding_index/ tag_embedding_index/ # optional warm caches (serving live-fills without them): python -m modal volume put music-crs-cache exp/analysis/rerank/q06_memo.json rerank/q06_memo.json python -m modal volume put music-crs-cache exp/analysis/rerank/raw_msg_store/ rerank/raw_msg_store/ ``` ## FULL path — retrain from scratch ```bash # 1. Build the catalog + tag index (FAST path, steps above). # 2. Run the RRF retrieval to produce per-turn trace shards (capture the RUN_ID it prints): python run_experiment.py --backend modal \ --tid state_ranker_v10_rrf_devset --batch_size 64 # 3. Build features per trace shard. Both paths call the SAME # features.compute_turn_features the server uses, so the parquet schema # matches the served model's meta.json by construction. # # (a) Recommended — on Modal (no local catalog/caches needed; reads volumes): python -m modal volume put music-crs-cache exp/ground_truth/devset.json rerank/ground_truth_devset.json # once modal run modal/app.py::run_build_features_for_ranker \ --lineage v10 --tid state_ranker_v10_rrf_devset --run-id --n-shards 50 # writes rerank/v10/features/ + rerank/v10/constraint_features.parquet + rerank/v10/label_weights.parquet. # # (b) Local alternative — needs the catalog on disk; warm caches make it local-only: python scripts/rerank/build_features.py \ --trace exp/inference/devset/_trace.jsonl \ --ground-truth exp/ground_truth/devset.json \ --db-uri cache/lancedb \ --tag-index cache/tag_embedding_index/qwen_0_6b.npz \ --embed-memo exp/analysis/rerank/q06_memo.json \ --branch-names models/reranker_v12_goalfree/branch_names.json \ --msg-store exp/analysis/rerank/raw_msg_store \ --out exp/analysis/rerank/v10/features # By default this fans out to 12 checkpointed shards with 4 local workers and # fills any missing feature embeddings into the cache. Use --offline only for # strict cache-only replay; override with --num-shards N --num-workers M when needed. # 4. Constraint sidecar + label weights — SKIP if you used the Modal builder in # step 3a: run_build_features_for_ranker already wrote BOTH sidecars to the cache volume. Only needed for the local # path 3b: python scripts/rerank/build_constraint_features.py --db-uri cache/lancedb \ --features exp/analysis/rerank/v10/features \ --out exp/analysis/rerank/v10/constraint_features.parquet python scripts/rerank/build_label_weights.py \ --trace-glob "exp/inference/devset/.run_.shard_*_trace.jsonl" \ --db-uri cache/lancedb \ --out exp/analysis/rerank/v10/label_weights.parquet # 5. Train on Modal (CPU): build matrix → 5 CV folds → finalize → full_model. # full_model is the stage that writes model_full.txt (finalize only reports # OOF metrics). Upload lockbox/ground-truth once, then: modal run modal/app.py::run_train_lgbm_ranker --lineage v10 # 6. Fetch + publish the full-data model into the committed bundle: modal volume get music-crs-cache rerank/v10/train/model_full.txt exp/analysis/rerank/v10/train/ modal volume get music-crs-cache rerank/v10/train/meta.json exp/analysis/rerank/v10/train/ modal volume get music-crs-cache rerank/v10/train/cat_maps_v9.json exp/analysis/rerank/v10/train/ cp exp/analysis/rerank/v10/train/model_full.txt models/reranker_v12_goalfree/model.txt cp exp/analysis/rerank/v10/train/meta.json models/reranker_v12_goalfree/meta.json cp exp/analysis/rerank/v10/train/cat_maps_v9.json models/reranker_v12_goalfree/cat_maps.json # 7. Serve + evaluate (FAST path). ``` > The Modal v10 trainer writes scratch under `music-crs-cache:/rerank/v10/train/` > (including the large feature matrix). Only the small published bundle in > `models/reranker_v12_goalfree/` is the active committed bundle. ## Train/serve parity The server (`lgbm_reranker.py`), the offline builder (`build_features.py`), and staged replay (`scripts/rerank/replay_lgbm.py`) all produce features through the SAME `features.compute_turn_features`, so the feature **schema** is identical by construction — there is no parallel offline schema that can drift, and the regenerated parquet's columns match the served `meta.json` exactly. For numeric reproduction also keep the other inputs consistent between build and serve: the catalog, the tag index, the warm embedding caches, and `pool_k` (the per-pool features — `z__score__*`, `ratio__*`, `pct_*` — are normalized over the pool truncated to `pool_k`). To keep build and serve aligned, `build_features.py` (`--pool-k`) and `run_build_features_for_ranker` (`--pool-k`) both default to **500**, matching the v10 serving configs (`qu_kwargs.ranking.pool_k: 500`). If you change the serving `pool_k`, rebuild features with a matching `--pool-k`. ## Related work: why not a cross-encoder reranker? A zero-shot Qwen3-Reranker cross-encoder was evaluated as a serving-time reranker/filter on top of this LightGBM model and found net-negative on nDCG@20 in every fusion mode tested (replace, RRF-fusion, filter, promote) — see [`docs/research/cross_encoder_exploration.md`](research/cross_encoder_exploration.md) for the full experiment writeup. It does have offline value as a data labeler/teacher for bi-encoder training; it is not part of the serving path.