Using Experimaestro-IR for complex Information Retrieval experiments
Modern Information Retrieval (IR) research has outgrown monolithic scripts and ad-hoc shell pipelines. Most ML pipelines in IR —such as knowledge-distilled cross-encoder re-rankers— often combine heavy lifting stages e.g. large-scale document stores building and indexation, teacher score or hard negatives mining, distributed multi-GPU training, and multi-benchmark evaluations. These pipelines are fragile, hard to reproduce, and often waste compute by re-running expensive first-stage retrievals for every new checkpoint or random seed. Here is how Experimaestro-IR addresses the combinatorial complexity, compute redundancy, and reproducibility challenges of neural IR.
The Struggle: Multi-Stage IR Workflows orchestration complexity (Shared by many other ML experiments)
Evaluating a single cross-encoder checkpoint is not a simple inference script. It requires an end-to-end multi-stage pipeline:
- Collection indexing: Building sparse lexical or learned representations (e.g. SPLADE-v3 or BM25) over millions of passages.
- Candidate pool generation: Retrieving top-k candidates per topic query.
- Hydration & Re-ranking: Looking up passage texts from disk-backed document stores, concatenating query-doc pairs, and running cross-attention inference.
- Metric aggregation: Computing NDCG@10, MRR@10, Recall, and MAP across varied benchmark distributions (MS MARCO, BEIR, NanoBEIR).
With all this complexity, it's too easy to re-run expensive first-stage retrievals for each training seed or checkpoint, mess up with cluster job dependencies across SLURM scripts, and lose track of the exact tokenization, batching, and optimizer settings used in published runs.
How Experimaestro-IR Solves This
📦 1. Declarative Tasks vs. Configurations
Strongly-typed Python abstractions. Tasks handle computation (training, indexing), while Configs define declarative parameters (losses, tokenizers). This allow complete serialization of all tasks for reproducibility, and automatic caching of intermediate results.
♻️ 2. Automated Compute Deduplication
Decouples first-stage retrieval from re-ranking. Inverted indexes and candidate pools are hashed, cached, and shared across all downstream models, seeds, and epochs—eliminating duplicate retrieval compute.
🔍 3. 100% Auditability
Every hyperparameter is serialized into deterministic, content-hashed directories. Changing any parameter invalidates only the affected subgraph, guaranteeing reproducibility without manual cache management.
🧩 4. Flexible Configurations
Explicit control over an experiment's identity. Param[T] contributes to the task hash,
while Meta[T] marks runtime-only options (paths, debug logs) that are ignored.
You can add new features with default values without invalidating already-completed runs.
⚡ 5. End-to-end Orchestration
All Evaluation Tasks are launched the very moment the Learner completes. Executes across local GPUs and SLURM clusters transparently with custom launchers depending on the compute environment.
🎯 6. Declarative Grid Searches
Explore hyperparameter spaces systematically with Experimaestro's Grid Search engine. Define Cartesian parameter sweeps across backbones, losses, and seeds while automatically reusing shared sub-paths.
Defining an Experiment Declaratively
Rather than writing monolithic command-line strings, experiments are defined cleanly in Python using Experimaestro's typed configuration primitives.
See src/cross_encoders_training/experiment.py for the complete pipeline used for training and evaluating cross-encoders.
Thanks to Experimaestro, we can reconstruct the full experimental DAG directly from the serialized experiments. We give below an example for a knowledge-distilled cross-encoder trained on MS MARCO and evaluated In Domain (MSMARCO, Trec DL19 and DL20).
The interactive diagram below was generated directly with the
experimaestro dag command,
tooltips were added on top to to detail each node.
Ettin-32M / MiniLM Distillation & Multi-Benchmark Evaluation
Anatomy of the Cross-Encoder Pipeline
The interactive graph above demonstrates a knowledge-distillation pipeline fine-tuning an Ettin-32M student cross-encoder:
PairwiseDistillationSamplesTSV + MsMarcoPassagesStore → DistillationPairwiseTrainer (with MSEDifferenceLoss and LinearWithWarmup) → Learner (Fabric multi-GPU training) → STModuleLoader → TwoStageRetriever (combining precomputed SPLADE-v3 candidate runs with cross-encoder scoring) → Evaluate (reporting benchmark metrics).
By clicking "Show Intermediate Configs" on the interactive diagram, experimenters can inspect how every single hyperparameter is anchored directly to the task that consumes it.