Deriver Training Pipeline

The Deriver pipeline builds training and evaluation datasets, fine-tunes and assembles a candidate model, serves it, compares it to the production model, and publishes the model and dataset after approval. Registeel records the inputs to data generation and training. Falinks supplies human labels for calibrating evaluation judges.

System flow

flowchart LR
    DATA["Minccino<br/>builds training and evaluation data"]
    TRAIN["Machamp<br/>trains and assembles the candidate model"]
    SERVE["Metagross<br/>serves the candidate model"]
    EVAL["Excadrill<br/>runs the selected benchmarks"]
    REVIEW{"Release decision"}
    PUBLISH["Publish model"]
    REJECT["Reject model"]
    REG["Registeel<br/>records data and training inputs"]

    DATA -->|training and evaluation datasets| TRAIN
    TRAIN -->|candidate model| SERVE
    SERVE -->|served candidate model| EVAL
    EVAL -->|candidate and production results| REVIEW
    REVIEW -->|approve| PUBLISH
    REVIEW -->|reject| REJECT

    DATA -. "data-generation manifest" .-> REG
    TRAIN -. "training manifest" .-> REG

Responsibilities and handoffs

Training sequence

The table presents the training sequence in execution order and identifies each stage’s repository or role, input, function, and output.

Step	Stage	Repository or role	Input	Function	Output
1	Select the run profile	MLTM	Committed MLTM training profile	Reads the selected profile	Selected Minccino recipe, starting model, served model name, evaluation setup, release criteria, and publication repositories
2	Build the dataset	Minccino	Committed recipe selected in step 1; external source data named by the recipe	Reads the source data, applies the recipe’s stages in order, and writes separate training and evaluation datasets	Training and evaluation datasets
3	Train the model	Machamp	Training and evaluation datasets; starting model selected by the profile	Fine-tunes the starting model and saves the learned LoRA weight changes	LoRA adapter
4	Assemble the candidate	Machamp	LoRA adapter; same starting model used for training	Combines the adapter with the starting model	Candidate model
5	Serve the candidate	Metagross	Candidate model; served model name selected by the profile	Starts vLLM with the candidate model	Model service used for evaluation
6	Evaluate	Excadrill	Candidate model (served in step 5); production model; benchmark suite selected by the profile	Runs the selected benchmarks	Results for the candidate and production models
7	Review	Release reviewer	Results for the candidate and production models; release criteria	Compares the results and approves or rejects the candidate	Release decision
8	Publish the release	Release operator	Approved candidate; training dataset	Uploads the model and dataset to the release repositories selected by the profile	Published release model and dataset

Supporting workflows

These workflows support the pipeline but are not release stages.

Workflow	Repository	Input	Action	Output
Create a provenance manifest	Registeel	Job configuration and project checkout	Records the configuration, Git state, Python packages, and selected input-file hashes in an immutable manifest	Manifest and content-derived ID
Compare job provenance	Registeel	Two manifests	Reports differences in configuration, code, environment, and input files while hiding run-specific noise	Provenance diff
Calibrate evaluation judges	Excadrill and Falinks	Deriver traces selected for calibration	Excadrill grades deriver outputs with an automated judge. Falinks collects human grades for a sample of the deriver outputs. Excadrill measures agreement between the automated and human grades.	Calibration report with judge agreement by criterion

Glossary

Term	Meaning
Trace	JSONL record of one Honcho model call’s inputs and outputs; Excadrill stores traces for auxiliary evaluation and later data generation
Recipe	Committed Minccino YAML that declares source inputs and an ordered sequence of processing and output stages
Manifest	Registeel JSON document containing secret-redacted resolved configuration, repository Git state, Python environment, and hashes of explicitly selected input files
Starting model	Pretrained model that Machamp fine-tunes and later combines with the adapter
Adapter	LoRA weight changes learned during training
Merged model	Model produced by combining the adapter with the same starting model used for training
Candidate model	Merged model being served and evaluated for release
Production model	Deployed deriver model
vLLM	Inference server used by Metagross to run the candidate model
Profile	Committed MLTM document selecting the recipe, starting model, served model name, evaluation setup, release criteria, and publication repositories
Excadrill job	Sealed directory containing one benchmark run’s identity and resolved parameters, final status and exit code, scores, failures, benchmark and harness logs, and Honcho traces
Evaluation setup	Models, benchmarks, code versions, and generation settings used for an evaluation
Evaluation judge	Model that assigns a score or verdict to a benchmark output
Judge calibration	Comparison of automated judge verdicts with resolved human labels

MLTM: training procedure

MLTM defines the procedure that coordinates the repositories in this pipeline. For each training run, an operator agent follows the SFT core runbook. The operator also selects a training profile that supplies the run inputs. The runbook defines the stage order, checks, and stop conditions.

The implementation for each stage resides in the relevant platform repository. Before operating a repository, the agent loads its Hermes skill. Each skill documents the repository’s interface and the source revision against which it was verified. The agent confirms that revision and maintains a run record containing each stage’s artifacts and results.

Minccino: dataset generation

Minccino generates and processes the data used for training and evaluation. Its recipes compose reusable stages that collect Honcho traces or read prepared examples, then filter, annotate, transform, combine, validate, and split those examples into training and evaluation datasets.

Each Minccino job retains its intermediate outputs and final datasets. The MLTM runbook reviews a small sample before running the full recipe. Machamp consumes the final training and evaluation datasets; the Registeel manifest links them to the recipe, source data, code, and environment that produced them.

Machamp: training and candidate assembly

Machamp provides model fine-tuning, experiment tracking, checkpointing, and artifact storage. In this pipeline, it fine-tunes the starting model on the Minccino datasets and writes a LoRA adapter. It combines that adapter with the same starting model to create the candidate model.

Machamp records training metrics, checkpoints, and artifact locations in an Aim run on the training machine. The Registeel manifest ID links that Aim run to the training inputs. Machamp can also store training artifacts in GCS or Hugging Face. Metagross serves the assembled candidate for evaluation.

Metagross: model serving

Metagross deploys model artifacts as GPU inference services. It configures and launches vLLM or SGLang on local hosts, in containers, or on Modal. It also provides health checks, endpoint tests, metrics, and load testing. This pipeline uses Metagross and vLLM to serve the candidate during evaluation.

Excadrill: candidate and production evaluation

Excadrill manages benchmark execution. It validates each benchmark request, starts the required test environment, runs the benchmark implementation, and stores the results, failures, logs, and traces as a sealed job that can be inspected or reproduced.

System benchmarks test Honcho end to end. Direct-context baselines test the model without Honcho. Auxiliary evaluations analyze traces produced by completed system benchmarks.

Review and publication

A reviewer compares the candidate and production results and approves or rejects the candidate. The operator records the decision, reviewer, rationale, and time in the MLTM run record.

Machamp can upload training artifacts to Hugging Face before release. After approval, the release operator uses the Hugging Face client in Machamp’s environment to publish the candidate model and Minccino dataset to the release repositories named by the profile. Production deployment is separate from the MLTM training procedure.

Falinks: judge-calibration labels

Falinks manages the human labeling used to calibrate Excadrill’s automated judges. This workflow runs separately from a model release.

Excadrill runs an automated judge on observations from selected deriver traces. Falinks draws a stratified sample of those observations, assigns them for blind review, resolves disagreements, and returns the resolved human verdicts to Excadrill. Excadrill measures agreement between the automated and human verdicts.

Registeel: run provenance

Registeel records the inputs and execution environment of data-generation and training jobs in an immutable provenance manifest. The manifest covers the resolved configuration, repository state, Python environment, package versions, and hashes of selected input files. Its content-derived ID detects changes to the manifest and links later artifacts and metrics to the conditions under which they were produced. Registeel records provenance; it does not store the job’s outputs.

Registeel compares two manifests and reports changes in configuration, source, environment, and input files while omitting expected run-specific differences.

See Infra for host assignments.