Excadrill is a registry-driven benchmark orchestration CLI for Honcho. It boots Honcho instances (via a Docker-based harness), runs benchmarks against them as subprocesses, and produces sealed, reproducible job artifacts.
What it does
- Runs system benchmarks against a live Honcho harness
- Runs direct-LLM baselines (no harness — for comparison against Honcho-mediated runs)
- Runs trace-dependent auxiliary evaluators on completed jobs
- Manages benchmark datasets (GCS download, SHA-256 verification, local cache, workspace symlinking)
- Parametric sweeps with parallel execution
- Job re-execution from saved configuration (full reproducibility)
Why it matters
Excadrill is the upstream signal in our ML pipeline. Its reasoning traces feed Minccino, which generates training data for Machamp, which produces models served by Metagross, which Honcho calls. A benchmark regression here reverberates through the whole pipeline.
Supported benchmarks
System (require Honcho harness)
| Key | Benchmark |
|---|---|
beam | BEAM — Beyond a Million Tokens (10 memory abilities, contexts up to 10M) |
longmem | LongMemEval (multi-session memory + haystack conversations) |
locomo | LoCoMo (long context, 5 question categories) |
evolmem | EvolMem (cognitive-driven multi-session dialogue, 7 abilities) |
memorybench | MemoryBench (continual learning from feedback; 28 subsets, 3 domains) |
oolong | OOLONG (out-of-distribution long-context) |
personabench | PersonaBench (personal info understanding) |
tremu | TReMu (temporal reasoning over multi-session dialogues) |
mab | MemoryAgentBench (4 competencies, AR, TTL, LRU, CR) |
Baselines (no harness)
beam_baseline, longmem_baseline, locomo_baseline, memorybench_baseline, oolong_baseline — full conversation history fed directly to the LLM.
Auxiliary (trace-dependent sidecars)
| Key | Evaluator |
|---|---|
action_judge | Action judge — eight criteria, including five with human labels |
molecular | Derivation quality — decontextuality + minimality |
coverage | Fact-extraction recall + information coverage |
deriver | Observation extraction quality vs. 9 failure categories |
Falinks provides the human labels that calibrate the action judge. It documents the five labeled criteria, sampling weights, and evals/judge_calibration report.
Running it
# Install
git clone https://github.com/plastic-labs/excadrill.git
cd excadrill && uv sync
cp .env.example .env
# Inspect
excadrill list
excadrill validate beam --context-length 100K
# Run
excadrill run beam --pool-size 1 --batch-size 1 --context-length 100K
# Sweep
excadrill sweep beam --param context-length=100K,1M,10M
# Reproduce
excadrill reproduce <job_id>Global flags are --verbose, --json-output, and --mode local|distributed.
Job artifacts
Every run creates an immutable directory under excadrill_runs/<timestamp>-<hash>/ with metadata.json, config.json, results.json, status.json, plus subdirs for results/, logs/, traces/, and artifacts/. Exit codes are 0 for success, 1 for benchmark failure, 2 for harness failure, and 3 for validation error.
Stack
Python 3.11+, uv-managed. Docker for system benchmarks. Optional extras are dev (pytest/ruff/pyright), ml (torch/transformers/dspy), and gcs (cloud-storage/boto3).
Adding a benchmark
Benchmarks are defined in registry/benchmark_registry.yaml and run as uv run python -m {module} subprocesses. No Python changes are required to add one — only registry entries plus a runner module that conforms to BaseRunner.