Minccino — Data Trace Generation

Repo: plastic-labs/minccino

Minccino is the data trace generation and preprocessing pipeline for building SFT/DPO/GRPO datasets out of a Honcho memory harness. It sits between Excadrill (which produces traces) and Machamp (which consumes datasets).

Core concepts

Concept	Meaning
DataItem	Dict-wrapped record flowing through the pipeline
Stage	Single transformation. Declares mode (map / reduce / ordered) and io_bound so the executor picks the right concurrency
Recipe	Ordered list of stage references with config overrides — the only unit of composition
Job	One execution of a recipe. Writes an immutable artifact directory with resolved recipe, per-stage configs, stats, output
ArtifactStore	Local filesystem or GCS — pluggable backend
HonchoHarness	Self-contained. Default source stage boots a harness pointed at a job-scoped traces directory and tails the reasoning-trace JSONL as it’s written. No external harness checkout required

Stage groups

Group	Intent
source/	Produce records (honcho_live_traces, trace_file, trace_dir, honcho_sessions, honcho_conclusions, jsonl)
extract/	Parse structured fields from raw records
filter/	Drop records by predicate
annotate/	Add fields without changing identity (llm_judge, reward_model, regex heuristics)
transform/	Reshape into training formats (to_sft, to_dpo, to_grpo)
synthesize/	Create new records from existing ones (llm_rewrite, paraphrase, self_instruct)
mix/	Combine multiple input streams
split/	Partition a stream
validate/	Assert properties, fail-fast
sink/	Persist output

Running it

uv sync
cp .env.example .env   # HONCHO_REPO and LLM keys
 
minccino list stages --format json
minccino describe stage honcho_live_traces
minccino validate --recipe honcho_sft_v1
minccino run --recipe honcho_sft_v1 --dry-run --sample 4
minccino run --recipe honcho_sft_v1 --job-id my_first_run
minccino jobs show my_first_run
minccino jobs replay my_first_run --override filter.length.max_tokens=4096

Adding a stage

Two-file operation — drop a Python module into minccino/stages/<group>/<name>.py and a matching YAML into conf/<group>/<name>.yaml. No core changes required.

from minccino.core.stage import Stage, register
from minccino.core.data_item import DataItem
 
@register("my_filter", group="filter")
class MyFilter(Stage):
    mode = "map"
    io_bound = False
 
    def __init__(self, config):
        self.threshold = config.threshold
 
    async def run_item(self, item: DataItem, ctx) -> DataItem | None:
        return item if item.content["score"] > self.threshold else None

Where it fits in the system

• Upstream: Honcho (live traces) or Excadrill (sealed job traces)
• Downstream: Machamp consumes the datasets Minccino produces
• See system map