Plastic Labs Internal Wiki

Recently Updated

Machamp — LLM Fine-tuning

3 min read

Repo: plastic-labs/machamp Agent doc: repos/machamp/AGENT.md (read this before touching code)

Machamp is Plastic Labs’ framework for rapid LLM fine-tuning. It wraps Unsloth (LoRA + full-FT kernels), trl (SFTTrainer, DPOTrainer), and Aim (experiment tracking) behind a Hydra configuration system so changing model, dataset, logger, or remote artifact sink is a one-flag operation.

What ships

  • Two pipelines: train_unsloth_sft and train_unsloth_dpo
  • Model-agnostic — works across Gemma, Qwen, Llama, Mistral, Phi, anything unsloth.FastModel supports
  • Pluggable artifact sinks: GCS and Hugging Face Hub (registry-based, lazy-imported)
  • Aim logger as the reference BaseLogger implementation
  • Pluggable via ABCs — adding a new logger / sink / pipeline is a single-file drop

Where it fits in the system

Machamp consumes datasets produced by Minccino and produces model weights that Metagross serves. See the system map for the flow.

House rules

No Hydra CLI overrides — ever

Every config change goes through editing yaml under conf/, not via pipeline.base_model=... overrides at the CLI. The yaml files are the source of truth for every run; CLI overrides hide intent and break reproducibility from repo state alone.

| Behavioral change | Where to edit | | --------------------------------- | ------------------------------------------------------------------- | --- | --- | ----- | | Swap pipeline (SFT ↔ DPO) | conf/config.yaml → defaults: - pipeline: train_unsloth_dpo | | Swap dataset | conf/config.yaml → defaults: - data: <name> | | Swap artifact sink | conf/config.yaml → defaults: - artifact: hf_hub | gcs | all | none | | Change base model | conf/pipeline/train_unsloth_sft.yaml → base_model: | | Change LR / LoRA rank / optimizer | conf/pipeline/train_unsloth_sft.yaml → train_args / peft_args | | Toggle full finetuning | conf/pipeline/train_unsloth_sft.yaml → full_finetuning: true | | Change HF target repo | conf/artifact/hf_hub.yaml → repo_id: | | Change GCS bucket | conf/artifact/gcs.yaml → bucket: / prefix: |

One invocation

uv run python -m machamp

Whatever pipeline/model/dataset/logger/sink runs is determined entirely by yaml state.

Architecture

  • machamp/pipeline/ — BasePipeline ABC + train_unsloth_sft.py and train_unsloth_dpo.py
  • machamp/logger/ — BaseLogger ABC + Aim implementation (the trainer callback lives here, not in pipeline code)
  • machamp/artifact/ — BaseArtifactManager ABC + gcs.py, hf_hub.py. Auto-imported via @register decorator.
  • machamp/data/ — BaseDataset + per-dataset modules
  • conf/ — Hydra configuration groups (composition root is conf/config.yaml)

Invariants

  • Use FastModel.from_pretrained only — never FastLanguageModel (legacy) or raw AutoModelForCausalLM. You lose Unsloth kernels otherwise.
  • get_peft_model() must be gated on full_finetuning == False. Calling it in full-FT mode double-wraps and breaks the forward pass.
  • attn_implementation: 'auto' resolves to 'sdpa', not 'flash_attention_2'. Gemma 4’s head_dim > 256 exceeds FA2’s limit.
  • Artifact upload failures must never raise. The 18-hour training run is sacred.
  • PatchDPOTrainer() must be called before from trl import DPOTrainer.

Optional extras

uv sync --extra flash-attention   # Prebuilt FA2 wheel for torch 2.8 / cu12 / py3.11
uv sync --extra gcs               # google-cloud-storage
uv sync --extra docs              # MkDocs Material
uv sync --extra scripts           # Dataset-prep helpers (anthropic, polars, jinja2)

Tests

  • tests/test_artifact_managers.py — sink unit tests (offline; SDK clients patched)
  • tests/test_logger_aim.py — Aim logger smoke against on-disk repo
  • tests/test_e2e_training.py — end-to-end smoke (skipped if no GPU)

See also

  • Excadrill — produces benchmark traces that flow into Minccino
  • Minccino — produces the datasets Machamp consumes
  • Metagross — serves the models Machamp produces

Graph View

Backlinks