Runbook: harness plugin VM pass

A throwaway pass for any Honcho harness plugin (opencode-honcho, claude-honcho, dsh-honcho, …): a fresh VM for the client side and a seeded local Honcho for the server side, so nothing touches your workstation and nothing touches memory you care about. The strategy and the reasoning behind it are in Harness plugin testing; this page is the procedure.

Everything here is host-agnostic. The steps that depend on a particular harness (how it installs a plugin, its command names, the mechanics of each test) are specific to the plugin repo.

Before you start

One-time aurorus setup, if you have not already:

cd /path/to/aurorus
cp .env.example .env                          # set AZ_DEV_ALIAS=<you>
ln -s "$PWD/launch.sh" ~/.local/bin/aurorus-vm
az extension add -n bastion && az extension add -n ssh
aurorus-vm ssh-config >> ~/.ssh/config        # adds a `Host *.dev` block; needed for scp

1. VM

Reuse a VM if you have one: aurorus-vm list shows what is in rg-dev-<alias>, and az vm start -g rg-dev-<alias> -n <vm> wakes a deallocated one. Otherwise create a throwaway:

ALIAS=<you>
VM=<plugin>-test                              # e.g. opencode-honcho-test
 
az vm create \
  --resource-group rg-dev-${ALIAS} --name ${VM} \
  --location eastus2 --image Ubuntu2404 --size Standard_D2s_v5 \
  --admin-username azureuser --ssh-key-values ~/.ssh/azure_dev.pub \
  --vnet-name vnet-dev-${ALIAS} --subnet snet-vms \
  --nsg "" --public-ip-address "" \
  --storage-sku StandardSSD_LRS --os-disk-size-gb 64 \
  --tags purpose=dev owner=${ALIAS}
 
az vm auto-shutdown -g rg-dev-${ALIAS} -n ${VM} --time 2200 --email you@plasticlabs.ai

--public-ip-address "" is not optional. Azure Policy denies public IPs in rg-dev-*; Bastion is the only way in.
Standard_D2s_v5 is on the SKU allow-list, is enough for the harness plus the five sandbox containers, and costs about $70/month if left running. Deallocate when done (step 7).

With the Host *.dev block installed the VM is ssh <vm>.dev, and scp works even though there is no public IP. A TUI harness (OpenCode, Claude Code) runs inside that session and needs nothing more. A harness with a web UI (dsh on 3080) binds loopback on the VM, so forward the port and keep the session open:

ssh -L 3080:127.0.0.1:3080 <vm>.dev

2. Honcho sandbox on the VM

The sandbox has to be on the same machine as the harness: the VM cannot reach your laptop.

sudo apt-get install -y git jq curl unzip docker.io docker-compose-v2
sudo usermod -aG docker "$USER" && exit      # then ssh back in so the group applies

curl -LsSf https://astral.sh/uv/install.sh | sh    # sandbox.sh seeds through uv
export PATH="$HOME/.local/bin:$PATH"
git clone https://github.com/plastic-labs/honcho ~/honcho
~/honcho/sandbox/sandbox.sh up                     # mock provider: deterministic, free

The api is on http://127.0.0.1:18000, Postgres on 15432, Redis on 16379. sandbox.sh reset returns it to the seeded state in under a second; sandbox.sh status shows what is running.

For the tests that need real model output (turn-1 injection, tool quality), copy sandbox/real.env.example to sandbox/real.env, add a provider key, and run sandbox.sh up --provider real. Each mode keeps its own snapshot, so switching is cheap after the first seed. Which tests need which mode is the table in Harness plugin testing.

3. Install the harness and the plugin

This is the host-specific step. Follow the plugin repo’s docs/vm-runbook.md. Two rules hold for every harness:

Test the packed artifact, not a source checkout. npm pack on your workstation, scp the tarball up, install from that. It is what users get.
Check the plugin actually loaded before running any test. A host may import a file-path plugin without installing its dependencies, and a plugin whose import throws can fail silently in headless mode. Every plugin logs a line on successful init; find it in the host’s log before going further. A “nothing landed in Honcho” result with no init line is an install problem, not a plugin bug.

4. Point the plugin at the sandbox

Plugins resolve configuration through @honcho-ai/harness-plugin-core, so the same env vars work everywhere and win over the config file:

cat > ~/.harness-test.env <<'ENV'
export HONCHO_BASE_URL=http://127.0.0.1:18000
export HONCHO_PEER_NAME=vm-test-peer
export HONCHO_WORKSPACE=<plugin>-vm-test
# plus whatever model-provider key the harness itself needs
ENV
chmod 600 ~/.harness-test.env
echo '[ -f ~/.harness-test.env ] && source ~/.harness-test.env' >> ~/.bashrc
source ~/.harness-test.env

No HONCHO_API_KEY. The sandbox runs with auth off, and a plugin that follows opencode-honcho treats a localhost / 127.0.0.1 base URL as configured without a key.
Spell the URL out. HONCHO_BASE_URL=local expands to port 8000, the stock dev port, not the sandbox’s 18000.
Hosted variant. For the rows that need hosted Honcho (auth behaviour, the cloud setup flow), unset HONCHO_BASE_URL, set HONCHO_API_KEY, and keep the throwaway workspace name. Delete that workspace at teardown; the sandbox cannot do it for you.
Tests that exercise the config file (setup flow, upgrade prompts, hosts.<name> keys such as sessionStrategy) need the env overrides unset for that test, since env wins. To keep the file itself throwaway, point HONCHO_CONFIG_PATH at one; that is harness-plugin-core’s override. A plugin that predates core may have its own (dsh-honcho still reads HONCHO_CONFIG_DIR).

5. Reading back what landed

No auth header is needed against the sandbox. Listing is a POST to /list, not a GET.

H=http://127.0.0.1:18000/v3
WS=<plugin>-vm-test
curl -s -X POST -H 'Content-Type: application/json' "$H/workspaces/$WS/sessions/list" -d '{}' \
  | jq -r '.items[].id'
curl -s -X POST -H 'Content-Type: application/json' "$H/workspaces/$WS/sessions/<id>/messages/list" -d '{}' \
  | jq -r '.items[] | "\(.peer_id): \(.content)"'

Or go straight to the database: psql -h 127.0.0.1 -p 15432 -U postgres honcho_sandbox.

Run ~/honcho/sandbox/sandbox.sh reset between tests. It drops the plugin’s workspace along with everything else and restores the seeded fixture, so each test starts from the same bytes.

6. What every plugin must pass

The acceptance criteria are the same for every harness; the mechanics differ and are in the plugin repo’s runbook. Blocking items first.

Test	Pass	Fail	Blocking
Capture source filter	Only real user and assistant text lands	Injected memory, system prompt, tool output, or command-template text captured as a turn	yes
Kill mid-turn	Completed turns land once each; the interrupted one may be absent	Duplicates, or missing completed turns	yes
Failure modes (bad key, egress lost mid-session, unreachable base URL, missing config)	Host boots, turns complete, status surfaces the error, later turns recover when the network does	Host crashes or hangs, outage is invisible in status, or failure is sticky for the process lifetime	yes
Turn-1 injection	A saved fact is used on the very first message of a fresh session	Missed on turn 1, correct on an identical turn 2 (hook ran before the runtime was ready)
Redaction	A pasted secret is redacted, or the gap is recorded as known	Recorded as a regression when the plugin never claimed it
Session naming	Each configured strategy yields the ids it promises (per directory, per session, per branch, global)	Two projects share an id, or a long path breaks Honcho’s 100-char id limit
Cross-integration	Default is isolation per plugin workspace; pointing two plugins at one workspace and peer shares memory	Plugins see each other by default
Tools and commands	Search returns messages and conclusions; a saved conclusion survives into a fresh session; status is accurate; config elides the key
Config surface	Every documented key has a visible effect; an unsupported or ignored key is logged with the reason	A key is silently ignored

Egress loss is easiest to stage against the sandbox: docker compose -p honcho-sandbox stop api mid-turn, then start it and confirm recovery. The bad-key row is meaningless against the sandbox (auth is off) and must run hosted.

7. Teardown

~/honcho/sandbox/sandbox.sh down          # optional; reset is what matters between passes
curl -X DELETE -H "Authorization: Bearer $HONCHO_API_KEY" \
  "https://api.honcho.dev/v3/workspaces/<plugin>-vm-test"   # only if you ran the hosted variant
 
az vm deallocate -g rg-dev-<alias> -n <vm>   # stop compute billing, keep the VM
az vm delete -g rg-dev-<alias> -n <vm> --yes # or remove it outright

Stopping the VM from inside the guest does not deallocate it; you keep paying. The nightly auto-shutdown from step 1 covers you if you forget.

8. Reporting

Record the result against the plugin’s Linear issue. For a failure, capture the plugin’s log lines, the status and config command output with the key already elided, the Honcho session id, and which backend (sandbox mock, sandbox real, hosted) the row ran on. Fold any new gotcha into the plugin repo’s runbook if it is host-specific, or into this page if every harness would hit it.

Plastic Labs Internal Wiki

Explorer

Recently Updated

Company

Operating Model — Decisions Log

Operating Model

Security & Compliance

ideal-reasoned-representations