Harness plugin testing strategy

How a Honcho harness plugin (opencode-honcho, claude-honcho, dsh-honcho, and the rest) is verified without contaminating either the tester’s machine or anyone’s Honcho memory. This page holds the strategy and the decisions behind it. The commands live next to the code and are linked below, not copied.

Two contaminations, two tools

A harness test can leak in two independent directions, and each needs its own isolation.

Contamination	What leaks	Isolated by
Client	The harness install, the plugin and its node_modules, ~/.honcho/config.json, the harness’s own state dirs	A throwaway VM in the tester’s aurorus sandbox
Server	Messages, sessions, and conclusions written into a Honcho workspace; derived memory that then shapes later answers	The committed Honcho sandbox: sandbox/sandbox.sh in plastic-labs/honcho

Before the sandbox landed (DEV-2461, honcho #1111), the VM runbook covered the server side by pointing the plugin at hosted api.honcho.dev with a throwaway workspace and peer, and deleting the workspace at teardown. That was isolation by namespace, not by instance: every pass spent real deriver calls, the baseline was whatever the last pass left behind, and cleanup was a DELETE against production. The sandbox replaced that with a local instance that resets to a seeded, byte-identical state in under a second.

The two tools are deliberately separate. The sandbox lives in the honcho monorepo and knows nothing about any harness; the runbook lives in the plugin repo and knows nothing about Docker. Neither should absorb the other.

The seam is one URL

Every harness plugin resolves its configuration through @honcho-ai/harness-plugin-core, which reads HONCHO_BASE_URL (also HONCHO_URL and HONCHO_ENDPOINT) and uses the value verbatim. Pointing a plugin at the sandbox is therefore:

export HONCHO_BASE_URL=http://127.0.0.1:18000

Three consequences of that:

• No API key. The sandbox runs with auth disabled. opencode-honcho treats any localhost, 127.0.0.1, or ::1 host as “Local / self-hosted” and reports itself configured with no key at all. Other plugins adopting harness-plugin-core should match that behaviour.
• Do not use the local shorthand. harness-plugin-core expands HONCHO_BASE_URL=local to port 8000, the stock dev port. The sandbox is on 18000 precisely so it coexists with a normal local stack, so spell the URL out.
• Colocate the sandbox with the harness. aurorus VMs have no public IP and cannot reach a laptop, so the sandbox runs on the VM itself: install Docker, clone honcho, sandbox.sh up. A D2s_v5 is enough for the five containers.

Which backend for which test

The sandbox’s default mock provider is deterministic and free but returns placeholder text from the dialectic and hash-derived embeddings with no semantic similarity. That decides the split:

Test class (numbering from the opencode runbook)	Backend	Why
Capture source filter (1), kill mid-turn (2)	Sandbox, mock	Assert on what landed; read back over the unauthenticated API or psql on 15432
Egress failure (3)	Sandbox, mock	Stop the api container mid-turn instead of an iptables rule
Bogus API key (3)	Hosted	The sandbox does not check auth, so the row is meaningless there
Turn-1 injection (4), tools (9)	Sandbox with --provider real, or hosted	Needs real dialectic output and real embeddings
Redaction (5), session naming (6), import (7), commands (10)	Sandbox, mock	Pure plugin behaviour; reset between runs gives a clean baseline
Cross-integration sharing (8)	Sandbox, mock	Two plugins, one local workspace, no production writes
/honcho:setup cloud branch (7, 10)	Hosted	The localhost URL takes the self-hosted branch, so the “requires a key” path is never exercised

Rule of thumb: assert against the sandbox’s seeded conclusions, which are identical bytes in both provider modes, and never against the text or level mix of derived ones. The sandbox README explains why at length.

Where the mechanics live

• Procedure: Runbook: harness plugin VM pass, host-agnostic. VM, sandbox on the VM, pointing a plugin at it, reading back, acceptance criteria, teardown.
• Sandbox internals: sandbox/README.md in plastic-labs/honcho. Provider modes, fixture format, how reset is fast.
• Host-specific steps: each plugin repo’s docs/vm-runbook.md. How that host installs a plugin, its command names, and the mechanics behind each acceptance criterion. opencode-honcho’s is the first.
• VM access: aurorus (aurorus-vm).

References

• Sandbox: DEV-2461, shipped in honcho #1111 under milestone M4 (E2E Sandbox) of the Integration Management Strategy project.
• The mock server spec (DEV-1655, backlog) is a different tool for a different job: an SDK/CI mock with no Postgres and no deriver. The sandbox does not replace it, and it does not replace the sandbox, since harness tests need derived memory to recall against.