CLI & Repo Restructuring (v2)

Status: Draft | Owner: vineeth | Last updated: 2026-03-25 Supersedes: cli.v1.md (2025-01-13 draft)

1. Problem Statement

Honcho is currently structured as a single-package monolith under src/ with co-located subsystems (API server, deriver background worker, dreamer consolidation agent, dialectic query engine) sharing a flat namespace. This creates several problems:

No standalone CLI. Running Honcho locally requires manual orchestration: starting PostgreSQL, running Alembic migrations, launching the FastAPI server, starting the deriver as a separate process, and configuring Redis. There is no single command to get a working local Honcho stack.
Monolithic packaging. The API server, deriver, dreamer, and dialectic are all installed as one package. Deploying the deriver (a background worker) pulls in all API router dependencies. There is no way to install only the components you need.
No workspace isolation. The pyproject.toml already uses [tool.uv.workspace] for the Python SDK, but the core server code is not factored into workspace packages. Shared utilities (models, config, DB, embedding client) are imported via src. prefix paths, coupling everything.
Docker Compose is manual. The existing docker-compose.yml.example must be copied, configured, and invoked directly. There is no tooling to generate, validate, or manage the compose stack lifecycle.
Developer onboarding friction. New contributors must read multiple docs, set up environment variables, install Docker, run migrations, and start multiple processes. The supabase CLI and openhands CLI demonstrate that a single init + up workflow dramatically reduces time-to-first-request.

2. Goals / Non-Goals

Goals

G1: Create a honcho CLI (Typer-based) installable via uv tool install honcho-cli that orchestrates local Honcho development.
G2: Restructure the repository into a uv workspace monorepo with distinct packages: honcho-shared, honcho-api, honcho-deriver, honcho-dreamer, honcho-cli.
G3: honcho init generates a project-local .honcho/ directory with config.toml and docker-compose.yml.
G4: honcho up / honcho down wrap Docker Compose to manage the full stack (PostgreSQL + pgvector, Redis, API, deriver).
G5: honcho serve runs the API server directly (outside Docker) for development with hot-reload.
G6: honcho deriver runs the deriver worker directly (outside Docker) for development.
G7: honcho doctor validates the local environment (Docker, uv, ports, DB connectivity).
G8: honcho status shows running services and health.
G9: honcho nuke tears down all containers, volumes, and local state for a clean reset.
G10: Configuration layering: ~/.honcho/config.toml (global) < .honcho/config.toml (project-local) < environment variables < CLI flags.
G11: The default honcho up stack uses PostgreSQL (via pgvector/pgvector:pg15 container), not SQLite. SQLite support is a separate spec.

Non-Goals

SQLite as a database backend (separate spec: spec-sqlite-support.md).
Managed cloud deployment (honcho deploy). This is a Groudon concern.
SDK scaffolding or code generation (honcho generate).
GUI or TUI dashboard.
Windows-native support (WSL2 is acceptable).
Production deployment orchestration (Fly.io, Kubernetes). The CLI targets local dev and self-hosted single-machine setups.
Mock server implementation (honcho mock will be a placeholder command reserved for a future mocking-server spec).

3. Design

3.1 Package Structure (uv Workspaces)

The repository root pyproject.toml becomes a workspace root. Each subsystem becomes an independent package with its own pyproject.toml, declaring only the dependencies it actually needs.

honcho/                              # repository root
+-- pyproject.toml                   # workspace root (no [project], only [tool.uv.workspace])
+-- uv.lock                         # single lockfile for entire workspace
+-- alembic.ini
+-- migrations/
+-- docker/
+-- Dockerfile
+-- config.toml.example
+-- packages/
|   +-- shared/                      # honcho-shared
|   |   +-- pyproject.toml
|   |   +-- src/
|   |       +-- honcho_shared/
|   |           +-- __init__.py
|   |           +-- models.py        # SQLAlchemy ORM models (from src/models.py)
|   |           +-- schemas.py       # Pydantic schemas (from src/schemas.py)
|   |           +-- config.py        # Configuration management (from src/config.py)
|   |           +-- db.py            # Engine creation, session factory (from src/db.py)
|   |           +-- dependencies.py  # DB dependency injection (from src/dependencies.py)
|   |           +-- exceptions.py    # Exception types (from src/exceptions.py)
|   |           +-- security.py      # JWT auth (from src/security.py)
|   |           +-- embedding_client.py
|   |           +-- vector_store/    # Vector store abstraction
|   |           +-- cache/           # Cache abstraction
|   |           +-- telemetry/       # Telemetry + metrics
|   |           +-- utils/           # Shared utilities
|   |               +-- __init__.py
|   |               +-- agent_tools.py
|   |               +-- clients.py
|   |               +-- filter.py
|   |               +-- formatting.py
|   |               +-- search.py
|   |               +-- shared_models.py
|   |               +-- types.py
|   |               +-- summarizer.py
|   +-- api/                         # honcho-api
|   |   +-- pyproject.toml
|   |   +-- src/
|   |       +-- honcho_api/
|   |           +-- __init__.py
|   |           +-- main.py          # FastAPI app (from src/main.py)
|   |           +-- routers/         # API routes (from src/routers/)
|   |           +-- crud/            # CRUD operations (from src/crud/)
|   |           +-- dialectic/       # Dialectic agent (from src/dialectic/)
|   |           +-- webhooks/        # Webhook system (from src/webhooks/)
|   +-- deriver/                     # honcho-deriver
|   |   +-- pyproject.toml
|   |   +-- src/
|   |       +-- honcho_deriver/
|   |           +-- __init__.py
|   |           +-- __main__.py      # Entry point (from src/deriver/__main__.py)
|   |           +-- consumer.py
|   |           +-- enqueue.py
|   |           +-- queue_manager.py
|   |           +-- agent/
|   +-- dreamer/                     # honcho-dreamer
|   |   +-- pyproject.toml
|   |   +-- src/
|   |       +-- honcho_dreamer/
|   |           +-- __init__.py
|   |           +-- agent.py         # DreamerAgent (from src/dreamer/agent.py)
|   |           +-- dreamer.py       # Legacy dreamer (from src/dreamer/dreamer.py)
|   +-- cli/                         # honcho-cli
|       +-- pyproject.toml
|       +-- src/
|           +-- honcho_cli/
|               +-- __init__.py
|               +-- main.py          # Typer app root
|               +-- commands/
|               |   +-- __init__.py
|               |   +-- init.py      # honcho init
|               |   +-- serve.py     # honcho serve
|               |   +-- deriver.py   # honcho deriver
|               |   +-- up.py        # honcho up
|               |   +-- down.py      # honcho down
|               |   +-- status.py    # honcho status
|               |   +-- doctor.py    # honcho doctor
|               |   +-- nuke.py      # honcho nuke
|               |   +-- mock.py      # honcho mock (placeholder)
|               +-- config.py        # CLI config loading/merging
|               +-- docker.py        # Docker Compose generation + orchestration
|               +-- templates/       # Jinja2 templates for docker-compose.yml, config.toml
|               +-- console.py       # Rich console output helpers
+-- sdks/
|   +-- python/                      # honcho-ai (Python SDK, already a workspace member)
|   +-- typescript/                  # @honcho-ai/sdk (TypeScript SDK, not a uv member)
+-- tests/                           # Integration tests spanning packages

3.2 Root `pyproject.toml` (Workspace Root)

The root pyproject.toml stops being a [project] and becomes purely a workspace coordinator:

[tool.uv.workspace]
members = [
    "packages/shared",
    "packages/api",
    "packages/deriver",
    "packages/dreamer",
    "packages/cli",
    "sdks/python",
]
 
# Workspace-level dev dependencies (linting, testing, etc.)
[dependency-groups]
dev = [
    "pytest>=8.2.2",
    "pytest-asyncio>=0.23.7",
    "pytest-cov>=6.2.1",
    "pytest-xdist>=3.8.0",
    "coverage>=7.6.0",
    "ruff>=0.11.2",
    "basedpyright>=1.29.4",
    "pre-commit>=4.2.0",
    "honcho-ai",
    "fakeredis>=2.32.0",
]
 
[tool.uv.sources]
honcho-shared = { workspace = true }
honcho-api = { workspace = true }
honcho-deriver = { workspace = true }
honcho-dreamer = { workspace = true }
honcho-cli = { workspace = true }
honcho-ai = { workspace = true }
 
[tool.ruff.lint]
select = ["E", "F", "UP", "B", "SIM", "I"]
ignore = ["E501", "B008", "COM812"]
 
[tool.ruff.lint.flake8-bugbear]
extend-immutable-calls = ["fastapi.Depends"]
 
[tool.pytest.ini_options]
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "session"
addopts = "--strict-markers -n auto --ignore=tests/alembic"
testpaths = ["tests"]
filterwarnings = [
    "ignore:Call to deprecated close\\. \\(Use aclose\\(\\) instead\\).*:DeprecationWarning",
]
 
[tool.basedpyright]
include = ["packages/*/src", "tests", "sdks/python/src"]
exclude = ["tests/**/disabled*.py"]
reportMissingTypeStubs = false
reportUnusedCallResult = false
reportAny = false
reportExplicitAny = false
reportImplicitOverride = false
reportImportCycles = false
 
[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

3.3 Package `pyproject.toml` Examples

`packages/shared/pyproject.toml` (`honcho-shared`)

[project]
name = "honcho-shared"
version = "3.1.0"
description = "Honcho shared models, config, and utilities"
requires-python = ">=3.10"
dependencies = [
    "sqlalchemy>=2.0.30",
    "pgvector>=0.2.5",
    "psycopg[binary]>=3.1.19",
    "pydantic>=2.11.7",
    "pydantic-settings>=2.10.1",
    "python-dotenv>=1.0.0",
    "nanoid>=2.0.0",
    "alembic>=1.14.0",
    "pyjwt>=2.10.0",
    "httpx>=0.27.0",
    "rich>=13.7.1",
    "tiktoken>=0.9.0",
    "typing-extensions>=4.11.0",
    "json-repair>=0.49.0",
    "openai>=1.99.7",
    "google-genai>=1.32.0",
    "groq>=0.31.0",
    "tenacity>=9.1.2",
    "turbopuffer>=1.8.1",
    "lancedb>=0.25.3",
    "pyarrow>=19.0.0",
    "redis>=7.0.0,<8.0.0",
    "cashews[redis]==7.4.4",
    "scikit-learn>=1.6.0",
    "prometheus_client>=0.21.0",
    "cloudevents>=1.12.0",
]
 
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
 
[tool.hatch.build.targets.wheel]
packages = ["src/honcho_shared"]

`packages/api/pyproject.toml` (`honcho-api`)

[project]
name = "honcho-api"
version = "3.1.0"
description = "Honcho API server"
requires-python = ">=3.10"
dependencies = [
    "honcho-shared",
    "fastapi[standard]>=0.131.0",
    "fastapi-pagination>=0.14.2",
    "sentry-sdk[anthropic,fastapi,sqlalchemy]>=2.3.1",
    "greenlet>=3.0.3",
    "pdfplumber>=0.11.7",
    "langfuse>=3.3.2",
]
 
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
 
[tool.hatch.build.targets.wheel]
packages = ["src/honcho_api"]
 
[tool.uv.sources]
honcho-shared = { workspace = true }

`packages/deriver/pyproject.toml` (`honcho-deriver`)

[project]
name = "honcho-deriver"
version = "3.1.0"
description = "Honcho background deriver worker"
requires-python = ">=3.10"
dependencies = [
    "honcho-shared",
    "uvloop>=0.21.0",
]
 
[project.scripts]
honcho-deriver = "honcho_deriver.__main__:main"
 
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
 
[tool.hatch.build.targets.wheel]
packages = ["src/honcho_deriver"]
 
[tool.uv.sources]
honcho-shared = { workspace = true }

`packages/dreamer/pyproject.toml` (`honcho-dreamer`)

[project]
name = "honcho-dreamer"
version = "3.1.0"
description = "Honcho memory consolidation dreamer"
requires-python = ">=3.10"
dependencies = [
    "honcho-shared",
]
 
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
 
[tool.hatch.build.targets.wheel]
packages = ["src/honcho_dreamer"]
 
[tool.uv.sources]
honcho-shared = { workspace = true }

`packages/cli/pyproject.toml` (`honcho-cli`)

[project]
name = "honcho-cli"
version = "0.1.0"
description = "Honcho CLI for local development and instance management"
requires-python = ">=3.10"
dependencies = [
    "honcho-shared",
    "honcho-api",
    "honcho-deriver",
    "typer>=0.15.0",
    "rich>=13.7.1",
    "jinja2>=3.1.4",
    "httpx>=0.27.0",
    "tomli-w>=1.0.0",
]
 
[project.scripts]
honcho = "honcho_cli.main:app"
 
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
 
[tool.hatch.build.targets.wheel]
packages = ["src/honcho_cli"]
 
[tool.uv.sources]
honcho-shared = { workspace = true }
honcho-api = { workspace = true }
honcho-deriver = { workspace = true }

3.4 CLI Commands

3.4.1 `honcho init`

Initializes a local Honcho project directory.

# packages/cli/src/honcho_cli/commands/init.py
 
import typer
from pathlib import Path
from honcho_cli.docker import generate_compose_file
from honcho_cli.config import generate_default_config
 
app = typer.Typer()
 
@app.command()
def init(
    directory: Path = typer.Argument(Path("."), help="Project directory"),
    docker: bool = typer.Option(True, help="Generate docker-compose.yml"),
    force: bool = typer.Option(False, "--force", "-f", help="Overwrite existing config"),
):
    """Initialize a new Honcho project directory."""
    honcho_dir = directory / ".honcho"
    honcho_dir.mkdir(parents=True, exist_ok=True)
 
    config_path = honcho_dir / "config.toml"
    if config_path.exists() and not force:
        typer.echo(f"Config already exists at {config_path}. Use --force to overwrite.")
        raise typer.Exit(1)
 
    generate_default_config(config_path)
    typer.echo(f"Created {config_path}")
 
    if docker:
        compose_path = honcho_dir / "docker-compose.yml"
        generate_compose_file(compose_path)
        typer.echo(f"Created {compose_path}")
 
    typer.echo("\nRun 'honcho up' to start the stack.")

Generated .honcho/config.toml:

# Honcho local development configuration
# Values here override ~/.honcho/config.toml (global)
# Environment variables override both.
 
[db]
connection_uri = "postgresql+psycopg://postgres:postgres@localhost:5432/honcho"
schema = "public"
 
[auth]
use_auth = false
 
[deriver]
enabled = true
workers = 1
provider = "google"
model = "gemini-2.5-flash-lite"
 
[cache]
enabled = false
# url = "redis://localhost:6379/0"
 
[llm]
# Set your API keys here or via environment variables
# anthropic_api_key = ""
# openai_api_key = ""
# gemini_api_key = ""

Generated .honcho/docker-compose.yml:

# Generated by `honcho init`. Managed by `honcho up` / `honcho down`.
# Do not edit manually unless you know what you are doing.
 
services:
  database:
    image: pgvector/pgvector:pg15
    restart: unless-stopped
    ports:
      - "${HONCHO_PG_PORT:-5432}:5432"
    environment:
      POSTGRES_DB: honcho
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_HOST_AUTH_METHOD: trust
    volumes:
      - honcho-pgdata:/var/lib/postgresql/data/
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres -d honcho"]
      interval: 5s
      timeout: 5s
      retries: 5
 
  redis:
    image: redis:8.2
    restart: unless-stopped
    ports:
      - "${HONCHO_REDIS_PORT:-6379}:6379"
    volumes:
      - honcho-redis-data:/data
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping"]
      interval: 5s
      timeout: 5s
      retries: 5
 
volumes:
  honcho-pgdata:
  honcho-redis-data:

3.4.2 `honcho up` / `honcho down`

Docker Compose wrappers inspired by supabase start / supabase stop.

# packages/cli/src/honcho_cli/commands/up.py
 
import subprocess
import typer
from honcho_cli.config import load_merged_config, find_project_root
from honcho_cli.docker import get_compose_path, wait_for_healthy
 
app = typer.Typer()
 
@app.command()
def up(
    detach: bool = typer.Option(True, "--detach/--no-detach", "-d", help="Run in background"),
    services: bool = typer.Option(True, "--services/--infra-only", help="Start API+deriver or just DB+Redis"),
):
    """Start the Honcho stack via Docker Compose."""
    compose_path = get_compose_path()
    if compose_path is None:
        typer.echo("No .honcho/docker-compose.yml found. Run 'honcho init' first.")
        raise typer.Exit(1)
 
    cmd = ["docker", "compose", "-f", str(compose_path), "up"]
    if detach:
        cmd.append("-d")
 
    if not services:
        cmd.extend(["database", "redis"])
 
    result = subprocess.run(cmd, check=False)
    if result.returncode != 0:
        typer.echo("Failed to start stack. Run 'honcho doctor' to diagnose.")
        raise typer.Exit(result.returncode)
 
    if detach:
        wait_for_healthy(compose_path)
        typer.echo("Honcho stack is running.")
        typer.echo("  API:      http://localhost:8000")
        typer.echo("  Postgres: localhost:5432")
        typer.echo("  Redis:    localhost:6379")

# packages/cli/src/honcho_cli/commands/down.py
 
import subprocess
import typer
from honcho_cli.docker import get_compose_path
 
app = typer.Typer()
 
@app.command()
def down(
    volumes: bool = typer.Option(False, "--volumes", "-v", help="Remove volumes (destroys data)"),
):
    """Stop the Honcho stack."""
    compose_path = get_compose_path()
    if compose_path is None:
        typer.echo("No .honcho/docker-compose.yml found.")
        raise typer.Exit(1)
 
    cmd = ["docker", "compose", "-f", str(compose_path), "down"]
    if volumes:
        cmd.append("-v")
 
    subprocess.run(cmd, check=True)
    typer.echo("Honcho stack stopped.")

3.4.3 `honcho serve`

Runs the API server directly (no Docker), for development with hot-reload.

# packages/cli/src/honcho_cli/commands/serve.py
 
import typer
import subprocess
import sys
 
app = typer.Typer()
 
@app.command()
def serve(
    host: str = typer.Option("0.0.0.0", help="Bind address"),
    port: int = typer.Option(8000, help="Port"),
    reload: bool = typer.Option(True, "--reload/--no-reload", help="Enable hot reload"),
):
    """Run the Honcho API server locally (outside Docker)."""
    cmd = [
        sys.executable, "-m", "uvicorn",
        "honcho_api.main:app",
        "--host", host,
        "--port", str(port),
    ]
    if reload:
        cmd.append("--reload")
 
    subprocess.run(cmd, check=True)

3.4.4 `honcho deriver`

Runs the deriver worker directly.

# packages/cli/src/honcho_cli/commands/deriver.py
 
import typer
import subprocess
import sys
 
app = typer.Typer()
 
@app.command()
def deriver():
    """Run the Honcho deriver worker locally (outside Docker)."""
    subprocess.run(
        [sys.executable, "-m", "honcho_deriver"],
        check=True,
    )

3.4.5 `honcho doctor`

Validates the local environment.

# packages/cli/src/honcho_cli/commands/doctor.py
 
import shutil
import subprocess
import typer
from rich.console import Console
from rich.table import Table
 
app = typer.Typer()
console = Console()
 
CHECKS = [
    ("Docker", lambda: shutil.which("docker") is not None),
    ("Docker Compose", lambda: _check_compose()),
    ("uv", lambda: shutil.which("uv") is not None),
    ("Python >= 3.10", lambda: _check_python_version()),
    ("Port 5432 available", lambda: _check_port(5432)),
    ("Port 6379 available", lambda: _check_port(6379)),
    ("Port 8000 available", lambda: _check_port(8000)),
    ("PostgreSQL reachable", lambda: _check_pg_connection()),
]
 
@app.command()
def doctor():
    """Check local environment for Honcho development."""
    table = Table(title="Honcho Doctor")
    table.add_column("Check", style="cyan")
    table.add_column("Status")
 
    all_passed = True
    for name, check_fn in CHECKS:
        try:
            passed = check_fn()
        except Exception:
            passed = False
 
        status = "[green]PASS[/green]" if passed else "[red]FAIL[/red]"
        if not passed:
            all_passed = False
        table.add_row(name, status)
 
    console.print(table)
    if not all_passed:
        raise typer.Exit(1)

3.4.6 `honcho status`

# packages/cli/src/honcho_cli/commands/status.py
 
import subprocess
import typer
from honcho_cli.docker import get_compose_path
 
app = typer.Typer()
 
@app.command()
def status():
    """Show the status of running Honcho services."""
    compose_path = get_compose_path()
    if compose_path is None:
        typer.echo("No .honcho/docker-compose.yml found. Run 'honcho init' first.")
        raise typer.Exit(1)
 
    subprocess.run(
        ["docker", "compose", "-f", str(compose_path), "ps", "--format", "table"],
        check=False,
    )

3.4.7 `honcho nuke`

# packages/cli/src/honcho_cli/commands/nuke.py
 
import shutil
import subprocess
import typer
from pathlib import Path
from honcho_cli.docker import get_compose_path
 
app = typer.Typer()
 
@app.command()
def nuke(
    confirm: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation"),
):
    """Destroy all Honcho containers, volumes, and local state."""
    if not confirm:
        confirm = typer.confirm(
            "This will destroy ALL Honcho containers, volumes, and .honcho/ directory. Continue?"
        )
        if not confirm:
            raise typer.Abort()
 
    compose_path = get_compose_path()
    if compose_path is not None:
        subprocess.run(
            ["docker", "compose", "-f", str(compose_path), "down", "-v", "--remove-orphans"],
            check=False,
        )
 
    # Remove local .honcho directory
    honcho_dir = Path(".honcho")
    if honcho_dir.exists():
        shutil.rmtree(honcho_dir)
        typer.echo("Removed .honcho/")
 
    typer.echo("Honcho environment nuked.")

3.4.8 `honcho mock` (Placeholder)

# packages/cli/src/honcho_cli/commands/mock.py
 
import typer
 
app = typer.Typer()
 
@app.command()
def mock():
    """[Placeholder] Start a mock Honcho server for testing."""
    typer.echo("Mock server is not yet implemented. See the mocking-server spec.")
    raise typer.Exit(0)

3.4.9 Main CLI Entry Point

# packages/cli/src/honcho_cli/main.py
 
import typer
from honcho_cli.commands import init, serve, deriver, up, down, status, doctor, nuke, mock
 
app = typer.Typer(
    name="honcho",
    help="Honcho CLI - local development and instance management",
    no_args_is_help=True,
)
 
app.command(name="init")(init.init)
app.command(name="serve")(serve.serve)
app.command(name="deriver")(deriver.deriver)
app.command(name="up")(up.up)
app.command(name="down")(down.down)
app.command(name="status")(status.status)
app.command(name="doctor")(doctor.doctor)
app.command(name="nuke")(nuke.nuke)
app.command(name="mock")(mock.mock)
 
if __name__ == "__main__":
    app()

3.5 Configuration Layering

Configuration is resolved in order of increasing precedence:

Built-in defaults (hardcoded in honcho_shared.config.AppSettings)
Global config (~/.honcho/config.toml) — user-level defaults across all projects
Project config (.honcho/config.toml) — project-specific overrides
Environment variables (DB_CONNECTION_URI, DERIVER_PROVIDER, etc.)
CLI flags (--port, --host, etc.)

The CLI config module merges these layers before constructing the AppSettings singleton:

# packages/cli/src/honcho_cli/config.py
 
import tomllib
from pathlib import Path
from typing import Any
 
GLOBAL_CONFIG_PATH = Path.home() / ".honcho" / "config.toml"
LOCAL_CONFIG_DIR = ".honcho"
LOCAL_CONFIG_FILE = "config.toml"
 
 
def find_project_root() -> Path | None:
    """Walk up from cwd to find a directory containing .honcho/."""
    current = Path.cwd()
    while current != current.parent:
        if (current / LOCAL_CONFIG_DIR / LOCAL_CONFIG_FILE).exists():
            return current
        current = current.parent
    return None
 
 
def load_toml(path: Path) -> dict[str, Any]:
    """Load a TOML file, returning empty dict if not found."""
    if not path.exists():
        return {}
    with open(path, "rb") as f:
        return tomllib.load(f)
 
 
def load_merged_config() -> dict[str, Any]:
    """Load and merge configuration from all layers."""
    config: dict[str, Any] = {}
 
    # Layer 1: Global config
    global_config = load_toml(GLOBAL_CONFIG_PATH)
    _deep_merge(config, global_config)
 
    # Layer 2: Project-local config
    project_root = find_project_root()
    if project_root:
        local_config = load_toml(project_root / LOCAL_CONFIG_DIR / LOCAL_CONFIG_FILE)
        _deep_merge(config, local_config)
 
    return config
 
 
def _deep_merge(base: dict, override: dict) -> dict:
    """Recursively merge override into base."""
    for key, value in override.items():
        if key in base and isinstance(base[key], dict) and isinstance(value, dict):
            _deep_merge(base[key], value)
        else:
            base[key] = value
    return base

3.6 Docker Integration

The docker.py module handles Docker Compose file generation and lifecycle management:

# packages/cli/src/honcho_cli/docker.py
 
import subprocess
import time
from pathlib import Path
from jinja2 import Environment, PackageLoader
 
COMPOSE_TEMPLATE = "docker-compose.yml.j2"
 
def get_compose_path() -> Path | None:
    """Find the nearest .honcho/docker-compose.yml."""
    current = Path.cwd()
    while current != current.parent:
        candidate = current / ".honcho" / "docker-compose.yml"
        if candidate.exists():
            return candidate
        current = current.parent
    return None
 
 
def generate_compose_file(output_path: Path, context: dict | None = None) -> None:
    """Generate a docker-compose.yml from the template."""
    env = Environment(loader=PackageLoader("honcho_cli", "templates"))
    template = env.get_template(COMPOSE_TEMPLATE)
    content = template.render(context or {})
    output_path.parent.mkdir(parents=True, exist_ok=True)
    output_path.write_text(content)
 
 
def wait_for_healthy(compose_path: Path, timeout: int = 60) -> bool:
    """Wait for all services to be healthy."""
    start = time.time()
    while time.time() - start < timeout:
        result = subprocess.run(
            ["docker", "compose", "-f", str(compose_path), "ps", "--format", "json"],
            capture_output=True,
            text=True,
        )
        if result.returncode == 0 and "unhealthy" not in result.stdout:
            return True
        time.sleep(2)
    return False

3.7 Import Path Migration

All internal imports change from src. prefix to package-specific imports:

Before (monolith)	After (workspace)
`from src.models import Message`	`from honcho_shared.models import Message`
`from src.config import settings`	`from honcho_shared.config import settings`
`from src.db import SessionLocal`	`from honcho_shared.db import SessionLocal`
`from src.crud.message import create_messages`	`from honcho_api.crud.message import create_messages`
`from src.deriver.consumer import ...`	`from honcho_deriver.consumer import ...`
`from src.dreamer.agent import ...`	`from honcho_dreamer.agent import ...`
`from src.routers.messages import router`	`from honcho_api.routers.messages import router`
`from src.dialectic.chat import ...`	`from honcho_api.dialectic.chat import ...`
`from src.utils.agent_tools import ...`	`from honcho_shared.utils.agent_tools import ...`

3.8 Alembic Migration Path

Alembic remains at the repository root. The migrations/env.py import path changes:

# migrations/env.py (updated)
from honcho_shared.config import settings
from honcho_shared.db import Base
import honcho_shared.models  # noqa: F401 -- register models

The alembic.ini stays at the repo root. The CLI wraps migration commands:

honcho db migrate     # alembic upgrade head
honcho db revision    # alembic revision --autogenerate
honcho db downgrade   # alembic downgrade -1
honcho db history     # alembic history

These are convenience wrappers; direct alembic usage remains supported.

4. Migration / Packaging Plan

4.1 Backward Compatibility

During the transition, a compatibility shim ensures existing src. imports continue to work:

# src/__init__.py (compatibility shim, temporary)
"""
Backward-compatibility shim. All code has moved to honcho_shared, honcho_api,
honcho_deriver, and honcho_dreamer packages. This shim will be removed in v4.0.
"""
import warnings
import sys
 
warnings.warn(
    "Importing from 'src' is deprecated. Use 'honcho_shared', 'honcho_api', "
    "'honcho_deriver', or 'honcho_dreamer' instead.",
    DeprecationWarning,
    stacklevel=2,
)
 
# Re-export from new locations for backward compat
import honcho_shared as _shared
sys.modules["src.models"] = _shared.models
sys.modules["src.config"] = _shared.config
sys.modules["src.db"] = _shared.db
# ... etc

4.2 Dockerfile Update

The Dockerfile must install from workspace packages:

FROM python:3.13-slim-bookworm
COPY --from=ghcr.io/astral-sh/uv:0.9.24 /uv /bin/uv
 
WORKDIR /app
 
ENV UV_COMPILE_BYTECODE=1
ENV UV_LINK_MODE=copy
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
 
# Copy workspace root files
COPY uv.lock pyproject.toml /app/
COPY packages/ /app/packages/
 
# Install dependencies
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --no-group dev
 
ENV PATH="/app/.venv/bin:$PATH"
 
# Copy remaining files
COPY migrations/ /app/migrations/
COPY alembic.ini /app/alembic.ini
COPY config.toml* /app/
 
RUN addgroup --system app && adduser --system --group app
RUN chown -R app:app /app
USER app
 
EXPOSE 8000
 
CMD ["uvicorn", "honcho_api.main:app", "--host", "0.0.0.0"]

4.3 Fly.io Deployment

The fly.toml process commands update:

[processes]
api = "uvicorn honcho_api.main:app --host 0.0.0.0 --port 8000"
deriver = "python -m honcho_deriver"

5. Implementation Phases

Phase 1: Workspace Scaffolding (Week 1-2)

Create packages/ directory structure with empty __init__.py files.
Create pyproject.toml for each package.
Update root pyproject.toml to workspace-only config.
Run uv sync to verify the workspace resolves.
All tests remain in tests/ at the root.

Phase 2: Extract `honcho-shared` (Week 2-3)

Move src/models.py, src/schemas.py, src/config.py, src/db.py, src/dependencies.py, src/exceptions.py, src/security.py, src/embedding_client.py to packages/shared/src/honcho_shared/.
Move src/utils/ to packages/shared/src/honcho_shared/utils/.
Move src/cache/ to packages/shared/src/honcho_shared/cache/.
Move src/telemetry/ to packages/shared/src/honcho_shared/telemetry/.
Move src/vector_store/ (if exists as module) to packages/shared/src/honcho_shared/vector_store/.
Update all imports within shared package to relative or honcho_shared. prefix.
Update migrations/env.py to import from honcho_shared.
Verify: uv run pytest tests/ passes.

Phase 3: Extract `honcho-api` (Week 3-4)

Move src/main.py to packages/api/src/honcho_api/main.py.
Move src/routers/ to packages/api/src/honcho_api/routers/.
Move src/crud/ to packages/api/src/honcho_api/crud/.
Move src/dialectic/ to packages/api/src/honcho_api/dialectic/.
Move src/webhooks/ to packages/api/src/honcho_api/webhooks/.
Update all imports.
Verify: uv run fastapi dev packages/api/src/honcho_api/main.py starts.
Verify: uv run pytest tests/ passes.

Phase 4: Extract `honcho-deriver` and `honcho-dreamer` (Week 4-5)

Move src/deriver/ to packages/deriver/src/honcho_deriver/.
Move src/dreamer/ to packages/dreamer/src/honcho_dreamer/.
Update entry points and imports.
Verify: uv run python -m honcho_deriver starts.
Verify: all tests pass.

Phase 5: Build CLI (Week 5-7)

Implement honcho init with template generation.
Implement honcho up / honcho down with Docker Compose orchestration.
Implement honcho serve and honcho deriver.
Implement honcho doctor, honcho status, honcho nuke.
Add honcho mock placeholder.
Add CLI-specific tests.
Verify: uv tool install --editable packages/cli && honcho init && honcho up works end-to-end.

Phase 6: Cleanup & Documentation (Week 7-8)

Remove src/ directory (or leave as deprecated shim for one release cycle).
Update Dockerfile and fly.toml.
Update CLAUDE.md, CONTRIBUTING.md, README.md.
Update CI workflows for workspace structure.
Tag release.

6. Files to Modify

New Files

File	Purpose
`packages/shared/pyproject.toml`	Shared package definition
`packages/api/pyproject.toml`	API package definition
`packages/deriver/pyproject.toml`	Deriver package definition
`packages/dreamer/pyproject.toml`	Dreamer package definition
`packages/cli/pyproject.toml`	CLI package definition
`packages/cli/src/honcho_cli/main.py`	CLI entry point
`packages/cli/src/honcho_cli/commands/*.py`	CLI command implementations
`packages/cli/src/honcho_cli/config.py`	Config layer merging
`packages/cli/src/honcho_cli/docker.py`	Docker Compose management
`packages/cli/src/honcho_cli/templates/*.j2`	Jinja2 templates for generated files

Modified Files

File	Change
`pyproject.toml` (root)	Convert from `[project]` to workspace-only
`migrations/env.py`	Update imports from `src.` to `honcho_shared.`
`migrations/utils.py`	Update imports from `src.` to `honcho_shared.`
`Dockerfile`	Update for workspace package layout
`fly.toml`	Update process commands
`alembic.ini`	Potentially update script_location if needed
`.github/workflows/*.yml`	Update CI for workspace structure
`CLAUDE.md`	Update project structure documentation
`CONTRIBUTING.md`	Update development guide

Moved Files (not exhaustive)

From	To
`src/models.py`	`packages/shared/src/honcho_shared/models.py`
`src/config.py`	`packages/shared/src/honcho_shared/config.py`
`src/db.py`	`packages/shared/src/honcho_shared/db.py`
`src/main.py`	`packages/api/src/honcho_api/main.py`
`src/routers/*.py`	`packages/api/src/honcho_api/routers/*.py`
`src/crud/*.py`	`packages/api/src/honcho_api/crud/*.py`
`src/deriver/*.py`	`packages/deriver/src/honcho_deriver/*.py`
`src/dreamer/*.py`	`packages/dreamer/src/honcho_dreamer/*.py`
`src/dialectic/*.py`	`packages/api/src/honcho_api/dialectic/*.py`

7. Risk Assessment

Risk	Likelihood	Impact	Mitigation
Import breakage across 50+ files	High	High	Automated refactoring script + comprehensive test suite. Run `ruff` and `basedpyright` after each phase.
Circular imports between packages	Medium	High	`honcho-shared` must have zero imports from `honcho-api`, `honcho-deriver`, or `honcho-dreamer`. Enforce with an import linter rule.
uv workspace resolution issues	Medium	Medium	Test `uv sync` on CI after each phase. Pin uv version in CI.
Docker Compose version compatibility	Low	Medium	Target Compose V2 (docker compose, not docker-compose). Test on Docker Desktop 4.x.
Alembic migration breakage	Medium	High	Run migration tests (`tests/alembic/`) after updating imports. Keep `alembic.ini` at root.
SDK tests break due to import changes	Low	Medium	SDK tests use HTTP calls, not internal imports. Verify the server starts before running SDK test suite.
Performance regression from package indirection	Low	Low	Python import overhead is negligible. No runtime impact.
Existing Fly.io deployments break	Medium	High	Phase the rollout: update Dockerfile and fly.toml together. Test on staging before production.

8. Verification Plan

8.1 Per-Phase Gates

Each phase must pass these gates before proceeding:

uv sync resolves without errors.
uv run ruff check passes across all packages.
uv run basedpyright passes.
uv run pytest tests/ passes (all existing tests).
uv run fastapi dev packages/api/src/honcho_api/main.py starts and serves /openapi.json.
uv run python -m honcho_deriver starts the queue loop (may exit immediately without DB, but no import errors).

8.2 CLI End-to-End Tests

# Test the full CLI lifecycle
honcho doctor                          # Should pass on a dev machine with Docker
honcho init --force /tmp/test-honcho   # Generate project files
honcho up                              # Start stack
honcho status                          # Show running services
curl http://localhost:8000/openapi.json # Verify API is alive
honcho down                            # Stop stack
honcho nuke --yes                      # Clean up

8.3 CI Integration

Add a GitHub Actions workflow:

- name: Workspace resolution
  run: uv sync
 
- name: Lint
  run: uv run ruff check .
 
- name: Type check
  run: uv run basedpyright
 
- name: Unit tests
  run: uv run pytest tests/ -x
 
- name: CLI smoke test
  run: |
    uv tool install --editable packages/cli
    honcho --help
    honcho doctor || true  # Docker may not be available in CI

9. Open Questions

Package naming on PyPI. Should the packages be published to PyPI? If so, honcho-shared, honcho-api, etc. may conflict with existing package names. Alternatives: honcho-core, honcho-server. Do we want pip install honcho-cli to work, or is uv tool install sufficient?
Dreamer as separate package vs. part of deriver. The dreamer is invoked from the deriver queue manager. Should honcho-dreamer be a standalone package, or should it remain inside honcho-deriver? Keeping it separate enables independent deployment (e.g., a dedicated dreamer worker process), but adds a dependency edge.
Config file format. The spec uses TOML (consistent with existing config.toml.example). Should the CLI also support YAML or JSON for the generated config? Recommendation: TOML only, matching Python ecosystem conventions.
honcho up with API in Docker vs. native. The current design generates a compose file with only infra (DB + Redis) by default, and expects honcho serve / honcho deriver to run natively. Should there be an option to include the API and deriver in Docker as well (honcho up --full)? The docker-compose.yml.example already includes both. Recommendation: support --full flag that adds API + deriver containers.
Migration story for existing users. Users who cloned the repo and run uv run fastapi dev src/main.py will need to update their workflows. How long should the src/ compatibility shim persist? Recommendation: one minor release cycle (3.1.x ships workspace, 3.2.x removes shim).
Dialectic placement. The dialectic agent is invoked synchronously from API routes (/peers/{peer_id}/chat). It clearly belongs in honcho-api. However, it shares agent infrastructure (agent_tools.py, clients.py) with the deriver and dreamer. This shared code lives in honcho-shared. Is this the right boundary?
Monorepo vs. polyrepo for SDKs. The TypeScript SDK is not a uv workspace member (it uses Bun). Should it remain in-tree under sdks/typescript/, or should SDKs move to separate repositories? This is out of scope for this spec but affects the directory layout.

Plastic Labs Internal Wiki

Explorer

Recently Updated

Company

Operating Model — Decisions Log

Operating Model

Security & Compliance

ideal-reasoned-representations

cli