Plastic Labs Internal Wiki

Recently Updated

configurable-embeddings

26 min read

SPEC: Embedding Pipeline Configurability

Status: Draft for discussion Author: vineeth (with claude) Date: 2026-05-11 Related backlog clusters: Cluster A (dimensions param), Cluster B (pgvector dim), Cluster C (model + base URL), Cluster D (default-client base_url for the embedding-on-LLM-key path) from HANDOFF.md.

1. Problem Statement

Honcho’s embedding pipeline today is configurable on paper but not in practice. The friction shows up at five places:

  1. src/models.py declares Vector(1536) for both MessageEmbedding.embedding and Document.embedding. SQLAlchemy uses this dim when building queries, so even if the DB column has been ALTERed, query placeholders still claim 1536 and pgvector throws expected 1536 dimensions, not N.
  2. src/config.py:1273 hard-rejects non-1536 dims with pgvector unless VECTOR_STORE.MIGRATED=True. This validation predates a real story for “what dim does the actual column have?”
  3. src/embedding_client.py does not forward dimensions= on the OpenAI path, so configured EMBEDDING_VECTOR_DIMENSIONS is validated against the response but never requested. Any OpenAI-compatible provider whose native size ≠ configured value fails. The Gemini path is correct.
  4. Migrations hardcode Vector(1536) in four places across three files (migrations/versions/a1b2c3d4e5f6_initial_schema.py:366, migrations/versions/917195d9b5e9_add_messageembedding_table.py:31, migrations/versions/119a52b73c60_support_external_embeddings.py:45,53), so fresh deploys at a non-default dim can’t use alembic upgrade head alone.
  5. External vector stores (Turbopuffer, LanceDB) have their own dim enforcement — no startup validation cross-checks them against EMBEDDING.VECTOR_DIMENSIONS.

Empirically: ~14 open PRs and ~6 open issues are downstream of these five gaps, including several near-duplicate one-line patches for the same symptom (Cluster A).

2. Goals

  • Self-hosted operators can bootstrap Honcho at any embedding dim that pgvector/Turbopuffer/LanceDB support, with any OpenAI-compatible or Gemini embedding model, against any base URL.
  • Dimension is machine-enforced. The runtime introspects the physical vector-store schema at startup and crashes if it doesn’t match EMBEDDING.VECTOR_DIMENSIONS.
  • Model identity is an operator-owned contract, not a machine-enforced invariant. Because this spec intentionally avoids new persistent metadata, same-dim model swaps (e.g. text-embedding-3-small@1536 → text-embedding-3-large@1536-truncated) cannot be detected by the runtime. Operators own this — see §9.
  • Bootstrap is a one-time, idempotent script — not an Alembic migration that re-runs on every deploy.
  • The fix collapses ~14 PRs and ~6 issues into one canonical change.

3. Non-Goals

  • Per-workspace embedding configuration. Deployment-wide only. Per-workspace would multiply complexity and is not a requested feature.
  • In-place model/dim migration. Out of scope; see §11. The official answer to “I want to switch from text-embedding-3-small to nomic-embed-text” is: stand up a new deployment, dual-write or backfill out-of-band, cut over. (Distinct from storage-backend migration, e.g. pgvector → Turbopuffer at the same dim/model, which is handled today by src/reconciler/sync_vectors.py gated on VECTOR_STORE.MIGRATED and remains supported.)
  • Auto-detecting the right embedding model. Operators state their intent via env; we validate it matches reality.
  • Rewriting the LLM transport layer. Out of scope; that’s Clusters E (structured output) and M (multi-provider) in HANDOFF.md.
  • Solving the broader “default-client base_url for chat” problem. Cluster D should land as its own small PR. This spec is embedding-only, though the patterns will be parallel.

4. Design Principles

  1. Dimension immutable after bootstrap, model immutable by operator contract. Once the deployment has written one vector at dim N, the schema pins N; the runtime enforces this. The choice of model M at that dim is deployment-wide and operator-owned; the runtime does not track or enforce it.
  2. Fail fast at startup. Detect any mismatch between configured dim and physical schema before serving traffic. Both API and deriver enforce.
  3. Schema introspection is the source of truth for the dim. No metadata table. Each store’s native schema is queried at startup. Tradeoff: a silent model swap at the same dim is undetectable — see §9.
  4. One-time bootstrap script, not Alembic. Alembic continues to produce a default vector(1536) schema. A separate configure_embeddings script ALTERs the still-empty vector columns to the configured dim; it is safe to re-run and never touches external-store namespaces (which are per-workspace lazy-created — see §5.4).
  5. Backwards compatible by default. A deployment that already runs at 1536 with text-embedding-3-small and never sets the new env vars must continue to work with zero operator action.

5. Architecture

5.1 Config surface

Today (src/config.py — verified):

  • EmbeddingSettings at src/config.py:667-688 uses the same nested model-config shape as the rest of the LLM settings (env_prefix="EMBEDDING_", env_nested_delimiter="__"). Concrete env surface:
    • EMBEDDING_VECTOR_DIMENSIONS (top-level, default 1536)
    • EMBEDDING_MAX_INPUT_TOKENS, EMBEDDING_MAX_TOKENS_PER_REQUEST (top-level)
    • EMBEDDING_MODEL_CONFIG__TRANSPORT (e.g. openai, gemini)
    • EMBEDDING_MODEL_CONFIG__MODEL (e.g. text-embedding-3-small, nomic-embed-text)
    • EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL
    • EMBEDDING_MODEL_CONFIG__OVERRIDES__API_KEY / EMBEDDING_MODEL_CONFIG__OVERRIDES__API_KEY_ENV
    • Resolution path: resolve_embedding_model_config() at src/config.py:438-457.
  • VectorStoreSettings (env prefix VECTOR_STORE_, src/config.py:1165-1190): exposes TYPE, MIGRATED, NAMESPACE, DIMENSIONS.
  • Cross-validator at src/config.py:1278 rejects non-1536 + pgvector unless MIGRATED=True.

Proposed changes:

  • One new field on the embedding model config, folded into the existing nested pattern: EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE: Literal["auto", "always", "never"] = "auto" (see §5.3). No flat env additions; everything else we need is already there.
  • Keep VECTOR_STORE.MIGRATED entirely. It is a runtime dual-write feature flag — referenced in 9 src sites (src/crud/message.py:298,818, src/crud/document.py:200,469,813, src/reconciler/sync_vectors.py:201,323, src/utils/search.py:30, src/config.py:1278) — used to upgrade legacy tenants from one storage backend to another with zero downtime via src/reconciler/sync_vectors.py. This spec does not touch the migration path. All 9 gated branches and reconciler semantics remain as-is. MIGRATED=False continues to mean “dual-write to pg + external during reconciliation”; MIGRATED=True means “external-only, reconciliation complete.”
  • Narrow the validator at src/config.py:1278 rather than remove it. Today it forbids non-1536 EMBEDDING.VECTOR_DIMENSIONS whenever TYPE=pgvector or MIGRATED=False. That clause blocks fresh deploys at non-1536 even when the operator has bootstrapped pgvector at the target dim. Replace it with the runtime schema validator (§5.5), which introspects actual column dim and is more accurate than an operator-asserted flag. The dim/MIGRATED coupling goes away; MIGRATED itself does not.
  • Consolidate VECTOR_STORE.DIMENSIONS into EMBEDDING.VECTOR_DIMENSIONS. The VECTOR_STORE.DIMENSIONS field (src/config.py:1182) has zero functional consumers — verified by grep, the only references are its own definition and the auto-populate/cross-check at src/config.py:1267-1270. Treat EMBEDDING.VECTOR_DIMENSIONS as the single source of truth. Correct deprecation mechanism (an earlier draft incorrectly invoked extra="ignore" here — that setting only swallows unknown fields, but DIMENSIONS is a declared field on VectorStoreSettings:1182, so it is always parsed regardless):
    1. Drop the raise ValueError(...) at src/config.py:1268-1270.
    2. In propagate_namespace (src/config.py:1260), check "DIMENSIONS" in self.VECTOR_STORE.model_fields_set. If present → log via logger.warning("VECTOR_STORE_DIMENSIONS is deprecated; EMBEDDING_VECTOR_DIMENSIONS is authoritative. Drop it from your .env.") for operator visibility at startup. Do not rely on warnings.warn(DeprecationWarning) alone — Python filters DeprecationWarning by default outside __main__ and tests, so operators would never see it. Optionally also call warnings.warn(..., DeprecationWarning, stacklevel=2) so pytest can assert the deprecation in unit tests. Always overwrite self.VECTOR_STORE.DIMENSIONS = self.EMBEDDING.VECTOR_DIMENSIONS regardless of whether the operator set it.
    3. The field stays in the model for one release as a vestigial int holding the resolved value. Drop the field and the commented .env.template:282 line in the release after.
  • Leftover VECTOR_STORE_MIGRATED=true in operator .env files is preserved — this remains a live runtime flag.
  • Document EMBEDDING.VECTOR_DIMENSIONS and EMBEDDING.MODEL as bootstrap-time-only for fresh tenants and deployments. Existing tenants migrate via the MIGRATED + reconciler path; that flow is out of scope here.

5.2 The Vector(1536) problem in models.py

Vector(N) in a SQLAlchemy column is read at import time. We need N to come from settings, not a literal.

# src/models.py — proposed
from src.config import settings
 
_VECTOR_DIM = settings.EMBEDDING.VECTOR_DIMENSIONS
 
class MessageEmbedding(Base):
    embedding: MappedColumn[Any] = mapped_column(Vector(_VECTOR_DIM), nullable=True)
 
class Document(Base):
    embedding: MappedColumn[Any] = mapped_column(Vector(_VECTOR_DIM), nullable=True)

This is essentially PR #593’s approach, made authoritative. Current hardcodes are at src/models.py:281, 389. The settings import is safe — verified that src/config.py imports only stdlib + pydantic with no src.* imports, so adding from src.config import settings to src/models.py cannot trigger a circular import.

5.3 OpenAI dimensions= forwarding

src/embedding_client.py already stores self.vector_dimensions (constructor at src/embedding_client.py:40, response validator at :79). The OpenAI path’s three embeddings.create(...) call sites at src/embedding_client.py:104, 141, 290 must conditionally forward dimensions=self.vector_dimensions. The Gemini path already passes config={"output_dimensionality": self.vector_dimensions} at the parallel sites src/embedding_client.py:98, 132, 277.

Behavior controlled by EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE (new field on ConfiguredEmbeddingModelSettings):

  • auto (default, safe): forward dimensions= on the OpenAI path only if the operator explicitly set EMBEDDING_VECTOR_DIMENSIONS (provenance: "VECTOR_DIMENSIONS" in EmbeddingSettings.model_fields_set — pydantic exposes this for free) and the configured model is not in a small known-rejecting allowlist (currently text-embedding-ada-002). Deployments on the default 1536 keep their existing behavior unchanged; deployments that opted into a non-default dim get the parameter forwarded automatically.
  • always: always forward. For OpenAI-compatible self-hosted providers that require it (e.g. some Ollama/TEI configs that ignore dimensions= but tolerate it), and for the same-as-default truncation case below.
  • never: never forward. Explicit opt-out for providers that reject the parameter.

Implementation note (provenance resolution at the settings layer). The auto-mode decision depends on whether EMBEDDING_VECTOR_DIMENSIONS was explicitly set, which _EmbeddingClient does not currently know — its constructor only receives vector_dimensions: int (src/embedding_client.py:40). The current resolve_embedding_model_config() signature (src/config.py:438) takes only ConfiguredEmbeddingModelSettings and has no access to the parent EmbeddingSettings.model_fields_set — do not reach back into the global settings singleton from inside the resolver, that breaks its purity and makes it untestable. Two acceptable shapes:

(a) Preferred — method on EmbeddingSettings. Add EmbeddingSettings.resolve_send_dimensions() -> bool that reads its own self.MODEL_CONFIG.dimensions_mode, "VECTOR_DIMENSIONS" in self.model_fields_set, and self.MODEL_CONFIG.model against the known-rejecting allowlist. The settings instance is the natural owner of “did the operator set this?” and the resolver doesn’t change.

(b) Alternative — change the resolver signature. resolve_embedding_model_config(configured, vector_dimensions_explicit: bool) -> EmbeddingModelConfig. The caller (which holds the full EmbeddingSettings) computes the bool and passes it in. Keeps the resolver pure but ripples through every call site.

Either way, the resolved send_dimensions: bool is passed explicitly into _EmbeddingClient at construction time. The client itself should never inspect mode or provenance.

Known edge case — same-as-default truncation. An operator running text-embedding-3-large with truncation to 1536, but leaving EMBEDDING_VECTOR_DIMENSIONS at its default, will fail loudly at the response-dim validator (src/embedding_client.py:79) — the API returns native 3072. The failure is loud and at startup-of-first-call, not silent corruption. Mitigation paths: (a) explicitly set EMBEDDING_VECTOR_DIMENSIONS=1536 (provenance-aware auto will then forward dimensions=), or (b) set EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE=always. Document both in docs/v3/operations/changing-embeddings.mdx.

Acceptable to land Phase 0 independently of the rest of this spec — it’s the smallest piece — provided the field is wired through the resolution path at resolve_embedding_model_config() and persisted by the existing ConfiguredEmbeddingModelSettings machinery.

5.4 The bootstrap script

Entry point: src/scripts/configure_embeddings.py, invoked as uv run python -m src.scripts.configure_embeddings. We do not add a honcho console script — the package itself is already named honcho (pyproject.toml:2) and there is no existing top-level honcho CLI entry point, so claiming uv run honcho ... would conflict with the package name and create ambiguity with the honcho-cli workspace member. If we later want a shorter invocation we can add an explicit subcommand to honcho-cli, but the canonical path stays the -m form.

Invocation:

uv run python -m src.scripts.configure_embeddings              # interactive: print diff, ask to confirm
uv run python -m src.scripts.configure_embeddings --dry-run    # print intended changes, exit 0
uv run python -m src.scripts.configure_embeddings --yes        # apply without prompt
uv run python -m src.scripts.configure_embeddings --report     # print full external-store namespace inventory (see §5.5)

Install ordering (this is the canonical bootstrap sequence, not “step 1 of install”):

  1. Provision DB and run alembic upgrade head — this creates the default vector(1536) schema (migrations/versions/a1b2c3d4e5f6_initial_schema.py:366, etc.).
  2. Run uv run python -m src.scripts.configure_embeddings to ALTER the still-empty vector columns to the configured EMBEDDING_VECTOR_DIMENSIONS.
  3. Start API (src/main.py) and deriver (src/deriver/__main__.py); both run the §5.5 validator on boot and refuse to serve traffic on mismatch.

Operators on default 1536 can skip step 2 — the validator passes with no action.

Operations, in order, all idempotent:

  1. Read settings.EMBEDDING.VECTOR_DIMENSIONS, settings.VECTOR_STORE.TYPE, settings.VECTOR_STORE.NAMESPACE.

  2. pgvector phase (always, since pgvector is the application DB regardless of VECTOR_STORE.TYPE):

    • Schema-qualified introspection (see §5.5): query pg_attribute joined through pg_class/pg_namespace for the current dim of {schema}.documents.embedding and {schema}.message_embeddings.embedding, where {schema} = settings.DB.SCHEMA (src/config.py:603).
    • If both already equal target dim: log “pgvector: already at dim N, skipping ALTER” and continue to step 3 (external store validation/report). Do not exit early — --report against an already-configured deployment still needs to walk external namespaces, and the operator may have run the script specifically for that.
    • Otherwise, in a single transaction (TOCTOU-safe sequencing — ALTER ... USING NULL would silently wipe embeddings, so the count gate must run inside the lock window):
      1. LOCK TABLE {schema}.documents, {schema}.message_embeddings IN ACCESS EXCLUSIVE MODE — prevents a worker from writing between the count and the ALTER.
      2. SELECT COUNT(*) FROM {schema}.documents WHERE embedding IS NOT NULL; and the same for message_embeddings. Check is “non-null embeddings,” not “any rows” — documents legitimately holds rows with embedding IS NULL.
      3. If either count > 0: roll back and exit non-zero with "you have {N1} non-null document embeddings and {N2} non-null message embeddings at dim M; this script only configures empty tables. Re-embed out-of-band, then re-run." Never destructive.
      4. Drop HNSW indices on the embedding columns; ALTER TABLE {schema}.{table} ALTER COLUMN embedding TYPE vector(N) USING NULL (safe now that the count gate held inside the lock); rebuild HNSW indices; commit.

  3. External store phase (only if VECTOR_STORE.TYPE != "pgvector"): best-effort validation only, no creation. External-store namespaces in this codebase are per-workspace and lazy-created, not deployment-wide. Verified at:

    • src/vector_store/__init__.py:76 — get_vector_namespace(namespace_type, workspace_name, observer?, observed?) hashes the workspace name (and optional peer pair) into the namespace string.
    • src/vector_store/turbopuffer.py:58-60 — _get_namespace is a plain tpuf.namespace(name) accessor; namespaces materialize on first write.
    • src/vector_store/lancedb.py:72-105 — _get_or_create_table lazy-creates per-workspace tables and pulls the dim from settings.EMBEDDING.VECTOR_DIMENSIONS at create time (line ~98).

    There is no canonical deployment-wide namespace to bootstrap. The script therefore validates only:

    • LanceDB: correct-by-construction for fresh deploys. Enumerate any pre-existing on-disk tables under VECTOR_STORE.NAMESPACE; refuse to proceed if any have a mismatching dim.
    • Turbopuffer: enumerate existing namespaces matching the workspace-hash prefix (if any) and report their dims; refuse to proceed on mismatch. On a fresh deploy with no workspaces, this is a no-op.

  4. Print a final summary table: target=N | pgvector=N | external=<per-workspace report, M scanned, K mismatched> | OK.

Idempotency: running on a correctly-configured deployment is a no-op. Running on a populated, mismatched deployment exits non-zero with a clear diagnostic — never silently rewrites data.

Existing 1536 deployments: the script ships as a no-op for them. No operator action required.

5.5 Startup validation

Both api (FastAPI) and deriver workers run the same validator on boot.

# src/startup/validate_embedding_config.py — pseudocode
def validate_embedding_config(settings):
    target_dim = settings.EMBEDDING.VECTOR_DIMENSIONS
    schema = settings.DB.SCHEMA  # respect DB.SCHEMA, src/config.py:603
 
    # Schema-qualified introspection — do not use bare ::regclass.
    # SELECT a.atttypmod FROM pg_attribute a
    #   JOIN pg_class c     ON a.attrelid = c.oid
    #   JOIN pg_namespace n ON c.relnamespace = n.oid
    #  WHERE n.nspname = :schema
    #    AND c.relname IN ('documents', 'message_embeddings')
    #    AND a.attname = 'embedding';
    rows = introspect_pgvector_dims(schema)  # returns dict {table_name: atttypmod}
    expected = {"documents", "message_embeddings"}
    missing = expected - rows.keys()
    if missing:
        raise StartupError(
            f"Required vector columns missing: "
            f"{', '.join(sorted(f'{schema}.{t}.embedding' for t in missing))}. "
            f"Run `alembic upgrade head` first."
        )
    for table in expected:
        atttypmod = rows[table]
        if atttypmod == -1:
            raise StartupError(
                f"{schema}.{table}.embedding has no declared vector dimension "
                f"(unbounded typmod). Run "
                f"`uv run python -m src.scripts.configure_embeddings`."
            )
        # pgvector encodes dim as atttypmod - VARHDRSZ (4 bytes)
        actual = atttypmod - 4
        if actual != target_dim:
            raise StartupError(
                f"{schema}.{table}.embedding dim ({actual}) does not match "
                f"EMBEDDING_VECTOR_DIMENSIONS ({target_dim}). Run "
                f"`uv run python -m src.scripts.configure_embeddings` "
                f"or fix EMBEDDING_VECTOR_DIMENSIONS."
            )
 
    # External stores: per-workspace, lazy-created (see §5.4).
    # Best-effort sampler — *can miss mismatches outside the sample window*.
    # Run `--report` from the script for full enumeration.
    if settings.VECTOR_STORE.TYPE in ("turbopuffer", "lancedb"):
        mismatches = sample_external_namespace_dims(target_dim, limit=10)
        if mismatches:
            raise StartupError(
                f"Existing external-store namespaces have dim != {target_dim}: "
                f"{mismatches}. Run `uv run python -m "
                f"src.scripts.configure_embeddings --report`."
            )

Best-effort vs. hard guarantee. The startup external-store check samples up to limit existing namespaces. For a multi-tenant deployment with thousands of workspaces, sampling keeps boot time bounded but means a mismatch in an unsampled namespace can escape startup. Treat this as best-effort.

The --report mode of the bootstrap script performs full enumeration via the application DB — the canonical, vendor-agnostic path. Each namespace category is derived from a specific source table, not from peer-pair combinatorics:

  1. Message namespaces — one per workspace. SELECT name FROM workspaces; then for each row, get_vector_namespace("message", workspace_name) (src/vector_store/__init__.py:76).
  2. Document namespaces — one per existing (workspace_name, observer, observed) triple. SELECT workspace_name, observer, observed FROM collections; (table at src/models.py:332-333). Do not enumerate the cartesian product of peers — collections are created on demand by the deriver, so only existing rows correspond to namespaces that could exist.
  3. Per-namespace dim check — query each derived namespace by name against the configured store and verify dim.

Policy for missing/present/mismatched namespaces in the --report output and exit code:

  • Missing namespace (derived name does not exist in the external store): OK. The workspace/collection exists in pg but hasn’t yet had a vector written. Recorded as missing in the report; does not fail the exit.
  • Present + matching dim: OK. Recorded as ok.
  • Present + mismatching dim: failure. Recorded as mismatch; --report exits non-zero. This is the actionable case — operator must either reconfigure the namespace out-of-band or change EMBEDDING_VECTOR_DIMENSIONS.

This is O(W + C) round-trips for W workspaces and C collections. It does not depend on a vendor namespace-listing API — Turbopuffer’s SDK enumeration support is not verified across the versions we use, and even where it exists, prefix-listing semantics may not align with our hash-based namespacing. Optional optimization: if the active store’s SDK supports listing/prefix-listing, batch via that path; otherwise fall through to the per-namespace queries. The pg-driven path is the canonical implementation.

Run --report in CI / pre-deploy gates. If the runtime ever needs a hard guarantee at startup, switch the sampler to full enumeration behind STRICT_EMBEDDING_VALIDATION=true and accept the boot-time cost.

Wire this into:

  • src/main.py lifespan / startup event for the API server (after DB pool is up — the external-store sampler needs to read workspace names from pg).
  • src/deriver/__main__.py for the deriver worker (same ordering: DB first, then validator, then embedding client).

Validation runs before the embedding client is instantiated, so a misconfigured deployment never makes an embedding call.

First-write-per-workspace pinning: because Turbopuffer namespaces (and LanceDB tables) are dim-pinned by the first vector written to them, a fresh deploy’s safety reduces to “pgvector dim matches EMBEDDING.VECTOR_DIMENSIONS and the embedding client honors that dim.” Phase 0 (forwarded dimensions=) and Phase 1 (models.py honors settings) plus the pgvector startup check are jointly sufficient — see §9 risk register for the residual.

5.6 External store details

StoreSchema introspectionBootstrap actionBootstrap on populated store
pgvectorSchema-qualified pg_attribute join through pg_class/pg_namespace (respect DB.SCHEMA, src/config.py:603); decode atttypmod - 4 → dim; handle unbounded (atttypmod == -1) and missing-table cases with actionable errorsALTER TABLE {schema}.{table} ALTER COLUMN embedding TYPE vector(N) USING NULLRefuse with clear error
TurbopufferNamespace(...).schema() returns the vector specNo-op — namespaces are per-workspace and lazy-created on first write by application code (src/vector_store/turbopuffer.py:58-60)Refuse if any sampled namespace dim ≠ target; suggest --report for full enumeration
LanceDBOpen table, read schemaNo-op — tables are per-workspace and lazy-created with settings.EMBEDDING.VECTOR_DIMENSIONS (src/vector_store/lancedb.py:72-105); script only validates any pre-existing on-disk tablesRefuse if any sampled table dim ≠ target

For Turbopuffer specifically: namespaces are dim-locked after first write. The runtime safety of the first write per workspace derives from EMBEDDING_VECTOR_DIMENSIONS being correct at the point the embedding client is called — guaranteed by the pgvector startup validator (§5.5), which crashes the process before any external write can occur on a misconfigured deployment. The bootstrap script is never the thing that creates a Turbopuffer or LanceDB namespace.

6. Implementation Phases

Phased so each lands independently and the backlog can drain incrementally.

Phase 0 — Cluster A standalone (1 PR, ~200 lines)

  • Add dimensions_mode: Literal["auto", "always", "never"] = "auto" to ConfiguredEmbeddingModelSettings. Wire it through resolve_embedding_model_config() (src/config.py:438-457).
  • Implement the three-mode behavior on the OpenAI path at src/embedding_client.py:104, 141, 290 (see §5.3).
  • Tests for auto (sends when EMBEDDING_VECTOR_DIMENSIONS is explicit, skips for ada-002, skips on default), always, never.
  • Close #602, #640, #624, #648, #642, #625, #601, #564(partial) when merged.
  • No dependency on the rest of the spec. Can ship today.

Phase 1 — Make models.py honor EMBEDDING.VECTOR_DIMENSIONS (1 PR, ~50 lines)

  • Replace Vector(1536) with Vector(_VECTOR_DIM) from settings at src/models.py:281, 389.
  • Keep the dim-vs-MIGRATED clause at src/config.py:1278-1282 for now. Deleting it here would open a release-gap window where non-1536 pgvector can start without any fail-fast guard — old guard gone, new startup validator (Phase 2) not yet shipped. Phase 2 atomically swaps the old guard for the new validator in the same PR.
  • Add a unit test that imports the module under a non-1536 dim env and asserts the column type. The test sets EMBEDDING_VECTOR_DIMENSIONS=768 plus the temporary escape hatch VECTOR_STORE_TYPE=turbopuffer + VECTOR_STORE_MIGRATED=true to satisfy the still-present guard at :1278. Phase 2 removes the need for those escape-hatch envs.
  • Closes #593, #595 when merged.

Phase 2 — Startup validator + dim-vs-MIGRATED swap + VECTOR_STORE.DIMENSIONS deprecation (1 PR, ~280 lines)

Single-PR atomic swap — adds the new fail-fast guard and removes the old one in the same release, so there is no window where non-1536 pgvector can start unprotected.

  • Introspection helpers per store (pgvector via schema-qualified pg_attribute; external via pg-driven workspace enumeration — see §5.5).
  • Validator hook into API (src/main.py lifespan) and deriver (src/deriver/__main__.py) lifecycle, after DB pool, before embedding client.
  • Fail closed on introspection failure after the retry budget exhausts (§9). Crash with "could not validate embedding schema: {error}" — uncertainty is not a green light.
  • Delete the dim-vs-MIGRATED clause at src/config.py:1278-1282 in the same PR as the new validator. Phase 1’s escape-hatch envs become unnecessary; remove from the test added in Phase 1.
  • Deprecate VECTOR_STORE.DIMENSIONS (src/config.py:1182, 1267-1270): drop the mismatch raise; in propagate_namespace, check "DIMENSIONS" in self.VECTOR_STORE.model_fields_set and warn-and-override. Field stays for one release; drop in a follow-up. Precise mechanism in §5.1.
  • The 9 dual-write branches (src/crud/message.py:298,818, src/crud/document.py:200,469,813, src/reconciler/sync_vectors.py:201,323, src/utils/search.py:30) remain untouched and load-bearing for the legacy-tenant migration path.
  • Tests covering match, mismatch, missing-namespace, “fresh deploy with no workspaces,” “MIGRATED=False dual-write still works at the new dim,” “introspection failure crashes fail-closed,” and “non-1536 + pgvector + MIGRATED=false now starts cleanly when schema matches” cases.

Phase 3 — configure_embeddings script (1 PR, ~400 lines)

  • src/scripts/configure_embeddings.py, invoked as uv run python -m src.scripts.configure_embeddings. No console-script entry in pyproject.toml to avoid collision with the honcho package name (pyproject.toml:2).
  • Dry-run, interactive, --yes, and --report modes (the last enumerates external-store namespaces in full — see §5.5).
  • pgvector ALTER respects DB.SCHEMA (src/config.py:603); no bare ::regclass.
  • Idempotency tests + “refuse-when-populated” tests + “refuse on unbounded typmod” test.
  • Documentation in docs/v3/contributing/configuration.mdx and a short README in src/scripts/.

Phase 4 — Documentation + backlog cleanup

  • Update docs/v3/contributing/configuration.mdx with the bootstrap flow
  • Add docs/v3/operations/changing-embeddings.mdx that explains: “you can’t, here’s the destroy + rebuild path”
  • Close the now-resolved backlog: dup PRs from Cluster A, Cluster B PRs, issues #601, #625, #564, #590, #585, plus #578 partially (model name was already in EmbeddingSettings; what’s left is the docs).

7. Open Questions

These are calls only you can make; I’ve put my read in italics.

  1. What does the script do when the operator’s settings match the schema exactly but the original model is unknowable? My read: print the schema state and the configured state, and emit a strong “we cannot verify which model was originally used — operator confirms” notice. Continue.
  2. Should the script ever attempt the destructive ALTER on a non-empty schema? My read: never. Always error out. Re-embedding belongs to a separate reconciler tool.
  3. Do we need a --force flag for emergency operator overrides? My read: no. Easier to add later than to remove the footgun.
  4. Should Phase 0 (Cluster A fix) gate on the spec, or can it land immediately? My read: ship Phase 0 immediately. It is small, independently useful, and 7 open PRs / 4 issues close behind it.
  5. Does the validator run inside the FastAPI lifespan (and thus block readiness) or as a separate honcho preflight command? My read: lifespan. We want it to be impossible to serve traffic with a misconfigured deployment. K8s readiness probes will hold traffic naturally.
  6. Plastic Labs–specific: what does Turbopuffer namespace creation cost / require for a 4000-tenant deployment? Resolved against code. src/vector_store/__init__.py:76 shows namespaces are per-workspace (workspace name hashed into the namespace string), and src/vector_store/turbopuffer.py:60 confirms they materialize on first write — there is no deployment-wide namespace. Bootstrap script is therefore pgvector-only (§5.4 step 3); external-store dim is implicitly pinned per-workspace at first write; the startup validator samples existing namespaces (§5.5). No upfront creation cost.
  7. Should we add a CLI subcommand honcho preflight that runs the validator without starting the server? My read: yes as a Phase 2 follow-up. Useful for CI and for k8s init containers.

8. Backwards Compatibility

Existing deployment shapeBehavior under this spec
1536 + text-embedding-3-small + pgvector + MIGRATED=FalseNo-op. Settings agree with schema. Validation passes.
1536 + text-embedding-3-small + pgvector + MIGRATED=True (post-Turbopuffer migration)No behavior change. MIGRATED remains a live runtime flag controlling dual-write semantics; only the dim/MIGRATED coupling at src/config.py:1278 is narrowed.
Tenant mid-migration (MIGRATED=False, reconciler running)No behavior change. Dual-write branches at src/crud/{message,document}.py and src/reconciler/sync_vectors.py continue exactly as today.
Operator explicitly sets VECTOR_STORE_DIMENSIONS=N in .envOne-release deprecation: startup warns, EMBEDDING.VECTOR_DIMENSIONS wins. Field removed in a follow-up.
1536 + text-embedding-3-small + TurbopufferNo-op.
Non-1536 via local patch (e.g. PR #590 backport)Validation now passes since schema matches settings. Local patch should be unwound.
Operator changes EMBEDDING_VECTOR_DIMENSIONS post-deploy without running the scriptStartup fails loudly with a schema-qualified, actionable instruction. Preferred over silent corruption.
Operator changes EMBEDDING_MODEL_CONFIG__MODEL to a same-dim different modelUndetected by design. Model identity is an operator-owned contract, not a machine-enforced invariant (§2, §4 principle 3). Documented limitation; see §9.

9. Risk Register

RiskSeverityMitigation
Silent model swap at same dim. Operator changes EMBEDDING_MODEL_CONFIG__MODEL from text-embedding-3-small to text-embedding-3-large@1536-truncated. New writes use the new model; old vectors use the old model; recall quality silently collapses.High — known limitation of the no-metadata constraintCannot machine-enforce under the “pure introspection” decision (see §4 principle 3, §2 goal 3). Document loudly in docs/v3/operations/changing-embeddings.mdx. Treat as an operator-owned contract, not a runtime invariant. Revisit only if it bites in practice — would re-open the metadata-table option.
Script run accidentally on populated DBHighAlways error if rows present, never destructive. Tests covering this case.
Startup validator cannot complete introspection (DB down, permission denied, timeout)HighRetry the introspection query with a small budget (3 retries, ~1s backoff, <5s total). Fail closed after the budget exhausts — crash with "could not validate embedding schema: {underlying error}". Uncertainty is not a green light; aligns with §4 principle 2 (fail fast at startup). K8s readiness probe holds traffic naturally.
Best-effort external-store sampler misses a mismatching namespace. With limit=10 sampling, a wrong-dim namespace in workspace #4001 escapes the boot check.MediumDocumented as best-effort (§5.5). Operators get full enumeration via uv run python -m src.scripts.configure_embeddings --report — recommended as a CI / pre-deploy gate. A future STRICT_EMBEDDING_VALIDATION=true env can switch the runtime to full enumeration with the boot-time cost.
Embedding API call passes dimensions= to a provider that rejects itLowEMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE="never" opt-out; auto default skips ada-002 and skips when EMBEDDING_VECTOR_DIMENSIONS is at the default.
First-write-per-workspace silently pins external namespace dim. Fresh workspace + wrong EMBEDDING_VECTOR_DIMENSIONS would pin the namespace forever at the wrong dim.MediumThe pgvector startup validator (§5.5) crashes before any first-write can occur on a misconfigured deployment, since pgvector schema is set by the bootstrap script from the same EMBEDDING_VECTOR_DIMENSIONS. Phase 0 (dimensions_mode forwarding) and Phase 1 (models.py honors settings) close the remaining “client says N, schema says M” gap.

10. Testing Strategy

  • Unit:
    • models.py Vector dim respects EMBEDDING_VECTOR_DIMENSIONS
    • Embedding client passes dimensions= when EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE="auto" and EMBEDDING_VECTOR_DIMENSIONS is explicitly set
    • Embedding client passes dimensions= when mode is "always"
    • Embedding client omits dimensions= when mode is "never" or when auto + model is on the known-rejecting allowlist (ada-002) or EMBEDDING_VECTOR_DIMENSIONS is at default
    • Schema-qualified pgvector introspection returns correct dim under non-public DB.SCHEMA; errors actionably on unbounded typmod and missing table
    • Validator raises on mismatch, passes on match, raises on missing column with the schema-qualified path included in the error
  • Integration:
    • Spin up Postgres+pgvector container at 1536; verify validator passes
    • ALTER columns to 768; verify validator now errors
    • Run bootstrap script on empty schema at 768; verify it ALTERs and validator passes
    • Run bootstrap script on populated schema; verify it errors without changes
  • Manual smoke (Plastic Labs deployment):
    • Dry-run script against production-like Turbopuffer namespace
    • Confirm script reports “skipping, already at 1536” cleanly

11. Out of Scope (Future Work)

  • In-place embedding model/dim migration. Dual-write + backfill + cutover for changing dims or models post-bootstrap. Likely its own spec. Hard problems: atomicity, cost of re-embedding millions of vectors, dialectic-during-migration semantics. (Storage-backend migration at constant dim/model is already supported by src/reconciler/sync_vectors.py + VECTOR_STORE.MIGRATED; that path is preserved unchanged by this spec.)
  • Per-workspace embedding configuration. Would require schema changes (workspaces would need their own collections / namespaces) and is materially harder.
  • Hybrid embeddings. Some queries embedded with model A, others with model B. Out of scope and not requested.
  • Auto-detection. Probing the embedding endpoint at startup to determine its native dim and offering to align settings. Possibly useful UX but not a safety necessity given the validator.
  • The chat / LLM base_url cluster (Cluster D). Parallel work but separate PR.

12. Success Criteria

This spec is successful if, after implementation:

  1. After alembic upgrade head, running uv run python -m src.scripts.configure_embeddings --dry-run is the documented next step for self-hosted installs at a non-default dim.
  2. Starting Honcho with an EMBEDDING_VECTOR_DIMENSIONS value that doesn’t match the actual pgvector schema produces a clear, actionable, schema-qualified error within 2 seconds of startup, before any HTTP route is served.
  3. The 14 open PRs and 6 open issues in Clusters A, B, C of HANDOFF.md are closed (merged, deduped, or rejected with a pointer here).
  4. A self-hosted operator running nomic-embed-text@768 against Ollama can complete the install without forking the repo or patching code, by setting EMBEDDING_MODEL_CONFIG__TRANSPORT, EMBEDDING_MODEL_CONFIG__MODEL, EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL, and EMBEDDING_VECTOR_DIMENSIONS=768.
  5. An existing Plastic Labs production deployment at 1536 + Turbopuffer requires zero operator action to upgrade past this change; MIGRATED semantics are preserved (see §5.1).

13. Backlog Items Subsumed

PRs to close on merge (with pointer to this spec):

  • Cluster A: #602 is Phase 0 — either merge it directly or land a clean replacement and close #602 as superseded (not both). Close on Phase 0 merge as superseded: #640, #624, #648, #642.
  • Phase 1 closes #593, #595.
  • Cluster B: covered by Phases 1 + 2.
  • Cluster C / D: model name already in EmbeddingSettings post-#535; this spec doesn’t add new env, just enforces and bootstraps.

Issues to close on merge:

  • #601, #625 (Cluster A) — Phase 0
  • #564, #590 (Cluster B) — Phases 1-3
  • #585 (Cluster C — also a user-support issue) — partial; pgvector dim now genuinely configurable

Out of scope for this spec but worth follow-up:

  • #578 — embedding-specific base URL and model name. Settings shape is in place; needs verification + docs only.
  • #554 — covered by #578 follow-up.
  • #608 — broader config-validation silence problem.

14. References

  • HANDOFF.md — triage source
  • PR #602 (canonical Phase 0)
  • PR #593 (canonical Phase 1 shape)
  • Issue #590 (canonical user need for non-1536 pgvector)
  • Issue #601 (canonical Phase 0 motivation)
  • pgvector docs on column dim ALTER and HNSW rebuild costs
  • Turbopuffer docs on namespace dim immutability

Graph View