Plastic Labs Internal Wiki

Recently Updated

conclusion-tagging

19 min read

Conclusion Tagging: Public Metadata & Read-Only Fields

Status: Draft | Owner: vineeth | Last updated: 2026-03-25

1. Problem Statement

Conclusions (internally stored as documents) are the atomic units of Honcho’s memory system. They represent facts, inferences, and observations derived from peer interactions. Today, the documents table has several valuable columns that are either hidden from the API entirely or lack extensibility:

  1. No public metadata field. Every other first-class resource (workspaces, peers, sessions, messages) exposes a user-writable metadata JSONB column. Conclusions do not. This prevents SDK users from attaching custom tags, categories, or application-specific annotations to conclusions.

  2. level, times_derived, and source_ids are invisible. These columns exist on the documents table and carry meaningful information about the provenance and confidence of a conclusion, but the Conclusion API response schema omits them entirely. Users cannot distinguish an explicit observation from a deductive inference, cannot see how many times a conclusion has been reinforced, and cannot traverse the reasoning tree.

  3. No filtering on conclusion metadata. Without a public metadata column, the existing filter system (which already supports JSONB metadata filtering for other resources) cannot be used to query conclusions by user-defined tags.

2. Goals / Non-Goals

Goals

  • G1: Add a public, user-writable metadata JSONB column to the documents table, following the exact same pattern as workspaces, peers, sessions, and messages.
  • G2: Expose level, times_derived, and source_ids as read-only fields on the Conclusion API response schema. These must not be settable via the create or update endpoints.
  • G3: Enable filtering on the new metadata column using the existing filter system (AND/OR/NOT, comparison operators, nested key access).
  • G4: Add a ConclusionUpdate schema and a PUT endpoint so users can update conclusion metadata after creation.
  • G5: Add a GIN index on the metadata column for efficient JSONB containment queries.
  • G6: Update both the Python and TypeScript SDKs to surface the new fields.
  • G7: Deliver this as SDK v2.1 — purely additive, non-breaking.

Non-Goals

  • Exposing or modifying internal_metadata (system-only, remains hidden).
  • Changing the soft-delete behavior.
  • Adding write access to level, times_derived, or source_ids via the public API. These are system-managed.
  • Changing the ConclusionCreate schema to accept metadata (this IS a goal; clarified in G1). However, metadata on create is optional and defaults to {}.
  • Backfilling existing documents with non-empty metadata. Existing rows get '{}'::jsonb from the server default.

3. Design

3.1 Schema Changes (DDL)

3.1.1 New column on documents table

Add a metadata column, mapped in SQLAlchemy as h_metadata (same convention as all other tables to avoid collision with Python’s built-in metadata).

ALTER TABLE {schema}.documents
ADD COLUMN metadata JSONB NOT NULL DEFAULT '{}'::jsonb;

3.1.2 GIN index for JSONB filtering

CREATE INDEX CONCURRENTLY ix_documents_metadata_gin
ON {schema}.documents USING gin (metadata);

3.1.3 SQLAlchemy model change

In src/models.py, add to the Document class:

h_metadata: Mapped[dict[str, Any]] = mapped_column(
    "metadata", JSONB, default=dict, server_default=text("'{}'::jsonb")
)

And add the GIN index to __table_args__:

Index(
    "ix_documents_metadata_gin",
    "metadata",
    postgresql_using="gin",
),

The full Document model field ordering should place h_metadata after id and before internal_metadata, matching the convention of other models. The complete updated __table_args__ tuple:

__table_args__ = (
    CheckConstraint("length(id) = 21", name="id_length"),
    CheckConstraint("length(content) <= 65535", name="content_length"),
    CheckConstraint("id ~ '^[A-Za-z0-9_-]+$'", name="id_format"),
    ForeignKeyConstraint(
        ["observer", "observed", "workspace_name"],
        ["collections.observer", "collections.observed", "collections.workspace_name"],
    ),
    ForeignKeyConstraint(
        ["observer", "workspace_name"],
        ["peers.name", "peers.workspace_name"],
    ),
    ForeignKeyConstraint(
        ["observed", "workspace_name"],
        ["peers.name", "peers.workspace_name"],
    ),
    ForeignKeyConstraint(
        ["session_name", "workspace_name"],
        ["sessions.name", "sessions.workspace_name"],
    ),
    Index(
        "ix_documents_embedding_hnsw",
        "embedding",
        postgresql_using="hnsw",
        postgresql_with={"m": 16, "ef_construction": 64},
        postgresql_ops={"embedding": "vector_cosine_ops"},
    ),
    Index(
        "ix_documents_source_ids_gin",
        "source_ids",
        postgresql_using="gin",
    ),
    Index(
        "ix_documents_metadata_gin",
        "metadata",
        postgresql_using="gin",
    ),
    Index(
        "ix_documents_sync_state_last_sync_at",
        "sync_state",
        "last_sync_at",
    ),
)

3.2 API Surface Changes

3.2.1 Response schema: Conclusion

Current Conclusion response in src/schemas/api.py:

class Conclusion(BaseModel):
    id: str
    content: str
    observer: str = Field(serialization_alias="observer_id")
    observed: str = Field(serialization_alias="observed_id")
    session_name: str | None = Field(default=None, serialization_alias="session_id")
    created_at: datetime.datetime

Updated Conclusion response:

class Conclusion(BaseModel):
    """Conclusion response - external view of a document."""
 
    id: str
    content: str
    observer: str = Field(
        description="The peer who made the conclusion",
        serialization_alias="observer_id",
    )
    observed: str = Field(
        description="The peer the conclusion is about",
        serialization_alias="observed_id",
    )
    session_name: str | None = Field(default=None, serialization_alias="session_id")
    h_metadata: dict[str, Any] = Field(
        default_factory=dict, serialization_alias="metadata"
    )
    level: str = Field(
        default="explicit",
        description="Reasoning level: explicit, deductive, inductive, or contradiction",
    )
    times_derived: int = Field(
        default=1,
        description="Number of times this conclusion has been independently derived",
    )
    source_ids: list[str] | None = Field(
        default=None,
        description="IDs of parent conclusions this was derived from",
    )
    created_at: datetime.datetime
 
    model_config = ConfigDict(
        from_attributes=True,
        populate_by_name=True,
    )

Key design points:

  • h_metadata is mapped from the ORM attribute h_metadata and serialized as "metadata" in JSON output. This follows the exact same pattern as Peer, Session, Message, and Workspace response schemas.
  • level, times_derived, source_ids are populated from the ORM model via from_attributes=True. They are present in the response but never accepted in create/update.

3.2.2 Create schema: ConclusionCreate

Add optional metadata field:

class ConclusionCreate(BaseModel):
    """Schema for creating a single conclusion."""
 
    content: Annotated[str, Field(min_length=1, max_length=65535)]
    observer_id: str = Field(..., description="The peer making the conclusion")
    observed_id: str = Field(..., description="The peer the conclusion is about")
    session_id: str | None = Field(
        default=None,
        description="A session ID to store the conclusion in, if specified",
    )
    metadata: _SanitizedMetadata = {}
 
    _token_count: int = PrivateAttr(default=0)
 
    @field_validator("content", mode="after")
    @classmethod
    def sanitize_content(cls, v: str) -> str:
        return v.replace("\x00", "")
 
    @model_validator(mode="after")
    def validate_token_count(self) -> Self:
        """Validate that content doesn't exceed embedding token limit."""
        encoding = tiktoken.get_encoding("o200k_base")
        tokens = encoding.encode(self.content)
        self._token_count = len(tokens)
 
        if self._token_count > settings.MAX_EMBEDDING_TOKENS:
            raise ValueError(
                f"Content exceeds maximum embedding token limit of {settings.MAX_EMBEDDING_TOKENS} "
                + f"(got {self._token_count} tokens)"
            )
        return self

The metadata field uses _SanitizedMetadata (same Annotated[dict[str, Any], BeforeValidator(_validate_metadata)] type used by all other create schemas), which enforces:

  • Max 100 root-level keys
  • Max nesting depth of 5
  • NUL byte sanitization

Default is {} (empty dict), making it fully backward-compatible.

3.2.3 New update schema: ConclusionUpdate

class ConclusionUpdate(BaseModel):
    """Schema for updating a conclusion's metadata."""
 
    metadata: _SanitizedMetadata | None = None

This follows the exact same pattern as PeerUpdate, SessionUpdate, and MessageUpdate. Only metadata is writable. level, times_derived, source_ids, and content are immutable via the API.

3.2.4 New endpoint: PUT /workspaces/{workspace_id}/conclusions/{conclusion_id}

Add to src/routers/conclusions.py:

@router.put(
    "/{conclusion_id}",
    response_model=schemas.Conclusion,
)
async def update_conclusion(
    workspace_id: str = Path(...),
    conclusion_id: str = Path(...),
    body: schemas.ConclusionUpdate = Body(
        ...,
        description="Conclusion update parameters",
    ),
    db: AsyncSession = db,
) -> schemas.Conclusion:
    """
    Update a Conclusion's metadata.
 
    Only metadata can be updated. Content, level, times_derived, and source_ids
    are system-managed and cannot be modified via this endpoint.
    """
    document = await crud.update_document_metadata(
        db,
        workspace_name=workspace_id,
        document_id=conclusion_id,
        metadata=body.metadata,
    )
    return schemas.Conclusion.model_validate(document)

3.2.5 Filter column mapping update

In src/utils/filter.py, the ALLOWED_EXTERNAL_TO_INTERNAL_COLUMN_MAPPING_DOCUMENTS dict must be updated:

ALLOWED_EXTERNAL_TO_INTERNAL_COLUMN_MAPPING_DOCUMENTS = {
    "session_id": "session_name",
    "workspace_id": "workspace_name",
    "observer_id": "observer",
    "observed_id": "observed",
    "metadata": "h_metadata",         # NEW: public metadata (was "internal_metadata")
    "level": "level",                  # NEW: allow filtering by level
    "times_derived": "times_derived",  # NEW: allow filtering by times_derived
    "created_at": "created_at",        # NEW: allow filtering by created_at
}

Critical change: The "metadata" key currently maps to "internal_metadata". This must change to map to "h_metadata" (the new public metadata column). Internal metadata filtering, if needed by the deriver or other internal systems, should use the internal column name directly (not go through the public filter API).

Impact assessment: The current mapping "metadata": "internal_metadata" is used when users pass {"filters": {"metadata": {...}}} in list/query endpoints. Since the Conclusion response currently does not expose any metadata field, users have no reason to filter on it today. However, if any internal code paths pass metadata filters for documents through apply_filter, those will break. We must audit all callers of apply_filter with model_class=Document and ensure none pass metadata filters expecting internal_metadata.

Callers to audit (from src/crud/document.py):

  • get_all_documents() — receives filters from src/routers/conclusions.py list endpoint
  • get_documents_with_filters() — receives filters from src/routers/conclusions.py list endpoint
  • query_documents() — receives filters from src/routers/conclusions.py query endpoint

None of these are called with metadata filters from internal code. The deriver and dreamer access documents directly via SQL, not through the filter system. This change is safe.

3.3 CRUD Changes

3.3.1 New function: update_document_metadata

Add to src/crud/document.py:

async def update_document_metadata(
    db: AsyncSession,
    workspace_name: str,
    document_id: str,
    *,
    metadata: dict[str, Any] | None = None,
) -> models.Document:
    """
    Update a document's public metadata.
 
    Args:
        db: Database session
        workspace_name: Name of the workspace
        document_id: ID of the document to update
        metadata: New metadata dict (replaces existing metadata entirely)
 
    Returns:
        The updated document
 
    Raises:
        ResourceNotFoundException: If the document is not found
    """
    stmt = select(models.Document).where(
        models.Document.id == document_id,
        models.Document.workspace_name == workspace_name,
        models.Document.deleted_at.is_(None),
    )
    result = await db.execute(stmt)
    document = result.scalar_one_or_none()
 
    if document is None:
        raise ResourceNotFoundException(
            f"Conclusion {document_id} not found in workspace {workspace_name}"
        )
 
    if metadata is not None and document.h_metadata != metadata:
        document.h_metadata = metadata
        await db.commit()
        await db.refresh(document)
    else:
        await db.commit()
 
    return document

3.3.2 Update create_observations to accept public metadata

In src/crud/document.py, the create_observations function constructs models.Document objects. Currently it sets internal_metadata={}. It must also set h_metadata from the new metadata field on ConclusionCreate:

# In create_observations(), inside the for loop:
doc = models.Document(
    workspace_name=workspace_name,
    observer=obs.observer_id,
    observed=obs.observed_id,
    content=obs.content,
    level="explicit",
    times_derived=1,
    internal_metadata={},
    h_metadata=obs.metadata if hasattr(obs, 'metadata') else {},  # NEW
    session_name=obs.session_id,
    embedding=embedding,  # (or omitted depending on store_embeddings_in_postgres)
)

Since create_observations receives Sequence[schemas.ConclusionCreate], and ConclusionCreate now has a metadata field defaulting to {}, this is straightforward. Both the store_embeddings_in_postgres and non-store branches must be updated.

3.3.3 Export new function

In src/crud/__init__.py, add update_document_metadata to the imports and __all__.

3.4 SDK Changes

3.4.1 Python SDK

sdks/python/src/honcho/api_types.py — Update ConclusionResponse:

class ConclusionResponse(BaseModel):
    """Conclusion API response."""
 
    model_config = ConfigDict(populate_by_name=True)
 
    id: str
    content: str
    observer_id: str
    observed_id: str
    session_id: str | None = None
    metadata: dict[str, Any] = Field(default_factory=dict)
    level: str = Field(default="explicit")
    times_derived: int = Field(default=1)
    source_ids: list[str] | None = None
    created_at: datetime.datetime

Add ConclusionUpdateParams:

class ConclusionUpdateParams(BaseModel):
    """Parameters for updating a conclusion."""
 
    metadata: dict[str, Any] | None = None

Update ConclusionCreateParams (in api_types.py):

class ConclusionCreateParams(BaseModel):
    """Parameters for creating a conclusion."""
 
    content: str = Field(min_length=1, max_length=65535)
    observer_id: str
    observed_id: str
    session_id: str | None = None
    metadata: dict[str, Any] | None = None  # NEW

sdks/python/src/honcho/conclusions.py — Update Conclusion class:

class Conclusion:
    id: str
    content: str
    observer_id: str
    observed_id: str
    session_id: str | None
    metadata: dict[str, Any]
    level: str
    times_derived: int
    source_ids: list[str] | None
    created_at: datetime.datetime
 
    def __init__(
        self,
        id: str,
        content: str,
        observer_id: str,
        observed_id: str,
        session_id: str | None,
        metadata: dict[str, Any],
        level: str,
        times_derived: int,
        source_ids: list[str] | None,
        created_at: datetime.datetime,
    ) -> None:
        self.id = id
        self.content = content
        self.observer_id = observer_id
        self.observed_id = observed_id
        self.session_id = session_id
        self.metadata = metadata
        self.level = level
        self.times_derived = times_derived
        self.source_ids = source_ids
        self.created_at = created_at
 
    @classmethod
    def from_api_response(cls, data: ConclusionResponse) -> "Conclusion":
        return cls(
            id=data.id,
            content=data.content,
            observer_id=data.observer_id,
            observed_id=data.observed_id,
            session_id=data.session_id,
            metadata=data.metadata,
            level=data.level,
            times_derived=data.times_derived,
            source_ids=data.source_ids,
            created_at=data.created_at,
        )

ConclusionCreateParams (in conclusions.py):

class ConclusionCreateParams(BaseModel):
    content: str
    session_id: str | None = None
    metadata: dict[str, Any] | None = None  # NEW

Add update() method to ConclusionScope:

def update(
    self,
    conclusion_id: str,
    metadata: dict[str, Any] | None = None,
) -> Conclusion:
    """
    Update a conclusion's metadata.
 
    Args:
        conclusion_id: The ID of the conclusion to update
        metadata: New metadata to set (replaces existing)
 
    Returns:
        Updated Conclusion object
    """
    self._honcho._ensure_workspace()
    body: dict[str, Any] = {}
    if metadata is not None:
        body["metadata"] = metadata
 
    data = self._honcho._http.put(
        routes.conclusion(self.workspace_id, conclusion_id),
        body=body,
    )
    return Conclusion.from_api_response(ConclusionResponse.model_validate(data))

Update build_conclusion_payload in the create() method to pass metadata:

def build_conclusion_payload(
    item: ConclusionCreateParams | dict[str, Any],
) -> dict[str, Any]:
    payload: dict[str, Any] = {
        "observer_id": self.observer,
        "observed_id": self.observed,
    }
    if isinstance(item, ConclusionCreateParams):
        payload["content"] = item.content
        if item.session_id is not None:
            payload["session_id"] = item.session_id
        if item.metadata is not None:
            payload["metadata"] = item.metadata
        return payload
 
    payload["content"] = item["content"]
    session_id = item.get("session_id")
    if session_id is not None:
        payload["session_id"] = session_id
    metadata = item.get("metadata")
    if metadata is not None:
        payload["metadata"] = metadata
    return payload

Add route for PUT conclusion to sdks/python/src/honcho/http/routes.py (it likely already has conclusion() for DELETE; verify it works for PUT too since it’s the same path).

3.4.2 TypeScript SDK

sdks/typescript/src/types/api.ts — Update ConclusionResponse:

export interface ConclusionResponse {
  id: string
  content: string
  observer_id: string
  observed_id: string
  session_id: string | null
  metadata: Record<string, unknown>
  level: string
  times_derived: number
  source_ids: string[] | null
  created_at: string
}

Add ConclusionUpdateParams:

export interface ConclusionUpdateParams {
  metadata?: Record<string, unknown>
}

sdks/typescript/src/conclusions.ts — Update Conclusion class:

export class Conclusion {
  readonly id: string
  readonly content: string
  readonly observerId: string
  readonly observedId: string
  readonly sessionId: string | null
  readonly metadata: Record<string, unknown>
  readonly level: string
  readonly timesDerived: number
  readonly sourceIds: string[] | null
  readonly createdAt: string
 
  constructor(
    id: string,
    content: string,
    observerId: string,
    observedId: string,
    sessionId: string | null,
    metadata: Record<string, unknown>,
    level: string,
    timesDerived: number,
    sourceIds: string[] | null,
    createdAt: string
  ) {
    this.id = id
    this.content = content
    this.observerId = observerId
    this.observedId = observedId
    this.sessionId = sessionId
    this.metadata = metadata
    this.level = level
    this.timesDerived = timesDerived
    this.sourceIds = sourceIds
    this.createdAt = createdAt
  }
 
  static fromApiResponse(data: ConclusionResponse): Conclusion {
    return new Conclusion(
      data.id,
      data.content,
      data.observer_id,
      data.observed_id,
      data.session_id,
      data.metadata ?? {},
      data.level ?? 'explicit',
      data.times_derived ?? 1,
      data.source_ids ?? null,
      data.created_at
    )
  }
}

Add update() method to ConclusionScope and update create() to pass metadata.

3.5 Filter Integration

After the column mapping change in Section 3.2.5, the following filter queries will work on the /conclusions/list endpoint:

// Filter by custom metadata tag
{"filters": {"metadata": {"category": "preference"}}}
 
// Filter by level
{"filters": {"level": "deductive"}}
 
// Filter by times_derived (most reinforced)
{"filters": {"times_derived": {"gte": 3}}}
 
// Compound filter
{"filters": {
  "AND": [
    {"observer_id": "alice"},
    {"level": {"in": ["deductive", "inductive"]}},
    {"metadata": {"importance": "high"}}
  ]
}}
 
// Filter by created_at
{"filters": {"created_at": {"gte": "2026-01-01"}}}

The level and times_derived columns are native SQL columns (not JSONB), so comparison operators (gte, lte, in, etc.) work directly via _build_comparison_conditions. The metadata column is JSONB and uses _build_nested_metadata_conditions for nested key access and contains for simple object matching, exactly like h_metadata on other tables.

4. Migration Plan

4.1 Migration file

File: migrations/versions/<new_hash>_add_public_metadata_to_documents.py

Revises: e4eba9cfaa6f (current HEAD — make_document_session_name_nullable)

"""add_public_metadata_to_documents
 
Add a public metadata JSONB column to the documents table with a GIN index,
following the same pattern as workspaces, peers, sessions, and messages.
 
Revision ID: <generated>
Revises: e4eba9cfaa6f
Create Date: 2026-XX-XX
 
"""
 
from collections.abc import Sequence
 
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import JSONB
 
from migrations.utils import column_exists, get_schema, index_exists
 
# revision identifiers, used by Alembic.
revision: str = "<generated>"
down_revision: str | None = "e4eba9cfaa6f"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
 
schema = get_schema()
 
 
def upgrade() -> None:
    """Add public metadata column with GIN index to documents table."""
    connection = op.get_bind()
    inspector = sa.inspect(connection)
 
    # Step 1: Add metadata column (non-nullable with server default)
    # This is instant on PostgreSQL since it has a DEFAULT value.
    if not column_exists("documents", "metadata", inspector):
        op.add_column(
            "documents",
            sa.Column(
                "metadata",
                JSONB,
                nullable=False,
                server_default=sa.text("'{}'::jsonb"),
            ),
            schema=schema,
        )
 
    # Step 2: Add GIN index for efficient JSONB filtering
    # Use CONCURRENTLY to avoid blocking writes on large tables.
    # Note: op.create_index does not support CONCURRENTLY directly.
    # We use raw SQL for this.
    if not index_exists("documents", "ix_documents_metadata_gin", inspector):
        # Cannot use CREATE INDEX CONCURRENTLY inside a transaction.
        # Alembic runs migrations inside a transaction by default.
        # We must end the transaction first.
        op.execute("COMMIT")
        connection.execute(
            sa.text(
                f"""
                CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_documents_metadata_gin
                ON {schema}.documents USING gin (metadata)
                """
            )
        )
        # Re-open a transaction for any subsequent operations
        op.execute("BEGIN")
 
 
def downgrade() -> None:
    """Remove public metadata column and its GIN index."""
    connection = op.get_bind()
    inspector = sa.inspect(connection)
 
    # Drop GIN index first
    if index_exists("documents", "ix_documents_metadata_gin", inspector):
        op.drop_index(
            "ix_documents_metadata_gin",
            table_name="documents",
            schema=schema,
        )
 
    # Drop column
    if column_exists("documents", "metadata", inspector):
        op.drop_column("documents", "metadata", schema=schema)

4.2 Backfill strategy

No data backfill is needed. The server_default=sa.text("'{}'::jsonb") ensures:

  • All existing rows get '{}'::jsonb as their metadata value.
  • PostgreSQL applies this default without rewriting the table (instant ADD COLUMN for NOT NULL + DEFAULT on PG 11+).
  • New rows without explicit metadata also get {}.

4.3 Deployment order

  1. Deploy migration — adds column and index. The application code can still run without changes since the new column has a default and is not read yet.
  2. Deploy application code — API starts returning and accepting metadata, level, times_derived, source_ids.
  3. Deploy SDK updates — new SDK versions read the new fields.

Steps 1 and 2 can be combined into a single deployment since the migration is backward-compatible (new column with default, new index).

5. Implementation Phases

Phase 1: Database (1 PR)

  • Add h_metadata mapped column to Document model in src/models.py
  • Add GIN index to __table_args__ in Document model
  • Create Alembic migration file
  • Test migration up/down on a local database

Phase 2: API schemas + CRUD (1 PR, depends on Phase 1)

  • Update Conclusion response schema to include h_metadata, level, times_derived, source_ids
  • Add metadata field to ConclusionCreate schema
  • Create ConclusionUpdate schema
  • Add to src/schemas/__init__.py exports: ConclusionUpdate
  • Update ALLOWED_EXTERNAL_TO_INTERNAL_COLUMN_MAPPING_DOCUMENTS in src/utils/filter.py
  • Create update_document_metadata CRUD function in src/crud/document.py
  • Export update_document_metadata from src/crud/__init__.py
  • Update create_observations to pass h_metadata from ConclusionCreate.metadata

Phase 3: Router endpoint (same PR as Phase 2)

  • Add PUT /{conclusion_id} endpoint to src/routers/conclusions.py
  • Add route tests for the update endpoint
  • Add route tests verifying level, times_derived, source_ids appear in responses
  • Add route tests verifying metadata filtering works
  • Add route tests verifying level and times_derived cannot be set via create/update

Phase 4: Python SDK (1 PR)

  • Update ConclusionResponse in sdks/python/src/honcho/api_types.py
  • Add ConclusionUpdateParams to sdks/python/src/honcho/api_types.py
  • Update ConclusionCreateParams in sdks/python/src/honcho/api_types.py and sdks/python/src/honcho/conclusions.py
  • Update Conclusion class in sdks/python/src/honcho/conclusions.py
  • Add update() to ConclusionScope and ConclusionScopeAio
  • Update build_conclusion_payload in create() to pass metadata
  • Update SDK tests

Phase 5: TypeScript SDK (1 PR)

  • Update ConclusionResponse interface in sdks/typescript/src/types/api.ts
  • Add ConclusionUpdateParams interface
  • Update Conclusion class in sdks/typescript/src/conclusions.ts
  • Add update() to ConclusionScope
  • Update create() to pass metadata
  • Update SDK tests

6. Files to Modify (exact paths relative to repos/honcho/)

Server

FileChange
src/models.pyAdd h_metadata column + GIN index to Document
src/schemas/api.pyUpdate Conclusion, ConclusionCreate; add ConclusionUpdate
src/schemas/__init__.pyExport ConclusionUpdate
src/crud/document.pyAdd update_document_metadata(); update create_observations() to pass h_metadata
src/crud/__init__.pyExport update_document_metadata
src/routers/conclusions.pyAdd PUT /{conclusion_id} endpoint
src/utils/filter.pyUpdate ALLOWED_EXTERNAL_TO_INTERNAL_COLUMN_MAPPING_DOCUMENTS
migrations/versions/<new>_add_public_metadata_to_documents.pyNew migration file

Python SDK

FileChange
sdks/python/src/honcho/api_types.pyUpdate ConclusionResponse, ConclusionCreateParams; add ConclusionUpdateParams
sdks/python/src/honcho/conclusions.pyUpdate Conclusion, ConclusionCreateParams, ConclusionScope; add update()
sdks/python/src/honcho/aio.pyAdd async update() to ConclusionScopeAio

TypeScript SDK

FileChange
sdks/typescript/src/types/api.tsUpdate ConclusionResponse; add ConclusionUpdateParams
sdks/typescript/src/conclusions.tsUpdate Conclusion, ConclusionScope; add update()

Tests

FileChange
tests/routes/test_conclusions.pyAdd tests for update endpoint, metadata in responses, read-only fields, filtering
tests/sdk/test_conclusions.pyAdd tests for SDK conclusion metadata, update, new fields
sdks/typescript/__tests__/conclusions.test.tsAdd tests for TypeScript SDK changes

7. Risk Assessment

Low Risk

  • Adding a NOT NULL column with DEFAULT. PostgreSQL 11+ handles this as a metadata-only change (no table rewrite). The '{}'::jsonb default is applied lazily on read for existing rows. This is instant even for large tables.
  • New response fields are additive. Existing SDK clients that do not expect metadata, level, times_derived, or source_ids will simply ignore them (standard JSON forward-compatibility).
  • ConclusionCreate.metadata defaults to {}. Existing callers that do not send metadata will get the same behavior as before.

Medium Risk

  • GIN index creation on large tables. The CREATE INDEX CONCURRENTLY avoids blocking writes but still requires a full table scan. For very large documents tables, this could take minutes. The migration handles this correctly by running outside a transaction.
  • Filter mapping change ("metadata" → "h_metadata" instead of "internal_metadata"). If any internal code path (deriver, dreamer, reconciler) passes {"metadata": {...}} as a filter through apply_filter for documents, it will break. Audit confirms this does not happen today (internal systems query documents directly via SQL, not the filter system). However, this should be verified in a staging environment before production deployment.

Mitigations

  • Run the migration on a staging database with production-scale data before deploying to production.
  • The migration uses IF NOT EXISTS guards for idempotency.
  • The column_exists / index_exists guards ensure the migration is re-runnable.

8. Verification Plan

Unit Tests

  1. Schema validation tests:

    • ConclusionCreate with metadata validates correctly (depth, key count, NUL sanitization).
    • ConclusionCreate without metadata defaults to {}.
    • ConclusionUpdate with None metadata is valid.
    • Conclusion response correctly serializes h_metadata as "metadata".
    • Conclusion response includes level, times_derived, source_ids from ORM attributes.

  2. CRUD tests:

    • create_observations stores h_metadata from the input.
    • update_document_metadata updates the metadata and returns refreshed document.
    • update_document_metadata raises ResourceNotFoundException for non-existent or soft-deleted documents.
    • update_document_metadata with metadata=None is a no-op.

Integration Tests (Route-level)

  1. Create with metadata:

    response = client.post(
    f"/v3/workspaces/{ws}/conclusions",
    json={"conclusions": [{
        "content": "User likes cats",
        "observer_id": "alice",
        "observed_id": "bob",
        "metadata": {"category": "preference", "confidence": 0.9}
    }]},
)
assert response.status_code == 201
data = response.json()
assert data[0]["metadata"] == {"category": "preference", "confidence": 0.9}
assert data[0]["level"] == "explicit"
assert data[0]["times_derived"] == 1
assert data[0]["source_ids"] is None

  2. Update metadata:

    response = client.put(
    f"/v3/workspaces/{ws}/conclusions/{conclusion_id}",
    json={"metadata": {"category": "updated"}},
)
assert response.status_code == 200
assert response.json()["metadata"] == {"category": "updated"}

  3. Read-only fields cannot be set:

    # level, times_derived, source_ids in ConclusionCreate are ignored
# (they are not fields on ConclusionCreate, so Pydantic strips them)
response = client.post(
    f"/v3/workspaces/{ws}/conclusions",
    json={"conclusions": [{
        "content": "test",
        "observer_id": "alice",
        "observed_id": "bob",
        "level": "inductive",  # should be ignored
        "times_derived": 99,   # should be ignored
    }]},
)
assert response.status_code == 201
assert response.json()[0]["level"] == "explicit"  # always explicit for user-created
assert response.json()[0]["times_derived"] == 1

  4. Filter by metadata:

    response = client.post(
    f"/v3/workspaces/{ws}/conclusions/list",
    json={"filters": {"metadata": {"category": "preference"}}},
)
assert response.status_code == 200
for item in response.json()["items"]:
    assert item["metadata"]["category"] == "preference"

  5. Filter by level:

    response = client.post(
    f"/v3/workspaces/{ws}/conclusions/list",
    json={"filters": {"level": "deductive"}},
)
assert response.status_code == 200

  6. List response includes new fields:

    response = client.post(
    f"/v3/workspaces/{ws}/conclusions/list",
    json={},
)
item = response.json()["items"][0]
assert "metadata" in item
assert "level" in item
assert "times_derived" in item
assert "source_ids" in item

Migration Tests

  1. Run alembic upgrade head on a database with existing documents and verify:

    • All existing documents have metadata = '{}'::jsonb.
    • The GIN index ix_documents_metadata_gin exists.
    • The column is NOT NULL.

  2. Run alembic downgrade -1 and verify:

    • The metadata column is removed.
    • The GIN index is removed.
    • All other columns are untouched.

SDK Tests

  1. Python SDK: Test that Conclusion objects have metadata, level, times_derived, source_ids attributes after creation and listing.
  2. Python SDK: Test ConclusionScope.update() method.
  3. TypeScript SDK: Test that Conclusion objects have the new properties.
  4. TypeScript SDK: Test ConclusionScope.update() method.

9. Open Questions

  1. Should ConclusionCreate accept metadata on the batch endpoint too? The current batch create schema (ConclusionBatchCreate) wraps a list of ConclusionCreate. Since ConclusionCreate gains a metadata field, batch creation automatically supports per-conclusion metadata. No additional work needed, but worth calling out.

  2. Should the ConclusionQuery (semantic search) endpoint also return the new fields? Yes — it returns list[schemas.Conclusion], and since we are updating the Conclusion schema, search results will automatically include metadata, level, times_derived, and source_ids. No additional work needed.

  3. Should metadata be included in the embedding? No. The metadata field is for user-defined tags and annotations, not semantic content. Embeddings are generated from content only. This is consistent with how metadata works on messages.

  4. Should the deriver/dreamer be able to set public metadata on conclusions they create? Not in this iteration. System-created conclusions (from the deriver and dreamer) will have h_metadata={}. A future iteration could allow the deriver to tag conclusions with metadata (e.g., {"auto_generated": true}), but that requires changes to the internal DocumentCreate schema and the deriver agent’s tool definitions, which is out of scope for this spec.

  5. Index creation time on large tables. The CREATE INDEX CONCURRENTLY for the GIN index on metadata could take significant time on tables with millions of rows. This is a one-time cost. If the documents table is exceptionally large, consider scheduling the migration during a low-traffic window. The application will function correctly without the index (queries will just be slower for metadata filters until the index is built).

Graph View

Backlinks