What is Honcho, and what are we optimizing?

1. Why memory matters

Agents without memory are expensive, slow, and badly aligned. Every conversation starts cold. Every preference has to be re-stated. Every clarifying question burns tokens and trust. The agent that has to ask “what did you mean by that?” three times is the agent the user abandons.

A memory layer fixes this by carrying state and inferences across turns, sessions, and even across agents. That’s the problem Honcho exists to solve.

2. What “memory” actually is

Memory is a skunked term. It’s cleaner to decompose it into two layers:

• State tracking — the facts. What happened, who said what, in which session, in what order. Retrieval-shaped. Solvable with good indexing, embeddings, and search.
• Entity modeling — the theories. Who is this entity, what does it want, how does it interpret things, what’s its current state. Inference-shaped. Has to be re-derived as evidence accumulates. Theory-of-Mind concepts live here.

The thesis: Honcho is the layer that produces ground truth on any entity. That entity is most often a user, but the abstraction is deliberately generic. An entity can be an AI agent, a team, a company, a brand, a codebase, a product. Anything you need to reason about over time and across context. Today the Peer paradigm is most often used to model users because Honcho’s roots are in chatbots, but the primitives are entity-agnostic by design.

Honcho currently leans on the entity-modeling side. The forward direction is widening Honcho’s view so it can be authoritative on both facts and theories about whatever entities the surrounding agent cares about.

3. What we optimize

Our mission is individual alignment: every agent should converge on the right answer for this specific user, in this specific context as fast as possible.

We measure that as cost-to-aligned-outcome, broken into two dimensions:

• Turns-to-aligned-completion (user-visible cost) — how many round-trips before the user accepts the outcome. Gated on correctness so “answer fast, answer wrong” doesn’t game the metric.
• Retrieval tool calls per turn (within-turn cost) — how often the agent reaches for context that should have been served by Honcho. Excludes functional tool calls (APIs, actions, code execution) — Honcho is not in the business of reducing those.

Both should drop. A single-turn task that required ten retrieval tool calls isn’t a win — Honcho should have surfaced what was needed up front. A five-turn task where the agent never looked outside Honcho is healthier than a two-turn task that leaned on three external memory systems. The first dimension says did the user get what they wanted with minimal friction; the second says was Honcho actually the source of truth, or just one cache among many.

This metric pair collapses both halves of Honcho:

• Bad entity modeling → extra clarifying turns.
• Bad state coverage → extra retrieval tool calls.

It moves whether we improve modeling or state, and it captures both what the user feels and what the agent has to do.

Mechanism levers — what we work on to move the cost down:

• Inference quality (how well we predict an entity’s state, intent, preferences)
• Recall precision/recall (how surgically we surface the right context)
• Coverage (how many event types Honcho can model over — see §5)

Confirming metrics — should co-travel; if they don’t, something’s wrong:

• Token efficiency per task
• End-to-end latency
• Cold-start reduction (new agent + returning entity)
• Cross-session and cross-agent coherence
• Retention/engagement (the product-level confirmation)

This reconciles “quality vs. efficiency” — they’re the two axes that collapse into a single rate: how fast we converge on the right answer with Honcho as the source of truth.

4. How we measure it

Public recall benchmarks (LoCoMo, LongMemEval, BEAM) are table stakes. We need to be competitive on them, but they’re not our moat. Anyone with a vector DB can iterate against these.

Our differentiation is the modeling layer, and there is no obviously-good public benchmark for it. Task evals like Terminal-Bench 2.0 and τ-bench produce too much variance to localize Honcho’s contribution.

The path we’re investing in:

1. Unified test bench (tests/unified/) — JSON-defined, step-based test cases with llm_judge assertions. The seed for enumerating the behaviors we actually care about (peer cards, representations, dialectic responses) in a reproducible way.
2. Synthetic datasets built around those enumerated cases — controlled scenarios that let us measure modeling quality without depending on real conversation volume we don’t yet have.
3. DRIFT — our designed internal benchmark for modeling quality. Still in design; this is where we expect to land the differentiated story.

Open question on public personalization benchmarks. PersonaMem (COLM 2025) and the newer PersonaMem-v2 target implicit user-persona inference over long conversations and look like the closest match to what we want to measure. Whether they actually capture the thing we care about — alignment, intent inference, engagement-driven personalization — is unverified. We owe ourselves an exhaustive review of personalization benchmarks (PersonaMem-v2, AlpsBench, O-MEM, RF-Mem, and whatever else surfaces) before claiming any of them as our scorecard.

On personality and willingness to pay. Consumer products like Sauna, Poke, and Endearing have flagged engagement-driven personalization as a core problem. Retention and DAU lift in real products are the cleanest evidence that this dimension is worth paying for, regardless of which benchmark we end up trusting.

5. What’s missing

To be authoritative on any entity, Honcho has to ingest more than peer-to-peer messages. Multimodal content and files are the immediate gaps. This connects directly to the retrieval-tool-call dimension in §3: every time an agent reaches for a file system, a screen state, or a tool output because Honcho can’t see it, that’s a coverage gap showing up in the metric. Coverage failures become quantitative, not vibes.

Roadmap implication: expand the event types Honcho models over, while keeping entity modeling always-on as the differentiator.

6. What Honcho is not

• Not a vector database.
• Not a RAG framework.
• Not a chatbot SDK.

It’s the layer that maintains an evolving, queryable model of any entity worth reasoning about — surfaced through retrieval (Dialectic) and refined in the background (Deriver, Dreamer). Everything else is implementation detail.