drift

DRIFT Benchmark

Abstract

Existing AI memory benchmarks — LoCoMo, LongMemEval, BEAM, and similar — conflate long-context retrieval performance with the value contribution of a dedicated memory system. They permit raw transcript access, use fixed context windows, and optimize for question-answering accuracy over synthetic conversation histories. These properties make them gameable by context expansion alone and structurally unable to measure what actually matters: whether a memory system improves agent behaviour over genuine temporal gaps where the original context no longer exists.

DRIFT (Dynamic Recall via Injected Fictional Trajectories) is a simulation-based benchmark designed to close these gaps. DRIFT generates synthetic latent profiles (i.e. ground truth identities) for typed entities — humans, assistant agents, coding agents, crawlers, and peer configurations — and produces realistic interaction trajectories across short, medium, and long temporal horizons. A hard budget gate destroys raw session transcripts after each turn, forcing all subsequent inference to operate exclusively on whatever the memory system chose to store. Evaluation is performed against the latent ground truth at structured checkpoints.

DRIFT treats token efficiency and latency as first-class metrics alongside capability. The Token Efficiency Score (TES) measures capability delivered per token spent relative to a reference compressor. The Latency Profile Score (LPS) measures read latency and time-to-first-relevant-token against reference production targets. These are reported as a separate DRIFT-E composite, keeping capability and efficiency surfaces independent so deployment trade-offs are explicit.

DRIFT is explicitly designed to generalize beyond the user-assistant dyad. It supports multi-peer session topologies, agentic paradigms (coding, crawling, tool-use), and mixed human-agent peer graphs, each with distinct entity profile schemas, event injection distributions, and probe types. A pluggable memory interface contract enables direct comparison across Honcho, Zep, Mem0, and other systems on the same evaluation harness.

---

1. Motivation and Problem Statement

1.1 The Structural Failure of Existing Benchmarks

Current AI memory benchmarks share a common architectural flaw: they evaluate memory system quality by presenting a model with a long document or conversation history and asking questions about it. This design makes the benchmark solvable without a memory system at all — a sufficiently large context window renders every existing benchmark trivially addressable.

The specific failures are:

• Context leakage. Raw transcripts remain accessible during evaluation. The memory system is never actually required to compress, abstract, or selectively store information.
• Fixed context horizon. Benchmarks use predefined document lengths, not genuine temporal gaps. A system cannot be tested on its ability to handle sessions separated by days, weeks or even years.
• QA proxy. Evaluation reduces to question-answering accuracy on synthetic transcripts, which rewards verbatim retrieval rather than belief modeling, preference tracking, or behavior adaptation.
• Goodhart vulnerability. Because the proxy metric is well-defined and bounded, systems can be optimized directly against it without improving the underlying capability. Claims of solved memory proliferate while real-world quality stagnates.
• Dyadic assumption. Every major benchmark assumes a single human interacting with a single AI assistant. Agentic systems, multi-agent pipelines, and tool-mediated workflows are entirely out of scope.

1.2 The Goodhart Problem in Memory Research

The correct diagnostic question is not “can this system answer questions about a conversation?” but “does this system make an agent demonstrably better over time, compared to an agent with no memory, and what fraction of that improvement is attributable to the memory system specifically?” No existing benchmark asks this question.

1.3 Design Principles

DRIFT is designed around six principles that directly address each failure mode:

1. Temporal irreversibility. Raw session transcripts are destroyed after each session. Evaluation proceeds exclusively from stored memory state.
2. Latent ground truth. Entity profiles are generated before any interaction transcript exists. Ground truth is the latent profile state at each checkpoint.
3. Contribution isolation. The primary metric compares the memory system against an amnesiac baseline at equal inference compute.
4. Information density variation. The majority of sessions contain no memorable signal. Selectivity is measured explicitly and penalized.
5. Paradigm generality. DRIFT supports user-assistant, coding agent, crawler agent, and multi-peer topologies through a typed entity factory.
6. Efficiency parity. Token cost and latency are first-class metrics, not footnotes.

---

2. System Architecture

DRIFT consists of three sequential phases: World Generation, the Memory Loop, and Evaluation. Each is implemented as an independent module with well-defined interfaces.

2.1 Phase Overview

Phase	Input	Output
01 · World Generation	Scenario specification	Latent profiles + event schedules + session transcripts
02 · Memory Loop	Session stream + memory system under test	Memory state snapshots per session boundary
03 · Evaluation	Memory snapshots + checkpoint probes + latent ground truth	DRIFT-C and DRIFT-E scores

---

3. World Generation

3.1 Scenario Specification Language

A DRIFT scenario is a declarative YAML graph. All transcripts, events, and ground truth annotations are derived from it synthetically.

id: ua-baseline-001
 
paradigm: user_assistant # user_assistant | coding_agent | crawler_agent | multi_peer
 
horizon:
  short: { sessions: 30, session_gap_hours: 8 }
 
  medium: { sessions: 200, session_gap_days: 3 }
 
  long: { sessions: 1000, session_gap_days: 14 }
 
noise_ratio: 0.70 # fraction of sessions with no signal events
 
entities:
  - id: principal
 
    type: human
 
    profile:
      big_five: { O: 0.7, C: 0.5, E: 0.4, A: 0.8, N: 0.3 }
 
      event_distribution:
        job_change: 0.02
 
        opinion_shift: 0.05
 
        habit_form: 0.10
 
  - id: assistant
 
    type: assistant_agent
 
    capability_profile:
      context_window: 200000
 
      tool_access: [search, code_interpreter]
 
topology:
  - { session_range: [0, -1], active: [principal, assistant], pattern: linear }
 
checkpoints:
  - { after_session: 30, probe_types: [recall, action] }
 
  - { after_session: 200, probe_types: [recall, action, revision] }
 
  - { after_session: 1000, probe_types: [recall, action, revision] }

3.2 Entity Type System

DRIFT uses a typed entity factory. Each entity type carries its own profile schema, event injection distribution, and ground truth representation.

3.2.1 Human / Principal

| Field | Description |

|---|---|

| stable_attributes | Big Five personality, cultural background, communication register, core values, long-term goals. Does not mutate. |

| mutable_state | Current role/employer, active relationships, health status, ongoing projects, beliefs on contested topics. Updated by life event injections. |

| event_distribution | Probability over: job transition, relationship change, opinion shift, habit formation, health event. |

| communication_style | Lexical preferences, verbosity, formality, tendency to reference prior context. |

| ground_truth_repr | Structured snapshot of mutable_state at each checkpoint. The label against which memory systems are evaluated. |

3.2.2 Assistant Agent

| Field | Description |

|---|---|

| capability_profile | Declared tool access, context window budget, response style defaults. |

| task_state | Active task graph: requested, completed, blocked. Working memory for the session trajectory. |

| principal_model | Agent’s accumulated model of the human: inferred preferences, communication style, known constraints. |

| event_distribution | Task completion, instruction revision, preference revelation, explicit correction. |

| ground_truth_repr | The correct principal_model at each checkpoint. |

3.2.3 Coding Agent

| Field | Description |

|---|---|

| project_graph | Repository topology, active modules, known dependencies, architectural decisions to date. |

| decision_log | Timestamped architectural choices, rejected alternatives, principal rationale where stated. |

| failure_surface | Known bugs, failing tests, anti-patterns encountered. |

| style_conventions | Inferred code style, naming conventions, testing philosophy. |

| event_distribution | Commit, test failure, architectural override, refactor request, dependency change. |

| ground_truth_repr | Correct project_graph + decision_log at each checkpoint. |

3.2.4 Tool / Environment Agent

| Field | Description |

|---|---|

| frontier_model | For crawlers: visited URL graph, observed patterns, traversal strategy. For test runners: suite state, flaky tests, coverage map. |

| objective_state | Current task objective and progress. Updated by completion and revision events. |

| environmental_delta | Changes in external environment since last session. |

| event_distribution | Objective completion, environment change, traversal dead-end, coverage milestone. |

| ground_truth_repr | Correct frontier_model and objective_state at each checkpoint. |

3.3 Event Injection

Events are classified into three tiers:

• Signal events (~10% of sessions). High-importance events that any competent memory system should store: job change, relationship end, diagnosis, architectural override, explicit instruction revision.
• Soft signal events (~20% of sessions). Moderate-importance accumulating evidence: a third mention of a coding preference, repeated requests for brief responses.
• Noise sessions (~70% of sessions at default noise_ratio: 0.70). No event of lasting significance. Routine task execution, small talk. The memory system should store nothing from these.

Noise ratio design note. Existing benchmarks implicitly use noise_ratio ≈ 0 — every session contains a fact worth recalling. DRIFT treats high noise as the default because real deployments look like this. Selectivity — correctly not storing noise — is explicitly penalised in the metric suite. A memory system with perfect recall but zero selectivity scores poorly regardless of recall performance.

3.4 Temporal Horizons

| Horizon | Simulated span | Primary memory operation |

|---|---|---|

| Short | Minutes to days (2–50 sessions) | Within-topic coherence, preference consistency, entity state tracking across a continuous task. |

| Medium | Weeks to months (50–500 sessions) | Habit accumulation, recurring preference consolidation, project knowledge evolution. |

| Long | Months to years (500–5000 sessions) | Belief revision under life events, decayed relevance of outdated facts, identity-level continuity. |

Long-horizon performance is weighted most heavily (weight 0.50 in the aggregate) because it is the hardest, least gameable, and most practically important class.

3.5 Transcript Synthesis

Given a latent profile, an event schedule, and a scenario topology, the world generator produces synthetic conversation transcripts using a prompted LLM. The synthesis prompt constrains the generator to:

• Be consistent with the entity’s communication style and mutable state at the time of generation
• Introduce signal events naturalistically — the event emerges through dialogue, not announcement
• Match the noise_ratio: noise sessions contain no events and no novel information
• Vary in length, formality, and topic distribution to prevent pattern-matching shortcuts

Transcript synthesis is the only stage in DRIFT that uses an LLM. All subsequent stages — event injection, ground truth annotation, probe generation, and metric computation — are deterministic.

---

4. Multi-Peer Topologies and Agentic Paradigms

4.1 Session Topology

 
@dataclass
 
class SessionSlot:
 
    session_index:       int
 
    simulated_time:      datetime
 
    active_entities:     list[EntityID]
 
    interaction_pattern: Literal[
 
        "linear",        # A -> B -> A
 
        "broadcast",     # A -> [B, C, D]
 
        "mesh",          # all-to-all
 
        "hierarchical",  # principal -> orchestrator -> workers
 
        "async",         # interleaved without strict turn order
 
    ]
 
    duration_turns: int
 

4.2 Memory Scope Model

| Scope | Visibility | Description |

|---|---|---|

| Private | Per-entity only | Each entity’s internal memory of its own trajectory. Inaccessible to other entities even if they shared sessions. |

| Shared session | All session participants | Information jointly established during a session: decisions made, facts agreed upon, tasks assigned. |

| Emergent | Derived across sessions | Collective world model that emerges from many sessions. Evaluated by cross-entity consistency probes. |

Scope leakage detection. DRIFT embeds canary facts: information disclosed by entity A in a session where entity B was not present. If entity B can recall a canary fact at evaluation time, the memory system has leaked cross-entity private state. Leakage events are logged and penalised in the Privacy Score (see §7.5).

4.3 Paradigm Definitions

4.3.1 User-Assistant

One human principal interacts with one assistant agent across a trajectory. Baseline paradigm; establishes compatibility with existing benchmarks.

Key evaluation questions:

• Does the assistant correctly update its model of the user after a life event?
• Does the assistant act differently based on correctly stored preferences vs a no-memory baseline?
• Does the assistant avoid re-asking for information it has already been given?

4.3.2 Coding Agent

One human principal interacts with one or more coding agents (Claude Code, Codex, OpenHands) over a software project trajectory.

Distinctive memory requirements:

• Decision persistence. Architectural decisions must survive across sessions. Contradicting a prior decision without acknowledgement is memory failure.
• Anti-pattern memory. Known bugs and failed approaches should surface when a similar situation recurs.
• Style convergence. The agent should progressively converge on the principal’s style conventions.
• Failure surface tracking. Current test suite state, including known failures, must be maintained accurately.

4.3.3 Crawler / Web Agent

An autonomous agent (OpenCrawl, browser agents) pursues a long-horizon objective across many sessions.

Distinctive memory requirements:

• Frontier maintenance. The agent must not revisit already-explored states.
• Objective coherence. Current objective and sub-objective structure must be preserved across sessions, including principal revisions.
• Environmental change detection. Information that was true at time T must be distinguishable from information that may have changed since T.

4.3.4 Multi-Peer

Two or more entities of any type participate in shared sessions. Introduces the full scope model and all probe types including consistency, attribution, handoff, and canary probes.

Example configurations:

• Two human principals + one shared assistant (collaborative task with competing preferences)
• One human principal + one orchestrator agent + three worker agents (hierarchical coding team)
• One human + one assistant + one crawler agent in periodic joint sessions (research workflow)
• Multiple peer AI agents with no human present (fully autonomous pipeline)

Fully autonomous pipelines. When no human is present, the “principal” is defined as the entity that issued the original objective. Its latent profile consists of the objective specification, success criteria, and declared constraints. Ground truth is evaluated against objective completion quality and behavioural consistency with declared constraints — not a human preference model.

---

5. The Budget Gate

5.1 Mechanism

After each session, the raw session transcript is removed from the evaluation harness’s accessible state. From that point forward, all inference — in subsequent sessions and in all probes — can only access the output of the memory system under test.

The sequence:

1. Session N transcript is generated by the world generator and passed to the memory system via store(session).
2. The memory system returns a MemoryState representing whatever it chose to store.
3. The transcript is discarded. Only the MemoryState persists.
4. Session N+1 is synthesised using the entity’s latent profile but evaluated using only the MemoryState from step 2.
5. At checkpoints, probes are answered using only the accumulated MemoryState chain.

Why this matters. Every existing memory benchmark allows the system under test to access the original conversation history during evaluation. This means retrieval-augmented approaches — which simply fetch relevant chunks from the verbatim transcript — can match or exceed dedicated memory systems. The budget gate makes this impossible.

5.2 Budget Gate Variants

| Variant | Constraint | Use case |

|---|---|---|

| Hard gate | Transcript fully discarded after store() returns. | Primary evaluation mode. |

| Soft gate | Transcript retained for K sessions before discarding. | Ablation: how much does a short-term buffer help? |

| Summary gate | Transcript replaced by a fixed-size (N-token) summary from a reference model. | Tests whether systems extract signal from pre-compressed input. |

5.3 Scope-Level Enforcement (Multi-Peer)

• Private scope gate. Each entity’s private transcript segments are discarded separately. Entity B cannot access entity A’s private memory state.
• Shared scope gate. Shared session content is discarded after the session. All participants must have stored what they need before the gate fires.
• Emergent scope. Derived at evaluation time from the ensemble of private stores. Implicitly constrained because each private store is budget-gated.

---

6. Probe Suite

Probes are structured queries issued to the memory system at checkpoint intervals. Each probe is answered using only the accumulated MemoryState — the budget gate ensures no other access is possible. Answers are scored against the latent ground truth snapshot at the checkpoint time.

6.1 Universal Probe Types

Recall Probes

Direct queries about entity state.

| Sub-type | Example |

|---|---|

| Fact recall | “What does this entity do for work?” |

| Relationship recall | “What is the current status of the entity’s relationship with entity X?” |

| Temporal recall | “When was the last time this entity mentioned their health?” Tests whether temporal metadata is preserved. |

| Negative recall | “Has this entity ever expressed a preference about X?” where true answer is no. Tests against hallucinated recall. |

Action Probes

Downstream task scenarios where the correct action is non-obvious without stored memory.

| Sub-type | Example |

|---|---|

| Preference-driven | “Recommend a venue for this entity’s team outing” where ground truth reveals they are sober and prefer outdoor settings — both mentioned once, 40 sessions apart. |

| Constraint-aware | “Draft a response to this message” where ground truth reveals a sensitive ongoing situation. |

| Style-adaptive | “Complete this code snippet” where ground truth specifies the principal’s preferred patterns. |

Belief Revision Probes

Probes issued immediately after a signal event requiring the memory system to update its stored model. Hardest and most important probes in the suite.

| Sub-type | Example |

|---|---|

| Fact update | Entity changed employers in session N. Probe at N+1: “Where does this entity work?” Correct answer uses new employer. |

| Preference reversal | Entity retracted a prior preference. Probe tests whether old preference was deprecated. |

| Belief half-life | Probes at increasing intervals after an event. Tracks how long the correct updated belief persists. |

6.2 Paradigm-Specific Probe Types

Frontier Probes (Coding + Crawler)

| Sub-type | Example |

|---|---|

| Coverage | “Has module X been refactored to use the new error handling pattern?” |

| Anti-pattern | “Propose an implementation for feature Y” where a known-bad approach exists in failure_surface. |

| Traversal | “What URLs matching pattern P have been successfully crawled?” |

| Objective coherence | “What is the current sub-objective?” after a revision session. |

Multi-Peer Probe Types

| Sub-type | Description |

|---|---|

| Consistency | Queries issued independently to two or more agents that shared a session. Consistent answers required across all respondents. |

| Attribution | “Who proposed the decision to use async/await?” Tests whether memory preserves authorship alongside content. |

| Canary | Queries about facts known only to entity A, issued to entity B. Correct answer: entity B cannot know this. Any recall = scope leakage. |

| Handoff | New entity joins mid-trajectory. Scored as time-to-accuracy: sessions until the new entity achieves full checkpoint accuracy. |

---

7. Metric Suite

DRIFT reports four capability metrics, two efficiency metrics, and one secondary safety metric. Capability and efficiency metrics are reported independently.

Why two scores, not one. A system with DRIFT-C = 0.90 and DRIFT-E = 0.30 is excellent for offline analysis where latency and cost are irrelevant. A system with DRIFT-C = 0.70 and DRIFT-E = 0.85 may be the correct choice for a real-time assistant at scale. Collapsing these into one number forces the benchmark authors to make a deployment trade-off that belongs to the deployer.

7.1 Memory Value Over Time (MVOT)

MVOT is the primary metric. It measures the cumulative improvement of the memory system over an amnesiac baseline, normalised by storage overhead.

Definition:

Let E_mem(t) = probe error rate of the memory system at checkpoint t

Let E_amn(t) = probe error rate of an amnesiac baseline at checkpoint t

Let C_mem = storage cost (tokens stored per session)

Let C_base = storage cost of the reference compressor

MVOT = [ Σ_t (E_amn(t) - E_mem(t)) / E_amn(t) ] / T  ×  (C_base / C_mem)

• MVOT = 1.0: theoretical maximum (perfect recall, minimum storage)
• MVOT < 0: the memory system performs worse than an amnesiac baseline
• The efficiency term (C_base / C_mem) penalises systems that achieve recall by storing everything

7.2 Selectivity Score

Measures the fraction of noise sessions correctly not stored (or stored at negligible weight).

Selectivity = (TNR_noise + (1 - FNR_signal)) / 2

Where TNR_noise = true negative rate on noise sessions, FNR_signal = false negative rate on signal sessions. Range [0, 1]. A system with perfect selectivity stores exactly the signal sessions and nothing from noise sessions.

7.3 Revision Accuracy

Measures correctness and speed of model updates after a signal event. Computed only over belief revision probes.

Two sub-components:

• Update correctness. Fraction of revision probes where the new ground truth is correctly reflected at the first checkpoint after the event.
• Deprecation correctness. Fraction where the old (now incorrect) belief is no longer retrievable.

Revision Accuracy = harmonic_mean(update_correctness, deprecation_correctness)

A system that adds the new belief but retains the contradictory old belief scores poorly.

7.4 Action Alignment

Given what the memory system knows, does the agent take the right action? Computed over action probes, scored by a reference judge (a separate LLM prompted with the latent ground truth).

The judge asks: “Given this entity profile [latent ground truth], is the following action better, equivalent, or worse than an agent with no memory?” Scores: +1 (better), 0 (equivalent), -1 (worse). Action Alignment is the mean score normalised to [0, 1].

7.5 Privacy Score (Secondary)

Computed only in multi-peer scenarios. Measures scope leakage rate.

Privacy Score = 1 - canary_probe_recall_rate

Perfect score (1.0) = no scope leakage detected. Reported separately; does not enter the DRIFT-C aggregate. It is a safety property, not a capability property.

7.6 DRIFT-C: Capability Composite

Weighted sum of the four capability metrics, varying by horizon class:

| Metric | Short weight | Medium weight | Long weight |

|---|---|---|---|

| MVOT | 0.35 | 0.40 | 0.45 |

| Selectivity | 0.25 | 0.25 | 0.20 |

| Revision Accuracy | 0.20 | 0.20 | 0.20 |

| Action Alignment | 0.20 | 0.15 | 0.15 |

Horizon weights in the aggregate: Short 0.20, Medium 0.30, Long 0.50.

7.7 Token Efficiency Score (TES)

Token Cost Components

Three components instrumented at the memory system interface boundary:

| Component | Definition |

|---|---|

| Write tokens (W) | Total tokens consumed by all LLM calls inside store() per session: synthesis, compression, consolidation, internal reasoning. |

| Storage tokens (S) | Size of the MemoryState representation in tokens after store() returns. |

| Read tokens (R) | Total tokens consumed by all LLM calls inside retrieve() per query: re-ranking, synthesis over retrieved chunks, chain-of-thought. |

Total token cost per session equivalent:

TC = W + S_delta + (R × Q_rate)

Where S_delta is the marginal storage increase per session and Q_rate is the average retrieve() calls per session.

TES Formula

Let TC_ref = token cost of the reference compressor (one LLM call per session, 512-token cap output, no retrieval LLM calls)

Let MVOT_ref = MVOT score of that same reference compressor

TES = (MVOT_sys / MVOT_ref) / (TC_sys / TC_ref)

• TES > 1.0: more capability per token than the reference compressor
• TES < 0.5: spending more than 2× the tokens for proportionally less gain than a simple summariser

Token Cost Curve

In addition to the scalar TES, DRIFT produces a token cost curve: MVOT plotted as a function of cumulative token budget across the trajectory. This reveals whether a system front-loads cost at write time (expensive compression, cheap retrieval) or back-loads it (cheap storage, expensive read tax at every inference call).

7.8 Latency Profile Score (LPS)

Latency Measurement Points

| Point | Definition |

|---|---|

| Write latency (WL) | Wall-clock time from store() call to return. Measured at p50/p95/p99. Async in most architectures; bounds next-session memory availability. |

| Read latency (RL) | Wall-clock time from retrieve() call to first result. On the critical conversational path — every response waits for this. Measured at p50/p95/p99. |

| Time to First Relevant Token (TTFRT) | End-to-end: start of new session turn → first token of a memory-informed response. TTFRT = RL + inference_latency_overhead. The actual user-perceived cost of memory. |

Benchmarking Conditions

• Reference hardware: 8-core CPU, 32 GB RAM, no GPU, NVMe SSD
• Concurrency: 4 parallel sessions to simulate realistic multi-user load
• Network: measured separately for local deployment (memory system hosted locally) and cloud API deployment
• Reported figure: p99 as primary headline; p50 as secondary. Mean is not reported (too sensitive to LLM call latency outliers)

LPS Formula

Let RL_ref = 50 ms and TTFRT_ref = 200 ms (reference targets for a production-viable memory system)

Read Score  = min(1.0, RL_ref / RL_p99)

TTFRT Score = min(1.0, TTFRT_ref / TTFRT_p99)

LPS = 0.40 × Read Score + 0.60 × TTFRT Score

TTFRT is weighted higher (0.60) because it is the user-perceived metric.

Dialectic Speed

A secondary latency metric specific to conversational paradigms: the full round-trip for a memory-informed exchange (retrieve → incorporate → generate → store). Captures the compounding latency of a system slow at both read and write. Reported as a standalone figure, not folded into LPS.

Production latency budget targets:

| Metric | Local p99 | Cloud API p99 |

|---|---|---|

| Read latency | < 50 ms | < 150 ms |

| Write latency | < 2000 ms | < 2000 ms |

| TTFRT | < 200 ms | < 400 ms |

| Dialectic speed | < 500 ms | < 800 ms |

7.9 DRIFT-E: Efficiency Composite

| Metric | Weight in DRIFT-E |

|---|---|

| Token Efficiency Score (TES) | 0.55 |

| Latency Profile Score (LPS) | 0.45 |

---

8. Harness Integration

8.1 Overview

The DRIFT benchmark harness lives at tests/bench/[harness.py](http://harness.py) in the Honcho repository. It implements the full DRIFT pipeline against Honcho’s SDK using the native Workspace / Peer / Session / Message primitives.

Honcho primitive mapping:

| DRIFT primitive | Honcho primitive |

|---|---|

| Scenario | workspace_id (one per scenario run) |

| Entity | Peer |

| Session | Session |

| Memory operation: store | session.add_messages() + background deriver |

| Memory operation: retrieve | [peer.chat](http://peer.chat)(query) |

| Memory state snapshot | peer.representation(session) or session.context(...) |

| Shared session scope | Session-level context via session.context() |

| Private peer scope | [peer.chat](http://peer.chat)() querying against that peer’s derived representations |

This alignment means a DRIFT evaluation run against Honcho is not a synthetic test — it is a real Honcho workload with annotated ground truth. The same infrastructure serves as the training environment for Honcho’s GRPO-based RL loop.

8.2 Plugging In an Alternative Memory System

To evaluate a non-Honcho memory system, implement the MemoryBackend protocol and register it with the harness:

 
from tests.bench.harness import MemoryBackend, DriftHarness
 
class ZepBackend(MemoryBackend):
 
    async def store(self, session_id: str, messages: list[dict]) -> None:
 
        # call Zep's add_memory API
 
        ...
 
    async def retrieve(self, session_id: str, entity_id: str, query: str) -> str:
 
        # call Zep's search API
 
        ...
 
    def token_counts(self) -> TokenCounts:
 
        # return write/storage/read token tallies
 
        ...
 
harness = DriftHarness(backend=ZepBackend(), scenario_path="scenarios/ua-baseline-001.yaml")
 
report = await [harness.run](http://harness.run)()
 

8.3 Running the Benchmark

 
# Run a single scenario against Honcho (default backend)
 
uv run pytest tests/bench/[harness.py](http://harness.py) -k ua-baseline-001 -v
 
# Run all canonical scenarios
 
uv run pytest tests/bench/ -v
 
# Run with a specific horizon only
 
uv run pytest tests/bench/[harness.py](http://harness.py) --horizon short -v
 
# Output results to JSON
 
uv run pytest tests/bench/[harness.py](http://harness.py) --drift-output results/run-001.json -v
 
# Compare two backends
 
uv run python tests/bench/[harness.py](http://harness.py) compare \
 
    --backend-a honcho \
 
    --backend-b zep \
 
    --scenario scenarios/ua-baseline-001.yaml
 

---

9. Implementation Reference

9.1 Technology Stack

| Component | Technology | Rationale |

|---|---|---|

| World generator | Python + LLM API (Claude / GPT-4o for synthesis) | Flexible scripting; LLM synthesis is the only non-deterministic stage |

| Entity profile store | PostgreSQL + pgvector | Structured profile fields + vector similarity for soft matching |

| Session transcript store | Object storage (S3-compatible) | Large volume; retrieved only by the generator, never by the memory system under test |

| Memory system interface | Python Protocol class | Language-agnostic; allows Honcho, Zep, Mem0 without code changes to harness |

| Budget gate | Session controller process; deletes transcript references after store() returns | Hard process-level enforcement; not bypassable |

| Evaluation harness | Python + asyncio probe runner | Parallelise probe evaluation; judge LLM calls are the bottleneck |

| Metric computation | Deterministic Python; no LLM calls except Action Alignment judge | All metrics except Action Alignment are reproducible |

| Result store | PostgreSQL; one row per (scenario, checkpoint, metric) | Enables aggregate reporting, ablation queries, longitudinal comparison |

9.2 Scenario Authoring

Scenarios are authored as YAML files and validated against the ScenarioSpec schema before generation. The full schema is defined in tests/bench/[schema.py](http://schema.py).

9.3 Adding a New Entity Type

1. Define a profile schema dataclass in tests/bench/[entities.py](http://entities.py)
2. Implement an event injection distribution in tests/bench/[events.py](http://events.py)
3. Define the ground truth representation and checkpointing logic in tests/bench/ground_[truth.py](http://truth.py)
4. Register the new type in the EntityFactory in tests/bench/world_[gen.py](http://gen.py)
5. Add any paradigm-specific probe types to tests/bench/[probes.py](http://probes.py)

---

10. Limitations and Open Problems

10.1 The Sim-to-Real Gap

DRIFT relies on synthetic transcript generation, introducing a sim-to-real gap. Mitigation strategies: fine-tuning the transcript synthesiser on real conversation corpora; injecting real fragments at low frequency; periodic validation studies comparing DRIFT scores to human-judged memory quality in deployed systems.

The sim-to-real gap is a known limitation, not a fatal one. Simulation-based benchmarks with this gap are standard in RL (sim-to-real transfer) and NLP (synthetic data augmentation). The benchmark’s value comes from the structural properties it enforces — temporal irreversibility, latent ground truth, contribution isolation — which hold regardless of transcript realism.

10.2 Judge Model Dependence

Action Alignment uses an LLM judge. DRIFT addresses this by specifying a fixed reference judge (a pinned model version) and releasing all judge prompts publicly so that score differences across judge versions can be audited.

10.3 Scenario Coverage

The benchmark’s quality is bounded by the diversity of its scenario library. The DRIFT specification includes a scenario diversity audit: for each new scenario added to the canonical suite, a diversity checker verifies it is not solvable by a strategy that already achieves high performance on existing scenarios.

10.4 Open Research Questions

• What is the right storage cost normalisation for the MVOT efficiency term? Token count is a proxy; the correct normalisation may depend on retrieval compute, not storage volume.
• How should the noise ratio be calibrated to match real deployment distributions? The 0.70 default is an estimate.
• Can belief revision be measured without an LLM judge for complex preference changes?
• How does multi-peer scope leakage interact with the DRIFT-C score? Systematic leakage may inflate recall scores in ways the current decomposition does not fully capture.
• What is the right temporal discount for the long-horizon weight?

---

11. Development Roadmap

| Phase | Deliverable | Target |

|---|---|---|

| v0.1 — Specification | This document. Scenario spec language, entity type system, probe suite, metric definitions, harness integration mapping. | Complete |

| v0.2 — Core harness | World generator for user-assistant paradigm. Budget gate. Recall and action probe runners. MVOT, Selectivity, TES, and LPS metrics. Token cost instrumentation middleware. | Q2 2026 |

| v0.3 — Baseline evaluation | Run v0.2 harness against Honcho, Zep, and Mem0 on 10 canonical user-assistant scenarios. Publish comparative results. | Q3 2026 |

| v0.4 — Agentic paradigms | Coding agent and crawler paradigm support. Frontier and consistency probe types. Revision Accuracy metric. | Q3 2026 |

| v0.5 — Multi-peer | Full scope model. Multi-peer topology support. Attribution, handoff, and canary probe types. Privacy Score metric. | Q4 2026 |

| v1.0 — Public release | Canonical scenario library (50+ scenarios across all paradigms). Public leaderboard. SDK for third-party memory system registration. | Q1 2027 |

11.1 Relationship to Honcho RL Training

DRIFT is developed in parallel with Honcho’s GRPO-based learned memory backend. The relationship is bidirectional:

• DRIFT informs Honcho’s reward design. The metric suite — particularly MVOT and Revision Accuracy — provides the reward signal structure for the GRPO inner loop. A memory system that maximises MVOT across DRIFT scenarios is, by construction, learning the right thing.
• Honcho’s training runs generate DRIFT data. Every GRPO training rollout is a DRIFT-compatible trajectory. The training infrastructure and the benchmarking infrastructure share the same session format, entity profile schema, and budget gate mechanism.
• DRIFT prevents Honcho from overfitting to its own training distribution. Scenarios from paradigms and topologies outside Honcho’s current training distribution provide an out-of-distribution test that keeps the research honest.