Plastic Labs Internal Wiki

Recently Updated

Oracles in Excadrill

10 min read

Oracles in Excadrill

This note describes the use of oracles in Excadrill: which are available and why, and how their data is generated. They are used to attribute performance among deriver, dreamer, dialectic and other pipeline stages.

System view

In System Overview, we have a description of the Honcho compound AI/ML system. We can describe the ML-based stages as:

  • Deriver
  • Dreamer
    • Retrieval: used in a tool call with a query.
  • Dialectic
    • Retrieval: used in a tool call with a query.
  • session/user.context
    • Largely deterministic but does do retrieval.
    • Retrieval is not based on a tool call here.
  • Summarization

An oracle is just something that has golden data that replaces a model call with this golden data. So, for the components above, it would be:

  • Deriver: explicit conclusions.
  • Dreamer: deductive and inductive conclusions, with provenance.
  • Dialectic: final evidence or the answer, depending on the meaning here.
  • Retrieval:
    • If no query, this is just a golden list of final evidence conclusions.
    • If has a query, no obvious way to oracularize. Could replace the semantic similarity search with a powerful LLM call tasked with retrieval.
  • Summarization is currently untouched.

Different oracle seams within the system

A deriver oracle is just something that contains golden explicit conclusions. It has a single data point per deriver run (typically either a user message or an entire conversational context) that consists of a list of strings.

A dreamer oracle is similar to the deriver except that:

  • It has to have a single data point for every dreamer run. In typical benchmarks this is a single data point that occurs at the end of the test case context ingestion.
  • It includes deductive and inductive conclusions with provenance. Note that, if the dreamer is replaced with an oracle but the deriver is not, deriver conclusions will typically not influence the results of the dreamer but will be accumulated as explicit conclusions. This may cause some data drift where the oracular higher-order conclusions do not follow from, or ignore some, explicit first-order conclusions.

A dialectic oracle can mean different things:

  • Replacing the output of the dialectic with an oracle. This will typically replace the entire Honcho part on a benchmark, meaning that this is becomes a sanity check of the benchmark: will the judge score a correct answer as correct?
  • Inject golden conclusions into the dialectic inputs.

Test-time oracle design and how it integrates into each seam

Oracles are currently implemented for BEAM, LoCoMo and LongMemEval. The structure of these benchmarks allows us to simplify the oracle design - we only need to prepare one set of oracular conclusions for each eval.

  • Deriver: list of explicit conclusions.
  • Dreamer: list of deductive and inductive conclusions, each conclusion with provenance backlinks.
  • Dialectic: list of final answers.

The non-obvious seam is the dialectic oracle’s second seam - injecting golden conclusions. In this structure, this is almost the same as both deriver and dreamer oracles. These two oracles fix the set of conclusions eligible for the dialectic to see. As long as the number of max_observations in the retrieval step is high enough, which it currently is for these 3 benchmarks, the dialectic is guaranteed to see them all.

Oracle data construction process

While we could, of course, human-generate the oracle data by hand, this is not scalable and quite error-prone. Instead we use an LLM - Opus 4.8 currently - with privileged knowledge about the benchmark that the Honcho models do not see. To wit, the oracle sees the context, questions, answers, supporting evidence and judging rubric if present.

The oracle LLM is asked to produce the following schema:

{
    "extracted_facts": list[{
        "id": str,          # short local id, e.g. "e0", "e1", ...
        "content": str,
    }],
    "derived_facts": {
        "deductive": list[{
            "conclusion": str,
            "provenance": list[str],   # ids of the extracted_facts it follows from
        }],
        "inductive": list[{
            "conclusion": str,
            "provenance": list[str],   # ids of the >=2 facts exhibiting the pattern
        }]
    }
}

Each extracted fact carries its own id in the model’s output so that each conclusion can name, in its provenance, the specific facts it draws from. Opus is fine at this: reasonable ids, no duplicates, sensible per-conclusion attribution. This keeps the whole memory in a single LLM call — no need to split extraction and derivation into separate passes. To keep the scheme simple, provenance may only reference extracted facts (not other conclusions).

Note that the schema above is the LLM output contract, not what is stored. When we map the model’s output onto Honcho observations, the model’s local fact ids are resolved and re-minted, so the stored sidecar differs in two ways:

  • The stored explicit observations are plain {"content", "level", ...} rows with no id field.
  • Each conclusion’s source_ids are strings of the form <item-id>-e<index>, combining the benchmark item id (the top-level key of the sidecar — a BEAM conversation id, a LoCoMo <sample_id>-<q_idx>, or a LongMemEval question_id) with the position of the supporting fact in that group’s explicit list (e0, e1, …; distractor groups in LongMemEval use d0, d1, …). The index is positional, the explicit rows carry no id of their own.

The referenced fact text is also copied into Honcho’s premises (deductive) / sources (inductive) fields. That premises/sources text is what the Honcho working representation actually renders under each conclusion at answer time — i.e. the provenance the dialectic reads on every retrieval (see DeductiveObservation.__str__). source_ids are stored but not rendered in the representation, so the dialectic never sees them directly; the get_reasoning_chain tool surfaces nothing. However, the dialectic always sees the premises and sources fields, so it sees provenance every time without having to call get_reasoning_chain.

The oracle prompts are constructed by borrowing from the deriver and dreamer prompts.

  • The extracted_facts are instructed to be decontextualized and minimal.
  • The deductive and inductive derived_facts keys are given instructions from the specialist dreamer agent prompts.

Every benchmark has slightly different ways of identifying golds, documented below. However, because the golds are specific to each question within a conversation in a benchmark (there are multiple questions per conversation), if we simply serve up the golds we are leaking answers into the test data. So, in addition to golds, we also produce distractors - valid facts that are not relevant to the current query.

BEAM

  • Gold: the LLM-as-judge grading rubric grounds which facts are needed, also conversation references exist for everything except abstention.
  • Distractors: all oracular conclusions for the entire conversation

Sample data:

{
    "content": "Enhanced documentation by structuring API endpoint details and architecture decisions in Confluence, incorporating tables and diagrams for collaboration and feedback.",
    "level": "explicit"
},
{
    "content": "There is a contradiction: the user claims to have never written Flask routes or handled HTTP requests, yet also states implementing a basic homepage route with Flask.",
    "level": "deductive",
    "source_ids": [
        "1-e13",
        "1-e14"
    ],
    "premises": [
        "Stated at one point that they have never written any Flask routes or handled HTTP requests in this project.",
        "Also stated implementing a basic homepage route with Flask."
    ]
},
{
    "content": "Follows a structured, phased development approach progressing from foundational features to security hardening and documentation.",
    "level": "inductive",
    "source_ids": [
        "1-e1",
        "1-e4",
        "1-e28",
        "1-e29"
    ],
    "pattern_type": "tendency",
    "confidence": "medium",
    "sources": [
        "The budget tracker's core functionality includes user authentication, expense tracking, and data visualization.",
        "Worked on enhancing security measures and improving authentication and authorization before deployment.",
        "Created a project schedule targeting delivery of the MVP by April 15, 2024, with phases covering authentication, transaction management, analytics, and deployment.",
        "Enhanced documentation by structuring API endpoint details and architecture decisions in Confluence, incorporating tables and diagrams for collaboration and feedback."
    ]
},

LoCoMo

  • Gold: the evidence key gives direct conversation sourcing.
  • Distractors: all oracular conclusions for the entire conversation.

Sample data:

{
    "content": "Caroline wants to support people with issues similar to her own.",
    "level": "explicit",
    "observer": "Caroline",
    "observed": "Caroline"
},
{
    "content": "Caroline would likely pursue education in psychology and counseling certification.",
    "level": "deductive",
    "source_ids": [
        "conv-26-2-e0",
        "conv-26-2-e1"
    ],
    "observer": "Caroline",
    "observed": "Caroline",
    "premises": [
        "Caroline plans to continue her education and explore career options.",
        "Caroline is keen on counseling or working in mental health."
    ]
},
{
    "content": "Caroline is motivated by a desire to help others facing struggles similar to hers, aligning her toward mental health fields.",
    "level": "inductive",
    "source_ids": [
        "conv-26-2-e1",
        "conv-26-2-e2"
    ],
    "pattern_type": "tendency",
    "confidence": "low",
    "observer": "Caroline",
    "observed": "Caroline",
    "sources": [
        "Caroline is keen on counseling or working in mental health.",
        "Caroline wants to support people with issues similar to her own."
    ]
},

The source_ids here are {conversation_id}-{question_id}-{explicit_fact}, same as in the benchmark. They’re not resolvable in this form to explicit conclusion provenance. Recall that they are currently not used.

LongMemEval

  • Gold:
    • Non-abstention: haystack_session_ids identified as answer_session_ids with turn-level has_answer marked when rendered for the oracle LLM;
    • Abstention: benchmark-provided “near-miss” answer_session_ids and a separate prompt to the oracle that these are abstention near-misses. Deductive and inductive conclusions are always excluded.
  • Distractors: randomly sampled turns from session_ids outside the answer_session_ids golds.

Sample data:

{
    "content": "Wants to focus on scene study and character development in acting classes.",
    "level": "explicit",
    "origin": "gold"
},
{
    "content": "The play attended at the local community theater was The Glass Menagerie.",
    "level": "deductive",
    "source_ids": [
        "58bf7951-e0",
        "58bf7951-e1"
    ],
    "origin": "gold",
    "premises": [
        "Attended a play at the local community theater.",
        "The play attended was a production of The Glass Menagerie."
    ]
},
{
    "content": "Has a strong personal interest and involvement in acting and theater.",
    "level": "inductive",
    "source_ids": [
        "58bf7951-e0",
        "58bf7951-e5",
        "58bf7951-e6",
        "58bf7951-e7"
    ],
    "pattern_type": "tendency",
    "confidence": "medium",
    "origin": "gold",
    "sources": [
        "Attended a play at the local community theater.",
        "Considering taking acting classes to improve acting skills.",
        "Recently auditioned for a role in 'The Crucible' and the audition did not go well.",
        "Wants to focus on scene study and character development in acting classes."
    ]
},

Beamer slides

How to run

Oracle data generation

These are smoke tests, remove the count restrictions to run full benchmarks.

Beam

uv run python scripts/populate_beam_observations.py --context-length 100K --limit 1 --output /tmp/beam_oracle_examples.json

LoCoMo

uv run python scripts/populate_locomo_observations.py --data-file data/locomo/v1/locomo10.json --test-count 1 --question-count 12 --output /tmp/locomo_oracle_examples.json

LongMemEval

uv run python scripts/populate_longmemeval_observations.py --test-file data/longmemeval/v1/longmemeval_s.json --limit 5 --distractors --distractor-ratio 1.0 --seed 0 --output /tmp/lme_oracle_examples.json

Benchmarks

These are smoke tests, remove the count restrictions to run full benchmarks.

These do NOT clear your workspace. If you ran an oracle and a non-oracle run against your main Honcho fleet (as opposed to the temporary Honcho that Excadrill spins up), you will cross-contaminate. Either delete your documents table or use --cleanup-workspace.

Beam

uv run python -m evals.offline --benchmark beam \
  --context-length 100K --conversation-ids 1 \
  --oracle-server-stage deriver --oracle-server-stage dreamer \
  --oracle-observations data/oracle-observations/beam/observations_100K.json \
  --skip-reasoning --pool-size 1 --batch-size 1 \
  --json-output /tmp/beam_oracle_run.json

LoCoMo

uv run python -m evals.offline --benchmark locomo \
  --data-file data/locomo/v1/locomo10.json \
  --sample-id conv-26 \
  --oracle-server-stage deriver --oracle-server-stage dreamer \
  --oracle-observations data/oracle-observations/locomo/observations_locomo10.json \
  --question-count 3 \
  --pool-size 1 --batch-size 1 \
  --json-output /tmp/locomo_oracle_run.json

LongMemEval

uv run python -m evals.offline --benchmark longmemeval \
  --test-file data/longmemeval/v1/longmemeval_s.json \
  --test-count 1 \
  --oracle-server-stage deriver --oracle-server-stage dreamer \
  --oracle-observations data/oracle-observations/longmem/observations_longmemeval_s.json \
  --skip-reasoning --pool-size 1 --batch-size 1 \
  --json-output /tmp/longmemeval_oracle_run.json

Graph View