ED Deriver

ED Deriver & Dialectic – Architecture Cheat-Sheet

This document summarises the two main memory workflows in Honcho that involve the ED components:

• Ingestion → Deriver (observation extraction & storage)
• Evaluation → Dialectic (answer synthesis from stored observations)

---

1 · Ingestion + Deriver workflow

High-level steps

1. lab/evals/locomo/ingest.py (or API) calls create_and_process() for every message.
2. EdDeriver.process_message:
   1. summarize_if_needed may add short / long summaries for the session.
   2. If a working representation already exists on the target Peer, it is reused; otherwise EdEmbeddingStore.get_relevant_observations(for_reasoning=True) performs a global semantic-search to supply an initial context.
      → This fallback is only executed when no cached working representation is available.
   3. critical_analysis_call (Anthropic, configurable) returns a ReasoningResponse containing thinking, explicit and deductive observations.
   4. New observations are embedded via EdEmbeddingStore.save_unified_observations and stored as Document rows.
   5. A fresh working representation snapshot is cached under Peer.internal_metadata["latest_working_representation"].
3. The structured ReasoningResponseWithThinking is returned to the caller.

flowchart TD
  subgraph "Ingestion Workflow"
    A["ingest.py iterate dataset"] --> B["create_and_process()"]
    B --> C["EdDeriver.process_message"]
    C --> D["summarize_if_needed"]
    C --> E["EdEmbeddingStore.get_relevant_observations<br/>(if needed)"]
    C --> F["critical_analysis_call → LLM"]
    F --> G["ReasoningResponse"]
    G --> H["_save_new_observations"]
    H --> I["EdEmbeddingStore.save_unified_observations"]
    G --> J["save_working_representation_to_peer"]
  end

sequenceDiagram
  participant Ingest as ingest.py
  participant Deriver as EdDeriver
  participant EmbStore as EdEmbeddingStore
  participant LLM as critical_analysis_call
  participant DB as Postgres
  Ingest->>Deriver: create_and_process(payload)
  Deriver->>DB: summarize_if_needed()
  Deriver->>EmbStore: get_relevant_observations()
  Note right of Deriver: "*Only executed when no cached<br/>working representation exists*"
  EmbStore-->>Deriver: initial context (explicit+deductive)
  Deriver->>LLM: critical_analysis_call()
  LLM-->>Deriver: ReasoningResponse
  Deriver->>EmbStore: save_unified_observations(new obs)
  EmbStore->>DB: batch insert Documents
  Deriver->>DB: save_working_representation_to_peer()
  Deriver-->>Ingest: ReasoningResponseWithThinking

---

2 · Evaluation + Dialectic workflow

High-level steps

1. lab/evals/locomo/evaluate.py runs question_eval, which calls fetch_answer.
2. Dialectic.chat (lab/dialectic/ed.py):
   1. Retrieves the peer’s cached working representation.
   2. Generates semantic-search queries (Groq → Gemini → rule-based) and uses EdEmbeddingStore.get_relevant_observations to pull additional historical observations.
   3. Builds a prompt containing Query + Working-Rep + Additional Context and calls dialectic_call (Anthropic).
   4. Returns the synthesised answer (string or stream).
3. evaluate.py scores the answer (rule-based F1, LLM-judge, Rumsfeld, judge-evaluator).

flowchart TD
  subgraph "Evaluation Workflow"
    A["evaluate.py question_eval"] --> B["fetch_answer()"]
    B --> C["Dialectic.chat"]
    C --> D["get_latest_working_representation"]
    C --> E["EdEmbeddingStore.get_relevant_observations"]
    D --> F["dialectic_call → LLM"]
    E --> F
    F --> G["answer returned"]
    G --> H["Scoring (rule-based / LLM judge / etc.)"]
  end

sequenceDiagram
  participant Eval as evaluate.py
  participant Dialectic as Dialectic.chat
  participant EmbStore as EdEmbeddingStore
  participant LLM as dialectic_call
  participant DB as Postgres
  Eval->>Dialectic: fetch_answer(query)
  Dialectic->>DB: get_latest_working_representation()
  Dialectic->>EmbStore: get_relevant_observations(search queries)
  EmbStore-->>Dialectic: historical observations
  Dialectic->>LLM: dialectic_call(prompt)
  LLM-->>Dialectic: synthesised answer
  Dialectic-->>Eval: answer string
  Eval->>Eval: scoring (F1 / LLM-judge / etc.)

---

Model & storage notes

• Embeddings: OpenAI text-embedding-3-small (batched).
• Deriver LLM: settings.LAB.deriver_model (Anthropic by default).
• Dialectic LLM: settings.LAB.dialectic_model.
• Observations stored in Document.embedding table along with metadata level, premises, message_id, session_name, etc.

---

Last updated: DEV-976