llm-client-refactor.v1

LiteLLM Migration Spec

This document captures the design decisions and implementation plan for replacing honcho_llm_call_inner in src/utils/clients.py with litellm as the provider abstraction layer.

Scope

Replace: honcho_llm_call_inner() (~1000 lines of provider-specific dispatch code) Keep: All application-level orchestration — tool execution loop, message truncation, streaming metadata, telemetry, retry/fallback logic

The goal is to use litellm as a Python SDK library (not the proxy server) to handle provider-specific message formatting, tool conversion, structured output, streaming, and cache control.

Current Architecture

honcho_llm_call()                    # Public API (~375 lines)
  ├─ _execute_tool_loop()            # Tool orchestration (~380 lines) — KEEP
  │    └─ honcho_llm_call_inner()    # Provider dispatch (~715 lines) — REPLACE
  ├─ _stream_final_response()        # Stream after tools (~65 lines) — KEEP (calls inner)
  ├─ truncate_messages_to_fit()      # Message truncation (~110 lines) — KEEP
  ├─ _format_assistant_tool_message() # Multi-turn formatting (~100 lines) — SIMPLIFY
  ├─ _append_tool_results()          # Tool result formatting (~60 lines) — SIMPLIFY
  └─ convert_tools_for_provider()    # Tool format conversion (~50 lines) — REMOVE

What litellm replaces in honcho_llm_call_inner

• Provider client initialization and management (CLIENTS dict, ~70 lines)
• Anthropic system message extraction and cache_control injection
• Tool format conversion per provider (convert_tools_for_provider)
• Structured output / response_model handling per provider (Anthropic JSON prefill, OpenAI .parse(), Gemini response_schema)
• Streaming per provider (4 different implementations)
• Thinking/reasoning parameter handling (Anthropic thinking, OpenAI reasoning_effort/verbosity)
• Token usage extraction including cache tokens
• Provider-specific response parsing

What stays as application code

• _execute_tool_loop() — iterative tool calling with executor callback
• truncate_messages_to_fit() — unit-based message truncation preserving tool pairs
• _format_assistant_tool_message() — can simplify to one format (litellm normalizes)
• _append_tool_results() — can simplify to one format
• HonchoLLMCallResponse — unified response type with cache tokens, thinking, iterations
• StreamingResponseWithMetadata — stream wrapper with tool-loop metadata
• IterationData callback — per-iteration telemetry hook
• Sentry @ai_track integration
• vLLM PromptRepresentation JSON repair shim (post-processing wrapper)
• Backup provider failover logic (adapt to use litellm fallbacks)

Config Changes

Design Principle: ModelConfig as the Universal Building Block

Every component that calls an LLM (deriver, summary, dream, dialectic levels) uses the same reusable config primitive. This replaces the current PROVIDER + MODEL + BACKUP_PROVIDER + BACKUP_MODEL pattern and the BackupLLMSettingsMixin.

class ModelConfig(BaseModel):
    """Reusable model configuration for any LLM-calling component.
 
    By default, credentials are auto-resolved from LLMSettings based on the
    litellm model prefix (e.g. "anthropic/" → ANTHROPIC_API_KEY). Setting
    api_key and/or base_url on an individual config overrides this and routes
    the call through an OpenAI-compatible endpoint.
    """
 
    model: str                          # litellm model string: "anthropic/claude-haiku-4-5"
    fallback_model: str | None = None
 
    # Per-slot credential overrides — when set, uses OpenAI-compatible format.
    # When unset, auto-resolves from LLMSettings based on model prefix.
    api_key: str | None = None
    base_url: str | None = None
 
    # Sampling parameters — None means "use provider default"
    temperature: float | None = None
    top_p: float | None = None
    top_k: int | None = None
 
    # Reasoning / thinking
    thinking_budget_tokens: int | None = None
    max_output_tokens: int | None = None

Key properties:

• Auto-routing by default: When api_key / base_url are None, the credential resolver falls back to LLMSettings global keys based on model prefix.
• Explicit override for custom endpoints: Setting api_key + base_url on any slot routes that call through an OpenAI-compatible endpoint, regardless of prefix. This makes it trivial to point any component at a private vLLM instance, a custom proxy, or any OpenAI-compatible API.
• Sampling params are first-class: temperature, top_p, top_k are universal enough to be explicit fields rather than buried in an extra_body dict. litellm passes them through natively to every provider.
• Flat reasoning config: thinking_budget_tokens lives directly on ModelConfig rather than in a nested ReasoningConfig. The request builder translates it to the right provider parameter (Anthropic thinking.budget_tokens, OpenAI reasoning_effort, etc.). Adding a second indirection layer for effort/verbosity/budget_tokens doesn’t pay for itself when each is really a different knob.

Model Specification

Merge PROVIDER + MODEL into a single litellm model string:

# Before
PROVIDER: SupportedProviders = "anthropic"
MODEL: str = "claude-haiku-4-5"
BACKUP_PROVIDER: SupportedProviders | None = None
BACKUP_MODEL: str | None = None
 
# After — each component embeds or extends ModelConfig
model: str = "anthropic/claude-haiku-4-5"
fallback_model: str | None = None

Types to Remove

• SupportedProviders literal type in src/utils/types.py
• LLMComponentSettings protocol (replaced by ModelConfig)
• BackupLLMSettingsMixin (replaced by fallback_model on ModelConfig)
• Provider validation in clients.py (SELECTED_PROVIDERS, BACKUP_PROVIDERS)

Settings Classes to Update

Class	Current Fields	After
DeriverSettings	PROVIDER, MODEL, BACKUP_PROVIDER, BACKUP_MODEL	Embeds ModelConfig fields (model, fallback_model, api_key, base_url, sampling, thinking)
SummarySettings	PROVIDER, MODEL, BACKUP_PROVIDER, BACKUP_MODEL	Same
DreamSettings	PROVIDER, MODEL, BACKUP_PROVIDER, BACKUP_MODEL, DEDUCTION_MODEL, INDUCTION_MODEL	Same, plus deduction_model and induction_model as full litellm strings
DialecticLevelSettings	PROVIDER, MODEL, BACKUP_PROVIDER, BACKUP_MODEL	Same

Dream Specialist Model Change

Currently DEDUCTION_MODEL and INDUCTION_MODEL inherit the parent PROVIDER implicitly (specialists.py:199 does settings.DREAM.model_copy(update={"MODEL": model})). With litellm, specialist models become fully qualified: deduction_model: str = "anthropic/claude-haiku-4-5".

config.toml Examples

# Before
[deriver]
provider = "google"
model = "gemini-2.5-flash-lite"
 
[dream]
provider = "anthropic"
model = "claude-sonnet-4-20250514"
deduction_model = "claude-haiku-4-5"
 
[dialectic.levels.minimal]
provider = "google"
model = "gemini-2.5-flash-lite"
 
[dialectic.levels.high]
provider = "anthropic"
model = "claude-haiku-4-5"
 
# After — standard providers (auto-resolved from LLMSettings)
[deriver]
model = "gemini/gemini-2.5-flash-lite"
thinking_budget_tokens = 1024
temperature = 0.0
 
[dream]
model = "anthropic/claude-sonnet-4-20250514"
thinking_budget_tokens = 8192
deduction_model = "anthropic/claude-haiku-4-5"
induction_model = "anthropic/claude-haiku-4-5"
 
[summary]
model = "gemini/gemini-2.5-flash"
 
[dialectic.levels.high]
model = "anthropic/claude-haiku-4-5"
thinking_budget_tokens = 1024
max_tool_iterations = 4
 
# After — custom OpenAI-compatible endpoint (per-slot override)
[dialectic.levels.minimal]
model = "openai/my-local-model"
base_url = "http://localhost:8000/v1"
api_key = "none"
temperature = 0.2
thinking_budget_tokens = 0
max_tool_iterations = 1
max_output_tokens = 250

API Key Management

litellm is used as a library — no proxy server. Keys are passed explicitly per call, not via litellm globals or environment variables.

Key Resolution

A two-tier resolver checks the per-slot config first, then falls back to global LLMSettings keys based on model prefix:

def resolve_credentials(config: ModelConfig) -> dict[str, str | None]:
    """Per-slot overrides take priority, then fall back to LLMSettings."""
    if config.api_key is not None or config.base_url is not None:
        # Explicit override — route as OpenAI-compatible
        return {"api_key": config.api_key, "api_base": config.base_url}
 
    # Auto-resolve from LLMSettings based on model prefix
    return _resolve_from_global_settings(config.model)
 
 
def _resolve_from_global_settings(model: str) -> dict[str, str | None]:
    """Map model prefix to credentials from the global LLMSettings store."""
    if model.startswith("anthropic/"):
        return {"api_key": settings.LLM.ANTHROPIC_API_KEY}
    elif model.startswith("openai/"):
        return {"api_key": settings.LLM.OPENAI_API_KEY}
    elif model.startswith("openrouter/"):
        return {"api_key": settings.LLM.OPENAI_COMPATIBLE_API_KEY}
    elif model.startswith("hosted_vllm/"):
        return {
            "api_key": settings.LLM.VLLM_API_KEY,
            "api_base": settings.LLM.VLLM_BASE_URL,
        }
    elif model.startswith("gemini/"):
        return {"api_key": settings.LLM.GEMINI_API_KEY}
    elif model.startswith("groq/"):
        return {"api_key": settings.LLM.GROQ_API_KEY}
    return {}

Custom Endpoints

With per-slot api_key + base_url, any component can target a custom endpoint without touching the global key store:

• Private vLLM / local models: model = "openai/my-model", base_url = "http://localhost:8000/v1", api_key = "none"
• OpenRouter: model = "openrouter/model-name" (litellm knows the base URL), or model = "openai/model-name" + explicit base_url
• Custom proxy or gateway: model = "openai/whatever" + base_url + api_key
• Standard providers: Just use the prefix (anthropic/, gemini/, etc.) — no api_key/base_url needed

LLMSettings Changes

LLMSettings stays as the global key store (fallback when per-slot overrides are not set). The EMBEDDING_PROVIDER field becomes EMBEDDING_MODEL:

class LLMSettings(HonchoSettings):
    # API Keys — unchanged
    ANTHROPIC_API_KEY: str | None = None
    OPENAI_API_KEY: str | None = None
    OPENAI_COMPATIBLE_API_KEY: str | None = None
    GEMINI_API_KEY: str | None = None
    GROQ_API_KEY: str | None = None
    VLLM_API_KEY: str | None = None
    VLLM_BASE_URL: str | None = None
    OPENAI_COMPATIBLE_BASE_URL: str | None = None
 
    # Embedding — simplified
    EMBEDDING_MODEL: str = "openai/text-embedding-3-small"
 
    # General
    DEFAULT_MAX_TOKENS: int = 2500
    MAX_TOOL_OUTPUT_CHARS: int = 10000
    MAX_MESSAGE_CONTENT_CHARS: int = 2000

Embedding Client

litellm supports embeddings via litellm.aembedding() with the same model string pattern. However, the ROI is lower — the provider dispatch code is ~60 lines while the batching/chunking infrastructure (~200 lines) stays regardless.

If adopting litellm for embeddings:

# Before (two code paths: genai.Client vs AsyncOpenAI)
if isinstance(self.client, genai.Client):
    response = await self.client.aio.models.embed_content(...)
else:
    response = await self.client.embeddings.create(...)
 
# After (one code path)
response = await litellm.aembedding(model=self.model, input=[query], **credentials)

Gemini output_dimensionality

Current code passes config={"output_dimensionality": 1536} to Gemini. Verify litellm maps the standard dimensions parameter to Gemini’s output_dimensionality.

Prompt Caching Improvements

litellm enables prompt caching optimizations that are currently harder with the custom client:

• cache_control passthrough: litellm passes cache_control blocks on messages natively
• Auto-injection: litellm.acompletion(..., cache_control_injection_points=[...]) can auto-place cache breakpoints
• Tool caching: Can add cache_control to the last tool definition for Anthropic (tools come before system in Anthropic’s cache hierarchy: tools → system → messages)

Current Caching Issues to Address

1. Deriver has no system message — instructions are concatenated with dynamic content into a user message. Split into system (static, cacheable) + user (dynamic).
2. Dialectic mixes static/dynamic in system prompt — peer cards and session history are appended to the system prompt. Split into: static instructions (first system message, cacheable) → dynamic context (second system message or user message).
3. Dreamer specialists same pattern — static agent instructions should be separated from dynamic peer card/hint content.

Risk Areas and Validation

Critical: Thinking Block Signatures (Anthropic)

The tool loop preserves ThinkingBlock.signature for multi-turn replay (clients.py:1772-1779). Anthropic requires signatures to be replayed in subsequent turns when using extended thinking.

Validation: Write a test that runs a multi-turn tool loop with thinking enabled through litellm and verify signatures round-trip correctly in the conversation history.

Critical: Gemini thought_signature

Similar to above — thought_signature in Gemini function call parts must be preserved for multi-turn (clients.py:2183-2190).

Validation: Test multi-turn tool loop on Gemini through litellm.

Medium: vLLM JSON Repair

The PromptRepresentation-specific JSON repair shim catches malformed JSON from vLLM. This becomes a post-processing wrapper around the litellm response:

response = await litellm.acompletion(...)
if provider == "hosted_vllm" and response_model:
    content = validate_and_repair_json(response.choices[0].message.content)
    # ... schema-aware repair for PromptRepresentation

Medium: Cache Token Extraction

HonchoLLMCallResponse tracks cache_read_input_tokens and cache_creation_input_tokens. Verify litellm exposes both fields consistently in response.usage across Anthropic, OpenAI, and OpenRouter.

Medium: Backup Provider Parameter Filtering

Current code strips thinking_budget_tokens when falling back to non-Anthropic, and reasoning_effort/verbosity when falling back to non-GPT-5. If using litellm’s fallbacks, verify it handles incompatible parameters gracefully or implement filtering in the wrapper.

Low: Anthropic response_model Pre-fill

Current code appends {"role": "assistant", "content": "{"} to force JSON. litellm handles structured output differently. Behavioral change to validate but likely an improvement.

Implementation Order

1. Add litellm dependency (uv add litellm)
2. Introduce ModelConfig base model in src/config.py
3. Create resolve_credentials() — two-tier resolver (per-slot override → global LLMSettings)
4. Replace honcho_llm_call_inner() with a wrapper around litellm.acompletion() / streaming, accepting ModelConfig
5. Remove convert_tools_for_provider() — litellm handles tool format conversion
6. Simplify _format_assistant_tool_message() — use OpenAI format only, litellm normalizes
7. Simplify _append_tool_results() — use OpenAI format only
8. Remove CLIENTS dict and provider initialization code
9. Validate thinking block round-trips on Anthropic and Gemini with multi-turn tool loops
10. Add vLLM JSON repair as post-processing wrapper
11. Update settings classes: Replace PROVIDER+MODEL+BACKUP_* with ModelConfig fields across DeriverSettings, SummarySettings, DreamSettings, DialecticLevelSettings
12. Remove SupportedProviders type, LLMComponentSettings protocol, BackupLLMSettingsMixin, provider validation
13. Update config.toml and environment variable documentation
14. Optionally: migrate embedding_client.py to use litellm.aembedding()

Files to Modify

File	Change
src/utils/clients.py	Replace honcho_llm_call_inner, remove provider dispatch, add litellm wrapper
src/utils/types.py	Remove SupportedProviders
src/config.py	Merge PROVIDER+MODEL fields, remove BackupLLMSettingsMixin
src/embedding_client.py	Optional: replace provider branches with litellm.aembedding
src/deriver/deriver.py	No change (calls honcho_llm_call)
src/dialectic/core.py	No change (calls honcho_llm_call)
src/dreamer/specialists.py	Update model_copy to not need provider injection
config.toml	Update model format to litellm strings
pyproject.toml	Add litellm dependency

Config Layer Design

Why Honcho keeps its own config layer above litellm

LiteLLM handles provider routing and request translation, but Honcho still needs a stable configuration surface that is decoupled from litellm’s parameter names:

• Provider capabilities differ meaningfully — ModelConfig gives Honcho a stable place to express fallback models, thinking budgets, and sampling params without coupling to litellm internals.
• Future migrations are cheaper if we need to swap transports or use native provider APIs for specific features.
• ModelConfig is the single reusable primitive shared across deriver, summary, dream, and dialectic. The request builder translates it into litellm calls at runtime.

Validation strategy

Move provider-specific validation out of config.py and into the request builder.

• Keep config validation structural: required fields, positive budgets, valid ranges.
• Do provider/model-specific filtering at runtime based on the selected model and litellm capabilities (e.g. strip thinking_budget_tokens for providers that don’t support it, translate it to reasoning_effort for OpenAI models).
• Log when a setting is ignored because the target model does not support it.

This is preferable to encoding provider assumptions directly in Pydantic validators, especially as providers shift from explicit budgets to effort-based reasoning or hybrid modes.

Fallback handling

Keep Honcho-managed fallback selection in the wrapper for the first migration instead of delegating directly to litellm fallbacks.

Reasons:

• Honcho currently filters incompatible reasoning parameters on fallback.
• Telemetry and retry behavior are already implemented at the application layer.
• Provider-specific request normalization may still need to differ between primary and fallback models.

litellm fallbacks can be revisited later once parameter filtering and telemetry requirements are proven compatible.

End state

• One shared ModelConfig primitive used by all LLM-calling components
• One request builder that maps ModelConfig → litellm acompletion() kwargs
• LLMSettings as the global credential store (fallback for per-slot overrides)
• litellm as the transport/provider abstraction, not the config schema