LLM Client Refactor

Status: Shipped | Owner: vineeth | Moved to done: 2026-05-08 History: llm-client-refactor.v1.md (Jan 2025 — original “use litellm as transport” plan)

This page describes what actually shipped. The earlier proposal used the litellm Python library as the unified transport behind a ProviderBackend abstraction. We rejected litellm during implementation and kept the abstraction with native SDKs underneath. See “Why litellm was rejected” below.

What shipped

The monolithic src/utils/clients.py (~2575 lines, ~715 of them in honcho_llm_call_inner) is gone. Non-embedding LLM orchestration now lives in a structured src/llm/ package whose public surface is honcho_llm_call.

src/llm/
├── __init__.py          # Public re-exports
├── api.py               # honcho_llm_call (public entrypoint)
├── backend.py           # ProviderBackend Protocol + CompletionResult / StreamChunk / ToolCallResult
├── backends/
│   ├── anthropic.py     # AnthropicBackend — native Anthropic SDK
│   ├── openai.py        # OpenAIBackend — native OpenAI SDK (also fronts OpenAI-compatible endpoints)
│   └── gemini.py        # GeminiBackend — native google-genai SDK
├── caching.py           # Provider-specific cache-control handling
├── conversation.py      # Conversation-level helpers
├── credentials.py       # default_transport_api_key, resolve_credentials
├── executor.py          # honcho_llm_call_inner — single-call path used by api + tool_loop
├── history_adapters.py  # AnthropicHistoryAdapter / GeminiHistoryAdapter / OpenAIHistoryAdapter
├── registry.py          # CLIENTS dict + factories: client_for_model_config, get_backend, …
├── request_builder.py   # execute_completion / execute_stream
├── runtime.py           # AttemptPlan, plan_attempt, effective_config_for_call, retry helpers
├── structured_output.py # response_model handling (Pydantic) + JSON repair
├── tool_loop.py         # execute_tool_loop (tool execution orchestration)
└── types.py             # HonchoLLMCallResponse, HonchoLLMCallStreamChunk, IterationData, …

Total: ~3.8K lines across 14 files (vs. one ~2.6K-line file).

Why litellm was rejected

The v2 proposal was “ProviderBackend Protocol with litellm as initial transport.” During the compatibility spike we kept hitting cases where the behavior we needed wasn’t directly exposed through litellm.acompletion() without dropping into provider-native code anyway. The four sticking points:

1. Anthropic thinking_blocks with signatures — preserving the raw thinking blocks (so we can replay them on subsequent turns) requires reading fields off the Anthropic SDK’s response objects directly. Going through litellm meant either monkey-patching its response normalization or attaching the raw response and re-parsing — both worse than just using the Anthropic SDK.
2. Gemini thought_signature — same problem in the other direction. Gemini tool calls need their thought_signature round-tripped through the conversation, and litellm’s tool-call normalization didn’t expose it stably across versions.
3. Gemini cached-content — explicit cachedContent references are a first-class feature in google-genai. Going through litellm’s prefix-cache path meant losing the explicit handle.
4. Provider-aware replay — Anthropic and Gemini need different conversation-history shapes (assistant tool-use blocks vs. function responses). Litellm normalizes inputs but not outputs in a way that preserved the round-trip. We ended up needing per-provider history adapters either way (history_adapters.py).

Once we had per-provider thinking handling, per-provider tool-call preservation, and per-provider history adapters, litellm was carrying ~one method of value (acompletion) at the cost of a heavy dependency and another normalization layer to debug. The native SDKs were already in the lockfile.

The ProviderBackend Protocol is the same shape that was originally going to wrap litellm — it just wraps the three native SDKs instead.

Architecture

ProviderBackend Protocol

src/llm/backend.py defines the transport-agnostic contract:

@runtime_checkable
class ProviderBackend(Protocol):
    async def complete(
        self,
        *,
        model: str,
        messages: list[dict[str, Any]],
        max_tokens: int,
        temperature: float | None = None,
        stop: list[str] | None = None,
        tools: list[dict[str, Any]] | None = None,
        tool_choice: str | dict[str, Any] | None = None,
        response_format: type[BaseModel] | dict[str, Any] | None = None,
        thinking_budget_tokens: int | None = None,
        thinking_effort: str | None = None,
        max_output_tokens: int | None = None,
        extra_params: dict[str, Any] | None = None,
    ) -> CompletionResult: ...
 
    def stream(self, *, ...) -> AsyncIterator[StreamChunk]: ...

CompletionResult, StreamChunk, and ToolCallResult are dataclasses that normalize across providers — token counts, finish reason, tool calls, thinking content, thinking blocks, reasoning details, plus a raw_response escape hatch.

Three concrete backends (AnthropicBackend, OpenAIBackend, GeminiBackend) each wrap a single native SDK client. They translate incoming messages / tools / params into provider-shaped requests and the response back into CompletionResult / StreamChunk.

Registry — the single owner of runtime objects

src/llm/registry.py is the only place that knows how to construct SDK clients, wrap them in backends, and pick history adapters:

• CLIENTS: dict[ModelTransport, ProviderClient] — module-level default clients populated at import time from settings.LLM.*_API_KEY. Tests patch this dict to inject mocks (patch.dict(CLIENTS, {...})).
• get_anthropic_client() / get_openai_client() / get_gemini_client() — lru_cache(maxsize=1) factories for the default clients.
• get_*_override_client(base_url, api_key) — lru_cache(maxsize=128) factories for per-call overrides (e.g., custom OpenAI-compatible endpoints).
• client_for_model_config(provider, model_config) -> ProviderClient — fast-path returns the default client; slow-path validates credentials and routes through the override factory.
• backend_for_provider(provider, client) -> ProviderBackend — wraps a raw client in the matching backend.
• history_adapter_for_provider(provider) -> HistoryAdapter — picks the right history adapter for assistant/tool message formatting.
• get_backend(config: ModelConfig) -> ProviderBackend — high-level one-shot factory; both production (honcho_llm_call_inner) and the live-test path go through this so credential validation stays consistent.

ModelConfig

ModelConfig lives in src/config.py (not in src/llm/) and is the unified config primitive that replaced the old PROVIDER / MODEL / BACKUP_PROVIDER / BACKUP_MODEL quadruple. Every non-embedding LLM-calling component now takes a ModelConfig. Backup configs are wired in at the same level — runtime.plan_attempt() plans primary vs. fallback per-attempt.

ModelTransport is the transport tag ("anthropic" | "openai" | "gemini"). Groq and the old vLLM branch are gone — Groq lives behind the OpenAI backend via the OpenAI-compatible endpoint path.

History adapters

history_adapters.py defines a HistoryAdapter Protocol with three implementations. The provider-aware replay format from the old clients.py is preserved:

• AnthropicHistoryAdapter — preserves thinking_blocks with signatures on assistant turns; converts tool calls/results to Anthropic content blocks.
• GeminiHistoryAdapter — preserves thought_signature on tool calls; emits Gemini function_call / function_response shapes.
• OpenAIHistoryAdapter — standard OpenAI tool-calling format. Default for any non-Anthropic non-Gemini transport.

This is the layer the original spec’s “non-goals: do not unify history encoding immediately” parked — and we kept it parked. There is no plan to unify these.

Public API surface

from src.llm import honcho_llm_call
 
response = await honcho_llm_call(
    model_config=ConfiguredModelSettings.dialectic(),
    prompt="…",
    max_tokens=4096,
    response_model=MyPydanticModel,  # optional structured output
    tools=[…], tool_executor=…,      # optional tool loop
    stream=False,
    thinking_budget_tokens=2048,     # Anthropic
    reasoning_effort="medium",       # OpenAI GPT-5
)

The signature did evolve from the pre-refactor honcho_llm_call. The biggest change is model_config: ModelConfig | ConfiguredModelSettings replacing the per-call provider / model strings.

Tests

Contract coverage lives under tests/llm/:

• tests/llm/test_backends/ — per-backend deterministic tests with mocked SDK clients
• tests/llm/test_history_adapters.py — provider-aware replay round-trips
• tests/llm/test_request_builder.py — request shape across providers
• tests/llm/test_credentials.py — credential resolution rules
• tests/llm/test_conversation.py — conversation-level helpers
• tests/llm/test_model_config.py — ModelConfig primitive

Live, network-touching coverage lives under tests/live_llm/ and is gated by API-key markers (requires_anthropic, requires_openai, requires_gemini).

Out of scope (kept out)

These were called out as non-goals in the v2 spec and remain non-goals:

• Embedding client migration — src/embedding_client.py is unchanged.
• Tool execution loop rewrite — tool_loop.py is the same execution model as the old _execute_tool_loop, just relocated.
• Runtime API-configurable models — model selection stays server-side.
• Heuristic max-token inflation for reasoning_effort.
• A unified history-encoding format across providers.

Open follow-ups

• Embedding client refactor — same treatment for src/embedding_client.py. Punt.
• OpenAI-compatible providers (Groq, OpenRouter, vLLM) — currently ride on OpenAIBackend via override clients. If any of them grows enough provider-specific behavior (e.g., OpenRouter reasoning_details edge cases) it may justify its own backend. Watch.
• Streaming + tool loop — StreamingResponseWithMetadata is the current bridge; revisit if the streaming-tools UX gets more complex.