deriver-performance

Parameters that affect Honcho Deriver performance

An overview of the key variables in Honcho (as of v2.4.0) that impact performance.

Note: this document does not include ML-related subjects such as choice of model for various LLM calls, thinking tokens, shape of structured output, model providers, context window size, etc. The focus is on Honcho as a software harness around these choices.

Note: Honcho “performance” can be defined many ways. The focus here is on deriver throughput: the volume of messages that Honcho can process to form peer representations. Many of these parameters have little to no impact on the performance of an individual API call or unit of work, and only take effect when Honcho is ingesting several messages at a time. This is critical since Honcho is primarily used in multi-user applications. Also, good throughput necessitates acceptably good “single-threaded” (one message at a time) performance.

Key: - - -

Configuration

Database

• DB_POOL_CLASS - Postgres config regarding how transactions are pool - UNKNOWN - ???

• DB_POOL_SIZE - Number of connections available to the application - MODERATE - More connections allow more transactions at once, in theory, but may introduce contention issues?

Deriver

• DERIVER_WORKERS - How many queue tasks can be processed in parallel - HIGH - The basic unit of parallelization for the deriver. Note that workers can process representation, webhook, summarization, or any other manner of task on the queue. Having several workers, at minimum 10, will ensure that different tasks don’t block one another. Note, however, that high amounts of workers (40+) will simply lead to bottlenecks elsewhere in the system (such as database connections).

• DERIVER_WORKING_REPRESENTATION_MAX_OBSERVATIONS - Effectively determines the number of tokens used on representation within the deriver LLM call - LOW - This does not actually impact the LLM call performance, because the tokens ingested by deriver is determined by DERIVER_MAX_INPUT_TOKENS. However, this does affect the database query which constructs the working representation, particularly if that query includes embedding search (semantic query).

• DERIVER_MAX_INPUT_TOKENS - Amount of tokens fed into deriver LLM call - HIGH - Not listed because of LLM performance (out of scope here), rather due to database query work needed to produce these tokens from our messages/summaries/documents/peer cards. More context takes more time to get.

Peer Card

• PEER_CARD_ENABLED - Whether or not the deriver will make peer cards at all - HIGH - Creating a peer card is currently done every time new observations are generated, meaning an entire second LLM call in that task. We should consider strategies to reduce the frequency of this call in order to reduce this feature’s impact on performance.

Dialectic Chat

Dialectic settings are not generally relevant to throughput since calls are made on a discrete basis and can be executed in parallel. Tuning related to database queries may still impact this call and its interactions with deriver work occurring in parallel.

Summaries

• SUMMARY_ENABLED - Whether or not summary tasks will be created - MODERATE - Summaries fill up the queue alongside other work, but can generally be done in parallel to other tasks.

• SUMMARY_MESSAGES_PER_SHORT_SUMMARY / SUMMARY_MESSAGES_PER_LONG_SUMMARY - The cadence at which short and long summaries are made within a session - MODERATE - Like the ENABLED flag, just impact the degree to which summaries get put on the work queue. Sufficiently high values mean the impact will be minimal. Note that extremely low values on these would lead to an extreme performance impact.

Dreaming

• DREAM_ENABLED - Whether or not Honcho will “dream” — produce background tasks to asynchronously improve representations - HIGH - Dream work is unstudied from a performance angle but has the potential to impact performance enormously across the entire system.

• DREAM_DOCUMENT_THRESHOLD / DREAM_IDLE_TIMEOUT_MINUTES / DREAM_MIN_HOURS_BETWEEN_DREAMS - The cadence of new documents at which dreams are activated in a collection - HIGH - Same logic as above

General App Settings

• EMBED_MESSAGES - Whether Honcho will produce embeddings - HIGH - I’m not sure this config is even respected anymore. Embeddings are critical to Honcho’s operation at this point for both derivation and dialectic, and constitute a majority of the database/computational “work” done in the system.

In-Code

• Whether or not Deriver includes message-at-hand as a semantic query in get_working_representation - Triggers both embedding creation (API call with network latency inside a DB operation, spooky) and embedding search on the documents table - MODERATE - This seems to have a surprisingly large impact on performance. It’s a desireable feature, so we should investigate how to enable it in a more performant way.

• How often peer card is generated - Currently on every deriver call that generates new observations. What if this wasn’t the case? - HIGH - Could turn peer card option into a low impact on performance.

In-Database

• Database embedding performance - How the database handles embeddings in storage, queries etc - HIGH - We don’t yet have a good idea of how badly this is slowing us down, but it could be the #1 performance improvement currently.

• MAX_CONNECTIONS - How many concurrent connections the database handles - HIGH - Could also be a #1 performance impact, but not really in an appropriate way, since we should have fewer/faster/smaller transactions to have less concurrent ones open at our current scale.