Plastic Labs Internal Wiki

Recently Updated

Load Testing

2 min read

This document goes over the load testing methodologies that we have used for Honcho.

Locust

Locust is an open-source load testing tool that allows us to simulate user behavior and test Honcho at scale. Our implementation of locust is in the loudred repository.

How to run

cd loudred
uv sync
source .venv/bin/activate
locust -f src/locustfile.py --host=honcho_server_url

Muk

Muk is our internal tooling to mock OpenAI API calls. It features error injection capabilities to simulate API failures, timeouts, and rate limiting. Muk is used in our load testing to explicitly mock all LLM usages with Honcho.

How to run

cd muk
uv sync
source .venv/bin/activate
uv run uvicorn src.main:app --host 0.0.0.0 --port PORT

Honcho

There are only two main changes that we need to make to Honcho to enable load testing.

  1. We need to point the database CONNECTION_URI to the load-test database in supabase.
  2. We need to mock all of the LLM calls to either directly use the OpenAI client or to use an OpenAI compatible provider from the ModelClient

See https://github.com/plastic-labs/honcho/tree/rajat/honcho-load-test-patch for an example of how to alter LLM calls in Honcho for load testing.

Fly

The above workflow details how to run load tests on locally running Honcho and muk instances.

To run load tests on a fly.io instance, we need to make the following changes:

  1. We need to deploy the patched version of Honcho to fly.io
  2. We need to deploy muk to fly.io and have the OPENAI_COMPATIBLE_BASE_URL point to the muk instance’s fly url.
  3. We need to run our locust file(s) in loudred against the Honcho instance in fly.io.

Graph View

Backlinks