Metagross — vLLM Inference for Tinybox

Repo: plastic-labs/metagross

Metagross is the inference service for serving Plastic Labs’ fine-tuned models on tinybox infrastructure. It wraps vllm serve with optimization parameters tuned for the Tinybox V1 Green (Porygon) — 6×4090, ~40GB each.

What ships

• OpenAI-compatible API (full compatibility with the OpenAI Python client)
• Multi-instance deployment: 1, 3, or 6 vLLM instances behind nginx
• LoRA adapter support (multiple adapters per instance)
• Reasoning-mode support (Qwen3, DeepSeek-R1 style)
• Prometheus metrics at /metrics
• Modal (serverless) deployment as an alternative to self-hosted
• Kubernetes manifests for larger-scale deployments

Tinybox parallelism strategies

For the Tinybox (6 GPUs), three sensible configurations:

Strategy	Tensor parallel	Layout	Best for
Max throughput (recommended)	TP=2	3 instances on (0,1), (2,3), (4,5)	Many concurrent requests
Max context / batch	TP=6	1 instance across all GPUs	Very long contexts, large batches
Ultra throughput	TP=1	6 instances	Many short requests (<512 tokens)

Recommended production config:

TENSOR_PARALLEL_SIZE=2
GPU_MEMORY_UTILIZATION=0.92
MAX_MODEL_LEN=8192
MAX_NUM_SEQS=256
ENABLE_PREFIX_CACHING=true
ENABLE_CHUNKED_PREFILL=true

Serving Neuromancer (nmxr-30b-a3b) on Porygon

Hand-run recipe for the Neuromancer XR checkpoint, outside the compose stack — for eval runs and one-off inspection. Weights live on Porygon’s local array, so no Hub fetch:

CUDA_VISIBLE_DEVICES=0,1,2,3 HF_HUB_OFFLINE=1 \
uvx --python 3.13 vllm serve /raid/models/nmxr-30b-a3b \
  --served-model-name nmxr-30b-a3b \
  --host 0.0.0.0 --port 7007 \
  --tensor-parallel-size 4 \
  --dtype bfloat16 \
  --max-model-len 40960 \
  --max-num-seqs 64 \
  --gpu-memory-utilization 0.90 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching --enable-chunked-prefill \
  --trust-remote-code --disable-custom-all-reduce \
  --reasoning-parser qwen3

Test it:

curl http://porygon:7007/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
  "model": "nmxr-30b-a3b",
  "messages": [{"role": "user", "content": "Say hello in five words."}],
  "max_tokens": 50,
  "temperature": 0
}'

Porygon has no public IP — reach it over Tailscale.

Why these flags

Flag	Why
CUDA_VISIBLE_DEVICES=0,1,2,3 + --tensor-parallel-size 4	Four of the six GPUs. Leaves 4 and 5 free for a second workload
HF_HUB_OFFLINE=1 + a /raid path	Weights are already on the box; don’t let vLLM reach for the Hub
--dtype bfloat16	Porygon is Ada (4090). NVFP4 checkpoints only run on Blackwell — those go on Varoom
--disable-custom-all-reduce	No NVLink on 4090s; TP traffic crosses PCIe and vLLM’s custom all-reduce misbehaves there
--reasoning-parser qwen3	Qwen3-family base, so reasoning content is split into reasoning_content
--port 7007	Keeps it clear of the compose stack’s :8000

OOM and slow-first-token fixes are the same as for the compose stack — see Troubleshooting below.

Where it fits in the system

Metagross serves the models Machamp trains. Honcho’s OPENAI_COMPATIBLE provider can be pointed at a Metagross instance via OpenAIBackend override clients, so a fine-tuned Plastic Labs model can be plugged in anywhere a stock OpenAI model would go. See system map.

Deployment options

• Docker (recommended for production): docker compose up -d (single) or docker compose -f docker-compose.yml -f docker-compose.multi.yml up -d (multi-instance + nginx)
• Modal: serverless GPU — ./scripts/deployment/modal/modal-deploy.sh deploy
• Kubernetes: manifests under docs/deployment.md
• Local dev: ./scripts/deployment/local/serve-dev.sh (Qwen2.5-1.5B-Instruct on a single GPU)

API

from openai import OpenAI
 
client = OpenAI(base_url="http://porygon:8000/v1", api_key=API_KEY)
 
response = client.chat.completions.create(
    model="qwen3-8b-sft",
    messages=[{"role": "user", "content": "Hello"}],
    max_tokens=100,
)

LoRA adapters are addressed by their adapter name as the model field.

Monitoring

• GET /health — health check
• GET /metrics — Prometheus (vllm:num_requests_running, gpu_cache_usage_perc, avg_prompt_throughput, avg_generation_throughput)
• docker compose --profile monitoring up -d — bundled Prometheus stack at :9090

Troubleshooting

OOM

Reduce GPU_MEMORY_UTILIZATION (0.85 → 0.80), reduce MAX_NUM_SEQS, reduce MAX_MODEL_LEN, or increase TENSOR_PARALLEL_SIZE.

Slow first token

Enable ENABLE_PREFIX_CACHING and ENABLE_CHUNKED_PREFILL. Reduce MAX_NUM_SEQS to prioritize throughput over batching.