README

DGX Spark Agent Setup Guide & Notes

This document includes notes from setting up a hermes agent on it.

A while back I looked into setting up qwen3.6 35b-a3b on the DGX using a docker run command and I used the following command:

docker run -d \
  --name qwen36-35b \
  --runtime nvidia \
  -p 100.97.228.103:8000:8000 \
  -v /home/vineeth/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:cu130-nightly \
  RedHatAI/Qwen3.6-35B-A3B-NVFP4 \
  --served-model-name qwen3.6-35b \
  --host 0.0.0.0 \
  --port 8000 \
  --quantization compressed-tensors \
  --moe-backend flashinfer_cutlass \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.87 \
  --max-model-len 131072 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 32768 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --language-model-only \
  --trust-remote-code

Improvement:

docker run -d \
  --name qwen36-35b \
  --gpus all \
  --restart unless-stopped \
  -p 100.97.228.103:8000:8000 \
  -p 127.0.0.1:8000:8000 \
  -v /home/vineeth/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:cu130-nightly \
  RedHatAI/Qwen3.6-35B-A3B-NVFP4 \
  --served-model-name qwen3.6-35b \
  --host 0.0.0.0 \
  --port 8000 \
  --quantization compressed-tensors \
  --moe-backend flashinfer_cutlass \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.87 \
  --max-model-len 131072 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 32768 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --language-model-only \
  --trust-remote-code