LLM tuning LLM
Having Claude Opus 5.5 tune vLLM and Qwen 3.8 for my actual usage
September 27, 2026
Opus 5.5 hasn't even been out for a week and I am already in love. It somehow moved the needle on being smart, fast, and cheap in one fell swoop. For my own personal tinkering I've been using it via OpenRouter, but in work land we're already seeing significant improvements in our Enterprise Claude Platform bills.
I decided to point Opus 5.5 at the DGX Spark cluster running Qwen and have it try its hand at optimizing it, using some of the open source deployment repos out there. It put its notes and testing scripts here for folks who are curious. Some things are little deployment fixes/gotchas (like control between the nodes going over the LAN and not the ConnectX-7 along side the NCCL traffic)
A lot of tuning for inference is a direct trade-off between quality and performance. There is a lot more info published on token rates, but very little published on intelligence for the various setup levers and quants out there.
The first big lever was that for the kinds of work I do (mid- to long context lengths coding) speculative decoding did not benefit from predicting a third token. So scaling MTP back from 3 to 2 saw a solid decode performance gain:
| Configuration | 1K ctx, T=0 (tok/s) | 1K ctx, T=0.6 (tok/s) | 64K ctx, T=0 (tok/s) | 64K ctx, T=0.6 (tok/s) |
|---|---|---|---|---|
| before (K=3, LAN control plane) | 45.1 | 42.5 | 40.5 | 37.6 |
| after, launch 1 | 46.1 | 43.4 | 44.6 | 48.0 |
| after, launch 2 (current) | 45.3 | 47.1 | 49.5 | 45.6 |
The even bigger lever was using the FP8 dense weights instead of the BF16 weights for the dense layers of Qwen 3.8 Flash Next. This resulted in minimal quality loss but big performance gains:
| decode tok/s (mean of 6 cells) | HumanEval+ | MBPP+ | GSM8K | NLL (nats/tok) | |
|---|---|---|---|---|---|
| A: nvidia BF16-dense (previous prod) | 44.4 | 95.1 | 79.9 | 96.5 | 0.1315 |
| B: RadixArk BF16-dense (control) | 39.0 | 93.9 | 79.4 | 96.4 | 0.1310 |
| C: RadixArk FP8-dense | 52.5 | 93.3 | 79.9 | 96.4 | 0.1340 |
In the end it all cost $31 of Opus 5.5 via pi.dev to get a 30% speed-up. The cost is mostly because the tests all took so long, and we basically had to re-pay for input tokens after each run. I think this pattern of using a model for a couple weeks and gathering enough data to tune to your use case is one I'll repeat for my next daily driver.