The vLLM doctor
Your GPU sits at ~100% whether it's doing real work or thrashing on KV cache and
recomputing preempted tokens — on a dashboard they look identical. InferPilot reads your live
/metrics and tells you which regime you're actually in, and whether a change like
fp8 KV cache is worth testing — or a waste of time.
Open source · MIT · no telemetry · no GPU needed for screening or the synthetic demo

GPU utilization sits near 100% whether you're genuinely compute-bound or just thrashing on KV cache. It can't tell these apart. Preemptions and queue depth provide a useful screening signal, which InferPilot uses to choose the next controlled experiment.
KV cache is full and the scheduler is preempting. Test fp8 under the same traffic and SLO.
A queue is building while KV has headroom, so this snapshot does not support KV capacity as the limit.
KV is full but not preempting yet. fp8 is pre-emptive insurance, not a measured win.
KV has headroom and nothing is queuing. If latency matters, that's a separate, low-load question.
Move the signals your vLLM server would report and watch the verdict change. Two servers can show the same ~99% GPU utilization and get opposite answers — that's the whole point.
What a measured window of your server reports.
Recomputed live — or it abstains.
A paired study on Qwen2.5-3B (A10G, vLLM 0.29.0) using an exact recomputed-token counter added to vLLM (an open PR to vLLM core). In the preempting regime, fp8 drove recomputed tokens to zero and throughput up. Total GPU cost: $0.81.
Honest caveat: in the later preregistered 7B study, throughput improved 33.6% on the pressured workload and 0.2% on the control, but recomputation did not fall as registered. The verdict was INCONCLUSIVE, not a mechanism win. Read the full result.
Pure Python, pydantic-based. The diagnosis path needs no GPU and no vLLM install.
Run with zero install
uvx inferpilot doctor --url …Keep it around
uv tool install inferpilotClassic
pip install inferpilotThen the daily loop: doctor to glance,
doctor --watch to monitor, capacity to turn a rate sweep into an SLO-capacity ceiling,
a $/token, and an action plan. No server yet? uvx inferpilot demo exercises the complete
decision contracts using clearly labelled synthetic metadata.