vLLM vs NVIDIA Flare vs myllmbox vs TensorFold on one DGX Spark: I measured six ways to serve Qwen3.8 Flash Next

TL;DR: The pinned vLLM default stays the default. myllmbox wins 8-user throughput, TensorFold wins code and scene decode, and Flare starts in 21-24 s.

One DGX Spark (GB10, 121 GB unified memory). One checkpoint: NVIDIA's Qwen3.8-Flash-Next-FP4-Local. Six ways to serve it. I wanted to know which one my open recipe should ship as the default. So I measured all of them on the same box, back to back: speed at every context size, one user and eight, peaks, cold starts, prefix caching, quality and, new this round, a one-shot scene-writing gauntlet.

Recipe: github.com/devopsfortunato/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark

The contenders

• Default: vLLM in eugr's spark-vllm-b12x image, pinned by digest. MTP with 3 draft tokens, FP8 KV cache, and the LM head read as MXFP8. This is what ./start.sh serves today.
• vLLM 0.32: upstream vLLM v0.32.0 plus a 21-patch series, with MTP at 3 draft tokens and an FP8 KV cache. Opt-in profile in the recipe.
• Flare + MXFP8 head: NVIDIA's new local inference engine (47581cc), with a 29-line patch of mine that reads the LM head as MXFP8 and the draft head as NVFP4. 3 drafts, fixed-width verification.
• Flare 5f37bd1 (stock): a later commit of the same engine, unpatched, with a BF16 LM head and 3 drafts at fixed width. No patch, a 35-minute export per update, and the most KV cache of the six.
• myllmbox v5.3: their vLLM 0.32 image and recipe. NVFP4 body with a BF16 output head, MTP up to 7 drafts.
• TensorFold Zig (Mia's recipe): their native Zig engine, which ports the Flash Next CUDA path, running in NVIDIA's PyTorch container. MTP drafts to depth 6, FP8 KV (lossy: ~98.8% top-1 agreement with BF16), vision on, 262k window. Same checkpoint, built locally. No prebuilt image exists for the current patch pins.

How I measured

Same Spark, thinking off, temperature 0, and every speed run checked for garbage output. Decode medians are over 5 runs per prompt. C=8 totals and the 8 x 8k prefill are over 2 trials, and the 10-second peak windows over 3 runs. Flare's output is not byte-repeatable at temperature 0, so its numbers carry that extra variance.

• Speed by context: llm_context_benchmarks, prose and code at 0.5k, 8k, 32k and 128k.
• Peak: the best context size, and the best 10-second window while writing a 3,000-token HTML/JS answer.
• One user vs eight (C=1 / C=8): 8 different prompts for 512 tokens each, one at a time and then all at once. Plus eight 8k-token documents at once for prefill.
• Prefix cache: a cold request, then a follow-up. The follow-up is the next question in the same chat, the same document re-sent with a new question, or the document in its own message. Sizes are 8k, 32k and 128k. The table shows 128k; every size is in the repo.
• Scene gauntlet (new): one-shot animated HTML pages, three runs each: the fish and pasture scenes from myllmbox's public gauntlet, plus a three.js soft-body "gummy bears" page of mine. This is the one thinking-on test. Requests ask for thinking at effort low and send no temperature, so each recipe's default sampling applies.
• Cold start: from docker run to the first answer, with the OS page cache dropped.
• Quality: tool-eval-bench 2.8.1 (97 scenarios; cells show two runs), sixcat-eval 0.8.3 (884 items), and perplexity where the server returns prompt logprobs.

What I found

1. Flare starts in 21-24 seconds. TensorFold takes 65-68 s. vLLM takes about 4 minutes for the default build and 8.3 for the 0.32 profile. myllmbox takes 523 s on its first boot and about 7 minutes after that. vLLM builds the model and captures its graphs on every start. Flare does that once, in a 17-35 minute export depending on the package. myllmbox carries a much larger, more heavily patched vLLM stack and pays for it at startup. Flare also fits 2.0-2.06M KV tokens next to the weights. That is 27-30% more than the default's 1.58M and 21-24% more than myllmbox's 1.66M.

2. Flare's defaults cost 29% of decode on a Spark. Its default is 5 drafts with adaptive verification: 31 t/s. Two run flags, 3 drafts at fixed width, bring stock Flare to 43.8 t/s prose. That is on par with the vLLM builds (41.9 for the default, 40.8 for 0.32). The MXFP8 head adds another 3 t/s on top.

3. The head layers were worth 5%. Flare keeps the LM head (248,320 x 2,560) in BF16 for exact verification. That is 1.27 GB read on every decode step. vLLM reads it as MXFP8, half the bytes. A 29-line patch to Flare's Qwen recipe does the same (and reads the draft head as NVFP4), and the decode step drops from 59 to 56 ms. Stock and patched are one engine commit apart, so a sliver of that 5% may belong to the commit, not the heads. That makes patched Flare the fastest single-user prose decoder of the six: 46.7 t/s against 43.8 for stock Flare, 41.9 for the default and 46.2 for myllmbox. tool-eval scored 88 and 89, and sixcat 90.2 against stock's 89.8.

4. Eight users flip the ranking. With 8 streams at once, myllmbox gives 136.9 t/s in total. TensorFold (128.5) and vLLM 0.32 (128.4) tie inside noise. The default gives 119.4, Flare + MXFP8 97.4 and stock Flare 93.1. Under load, Flare's eight 8k prompts all get their first token after about 30 s. myllmbox and vLLM 0.32 serve the first ones in 16 s, the default in 22 s.

5. Prefill is a near-tie at the top. vLLM 0.32 and myllmbox tie on 8 x 8k (2,355 and 2,351 t/s, inside run-to-run noise). The default does 2,203 and Flare 2,126-2,142. For a single cold 128k prompt, myllmbox is fastest at 56.2 s, vLLM 0.32 next at 58.1 s, then Flare at 61.4-61.9 s and the default at 62.1 s. myllmbox has the best single-stream prefill medians (2,328 prose / 2,366 code), with vLLM 0.32 next (2,295 / 2,346).

6. Prefix caching depends on how the client sends the follow-up. A normal chat sends its history, and every engine reuses it for the next question. TensorFold is the best of all six at 0.60 s, because it keeps a state one token short of every prompt it has served. The default takes 0.65 s, Flare 0.85-0.91 s and vLLM 0.32 1.38 s, but myllmbox takes 5.2 s.

Some clients re-send the same document with a new question in one user message, with no history. They get nothing back on Flare or TensorFold: both recompute the whole prompt, about 60 s on Flare and 64.3 s on TensorFold. The default needs 4.0 s, vLLM 0.32 1.35 s and myllmbox 5.18 s. Flare keeps states only at message boundaries, and TensorFold only for prompt-identical repeats. Either way, the shared part ends inside that message.

If the document goes in its own message and the question follows in a new one, stock Flare reuses it in 0.57 s and myllmbox in 5.07 s, its usual reuse speed. Flare + MXFP8 and the two vLLM builds were not run on this variant.

7. Quality: no engine is clearly better, but TensorFold's tool-eval dipped. The default: tool-eval 89 and 90, sixcat 92.1. Flare with the patch: 88 and 89, 90.2. vLLM 0.32: 88 and 87, 90.6. myllmbox: 88 and 87, 90.0. Stock Flare: 85 and 88, 89.8. TensorFold: 86 and 86, with sixcat 90.5. It is the only lane whose tool-eval never reached 88.

Perplexity was available on four of the six. Flare's prompt logprobs are an opt-in export flag (the patched package was built without it, the stock one with it), and TensorFold's API returns no prompt logprobs at all. myllmbox and vLLM 0.32 are marginally lower (1.086-1.088 over all windows) than the default (1.0902). Stock Flare is 1.0890. Perplexity is diagnostic rather than scored, and lower is better. The difference is small enough that the task scores decide it. With 97 scenarios and two to four runs, gaps of 1-3 points sit at the edge of eval noise. The honest read: no engine is clearly better on quality, and the default gives up nothing.

8. About "peak decode". This is the best 10-second window while a request writes code. myllmbox hits 86.5 t/s with drafts up to 7 deep. That is the figure it advertises, reproduced here. Then come TensorFold at 73.9, Flare + MXFP8 at 64.7, vLLM 0.32 at 61.0, stock Flare at 59.9 and the default at 59.4. On the same 8 mixed prompts, myllmbox does 48.5 t/s against 46.6 for Flare + MXFP8, 46.2 for TensorFold, 45.0 for the default and 44.4 for vLLM 0.32. Peak numbers tell you as much about the workload and the draft depth as about the engine.

9. Flare's throughput flags did not help this workload. I tried two obvious knobs: --scheduler-policy throughput, and a package re-export with --mixed-prefill. On stock Flare at 3 drafts, the throughput policy gained 0.8% at C=8 (93.43 against 92.71 total in the same session; the table's 93.1 is an earlier run of the same build). But it lost 4.1% at C=1 and added 1.3 s to median TTFT. Mixed prefill gave no C=8 gain (-0.1%) and cost 3.0% of prefill and 9% of KV capacity. Both together were worse than either. The defaults stay.

10. TensorFold writes code and scenes fastest, and it is the only lane that honors per-request thinking effort. Its code median is 67.2 t/s and its best-context peak 97.9. Every other lane sits in 53.8-60.4 on both. On the new scene gauntlet, it wrote the fish and pasture pages at 65.6 and 68.7 t/s whole, against the default's 56.4 and 57.9. On a 24k-token three.js soft-body gummy-bears page, it reasoned for 10.2k tokens first (50.3 t/s whole, against 56.1 for the default, which did not think). It still finished complete HTML in all three runs. The default's three gummy answers all ended without a closing </html>.

The interesting wrinkle: thinking effort is not an API parameter the model reads. It is a line the chat template writes into the prompt ("Reasoning effort is set to low. Keep your thinking brief..."), and only TensorFold passes the kwargs through to the template. The vLLM-based lanes render thinking off server-side, no matter what you send. On myllmbox, I had to render the template myself and post to /v1/completions to get it to think at all. TensorFold also gives the fastest same-chat follow-up (0.60 s at 128k, finding 6). But it recomputes a re-sent document in full, serves no prompt logprobs, and its FP8 KV is lossy (~98.8% top-1 agreement with BF16).

The final table

Best per row in bold. On 8 x 8k prefill, vLLM 0.32's 2,355 and myllmbox's 2,351 are inside run-to-run noise. On C=8 total, TensorFold's 128.5 and vLLM 0.32's 128.4 tie for second, 0.1% apart. Flare stock numbers use 3 drafts at fixed width. The package defaults (5 drafts, adaptive) are 29% slower. C=8 per stream is the median of the eight streams' own rates. The total counts all streams over one wall-clock window, so it lands below 8x the per-stream median. The two Flare builds are different engine commits (the patch no longer applies to 5f37bd1), hence their different KV capacities. KV capacity is per build, not per engine: the two vLLM profiles sit 11% apart on the same declared quantization. TensorFold's KV count is computed from its logged FP8 pool (25.97 GiB at 17,888 bytes a token). It is the only engine that reports GiB instead of tokens, and its FP8 KV is lossy. The flags --mixed-prefill and --scheduler-policy throughput were measured separately and did not improve this workload (finding 9).

The verdict

The pinned vLLM default stays the default. It has the top task scores (margins inside eval noise, but nothing beats it) and good prefix reuse everywhere. The perplexity spread is too small to override them (finding 7).

• vLLM 0.32 matches myllmbox on 8 x 8k prefill (2,355 against 2,351, inside noise). It is the best open-stack pick for prefill-heavy work and for clients that re-send long documents with each question. It stays opt-in until its tool-eval gap (87-88 against 89-90) is explained.
• Flare with the MXFP8 head is the pick for fast restarts, one-user prose decode and long-context decode, and it has the fastest Flare decode step. It stays an opt-in patch.
• Flare 5f37bd1 (stock) is the no-patch Flare option: a 22 s restart and 2.06M KV, the most of the six. It is also the only engine that reuses a document sent as its own message near-instantly (0.57 s at 128k; measured on two of the six, finding 6). It gives up about 3 t/s of prose decode to the patched build. Its first tool-eval run also came in 3 points lower (85 against 88). But a lower-precision head scoring higher than an exact one looks like run-to-run noise, not a cost.
• myllmbox v5.3 takes three of the six decode rows (the 10 s peak, C=8 per stream and C=8 total) and the best single-stream prefill. It pays for that with a 523 s first boot, about 7 minutes on later starts, 5.2 s prefix reuse at 128k, and closed binary libraries. Good for a dedicated single-model box, not for the default recipe.
• TensorFold Zig (Mia's recipe) is the code and scene writer. It has the fastest code decode by 10 t/s (67.2), a best-context peak of 97.9 that no other lane is near, C=8 in a dead heat with vLLM 0.32, and the fastest same-chat follow-up (0.60 s). It is also the only engine that honors per-request thinking effort. The costs: tool-eval at 86 · 86 (the only lane under 88 on both runs), lossy FP8 KV, no prompt logprobs, and a full recompute when a client re-sends a document. Opt-in, and a very promising young engine.

Flare: the good and the not-yet

NVIDIA's Flare compiles the model once into a package and serves it from a small Rust runtime, and it shows. It starts in 21-24 s, against 4 to 8.3 minutes for the vLLM builds and 523 s for myllmbox's first boot. It fits about 2.0-2.06M KV tokens on the same Spark (+27-30% over the default), and its decode steps are lean. With the right flags, it matches vLLM single-stream. With an MXFP8 LM head, it has the leanest decode step measured here (56 ms) and the fastest long-context decode (44 t/s at 128k against a next-best 39; from the context sweep, in the repo). It reports cached tokens and per-depth draft acceptance out of the box. Its message-boundary prefix checkpoints reuse a document sent as its own message in 0.57 s, the only measured win there.

Where it can improve:

• Defaults tuned for the Spark. 5 drafts with adaptive verification cost 29% of decode.
• A lower-precision LM head as an option rather than a patch.
• Concurrency. 8 streams give 93-97 t/s against vLLM's 119-128 and myllmbox's 137.
• Prefix reuse when a client re-sends the same document with a new question in one message. A 128k document is recomputed: 60 s against 1.4-5.2 s.
• Repeatable output at temperature 0.
• A prompt-logprobs export for evaluation (the patched package was built without it).
• Setup: a 17-35 minute export, about 190 GB of free disk, and development-key and file-permission friction.

For a one-day-old engine, a very strong start.

Everything is in the repo: the numbers, the methods, the Flare patch and the images they ran on. github.com/devopsfortunato/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark