I served the same model four different ways on one DGX Spark, back to back, and ran every recipe through the same benchmarks. The results surprised me in three places. Answer quality barely differs. Speed and memory differ a lot. And one recipe's engine rejects structured output outright.

The setup

I used one DGX Spark (GB10, 121 GB unified memory) and one model, Qwen3.8-Flash-Next. Each recipe ran on its own defaults, one server at a time, with a 262,144-token window, thinking off and temperature 0. My recipe's quality benchmark runs used its earlier KV pool sizing. The pool size doesn't change answers.

My recipe is github.com/devopsfortunato/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark. These are the four contenders:

  • My recipe, fast: vLLM with b12x kernels, NVIDIA's NVFP4 FP4-Local checkpoint, FP8 KV cache, MTP 3. These are the model card's activation settings, and fast is now the default.
  • My recipe, quality: the same, plus a second FP4 term for the prefill rounding residual and BF16 activations in the dense layers. It is opt-in.
  • hibrid48 by myllmbox: vLLM, Marlin W4A16 MoE kernels, its own hibrid48 NVFP4 weights, BF16 KV cache, MTP up to 7.
  • TensorFold by MiaAI-Lab: a native Zig engine serving the same NVIDIA FP4-Local checkpoint as mine, FP8 KV cache, MTP drafts.

What I measured:

  • tool-eval-bench 2.8.1: 97 agentic tool-calling scenarios including Hard Mode, two full runs per recipe.
  • sixcat-eval 0.8.3: 884 items across knowledge, math, truthfulness, instruction following, executed code and tools.
  • Perplexity: the same texts scored through each server with prompt logprobs, including fresh 2026 texts the model cannot have seen.
  • Speed: llm_context_benchmarks, prose and code from 0.5k to 128k tokens.

TL;DR

  • Answer quality is close. My recipe scored highest on every quality measure: tool-eval, sixcat and perplexity. The margins are small, within run-to-run noise, but it never lost on an overall quality score. The top scores were 90 on tool-eval and 92.1 on sixcat.
  • hibrid48 is the fastest, with 54.3 t/s decode against 45.8 for mine (geometric means). TensorFold is second.
  • Memory splits three ways. hibrid48 decodes up to 16 requests at once but has the smallest KV pool. TensorFold has the largest default pool, and mine sits between, with room for 5 full 262k windows.
  • The "more precise" setting is twice as close to full precision token by token, and that buys no benchmark points. So fast is my default.
  • TensorFold's engine rejects response_format. It runs the same weights as mine but scores 6/12 on structured output against 10/12.

Quality, speed, and memory overview of four Qwen3.8-Flash-Next recipes on one DGX Spark

1. Quality: closer than the leaderboard makes it look

My recipe scored highest on every quality measure: tool-eval, sixcat and perplexity. The margins are small, within run-to-run noise, but it never lost on an overall quality score.

On tool-eval-bench, the three vLLM recipes land within 2 points of each other. A recipe's own two runs differ by up to 4 (hibrid48 scored 86, then 90). A 1-2 point gap from one run is noise.

tool-eval-bench category scores for the four Qwen3.8-Flash-Next recipes

sixcat is more decisive in one place. hibrid48 scores 70 in the Tools category, against 85-90 for the others, and that pulls its overall score down to 87.8. On knowledge, math, truthfulness and code, all four are within a few points.

sixcat-eval category scores for the four Qwen3.8-Flash-Next recipes

Perplexity tells the same story in finer detail. My two settings and hibrid48 split the texts into exactly the same tokens, so the numbers compare directly. On fresh 2026 texts, fast scores 2.205, quality 2.227 and hibrid48 2.247. TensorFold's server refuses logprobs, so it has no perplexity number.

2. Speed: hibrid48 and TensorFold pull ahead

  • Decode, geometric mean: hibrid48 54.3 t/s, TensorFold 51.6, fast 45.8, quality 44.4.
  • Prefill, geometric mean: hibrid48 1,936 t/s, TensorFold 1,858, fast 1,717, quality 1,362.
  • Time to first token on a 128k prose prompt: hibrid48 60.2 s, TensorFold 63.9 s, fast 65.5 s, quality 86.0 s.

hibrid48 gets there with Marlin W4A16 kernels and up to 7 speculative tokens. TensorFold is the fastest of all on short code, with 67.9 t/s at 0.5k tokens. Every recipe stays above 38 t/s even at 128k.

Decode and prefill speed from 0.5k to 128k tokens for the four recipes

3. Memory: how much fits, and how many at once

Two numbers decide how many users or agents a recipe can serve at the same time:

  • The KV pool: how many tokens of context fit across all live requests.
  • The request cap: how many requests the server decodes together, however short they are.

As each recipe ships:

  • My recipe, fast or quality: 1,365,333 tokens of FP8 KV in a fixed 27 GB pool, with a cap of 5. The pool holds five full 262k windows, so every running request has room to reach the full window, like TensorFold's 5 streams. I tested it with five prompts of 246k-253k tokens, each generating 3,000 tokens. All five decoded together with no preemption. Free memory never went below 11.4 GiB in any test.
  • hibrid48: 877k tokens, because its KV is BF16, with a cap of 16. That fits 16 requests of up to about 54k each, 13 at 64k, 6 at 128k and 3 at 262k.
  • TensorFold: about 1.57M tokens of FP8 KV, because it takes all memory but 10 GiB, with a cap of 5. That is 5 requests at any length, all 5 at the full 262k window. Its server logs 26.2 GiB and no token count. The token figure uses its recipe's own sizing of 17,888 bytes a token.
  • My recipe with PROFILE=kv-max: 1.72M tokens, the largest pool of the four, with a cap of 6, so 6 requests fit at the full 262k window. It costs no quality (perplexity within 0.1% of the default), only free memory, which went down to 4.4 GiB while serving.

The caps are plain settings (MAX_NUM_SEQS on mine, max-num-seqs on hibrid48, PARALLEL on TensorFold). Raise one and more short requests fit, but the pool still limits long ones.

These capacities come from each server's boot log and settings. Requests decoded together share the GPU, so each one decodes slower. A load test with 1 to 16 requests of 8k and 32k tokens showed how that plays out:

  • hibrid48 delivers the most total output at every load (82 t/s with 16 requests of 8k against 51 for mine), then TensorFold, then mine. TensorFold's 5 streams are a bit faster than mine (56 against 51 t/s with 8 requests of 8k), and up to 5 requests quality comes last because it reads prompts slower.
  • The cap of 5 has a price. With my old cap of 8, mine gave 63 t/s with 8 or 16 requests of 8k. With 5 it gives 51, because extra requests wait for a free slot. Nothing fails past the cap, and every running request has room to reach the full window, so I keep 5.
  • Reading prompts stays at about 2,000-2,200 t/s whatever the load, but generating scales, with five streams at 91-117 t/s combined against about 40 for one.

KV cache pool size against how many requests each recipe decodes at once

4. The precision setting that does not pay

My recipe has a PRECISION switch. The quality setting adds a second FP4 term for the prefill rounding residual and keeps the dense layers' activations in BF16. Against a full-precision reference over 156k tokens, its token-level error halves. Mean |Δ logprob| drops to 0.020 from 0.039, and top-1 agreement rises to 99.23% from 98.59%.

Then I measured what users feel:

  • tool-eval: quality 90, fast 89.
  • sixcat: fast 92.1, quality 91.7.
  • Prefill: fast is 26% faster. On a 128k prompt you wait 66 s instead of 86 s.

So quality gives twice the fidelity, no benchmark gain and 20 seconds more waiting. Fast is the default, and quality is one environment variable away for anyone who needs that fidelity.

5. TensorFold and structured output

This is the finding I did not expect.

TensorFold's engine answers every request that carries response_format with HTTP 400: "structured output (response_format and the guided_* fields) is not supported by this engine yet".

tool-eval-bench 2.8.1 handles that now. When a server rejects the schema, the benchmark retries without it and grades the model on the schema already written in the prompt. Two things happen:

  • TC-64 now passes. It asks for schema-shaped JSON directly, and TensorFold writes valid JSON from the prompt alone. Under 2.7.1, the benchmark threw this scenario out as a server error.
  • TC-65 to TC-69 still lose points. There the JSON has to come after a tool call, and the model answers in prose instead: "Here's the current weather in Berlin: ...". On the vLLM recipes, constrained decoding keeps that answer valid JSON. None of their runs needed the fallback.

On Structured Output, TensorFold scores 6/12 and every vLLM recipe 10/12. TensorFold serves the same checkpoint as mine, so the gap is the engine, not the weights. If your agent relies on response_format or a JSON schema, plan for it on TensorFold today. Validate and retry, or prompt harder.

I opened a PR that adds structured output to TensorFold's engine and enforces response_format and the guided_* fields.

6. Benchmarks move too

Halfway through, tool-eval-bench jumped from 2.7.1 to 2.8.1. That brought 5 new document-workflow scenarios (92 to 97) and grading fixes across many others. I reran everything.

hibrid48 moved the most, from 91 · 91 on the old version to 86 · 90 on the new one. Its first new run also called run_code in two scenarios that did not ask for it, TC-03 and TC-17, and its second run passed both. TC-17 flips between runs for quality too.

Two lessons:

  • Run at least twice.
  • Don't rank recipes 2 points apart.

7. What did not make the cut

  • BF16 KV cache gave no measurable gain and a smaller KV pool.
  • BF16 LM head gave no gain and 6% slower decode.
  • MTP 4 with block verification gave 5-9% faster decode, but one tool-eval run scored 87, under my bar of 88. I documented it as an option.
  • Four b12x kernel levers were inside run-to-run noise, slower, or worse on perplexity.
  • There is no newer b12x image yet. nightly-20261015 is still latest.

Summary of four ways to serve Qwen3.8-Flash-Next on one DGX Spark

Which one should you run?

  • Several long requests at once: my recipe, fast. It holds 5 requests at the full window, with structured output. PROFILE=kv-max gives it the largest pool of the four, 1.72M tokens, and 6 full windows.
  • The fastest single stream, or many short requests: hibrid48, which decodes up to 16 at once. Expect less KV and weaker sixcat tool calls.
  • A light native engine without vLLM: TensorFold is quick, scores well and holds 5 full windows at once. For now, skip it if you depend on response_format.
  • Maximum fidelity to full precision: my recipe with PRECISION=quality, at the cost of 21% less prefill.

Reproduce it

Thanks to

  • myllmbox and MiaAI-Lab for their recipes
  • eugr for the b12x container
  • Tim Messerschmidt (SeraphimSerapis) for tool-eval-bench
  • vcruz305 for sixcat-eval
  • Ivan Fioravanti for llm_context_benchmarks