Qwen3.8-Flash-Next on 8 3090s

Eric Hartford, QuixiAI - August 2026

Qwen3.8-Flash-Next wasn't designed for this.

It is 125 billion parameters of Gated DeltaNet linear attention, Qwen Sparse Attention, a four-branch gated residual stream, 51 GB of n-gram embedding tables, and a built-in speculative drafter, with one full-attention layer in every four. It is a genuinely strange architecture. It was built for datacenters with fast interconnects and FP8 tensor cores.

We serve it on eight RTX 3090s. Five-year-old GeForce cards. 24 GB each. No FP8 hardware. No NVLink. And we serve it at its native 262,144-token context, with exact bf16 KV cache, and a 512 GiB pinned-RAM second tier that lets a conversation evicted from GPU memory resume in under a second instead of re-prefilling minutes of history.

Here is what that looks like, measured with an exact-token harness (1,000 tokens in / 2,000 out per request, the model's shipped sampling defaults, seeded):

Peak: 1,199.8 tok/s at 32 concurrent requests.

Our first working configuration managed 409.7 tok/s, at half the context, with quantized KV that turned out to be quietly corrupting answers. The campaign bought a net 2.9x while doubling context to the model's native maximum and removing a quantization we had shipped on purpose. None of the individual tricks is exotic. What follows is the order they hurt us in, more or less.

A note on the harness: everything here is a SlimServe profile (qwen38fn-fp8-8), our simplicity-first serving engine. The one idea you need for now is that every legal configuration is a profile - model x quant x platform x config, recorded in a registry with every setting annotated by the measurement that put it there. You type slimserve qwen38fn-fp8-8 --serve and get the one configuration tuned and validated on that hardware. Anything else is refused with a readable reason. More on why at the end.

1. The driver is leaving a third of the machine on the table

GeForce cards ship with peer-to-peer DMA disabled. Tensor-parallel all-reduce therefore bounces every tensor through system RAM, and on an 8-way TP model that is fatal: our bring-up throughput was capped by collectives, not compute.

We run QuixiAI's patched open-gpu-kernel-modules, which re-enables large-BAR P2P on GeForce. The recipe is a per-boot 32 GiB BAR1 setup, iommu=pt, and NCCL_P2P_LEVEL=SYS to allow P2P across root complexes on our dual-socket EPYC. Measured on this exact profile:

• c1: 109.6 -> 148.8 tok/s (+36%)
• c8: 409.7 -> 647.9 tok/s (+58%)

The stock driver still works - the profile only recommends the patched one. But more than a third of the machine is unreachable without it.

2. The KV quantization that was changing the model's answers

We originally shipped TurboQuant-compressed main KV. It buys 2.25x KV capacity, the benchmarks were fine, and we moved on.

Then multi-turn agentic testing surfaced something worse than a perf bug. Occasionally, the model would answer the previous turn's question. Not garble it, not hallucinate - answer the wrong question, fluently, with full confidence. With only 6B active parameters, this model appears genuinely sensitive to KV precision, and a compressed cache was enough to blur which turn it was in.

Main KV is now always exact bf16 on this platform, enforced by a registry test, at the model's native 262,144 context. Measured cost: nothing. bf16 is ~8% faster at c1 (no dequant on the attention path) and at parity from c8 to c32.

The attention kernel had to follow the dtype, though. The QSA sparse-gather kernel's tile profile depends on the KV format: narrow 16-column tiles keep in-register dequant alive for quantized KV, while wide tiles suit bf16. Pick the wrong profile and long decode steps collapse by 3.8x to 23x, depending on format. That tile pinning is now part of QuixiCore, our kernel library (more below).

Draft-model KV stays TurboQuant-compressed everywhere. Rejection sampling verifies every draft token against the bf16 target, so draft precision can only affect acceptance rate, never output content.

3. Prefix caching was silently off, and deep conversations were killing the engine

vLLM silently disables prefix caching for hybrid (linear-attention) models. We shipped that default without noticing, and paid full-history re-prefill on every chat turn. One agent with a deep conversation could degrade the entire engine for everyone.

On agentic traffic - long shared prefixes extended turn over turn, which is ~99% of our production load - prefix caching approaches a 100% hit rate. Its absence isn't a marginal loss. It's catastrophic.

The hybrid "align" cache mode makes it work: the engine snapshots the linear-attention state at block-aligned boundaries, so a cached prefix can resume with both its attention KV and its recurrent state. It is now policy, enforced by tests: every profile states prefix caching explicitly. No more silent defaults.

4. The 27-slot ceiling

At 32 concurrent requests the KV meter read 100% and the scheduler capped at 27 running sequences. It did not say why.

The cause: this hybrid's packed KV slab charges every request roughly 13-14 blocks at chat context - 1,056-token QSA blocks, a compressor ring block, GDN state blocks at k=2, plus the drafter's cache group. The default 0.9 GPU-memory utilization sized a pool that could not hold 32 of those. Raising it to 0.96 (the profile's value, with the activation and graph reserves re-profiled) fixed it. +29% at c32, from scheduling alone.

On top sits admission control: an ASGI middleware caps in-flight generation requests at 96 (3x the decode slots) and answers 429 + Retry-After beyond that. We measured this at 64 concurrent sessions: oversubscription past 32 buys zero throughput (780.5 vs 779.7 tok/s) and only queueing delay (TTFT p50 40.2 s vs 11.6 s). The cap sheds pathological load without policing bursts.

5. FP8 weights on hardware with no FP8

SM86 has no FP8 tensor cores, and Triton's FP8 MoE path wants SM89+. So the experts run Marlin W8A16 block-FP8 kernels instead: weights stay FP8 in memory - the bandwidth win is what matters for a 6B-active MoE - and decode in-register to bf16 compute.

This is the argument we keep making about weight formats. A format is just bits and a scale layout. There is nothing about it that belongs to one vendor's tensor cores, and the kernels can close the gap years after the silicon shipped.

One subtlety cost us a debugging session, and it is the kind that produces wrong answers rather than crashes: expert parallelism is a correctness requirement here, not a tuning choice. The MoE intermediate size is 640 and the FP8 block scales are 128x128. A tensor-parallel shard of 640/8 = 80 cannot carry exact block scales - the scale grid doesn't tile. EP keeps 64 whole experts per rank (640 = 5 x 128) and the math stays exact.

6. 51 GB of embedding tables that never touch the GPU

The model's n-gram "PLE" tables are 47.7 GiB. That's two full 3090s of VRAM. But each token reads only 16 rows - 2,560 bytes.

So we pin the table once, in a /dev/shm segment shared by all eight ranks, and gather those rows over UVA from inside the forward pass. The gather is CUDA-graph capturable, so it costs nothing extra under graph replay. And because every rank sees the whole table, we could delete a per-step embedding all-reduce that this no-NVLink box could not afford.

GPU memory holds weights and KV. Nothing else.

7. The CUDA graph capture-size trap

Decode runs under FULL_DECODE_ONLY CUDA graph capture. The trap: with MTP speculation at k=2, every decode step is num_seqs x 3 query tokens, so the graph capture size must be at least 3x max_num_seqs. Ours is 96 for 32 sequences.

Undersize it and the largest batches silently fall back to eager. We measured that cliff at 425 tok/s eager vs 880 graphed at c32 before we pinned the rule down. If your throughput curve has a mysterious dent at high concurrency, check whether your biggest batches are actually replaying graphs.

8. Speculating with the model's own drafter

Flash-Next ships a single-layer QSA MTP head. We run it at k=2 with index sharing across MTP iterations.

We also built dynamic-k: k=3 below the concurrency crossover, k=2 above. It measured +19.5% at c4 and -2.6% at c32. We shipped it disabled. c32 is this profile's operating point, and the peak belongs to static k=2. Building a feature, measuring it honestly, and turning it off is most of what this campaign was.

A methodology note that will save you an afternoon: never benchmark speculation with synthetic repeated-token prompts. Acceptance collapses to ~0% on degenerate text and looks exactly like a serving bug. Use natural text.

9. 4.8 million tokens of warm conversations in RAM

The GPU pool at 0.96 utilization holds ~270K tokens. That is barely one max-length conversation. Production is many agents with deep histories, so we built a second tier: 64 GiB of pinned host RAM per rank - 512 GiB total, ~12,150 block slots, ~4.8M tokens - behind the GPU pool.

The design rides the engine rather than fighting it.

Offload is free. KV blocks are immutable once filled. A dedicated copy stream DMAs each newly-filled block to its pinned slot, off the critical path. A trajectory-centric index maps hash-chained prefixes to slots.

Hybrid state is the hard part. You cannot snapshot linear-attention state whenever you like. It is updated in place and covers an unaligned token count at any given moment. Every naive scheme we tried - copy the live block at finish, copy the penultimate block, freeze-previous during prefill - produced subtly or spectacularly garbled resumes. The correct source is the engine's own align-mode machinery from trick #3: when a request crosses a block boundary, the boundary state is frozen, hashed into the prefix cache, and never written again. The tier looks that immutable snapshot up by its boundary hash and saves that. Torn copies become impossible by construction, and restores land exactly where the runner's state-index seed expects them.

Resume points exist at chunk ends. Frozen states only materialize where a scheduling chunk ended (~2K tokens apart), so the saver scans down to the deepest boundary every state group actually has. The remainder re-prefills on resume.

Byte-verified, then behavior-verified. A SHA-checking debug mode proved every offload/restore round-trip bit-exact. A marker-recall battery proved the semantics: conversations at 8K, 24K, and 42K depth, fully evicted from the GPU pool by churn, resumed in 0.5 s versus 9.0 s of cold prefill at 42K, with restored outputs byte-identical to GPU-cache-hit controls.

And it costs nothing. With the tier live, the deployed service measures c1 ~140 / c8 600.5 / c32 up to 1,199.8. The c32 peak was set with the tier enabled.

An agent returning to a conversation the GPU evicted an hour ago gets its first token back in under a second.

10. Sane defaults, enforced

The serving policy is fixed and test-enforced: automatic prefix caching, automatic tool calling, and thinking always on. Never greedy sampling - benchmarks run the model's shipped sampling defaults, seeded.

The chat template defaults to reasoning_effort: low. The shipped default of xhigh spends hundreds of thinking tokens on trivial turns. Any request can override it back up when the task deserves deep reasoning.

The bottom line

None of these tricks is exotic on its own. The throughput came from measuring one variable at a time, keeping the receipts, and refusing to call the job done until the workload that exposed the last failure passed clean.

Why every configuration is a profile

General-purpose serving engines expose hundreds of flags and let you discover, eight hours into a 244 GiB download, that your combination doesn't fit, silently disables the feature you needed, or serves at a tenth of the hardware's potential. Three of the ten tricks above were exactly that: a default that was quietly wrong for this model, on this hardware.

SlimServe refuses to be general-purpose. It's vLLM's engine underneath, ds4's interface philosophy on top, and every bleeding-edge performance trick we can land in between - but only for configurations we've actually tuned. Support for models and platforms outside the tested set has been deleted, so the rest can be tuned hard. The policies in this post - prefix caching, tool calling, thinking always on, never greedy, exact bf16 main KV - aren't documentation. They're registry-level defaults enforced by tests. A profile cannot silently regress them.

Underneath the profiles sits QuixiCore, our kernel library. It exists because the models we serve and the hardware we serve them on are both outside the mainstream's tuning envelope; nobody upstream is optimizing sparse-attention gather tiles for FP8-less Ampere GeForce cards, so we do. Kernels are developed in the serving repo first, against the real profile and the real workload, then ported back into the library. The rule is strict: only kernels on an actual serving path get vendored, and every one is tuned against measurements from the profile that uses it. The QSA tile pinning in trick #2 and the Marlin-lineage FP8 work in trick #5 are that process in action. The same library is what lets NVFP4 checkpoints serve on AMD MI300X and Apple Silicon - hardware with no FP4 support at all - because, again, a format is just bits.

SlimServe is the inference-serving component of SovereignStack, our stack for running frontier-class AI entirely on hardware you own. The premise is that serious AI capability shouldn't require renting it from a hyperscaler - and the premise only holds if the serving layer can extract everything the owned hardware has. A rack of five-year-old GeForce cards serving a 125B model at 1,200 tok/s with 262K context and half a terabyte of conversation cache is that premise made concrete. This is what sovereignty costs. It is a lot less than people think.

The profile (qwen38fn-fp8-8), kernels, KV tier, and benchmark harness from this post are all in the SlimServe repository.