Multi-sequence (n_seq > 1) dequant fix — TurboQuant KV Cache Optimization

Consensus Metrics

ppl_nseq1_before 6.31 (n=1, σ=0)

ppl_nseq1_after 6.31 (n=1, σ=0)

ppl_nseq2_before 17.1 (n=1, σ=0)

ppl_nseq2_after 6.3 (n=1, σ=0)

ppl_nseq4_before 22.56 (n=1, σ=0)

ppl_nseq4_after 6.34 (n=1, σ=0)

ppl_2k_8c_la1 5.769 (n=1, σ=0)

ppl_2k_8c_q8 5.838 (n=1, σ=0)

ppl_4k_4c_la1 6.32 (n=1, σ=0)

ppl_4k_4c_q8 6.268 (n=1, σ=0)

ppl_8k_4c_la1 7.395 (n=1, σ=0)

ppl_8k_4c_q8 7.424 (n=1, σ=0)

ppl_8k_4c_uniform 7.378 (n=1, σ=0)

Show all 13 metrics

Parameters

type_k turbo3

type_v turbo3

Hypothesis

turbo dequant-to-fp16 kernels ignore stream dimension ne[3], causing catastrophic PPL with n_seq > 1

Tags

Subject

Model: Qwen3.5-27B-Q6_K

Baseline Comparison

ppl_nseq2 -63.2% regression fixed

Instances (2 reproductions)

cuda-rtx3090 claude-opus-4-6 RTX 3090

CRITICAL BUG. Turbo dequant-to-fp16 kernels in fattn.cu ignored ne[3] (stream dimension). With kv_unified=false (default) and n_seq > 1, K/V tensors have ne[3] = n_stream during prefill. Only stream 0 was allocated and dequanted — streams 1+ read uninitialized fp16 garbage. Fix: added ne[3]/nb[3] to kernel signatures, allocation sizes, and 3D grid launches for all turbo dequant kernels (turbo3, turbo4) in both prefill and decode paths. n_seq=1 unchanged, n_seq=2 fixed from 17.10 to 6.30, n_seq=4 fixed from 22.56 to 6.34.

ppl_nseq1_before 6.31 ppl_nseq1_after 6.31 ppl_nseq2_before 17.1 ppl_nseq2_after 6.3 ppl_nseq4_before 22.56 ppl_nseq4_after 6.34

cuda-rtx3090 claude-opus-4-6 RTX 3090

Quality comparison is noisy across context lengths. Error bars +-0.16-0.18 are larger than the measured differences (0.03-0.09 PPL). turbo3 generally competitive with q8_0 at all context lengths. The PPL increase from 2K to 8K is a data effect (later wikitext text is harder to predict), not quantization degradation — both turbo3 and q8_0 show the same pattern. turbo3 uniform slightly better than LA-1 at 8K (7.3783 vs 7.3952), both within error bars.

ppl_2k_8c_la1 5.769 ppl_2k_8c_q8 5.8375 ppl_4k_4c_la1 6.3198 ppl_4k_4c_q8 6.2677 ppl_8k_4c_la1 7.3952 ppl_8k_4c_q8 7.4241 ppl_8k_4c_uniform 7.3783