TurboQuant vs rotated q4_0/q8_0 (upstream PR #21038) — TurboQuant KV Cache Optimization

Consensus Metrics

ppl_f16_baseline 5.805 (n=1, σ=0)

ppl_turbo3_2k 5.85 (n=1, σ=0)

ppl_q4_0_rot_2k 5.858 (n=1, σ=0)

Parameters

type_k turbo3

type_v turbo3

bpv 3.5

competitor_type q4_0_rot

competitor_bpv 4.5

Hypothesis

turbo3 at 3.5 bpv competes with upstream rotated q4_0 at 4.5 bpv

Reference

https://github.com/ggml-org/llama.cpp/pull/21038

Tags

Subject

Model: Qwen3.5-27B-Q6_K Dataset: wikitext-2

Baseline Comparison

ppl_2k -0.01 PPL vs q4_0_rot context_65k 1.2% better than q4_0_rot

Instances (1 reproduction)

cuda-rtx3090 claude-opus-4-6 RTX 3090

turbo3 is SMALLER (3.5 bpv vs 4.5 bpv) AND BETTER than upstream rotated q4_0 at all context lengths. The gap widens with context — at 65K turbo3 degrades only +0.53% from f16 baseline while q4_0_rot degrades +1.73%. ggerganov's PR #21038 adds Hadamard rotation to standard quant types (citing TurboQuant by name), but their approach applies rotation to existing block quantization formats. Our tighter Lloyd-Max codebook + norm correction combo outperforms at lower bit rate.

ppl_f16_baseline 5.8048 ppl_turbo3_2k 5.8501 ppl_q4_0_rot_2k 5.8578 ppl_turbo3_65k_delta "+0.53%" ppl_q4_0_rot_65k_delta "+1.73%"