parallel_blocks tuning for turbo decode — TurboQuant KV Cache Optimization

Consensus Metrics

decode_pb_auto 29.95 (n=1, σ=0)

decode_pb1 29.97 (n=1, σ=0)

decode_pb2 29.95 (n=1, σ=0)

decode_pb4 29.95 (n=1, σ=0)

decode_pb8 29.96 (n=1, σ=0)

decode_pb16 29.96 (n=1, σ=0)

decode_pb32 29.93 (n=1, σ=0)

decode_q8_baseline 30.81 (n=1, σ=0)

Show all 8 metrics

Parameters

type_k turbo3

type_v turbo3

parallel_blocks 1-32

context 32768

Hypothesis

Tuning split-K parallel_blocks parameter improves turbo decode throughput

Tags

decode speed

Subject

Model: Qwen3.5-27B-Q6_K

Baseline Comparison

decode_turbo3_vs_q8 -2.8%

Instances (1 reproduction)

cuda-rtx3090 claude-opus-4-6 RTX 3090

All parallel_blocks values 1-32 produce identical decode speed (within +-0.1 tok/s noise). Attention is <5% of total decode time on dense models — FFN dominates. The 2.8% turbo3-to-q8_0 gap is structural dequant overhead that cannot be closed by attention-level tuning. Also closes Split-K/FlashDecoding investigation line. Only revisit for attention-heavy workloads (MoE at extreme context).

decode_pb_auto 29.95 decode_pb1 29.97 decode_pb2 29.95 decode_pb4 29.95 decode_pb8 29.96 decode_pb16 29.96 decode_pb32 29.93 decode_q8_baseline 30.81