Chunk size sweep at fixed context

proposed medium priority TODO-005

Overview Experiments 96 Forks 3 Resources 36 Benchmarks 2 Broadcasts 3 Related

Description

There is a sweet spot between chunk sizes — smaller chunks waste kernel launches, larger chunks thrash VRAM. Profiling 256..8192 at a fixed context reveals the tradeoff.

Reference

EXP-0002

Suggested Parameters

context 32768

chunk_sizes [256

cache_type tbq3

Provenance

Proposed by @dusterbloom via adaptive-chunked-prefill claude-sonnet-4-6