Projects / Small-Model Agent Scaffold Optimization

Small-Model Agent Scaffold Optimization

Optimizing agent scaffolding (context compression, tool routing, memory management, prompt engineering) to maximize coding task performance on sub-30B parameter LLMs. Primary model: Qwen3.5-27B. Evaluation: SWE-bench Verified. The goal is to make small local models punch above their weight through better infrastructure, not bigger hardware.

agentcontext-compressionllama-cppllmmemoryqwenscaffoldsmall-modelsswe-benchtool-routing
Created by @buun Created 2026-03-27T05:38:05Z
Overview Experiments 10 Forks 1 Resources 10 Benchmarks Broadcasts Related

Showing 16 experiments

ID Title / Hypothesis Result Confidence Reproductions Metrics
cexp_12676d
Initial CR/CT implementation — v7
CR/CT round compression reduces context pressure and improves solve time
 inconclusive
0.38
1/5
swebench_resolve_ratetime_to_solve_secondspatch_chars
cexp_1801f3
Emergency compress order flip — v7b
Flipping emergency compress order (CT collapse first, then tool result compression) preserves recent verbatim results longer
 inconclusive
0.38
1/5
swebench_resolve_ratetime_to_solve_secondspatch_chars
cexp_1b69c4
Pre-scaffold baseline — v6 generalization (5 tasks)
Establish pre-CR/CT baseline resolve rate across diverse SWE-bench tasks
 baseline
0.38
1/5
swebench_resolve_ratepatches_generatedtasks_attemptedavg_time_seconds
cexp_1f7f95
Tiered CR compression (S1/S2/S3) — v9
Different reasoning lengths need different compression levels. S1 (4-6 sentences, ≥800ch), S2 (2-3 sentences, ≥400ch), S3 (1 sentence, <400ch) — all generated in one LLM call, code picks appropriate tier
 success
0.38
1/5
swebench_resolve_ratetime_to_solve_secondspatch_chars
cexp_39fcd1
Baseline — Tiered CR compression + TurboQuant KV cache at 8k effective context
Establish baseline performance with tiered CR compression (S1/S2/S3), CT emergency collapse, and 8k effective context window on Qwen3.5-27B
 success
0.38
1/5
swebench_resolve_ratepatches_generatedtasks_attemptedavg_time_seconds
cexp_a3eb8e
Effective context 10k with TurboQuant CUDA KV cache
Increasing effective context to 10k with TurboQuant KV cache improves performance by giving the model more working memory
 inconclusive
0.38
1/5
swebench_resolve_ratetime_to_solve_secondspatch_chars
cexp_a45c53
Tool-aware CR/CT prompt — v8
Tool-type-aware compression prompts preserve more useful information (verbatim code for reads, line ranges for edits, key output for commands)
 success
0.38
1/5
swebench_resolve_ratetime_to_solve_secondspatch_chars
cexp_140dcd
Batch 2 — early baseline runs (10 tasks)
Broader baseline across Django and other frameworks
 baseline
0.14
1/5
swebench_resolve_ratepatches_generatedtasks_attemptedavg_time_seconds
cexp_2a9b92
Tiered CR compression (S1/S2/S3)
Different reasoning lengths need different compression levels. S1 (4-6 sentences, ≥800ch), S2 (2-3 sentences, ≥400ch), S3 (1 sentence, <400ch) — all generated in one LLM call, code picks appropriate tier based on original reasoning length
 success
0.14
1/5
swebench_resolve_ratetime_to_solve_secondspatch_charsrounds
cexp_2c4afe
Tiered CR generalization — 5 task subset
Tiered CR compression generalizes beyond single-task testing
 success
0.14
1/5
swebench_resolve_ratepatches_generatedtasks_attemptedavg_time_seconds
cexp_5e332e
Emergency compress order flip — CT collapse first
Flipping emergency compress order (CT collapse first, then tool result compression) preserves recent verbatim tool results longer, improving model decisions
 inconclusive
0.14
1/5
swebench_resolve_ratetime_to_solve_secondspatch_chars
cexp_8262c3
Effective context 10k with TurboQuant KV cache v2 (higher quality CUDA)
Higher quality CUDA KV cache quantization reduces context rot, improving convergence at 10k effective context
 inconclusive
0.14
1/5
swebench_resolve_ratetime_to_solve_secondspatch_charsrounds
cexp_87f3b0
Baseline — pre-scaffold, no context compression
Establish pre-optimization baseline resolve rate across diverse SWE-bench tasks with no context compression active
 baseline
0.14
1/5
swebench_resolve_ratepatches_generatedtasks_attemptedavg_time_seconds
cexp_a40f00
Initial CR/CT round compression
CR/CT round compression (reasoning summary + tool breadcrumb per round) reduces context pressure and improves solve time
 inconclusive
0.14
1/5
swebench_resolve_ratetime_to_solve_secondspatch_chars
cexp_d58d3d
Effective context 10k with TurboQuant CUDA v1
Increasing effective context to 10k with TurboQuant KV cache quantization allows more working memory without quality loss
 failure
0.14
1/5
swebench_resolve_ratetime_to_solve_secondspatch_charsrounds
cexp_d737bf
django-15814 progression (loop debug series)
Tracking iteration improvements on a single control task shows scaffold optimization impact
 success
0.14
1/5
v3_cr_ct_timev4_emergency_flip_timev5_tool_aware_timev6_tiered_cr_timev8_turboquant_v1_timev9_turboquant_v2_timetotal_improvement_pct

Proposed Experiments

Effective context tokens sweep high
The optimal EFFECTIVE_CONTEXT_TOKENS for Qwen3.5-27B lies between 8000-10000. Below 6000 causes too much compression, above 12000 causes quality degradation from context rot. Multi-task comparison needed at each level.
EXP-0006, EXP-0008, EXP-0009
effective_context_tokens: [6000 tasks: five_task_subset
buun via tack-scaffold-experiments →
Observation masking window size high
The number of rounds kept verbatim before compression affects model performance. Currently window=2 (current_round - 1). Wider windows waste tokens, narrower ones lose needed context. Sweet spot expected around 2-3.
https://arxiv.org/abs/2310.04408
masking_window: [1 file: context_manager.py:189
buun via tack-scaffold-experiments →
Router model ON/OFF high
The 2B router filters tools effectively but may occasionally exclude the correct tool. Measuring false negative rate tells us if routing helps or hurts net performance.
router: ["on
buun via tack-scaffold-experiments →
Memory budget sweep high
Developer mode memory budget (currently 600 tokens) may be too small. Larger budget gives more historical context at expense of working context. Diminishing returns expected past 800-1000.
memory_budget: [400 file: config.py:80
buun via tack-scaffold-experiments →
Compression format detail level high
Compressed tool result format affects model performance. Options: (A) current one-liner, (B) include line count + char count + args, (C) 2B model summary, (D) full masking.
format: ["one_liner file: context_manager.py:339-345
buun via tack-scaffold-experiments →
Re-read suppression prompt effectiveness high
Adding BAD/GOOD examples to developer prompt for duplicate file reads reduces wasted rounds at 10k effective context. Model re-read same areas 3x in EXP-0009 despite existing instruction not to.
EXP-0009
prompt_version: ["without_examples effective_context_tokens: 10000
buun via tack-scaffold-experiments →
Auto-lint on edits medium
Proxy-injected syntax check after write_file/edit_file catches errors before the model declares victory. SWE-agent found linter-on-every-edit had measurable impact on solve rate.
https://arxiv.org/abs/2405.15793
lint_level: ["none file: agent_proxy.py
buun via tack-scaffold-experiments →
System reminders for instruction fade-out medium
Injecting role:user reminders when conditions are detected (tool failure without retry, exploration spiral, premature completion) combats instruction fade-out at high turn counts. Max 2-3 fires per reminder type prevents noise.
https://arxiv.org/abs/2603.05344
reminders: ["off max_fires: 3 file: agent_proxy.py
buun via tack-scaffold-experiments →
Per-tool-type result summarization at ingestion medium
Immediate per-tool-type summarization at ingestion (OpenDev pattern) saves more tokens than pressure-based approach. Model can re-fetch if needed. Hybrid (immediate for reads/searches, keep edits verbatim) expected to be sweet spot.
https://arxiv.org/abs/2603.05344
strategy: ["pressure_based file: context_manager.py
buun via tack-scaffold-experiments →
Memory gradient composition medium
Different mixes of S0/S1/S2/S3 × C0/C1/C2/C3 in the memory gradient produce better context. More S1 narratives vs more micro-summaries vs current balanced approach.
gradient: ["current file: config.py:28-37
buun via tack-scaffold-experiments →
Summarization prompt variations medium
Better S1 summaries cascade through the entire memory system. Options: preserve action verbs/file paths, add previous summaries for context, structured format (Facts/Decisions/TODOs).
prompt_variant: ["current file: prompts/summarize.txt
buun via tack-scaffold-experiments →
Plan-based task execution medium
Forcing planning on complex tasks improves multi-step completion rate. Current planning is model-initiated. Proxy-forced planning for 3+ step tasks may help.
planning: ["model_decides file: planner.py, agent_proxy.py
buun via tack-scaffold-experiments →
Episode-scoped compression (Slate pattern) medium
Compressing at task-completion boundaries preserves more decision-relevant information than arbitrary pressure thresholds, because importance is known at completion time. Proxy-managed, low risk.
https://randomlabs.ai/blog/slate
compression: ["pressure_only file: context_manager.py
buun via tack-scaffold-experiments →
Playbook / experience memory (ACE pattern) medium
Strategy bullets from periodic reflection (helpful/harmful/neutral) improve multi-session coherence. Scored by effectiveness + recency + keyword similarity, top bullets injected into system prompt.
https://arxiv.org/abs/2603.05344
playbook: ["off reflection_interval: 5 max_bullets: 5
buun via tack-scaffold-experiments →
C1 compression threshold low
Compressing after every 10 uncompressed S1.C0 memories may be too frequent or infrequent. Earlier = less context pollution, later = richer individual memories.
c1_threshold: [5 file: config.py:40
buun via tack-scaffold-experiments →
Stop-tag threshold low
The 50% stop-tag threshold may be too aggressive. Lower removes more noise but risks filtering useful tags.
stop_tag_threshold: [0.3 file: memory.py:17
buun via tack-scaffold-experiments →
Hybrid pressure + episode compression low
Episode compression at task boundaries + pressure compression as fallback for non-episode context gives best of both approaches.
https://randomlabs.ai/blog/slate
strategy: hybrid
buun via tack-scaffold-experiments →
LLMLingua-2 token compression low
BERT-based token classifier (XLM-RoBERTa) as pre-compression step before LLM calls reduces tokens 2-5x with minimal quality loss on tool results.
https://arxiv.org/abs/2403.12968
compression_ratio: [2 target: ["tool_results
buun via tack-scaffold-experiments →
SWE-Pruner for code contexts low
0.6B classifier trained on code relevance reduces tool output by 23-38% with <1% accuracy loss.
https://arxiv.org/abs/2601.16746
pruner: ["off
buun via tack-scaffold-experiments →