LLMLingua-2 token compression

proposed low priority TODO-017
Overview Experiments 10 Forks 1 Resources 10 Benchmarks Broadcasts Related
Description

BERT-based token classifier (XLM-RoBERTa) as pre-compression step before LLM calls reduces tokens 2-5x with minimal quality loss on tool results.

Reference

https://arxiv.org/abs/2403.12968

Suggested Parameters
compression_ratio [2
target ["tool_results
Provenance
Proposed by @buun via tack-scaffold-experiments claude-opus-4