NexusQuant: E8 lattice KV cache compression

Calibration-free KV cache compression. Same model, same prompt, 3 cache modes. Watch quality hold as compression increases.

pip install nexusquant-kv | GitHub | Papers

FP16 is the baseline. E8 K3V2 (key=3-bit, value=2-bit) is 6.1x compression. E8 K2V2 is 7.5x compression. Both are calibration-free.

Prompt

FP16 (baseline, 16 bpe)

E8 K3V2 (2.625 bpe, 6.1x)

E8 K2V2 (2.125 bpe, 7.5x)

Memory savings

Headline: +0.276% PPL (Mistral-7B-v0.1, wikitext-2, n=60 KIVI-iso) at 5.83x compression (Mistral-Inst-v0.3). Validated on 10 architecture families. Source data