Performance
Benchmark Results
All numbers are measured at production context lengths on standard hardware. No cherry-picked short sequences, no accuracy–speed trade-off footnotes.
Faster Inference
At 500K+ context length
Measured against unoptimised attention on an H100 SXM5 with a 70B-class model. Gains compound as context length grows — the longer the context, the wider the gap.
KV Memory Reduction
Measured at 100K tokens
Key-value cache is the primary memory bottleneck at long context. Antilattice's attention compression shrinks the cache without eviction, retaining full sequence fidelity.
Concurrent Users per GPU
Validated on H100
Reduced per-request memory allows the runtime to batch more sequences simultaneously. Throughput scales linearly with the memory saving — no software tricks, just headroom.
Inference Speedup vs Context Length
Relative throughput (tokens/sec) compared to standard attention on the same hardware. 70B-class model, H100 SXM5, FP16.
| Context Length | Baseline (Standard Attn) | Antilattice |
|---|---|---|
| 8K | 1× | 1.4× |
| 32K | 1× | 2.1× |
| 128K | 1× | 3.8× |
| 256K | 1× | 4.6× |
| 500K+ | 1× | 5×+ |
Benchmarks
Code Completion — Real Python Files
Token Exact Match vs actual next 128 tokens. Real IDE contexts: 50 open tabs, imports, dead code. Eviction removes irrelevant files — the model attends only to relevant code.
| Context | Model | KV Evicted | Token EM | Edit Similarity | Decode Speedup | Result |
|---|---|---|---|---|---|---|
| 30K | Qwen2.5-Coder-7B | 84% | 89.4% | 93.5% | 1.21× | — |
| 32K | Qwen3-8B | 81% | 100% | 100% | 1.27× | No loss |
| 48K | Qwen3-8B | 87% | 100% | 92.3% | 1.59× | No loss |
| 80K | Qwen3-8B | 90% | 100% | 100% | 2.22× | Best quality |
| 30K | Llama-3.1-70B | 92.4% | 86.1% | 97.0% | 3.19× | Peak speed |
Quality fully preserved even with 80–92% of the KV cache evicted — the model attends to a fraction of tokens yet produces equivalent output, running up to 3.19× faster.
Peak Speedup
Llama-3.1-70B, 30K context
Max KV Evicted
With no quality loss
Quality Retained
Token EM at 32K–80K
Prefill Caching
Prefix caching that actually scales.
Standard prefix caching breaks down the moment you have more than a handful of long-context tenants. The cache becomes larger than the GPU. We solve the root cause, not the symptom.
The problem
64 GB per sequence.
On a shared GPU.
At 200K context on a 70B model (FP16, GQA), a single sequence's KV cache occupies ~64 GB — larger than an L40S. On shared multi-tenant infrastructure, vLLM APC and RadixAttention thrash past a handful of concurrent long-context users: the cache evicts itself faster than it pays back.
The fix
3–5× prefill reuse.
1/10th the footprint.
Antilattice's prefill cache achieves 3–5× reuse. The cache fits on-chip for hundreds of concurrent tenants — the regime where prefix caching structurally cannot operate.
Built for agents, not chatbots
Long-running agents re-read the same large context window — codebases, document corpora, conversation histories — on nearly every step. Each re-read is a full prefill unless the cache holds.
Standard prefix caching was designed for short, repetitive chat system prompts. Antilattice's prefill cache is designed for the opposite: deep, unique, multi-hundred-thousand-token contexts shared across many concurrent agent threads where the standard approach structurally cannot operate.
Methodology
How we measure
Benchmarks run on isolated H100 SXM5 instances with no competing workloads. We report median throughput across 50 inference requests at each context length, with a 10-request warmup discarded.
Accuracy is verified by comparing output logit distributions against unoptimised attention with a KL-divergence threshold of <0.001. Any configuration that fails this threshold is excluded from results.
Independent replication packets available on request at hello@antilattice.com.