Joined October 2025
- Tweets21
- Following66
- Followers63
- Likes40
Pinned Tweet
What happens when we compress the KV cache of prompts with multiple instructions? 🤔
Existing compression methods can lead to some instructions being ignored. 🙀
We propose simple changes to KV cache eviction that fix this problem alongside other pitfalls to be aware of. 💯
Alex Chen retweeted
What happens if you interpret attention scores as probabilities? We can then use sampling techniques in KV cache eviction. Check out our new paper!
“A Probabilistic Interpretation of KV Cache Eviction”
Most KV eviction methods use top-k heuristics to decide what to keep, but this can heavily distort attention after tokens are removed.
This paper reframes KV eviction as expectation estimation, then uses probabilistic sampling and importance weighting to correct for missing KV entries during decoding.
This gives a principled bias-variance tradeoff that stays competitive with existing methods while being more robust.
alphaxiv.org/abs/2608.28293
Alex Chen retweeted
Every week, the price-intelligence-latency frontier shifts, and we expect this trend to continue
Across 100+ use cases in our product, keeping all up to date with the right model is a challenge -
Either we're losing out on intelligence for the dollars we spend, or we're spending too much money for the intelligence the feature needs
Ramp Router lets you benefit immediately without rewriting your application. We cut our LLM costs by 30%, while also making our features smarter and faster.
We built it for Ramp. Now we’re opening it up to everyone.
get access here: ramp.com/router?utm_source=X…
Alex Chen retweeted
Introducing PorTAL: Portable Task Adapters for LLMs. A novel recipe to cheaply port fine-tuning between models. It matches per task LoRA accuracy at half the cost, lowering the switching overhead of adapting tasks across LLMs.
At Ramp, every new model release used to mean retraining our fine-tunes from scratch. PorTAL learns the task once, then efficiently refits it onto any new base model, even across model families.
Alex Chen retweeted
(1/n)🚀 Introducing GCPO (Guidance Contrastive Policy Optimization), a new RL algorithm for visual and language generative models. Unlike existing methods, GCPO assigns per-token credit by compare the model's predictions with contrasting prompts, and emphasize key tokens.
Alex Chen retweeted
Check out our most recent work on dLLM!
One of the biggest promises of Diffusion LLMs is parallel generation: predicting multiple tokens at once to bypass the sequential bottleneck of autoregressive models.
However, parallel generation comes with a price. For example:
Should the sentence “He is from [MASK] [MASK]” be filled with [New] [York] or [San] [Diego]?
If a diffusion model predicts both at the exact same time, it assumes independence and may produce... [San] [York]. 🤦♂️
We argue this arises from a structural misspecification: models are restricted to fully factorized outputs because parameterizing the full joint distribution would require a prohibitively massive output head.
This is the Factorization Barrier crippling parallel generation. Here is how we broke it with CoDD.
Alex Chen retweeted
Introducing 💡On-Policy Self-Distillation💡, a simple method that enables LLM to teach itself with dense per-token feedback on its own on-policy generations—achieving 4-8x more token efficiency vs. GRPO and outperforming both GRPO and SFT/Off-Policy Distillation.
Key insight: like a student reviewing solutions, rationalizing them, and correcting prior mistakes, an LLM can be conditioned on privileged info (e.g., correct solution or a reasoning trace) and supervise its weaker self—the version without such access—by matching the privileged-info-induced distribution from itself.
🌐Blog: siyan-zhao.github.io/blog/20…
🧵👇
Alex Chen retweeted
(1/8) We introduce Sparse-LaViDa, a new framework that accelerates the inference speed of unified multi-modal diffusion language models via a novel sparse parameterization. It achieves up to 2.8x speed up on tasks including image generation, editing, and visual math reasoning.
Alex Chen retweeted
"An hour of planning can save you 10 hours of doing."
✨📝 Planned Diffusion 📝 ✨ makes a plan before parallel dLLM generation.
Planned Diffusion runs 1.2-1.8× faster than autoregressive and an order of magnitude faster than diffusion, while staying within 0.9–5% AR quality.
We propose Fair Eviction Policies: forcing each instruction to lose KV entries at equal rates. Similarly to whitelisting, fair eviction is able to lessen the degradation of defense at only a small cost to directive degradation.
Read the paper for more details! Done in collaboration with an awesome team: @danielmisrael, @renatogeh, and advisors @guyvdb and @adityagrover_
Project website: pitfalls-of-kv-cache-compres…
Paper: arxiv.org/abs/2510.00231
Github: github.com/Itisalex2/pitfall…