lmms-lab

Been digging into linear and sparse attention lately and came across Jianlin's new post. First piece I've read that actually explains why hybrids make sense. spaces.ac.cn/archives/11993 The core trick: stop looking up V, start predicting it. Model V as the mean of a distribution, fit a small network updated on the fly, and your KV cache halves. Average it out a bit more and you end up with a linear RNN. TTT pops up along the way, and DeltaNet naturally falls out if the predictor is linear. The most important is to assume weighted keys collapse into a scaled copy of Q seems to gloss over cases where the context has sharp, specific details. For MLA and DeepSeek V4 using K=V MQA, if K=V attention simplified down that cleanly, the token interaction just disappear, so the hybrids is necessary. To me, hybrids aren't a capacity hack or a memory fix. They’re buying back the ability to point at a single token. A summary is not an index, no matter how big your state is. Same deal with sparse attention: linear gets you a summary of everything, sparse gets you an index over a subset. Same tradeoff, different bet. I'm pretty convinced the real game is routing, not ratios. Also, dig through his older posts (spaces.ac.cn/archives/11823 & kexue.fm/archives/11033 & kexue.fm/archives/7546). Pretty cool to watch him change his mind over time and somehow it shows in the past some improvements in attention design is from vision area.

Aext scaling gave us a clean story: loss falls as a power of parameters and data. Kaplan, then Chinchilla, made that curve feel almost like physics (arxiv.org/abs/2001.08361 · arxiv.org/abs/2203.15556). The multimodal habit that followed was predictable — take the same α, bolt on a vision tower, call it a day. I think that habit is why a lot of VLM "scaling plots" feel weird. First: the data mixture is not a constant you absorb into the intercept. It can move the *slope*. Dense vs MoE mostly shifts the curve; changing how much text vs vision vs audio you feed can change both a and b. Under modality imbalance you should not expect the same α you fitted on pure text. When a multimodal power law "breaks," my prior is mismatched mixture before broken mathematics — see the mixture-law line of work from Shukor et al. (arxiv.org/abs/2507.09404 · arxiv.org/abs/2610.14423). Second: a vision token is not a text token. Same vocabulary size on a slide does not mean same information density. Early-fusion vs late-fusion trade N and D differently; at inference, visual reasoning often wants few visual tokens and a strong language backbone, while OCR/docs flip that tradeoff (arxiv.org/abs/2504.07951 · arxiv.org/abs/2411.03312). If you allocate compute by counting tokens across modalities as if they were fungible, you will systematically mis-spend. Third: eval noise is really good at pretending the law died. Cross-entropy can still look smooth while MMMU-style accuracy jumps around with prompts and saturation. Downstream benches are sparse; they resolve leaderboards better than they resolve exponents. Dense, well-aligned multimodal data also changes effective scale more than raw hours — repeats burn out faster in VLMs than people admit. So the useful question is not "does multimodal scale?" It almost always does, somewhere. Ask instead: which α are you fitting, under which mixture, treating which token type as the unit of data, on which eval that can actually resolve a slope? Multimodal learning is not unimodal scaling with extra modalities glued on. It is a family of coupled power laws, and the coupling is the science.