shivamm

i spent the last 10 days trying to build a language model that never runs out of memory, on a MacBook Air. it works. it's faster, it uses less energy, and it writes better tool calls than a transformer of the same size. but last night it failed the one test I cared about most. here's the full story, with real numbers. the goal transformers have a dirty secret: the longer they read, the heavier they get. every word they've seen sits in memory (the "kv cache"), and every new word has to look back at all of them. a robot or an agent that runs for hours can't live like that. so i set out to build a small model where: • memory stays the same size no matter how long it runs • cost per word stays the same at word 10 and at word 10 million • quality matches or beats a transformer of the same size everything was trained from scratch on one 16 gb m4 macbook air, using apple's mlx: 11 languages (including hindi, chinese, arabic and swahili), plus tool-call data. each model was ~14m parameters and saw 49m tokens, with each run taking ~2 hours. every number below comes from a logged run. what i built a hybrid: 9 gated deltanet layers (a fixed-size recurrent memory, from existing research) plus 3 layers of my own idea, which i call priority memory. priority memory is a small bank of exact "slots": 64 per head. every word gets a learned importance score. the bank always holds the highest-priority words seen so far, and the rest get evicted. it's like a notebook with a fixed number of pages where you only keep what matters. compared against: • a tuned transformer (same size, same data, same everything) • the same model with my layer removed (pure deltanet), to check whether my idea actually does anything — round 1: efficiency (big win) memory needed to keep reading: • transformer: 13 mb at 512 tokens → 805 mb at 32k tokens (and still growing) • mine: 1.07 mb. at every length. forever. total ram at 32k tokens: 2.5 gb vs 176 mb. speed generating text: • transformer: 585 tokens/sec at 512 → 124 at 8k → 30 at 32k • mine: ~326 tokens/sec at every length the transformer is faster on short text. but at 8k tokens mine is 2.6x faster, and at 32k it's 11x faster. energy (measured with powermetrics, above idle): • generating: 23.6 → 17.9 mj per token (−24%) • reading: 0.83 → 0.32 mj per token (−61%) — round 2: quality (small win) bits per byte on held-out text, lower is better: • transformer: 1.642 • deltanet only: 1.636 • mine: 1.619 mine won on all 11 languages, plus tool-call text (−4%). tool calls, fully correct json with the right name and arguments: transformer 38%, deltanet only 37%, mine 44% — a lesson in between: short runs lie before the full runs, i did quick 12m-token tests to tune learning rates. in those, the deltanet-only model beat mine. i honestly thought my layer was useless. then i ran the full 49m tokens. deltanet-only started faster, but mine kept improving, overtook it around step 3,500, and finished ahead on every language. if i had trusted the short runs, i would have thrown away the idea lol. — round 3: memory for facts (where it gets interesting) the test: plant a fact ("the secret code for x is 48213"), add filler text, then ask for it. the score is how many bits more the model prefers the true code over a fake one. at distance 0: • transformer: 10.6 • mine: 6.3 • deltanet only: 1.6 so my layer gives ~4x the recall of pure deltanet. that's the job it was designed for. the transformer was still better, but every model collapsed beyond 512 tokens, which is the length they were trained on. fair enough: 512 tokens is too short for a long-term memory to matter. so i ran the real test. — round 4: the long-text test (the failure) all three models got 12m more tokens of training on 2,048-token text. a quarter of it was recall practice: facts planted early, questions much later. the wording and names were deliberately different from the test, so they couldn't just memorise phrasing. my expectation: the slot memory finally gets a job, learns what to keep, and pulls ahead. recall afterwards (before → after): transformer: • 0 tokens away: 10.6 → 15.9 • 512 away: 1.5 → 9.9 • 1,024 away: 0.2 → 4.6 • 1,800 away: 0.03 → 1.7 mine: • 0 away: 6.3 → 3.6 • 512 away: 1.5 → 1.2 • 1,024 away: 0.9 → 0.5 deltanet only: • 0 away: 1.6 → 1.0 the transformer learned to remember across its whole window. both fixed-memory models got worse. text quality ended up tied (1.617 mine vs 1.619 transformer). tool calls: still mine, 42% vs 38%. and nobody recalled anything beyond 4k tokens (beyond the training length). — why i think it failed my best guess: my memory decides what to throw away, but nothing in training ever tells it "you threw away something you needed later". the keep/evict decision is a hard cut-off, so no learning signal flows through it. it literally cannot learn from its own forgetting. two smaller suspects: • 3 of my 4 memory heads have a fixed bias toward recent words, so only one head can hold a fact for 1,000+ tokens • ~1,500 recall examples is probably too little practice a transformer doesn't have this problem. it never forgets anything inside its window, so it just has to learn where to look. a fixed-memory model has to learn what to keep, and that is a much harder lesson. — what i take away from this constant memory is the easy part. you get the efficiency for free, by construction. recall has to be earned, and the hard part isn't the memory structure. it's the training signal that teaches the model what's worth keeping. judge architectures on full runs and on the task they're built for, not on short runs or average loss. you can do real architecture research on a laptop. caveats, because they matter: tiny models (14m), one seed each, single machine. this is a lab notebook, not a paper. if you work on recurrent models, state-space models, or long-context memory, i'd love to hear how you'd attack the "learning what to keep" problem.

i gave claude opus 5 random photos and asked it to doodle on them. no image gen, every wobbly line is javascript on a canvas ✏️ it draws them stroke by stroke, then they come alive. even the music is code 🎶 check below 🔽

🧵built a local visual explorer for pgvector embeddings. paste your postgres connection string → pick a table → see your embeddings as a 2D scatter plot in seconds. no cloud. no auth. no bs. runs entirely on your machine github.com/rishabhguptajs/pg…

i was building a research agent, and when it felt completed, there was no better topic in my mind that this one fs! XD some details regarding the agent : - tavily api for search - openrouter for genai - nodejs backend / nextjs frontend (with ts) - mongodb, qdrant, & clerk auth.