
curious about ai, robotics and art. design engineer at @heyglif. fun: https://xitter.cf/t.co/mEqjOEj5IJ
Berlin, Germany
Joined November 2013
- Tweets6.3K
- Following1.7K
- Followers3.2K
- Likes39.4K
Astra’s spatial understanding is mindblowing to watch.
three uncalibrated cameras. no intrinsics, no extrinsics.
it nudged the robot arm, learned how each camera sees motion, worked out the brush tip’s 3d offset, and verified it at new angles -- all from one prompt and a tiny follow up.
final result was under 0.2 mm of camera-measured tip movement!
i know it's not really the same, but all the neuron editing and exploration posts of Qwen remind of this short story by qntm: qntm.org/lena
one reason i love painting as a long-term project for my robot + the agents operating it is,
- infinite depth in technique and finesse.
- infinite breadth in creations.
it’s an eval + a creative outlet + a growing body of work, all at once.
the arm has 7 dof and ±0.1mm repeatability. high-res cameras, top-grade brushes, paper and pigments, a well-calibrated indoor workspace. in principle, i think something of Hokusai's or Monet’s calibre is possible with this setup.
you can see the progression here, bottom to top: calibration runs and basic brush tests, then colour mixing, wet-in-wet, glazing, angled strokes...
just getting started! so much more to learn + create 🦾
nil retweeted
Replying to @dhvanil
the crux of the "memory problem" is not how you store things (embeddings/markdowns) or how you retrieve. all memories are associative and depend on the task at hand. i bet task planning/decomposition and memory systems will morph together.
my biggest frustrations with ai tools rn is their opaque memory systems.
chatgpt knows some things about me, claude remembers smth else, claude code has a different system, codex feels like a sprawling web of long-forgotten chats, and codebase docs + skills are yet another system.
i DO want these memories to be contextual and siloed. different models and harnesses have different strengths. i want their memories to reflect that.
but a few things that really bother me:
1. Opacity: how are memories created, curated, updated, and forgotten? i have almost no visibility into this.
2. Context rot: what random memories and old chats are seeping into the current context window?
3. Curation: no tools to curate memories. i'd like to have powerful tools to inspect, edit, prune, and shape these memories.
nil retweeted
kay neurons are interpretable enough that models can modify their own weights toward some desiderata.
blogpost incoming
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
100%! especially with complicated creative work -- long videos, games, 3d scenes, -- there’s such a huge variety in output. you can’t really capture all that nuance in a few hundred words of prompts. plus, adding more detail often makes the models overindex on some stray detail.
what actually works much better:
- giving them lots of tasteful references and some high level pointers
- setting up tools so they can see what they’re making
- asking them to regularly critique their own work
- nudging them slightly over time (but not micromanaging)
A big part of eliciting interesting / novel outputs from bit models is what you ~don’t~ prompt. If you curate enough references and give rough directional guidance that is properly pointed, the models can interpolate style in ways you didn’t anticipate and that’s where the magic happens.
So in the case of putting them to work together, the only way to have them reliably use one another is to lose that instruction gap that allows creativity to shine through. In other words, it’s useless to force it for my purposes.
asked Claude to compose music like "Ryuichi Sakamoto's" for this video, and it spent hours going thru his catalog, analysing spectrograms, downloading scores, asking Gemini 100s of questions cuz it can "understand audio", using audio gen models, writing MIDI directly, what not.
i asked Claude Opus to turn this into a Chrome extension.
it basically works on any website. all credit for the original idea, code and motion to @notdetails 🫡
chrome extension (open-source) in 🧵













