curious about ai, robotics and art. design engineer at @heyglif. fun: https://xitter.cf/t.co/mEqjOEj5IJ

Berlin, Germany
Joined November 2013
Pinned Tweet
Astra’s spatial understanding is mindblowing to watch. three uncalibrated cameras. no intrinsics, no extrinsics. it nudged the robot arm, learned how each camera sees motion, worked out the brush tip’s 3d offset, and verified it at new angles -- all from one prompt and a tiny follow up. final result was under 0.2 mm of camera-measured tip movement!
13
25
5
401
21,381
"Soft Voltage" by Astra asked it to expand more, and it said: "A little electricity, a little drifting underwater -- the robot’s movement turning into something soft and alive."
2
2
23
1,337
one reason i love painting as a long-term project for my robot + the agents operating it is, - infinite depth in technique and finesse. - infinite breadth in creations. it’s an eval + a creative outlet + a growing body of work, all at once. the arm has 7 dof and ±0.1mm repeatability. high-res cameras, top-grade brushes, paper and pigments, a well-calibrated indoor workspace. in principle, i think something of Hokusai's or Monet’s calibre is possible with this setup. you can see the progression here, bottom to top: calibration runs and basic brush tests, then colour mixing, wet-in-wet, glazing, angled strokes... just getting started! so much more to learn + create 🦾
1
3
273
sometimes i feel sad for my robot friends
1
2
368
Replying to @dhvanil
the crux of the "memory problem" is not how you store things (embeddings/markdowns) or how you retrieve. all memories are associative and depend on the task at hand. i bet task planning/decomposition and memory systems will morph together.
1
1
89
my biggest frustrations with ai tools rn is their opaque memory systems. chatgpt knows some things about me, claude remembers smth else, claude code has a different system, codex feels like a sprawling web of long-forgotten chats, and codebase docs + skills are yet another system. i DO want these memories to be contextual and siloed. different models and harnesses have different strengths. i want their memories to reflect that. but a few things that really bother me: 1. Opacity: how are memories created, curated, updated, and forgotten? i have almost no visibility into this. 2. Context rot: what random memories and old chats are seeping into the current context window? 3. Curation: no tools to curate memories. i'd like to have powerful tools to inspect, edit, prune, and shape these memories.
16
1
2
16
1,258
Right to be Forgotten (by your agents)
250
kay neurons are interpretable enough that models can modify their own weights toward some desiderata. blogpost incoming
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
19
37
4
653
24,501
some calibration issues still need to be ironed out, but loving this wet-in-wet watercolor bleed style! titled "Slight Drift" by Opus
5
21
1,277
wondering if i should just run a 24/7 agent on my computer to keep checking for and patching vulnerabilities!
2
293
having such a bizarre interaction in codex about printing a webpage where Astra and the tab autocomplete model are arguing with each other. (the previous user messages were also just me tab auto-sending)
1
1
277
this feels so weird
1
1
114
and it continues...
56
software is going to be a lot lot lot more malleable than we can imagine today
9
420
the new rocket booster roadster would be pretty cool at 0.16G on the moon, i guess
1
3
370
100%! especially with complicated creative work -- long videos, games, 3d scenes, -- there’s such a huge variety in output. you can’t really capture all that nuance in a few hundred words of prompts. plus, adding more detail often makes the models overindex on some stray detail. what actually works much better: - giving them lots of tasteful references and some high level pointers - setting up tools so they can see what they’re making - asking them to regularly critique their own work - nudging them slightly over time (but not micromanaging)
A big part of eliciting interesting / novel outputs from bit models is what you ~don’t~ prompt. If you curate enough references and give rough directional guidance that is properly pointed, the models can interpolate style in ways you didn’t anticipate and that’s where the magic happens. So in the case of putting them to work together, the only way to have them reliably use one another is to lose that instruction gap that allows creativity to shine through. In other words, it’s useless to force it for my purposes.
4
291
asked Claude to compose music like "Ryuichi Sakamoto's" for this video, and it spent hours going thru his catalog, analysing spectrograms, downloading scores, asking Gemini 100s of questions cuz it can "understand audio", using audio gen models, writing MIDI directly, what not.
2
5
408
i asked Claude Opus to turn this into a Chrome extension. it basically works on any website. all credit for the original idea, code and motion to @notdetails 🫡 chrome extension (open-source) in 🧵
what if your website built itself
17
7
2
207
12,661
we are slowly getting better at watercolor (and video editing, and music composition)
1
11
666
got to do smth about the water pooling problem though. this rough handmade paper makes the matter worse.
93