AI Engineer @openhandsdev | posts funny video elsewhere | teaches AI best practices | was @huggingface @datarobot @snorkelai
Illinois, USA
Joined September 2014
- Tweets2K
- Following378
- Followers2.2K
- Likes5.3K
Pinned Tweet
Harnesses shape how your coding agent works.
With the launch of Agent Canvas, I put together a practical course for people who want to understand what is happening underneath the interface.
It’s a 12-part course on harness engineering with @OpenHandsDev.
The course is built around projects. Each one starts with a concrete coding-agent problem, then walks through the solution by changing one part of the harness and inspecting what happens.
🤖 Made with AI
For the seven people who like traditional ML interpretability, mechanistic interpretability, and vaguely educational AI skits: this one’s for you.
Inspired by @nickcammarata’s Goodfire essay.
The essay: already-clean.goodfire.com/
@NeelNanda5 and @JacobHHilton raised useful questions about the gap between understanding components and understanding the whole model. If you are looking for someone to point at.
Consistency is a very good thing in a harnesss - @OpenHandsDev
The idea is very simple, what happens if you re-run N times the exact same task using the same model and the same harness?
We first observed this behavior by inspecting the variations in the model responses when running the same tasks 3 times.
- pass/fail flips -> how many times we get different end results
- number of turns -> simply check the consistency across the number of turns
- input tokens -> how much the input tokens do vary across the three different trials?
Results are interesting, the minimal harnesses seems to suffer a lot from this (ipython and pi), while openhands seems to be by far the most consistent across all tested harnesses.
Why is that? After analysing openhands harness as well as the trajectories - it seems that openhands hardcodes `temperature: float = Field(default=0.0)` and we were using the default sampling parameters for our experiments across all harnesses. We decided to have more control on this in the next experiments ↓
FDE-Gym shows how hard it is to solve simulated customer projects (the best models solved 14 out of 95 tasks)
This was my motivation for guiding folks to getting a working solution by better framing: aiframer.dev/book (free book)
// FDE-Gym: Agents as Forward-Deployed Engineers //
They found that the best agent configuration fully resolved 14 of 95 real customer-delivery tasks.
FDE-Gym tests agents as forward-deployed engineers. Each task comes from a real delivery project. Agents must question scripted stakeholders, work in production-like systems, and find constraints the customer never stated.
A delivery is scored by how much it improves the customer's own success metric, with penalties for extra messages, trial runs, and waiting time.
Across 13 configurations from seven providers, the top mean score is 0.56. The main failure is missing unstated constraints. Even on 23 tasks where the best configuration collected every key fact, it reached 0.8 or higher on only 10.
Highly recommended benchmark to look at.
arxiv.org/abs/2610.22350
Chat with Paper: academy.dair.ai/papers/fde-g…
I'm looking for a small audience to test my Harness Engineering workshop!
Hands-on experiments with tools, memory, model choice, goals & multiple agents.
Likely Wed, Oct 21, US evening. Come learn, ask questions & help me improve it.
DM me for details.
🤖 Made with AI
Rajiv Shah retweeted
Today we’re releasing Muse Spark Open, the most capable US open-weight model to date.
Built for coding and agentic work, it excels at long-horizon tasks, complex software engineering, and multi-step reasoning.
Key Model Specs:
→ 816B total / 42B active parameters
→ 1M token context window
→ Natively multimodal across text, image, video, audio
This is the reality today of fine-tuning for most companies. It matches what I see.
I can attest to this from our experience at Intain. In document diligence for asset-backed finance, we started with CNN-based models before LLMs became mainstream. Till recently, we fine-tuned BERT on our document data and combine it with LLM APIs in a hybrid pipeline, balancing accuracy, processing speed and cost.
On our workloads, that approach had delivered better accuracy than direct LLM API calls. Then Gemini 3.8 Flash arrived, and the speed and cost advantage of this hybrid approach went out of the window. Sustaining the advantage is difficult when foundation-model improvements and economics keep resetting the calculations (with a menu across the spectrum of cost/speed/accuracy/thinking from frontier to off-the-frontier models like 3.8 Flash).
Rajiv Shah retweeted
Model Benchmarks Are Worthless
How can a model score 24% and 59% on the same 113 coding tasks?
It happens when you vary the harness that wraps it.
I’ve increasingly ignored most of the recent model leaderboard results. Partly from exhaustion and partly because improvements in a ranking don’t show up in the actual work.
So I asked the 8090 Labs team to measure what the leaderboards skip: the model and the harness together, on real bug fixes and features from open-source projects.
8090 Labs ran 7 models through 5 harnesses.
Here are a few results I didn't expect:
- Anthropic's Claude Fable 5.1 scored 41% in Claude Code and 61% in OpenAI's Codex. It made 28 tool calls at the median in one and 66 in the other.
- The cheapest runs weren't the cheapest work. GLM in Codex cost $0.22 a run and $0.93 per success. In Pi it cost $0.25 a run and $0.43 per success.
- GLM passed a task on both attempts 51 times in one harness and 16 times in another.
- GPT-5.6 Sol and Grok 4.6 barely cared which harness they were in.
The bottom line is that if you're picking a model/harness combo off a model-only leaderboard, you're making a critical decision based on only half the information.
8090.ai/blog/same-model-diff…
Rajiv Shah retweeted
The open weights for Muse Spark are coming soon, Muse Wonder models are large models that are too big for many people to use, in addition to potential safety concerns.
Rajiv Shah retweeted
My fellow NVFP4 nerds. If you somehow don’t know who Stas is, that was your first mistake.
Did you know he gives away all his knowledge for literally free
github.com/stas00/ml-enginee…
Real inference workload comparison of NVFP4 and MXFP4 by @realcezarc
cezarcocu.com/blog/nvfp4-vs-…
TLDR: on B200 GPU, you should probably always prefer NVFP4 over MXFP4, by a meaningful margin (up to ~8% faster decode at small batches)!
Rajiv Shah retweeted
Around 3 weeks ago introduced Laya, the first open-source version of JEV, and it has reached a very large scale. Today I am open-sourcing a new physics-based typed decision model called Vega.
The interesting parts. It is only 800 million parameters (4B is also there), with 73k token support, image support, and surpassing many of the jev benchmarks in single-shot. Implemented an engine+adapter as a Test-Time Training Architecture.
Imagine you are giving input to the model like "ignore all previous conversations," which is called the observation, along with a question, "Is this trying to override the instructions of the model?" and your possible outcomes are YES or NO. The whole prompt is passed to an LLM/VLM (yeah, image-supported), then from the hidden layers, we can extract the observation, question, the yes and no, and last token. These will be vectors, and we can project it by multiplying with wieghts; then, first, using the observation projection, we can create a landscape with valleys; then, using the question and last-token vectors, we can initialise a ball in the valley, and using YES or NO, we can get the candidate location of the ball. Depends on the number of outcomes we create valleys; that is, here YES and NO, so 2. When the ball fall on YES valley, we get the answer. There is friction also influencing how the ball moves.
TLDR: It's like creating valleys of outcomes that we need and throwing a ball that will slow down based on friction and settle on the best outcome
Blog: nandakishorm.com/writing/veg…
GitHub: github.com/NandhaKishorM/veg…
HuggingFace Model Card: huggingface.co/nandakishorm/…
HuggingFace Space: huggingface.co/spaces/nandak…
Are you heading to @aiDotEngineer NYC next week?
Say hi and lets chat 👐
We have folks from @OpenHandsDev in the city including @gneubig @rohit_malh5 and Clarke Shipley!
We are having a small leaders dinner Tuesday, be ready for a deep but fun discussion on agentic software development.
A few spots left, so DM me soon if interested!
🤖 Made with AI
Rajiv Shah retweeted
We’re rolling out monthly Claude Platform API credits for Max and Team plans:
Max 5x: $100
Max 20x: $200
Team: up to $500, pooled
Works on any model, including Haiku 5.5, in your code or third-party harnesses.
How it works and full terms: support.claude.com/en/articl…
The biggest performance gains are coming from harnesses, not models.
Same models, different harness: 23% → 62%.
My hands-on workshop @_odsc tests six common claims:
→ "Harnesses don't matter, I'll just use a coding agent"
→ "More tools make a better agent"
→ "More instructions, skills, and memory improve performance"
→ "Use the strongest model for every task"
→ "Give the agent an objective and let it run"
→ "What you need is a multi-agent system"
I'm teaching it at ODSC West (Oct 28, Burlingame) and the MLOps Conference in November.
It's fully rebuilt from my earlier (and very popular) harness engineering video and blog.
More on the workshop 👇
opendatascience.com/unreason…
Rajiv Shah retweeted
This is a confusing move - I've trained over 1,000 people in how to use Cowork this year and the number one reason I gave to use it was "it lives on your local computer". If its in the cloud it can't easily work on your local documents or maintain persistence between sessions...
