Elev retweeted
Introducing *HackBench* 🤡💀⏱️📊 ⏲️ 💪🔧 🔥
- Detecting & localizing reward hacking in autoresearch agents
- Frontier models fail
- Dataset on HuggingFace
- Recipe to collect natural & synthetic hacks
Blog post: facebookresearch.github.io/R…
🧵1/5
Elev retweeted
What follows is the story of a founder with zero distributed systems experience who shipped a long-running AI support agent that survives crashes, missing approvals, and payment timeouts.
His name is Chris. His agent checked an order, proposed a refund, waited for approval, and issued the refund.
Everything was fine in the demo, but in production, some tickets took up to 2 days. And in 2 days:
The process crashed while checking the order.
The approval never arrived.
The refund API timed out after the payment already went through.
He learned a few painful lessons.
Here's what's funny about Chris: he knew nothing about queues, retries, or the patterns around them.
Zero. Nada. Zilch.
But before he wrote more code, he asked one question:
"What happens when it stops halfway through?"
So he designed for it:
1. He made the order lookup and refund proposal separate steps, so a crash resumes instead of repeating.
2. He used step.waitForEvent() for the approval, with a deadline and an explicit timeout path.
3. He built the refund as its own step, with the payment provider's idempotency key.
4. He got one trace that shows which steps completed, waited, or failed.
That's durable execution, and Inngest (supporting this post) runs it.
You define the steps in your code. Inngest keeps the state, handles the waiting, and retries what fails. No queue to babysit.
Anyway….
Before you hand real work to an agent, ask his question.
inngest.link/system-design-c…
🤝 Paid partnership
Brilliant paper on scaling laws for agentic RL data.
It's a great paper worth reading if you are synthesizing data for agentic RL.
(bookmark it)
They fit scaling laws for agentic RL data along three axes: tasks, environments, and trajectories. Across model sizes, families, domains, and both PPO and GRPO, the exponents order the same way, with environments above tasks and tasks above trajectories.
The environment advantage seems to come mainly from world diversity, meaning genuinely different settings. Varying the structure of the same world helps about as much as adding tasks.
The laws also give each synthesis method a discount factor and a synthetic-to-real ratio at which training starts to collapse. Those ratios differ by an order of magnitude across axes.
In addition, allocating the budget with their recipe gives up to 1.7x compute efficiency over the common practice of generating more trajectories.
arxiv.org/abs/2610.23946
Chat with Paper: academy.dair.ai/papers/how-d…
Elev retweeted
OpenClaw deleted around 400k LOC of its own tests without much change in code coverage. Modern models just love writing tests for every tiny change, even if they aren't useful. This skill helped. github.com/openclaw/openclaw…
Elev retweeted
The main appeal of open-source agent harnesses isn't that they are free, but that we can inspect what they are doing on our computers.
Jev + Kimi K3 for fraud detection!
TLDR: Jev classified 100 emails in 1.42 seconds, then I routed the uncertain cases to Kimi K3. The full pipeline got 96/100 correct for only ~$0.07.
Video is not sped up, check out the live run!
Here was my process:
I gave Jev 100 emails to classify (a mix of 50 legit & 50 fraudelent emails). It classified all of them in 1.42 seconds.
An underrated feature about Jev is it will give you the confidence score for a classification, so I routed any prediction under 95% confidence to Kimi K3 to be fully sure.
31 emails fell below that threshold. After routing those to Kimi K3, the combined pipeline reached 96% accuracy.
The full run took 16 seconds & ~$0.07 in inference costs:
- $0.068 from Kimi K3 on @togethercompute
- $0.003 (1/3 of a cent) from Jev on @typesafeai.
I think this is a really interesting pattern: use a fast specialized model like Jev for the narrow task, then route the uncertain cases to a larger LLM.
I feel like this kind of approach could be a game changer for use cases like fraud or anything realtime. You can use the speed & low cost of Jev while having a larger LLM as a fallback to ensure high accuracy.
We're sharing our new framework for tracking, investigating, and disclosing instances of model misalignment at OpenAI.
The framework sets criteria and timelines for public disclosure, including when we haven’t yet fully explained or mitigated the behavior. More complex cases may require longer investigation or coordination with third parties.
We’ll prioritize examples that reveal new misalignment mechanisms, meaningful changes in known behavior, or findings that challenge assumptions about safety or mitigation.
Alongside the framework, we’re publishing six reports on instances of misaligned behavior we’ve observed during the training or evaluation of our models in the last six months.
This is a starting point. We’ll refine the process through experience and public feedback, and share more reports on an ongoing basis.
openai.com/index/model-misal…
Claude Cowork and chat are merging into one Claude.
Ask a quick question or hand over a report, and Claude takes it from there, even after you close your laptop. If something's unclear, Claude asks—you keep the final say.
Rolling out to Pro and Max over the next few weeks.
Elev retweeted
‼️ A browser extension could hijack AI agents in Comet, Edge, Opera Neon and Claude, while abusing Chrome’s Gemini integration to reach local files, the camera and mic.
Researchers used the same underlying trust-boundary weakness across all five products after the extension was already installed.
Here's the technique → thehackernews.com/2026/09/on…
Elev retweeted
TypeSafe just priced output tokens at zero.
Input is $0.042 per million tokens, per the launch. Every yes or no from a language model arrives inside a sentence it had to write one token at a time, and you pay for the whole sentence. Pricing output at zero follows from a model that writes almost none of it.
Diogo's launch post has the mechanism, and the waitlist. TypeSafe says Jev, its first public System One Model, answers many structured questions at once and returns a typed decision with a probability attached, so the calling code decides what happens next.
So an LLM call becomes something your code can branch on directly.
The launch example is support tickets. One long prompt splits into the judgments the app needs, is this urgent, is it billing, does it need a human? Each answer comes back typed with a confidence score, you pick a threshold, and anything under it goes to a person.
TypeSafe's phrase for it is smart if statements.
The confidence number is the piece I'd watch. Cheap tokens were never the whole blocker, since an output you have to double check still waits on a human before it touches production data. A probability that holds up the way TypeSafe describes would be the signal for when to let things run unattended.
Once a judgment is cheap and legible it lands in form handlers, notification checks, and every small place nobody would have written a prompt for. Threshold tuning turns into a real engineering skill, and the teams who learn it early automate the boring half of their stack.
🤝 Paid partnership
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?
I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev
• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions
AFAICT the shortest path to AI-based economic revolution
Is Bash All You Need?
Interesting paper from Microsoft.
If you are deciding which tools to give an enterprise agent, you might want to check this out.
(bookmark it)
They compared five tool interfaces on TheAgentCompany and APEX-Agents with Opus-4.8 and GPT-5.5. The options range from a catalog of typed tools, to bash alone, to programmatic tool calling where the agent writes code against a fixed catalog.
Bash alone scored 21.8 to 24.5 points higher than typed tools on TheAgentCompany and 4.8 to 7.4 points higher on APEX-Agents, while using 19% to 72% fewer tokens. Adding typed tools or agent-written tools on top of bash gave no measurable gain.
The authors recommend bash when execution can be sandboxed, and programmatic tool calling when compliance requires a fixed tool list.
Paper: academy.dair.ai/papers/is-ba…
Elev retweeted
Live now: our Harness Engineering Track from AI Engineer World's Fair 2026.
Ports, proofs, kill switches, and what is left of an agent when you remove the model.
Thesis: when an agent breaks in production, the model is rarely the part that failed.
youtube.com/watch?v=PXj0p_mW…
- 2026 State of AI Engineering: @barrnanas, Amplify Partners
- The Unreasonable Effectiveness of Separating the Task from the Model: @MaximeRivest & @isaacbmiller1, DSPy
- How Anthropic Builds: @mikeyk, Anthropic
- Tokens Should Have Jobs: @katelyn_lesse & @angjiang, Anthropic
- Building the Production Cage for Powerful Domain Agents: Mike Chambers, AWS
- Loophole: @brendanh0gan, Morgan Stanley
- Every step you take, every call you make: @vdgiselle, Restate
- Agent Frameworks Considered Harmful: @remilouf, .txt
- We let an AI agent execute Bash and lived to talk about it: Sarah Sanders, PostHog
- No Memory, No Harness: Kay Malcolm, Oracle
- How we Solved Agent Building: @andrewqu, Vercel
- Agents Without Code: @_philschmid, Google DeepMind
- "I've never seen anything scarier than an LLM with tool calls." @headinthebox
GPT-5.3-Codex-Spark launched on February 12, 2026.
Can you believe AI models are getting retired within 7 months of launch? 🤯
Elev retweeted
Custom GPTs in ChatGPT will likely be retired on December 11
"Migrate your GPTs by December 11 - GPTs that aren’t migrated won’t be available after December 11. Migrate yours so people can keep using them as plugins."
"Create a plugin instead - Plugins are replacing GPTs as the new home for custom tools. Create a plugin now, or continue creating a GPT during the two-week transition."
The real architecture behind "a team of AI agents" - not the hyped version:
One supervisor agent. It doesn't do the work - it just decides who does.
A handful of specialist workers, each with its own tools and exactly one job: a research agent, a writer agent, a code agent.
Every handoff between them is one step on a graph. The graph tracks the shared state, so if the whole thing crashes mid-task, it resumes from the last completed step instead of starting over from message one.
This is what LangGraph's supervisor pattern actually is. It's a real, open-source framework, not a secret from a paid course.
The part almost nobody posts about: the engineers who actually run this in production say most tasks people build a "team of agents" for should have been one well-prompted agent with good tools instead.
Multi-agent is genuinely more capable for the right problem. It also costs several times more in tokens, and adds a new failure mode - a supervisor stuck in a routing loop, bouncing a task between two workers who each keep deferring back to the other.
Use the pattern when the task genuinely needs different domains of expertise working together. Skip it when one agent with the right tools would just work.





























