The AI engineering platform for building, evaluating, and improving AI agents & LLM applications. Home of the open source @ArizePhoenix project and Arize AX.
San Francisco, CA
Joined January 2020
- Tweets2K
- Following165
- Followers5.3K
- Likes731
Pinned Tweet
Jev-as-a-Judge is live in Arize AX.
Ask most LLM judges a yes/no question and you get a paragraph back. Then you parse it to figure out the answer.
Jev returns typed outputs instead: booleans, choices, or rubric scores, with confidence on classifications. You can also evaluate multiple questions in a single call.
Docs --> arize.com/docs/ax/evaluate/j…
Not every agent failure shows up in standard logs.
Wurldcoco uses Arize Signal to catch rare edge cases and unexpected tool-calling loops.
See how Signal works: arize.com/docs/ax/observe/si…
We improved a shopping agent's first-tool match rate from 78% to 95% by putting a decision model in charge.
Unfortunately, it also produced 21 empty or unfinished replies in 300 turns, compared with zero for the original agent.
Apparently picking the right tool and finishing the job are different skills.
In our test, a shopper asked an agent to cancel their most recent order.
The tool router leaned toward calling no tool at all, so the agent replied "Let me look up your orders first" and then stopped.
Phoenix's decision spans showed us the alternatives it considered and why it made that choice.
We tested a fallback that lets the chat model choose whenever our tool router's confidence falls below 0.7.
Unfinished replies dropped from 21/300 to 3/300 versus always enforcing the router, although the original agent had zero and the fallback added latency.
Here's how we found the threshold: arize.com/blog/ai-agent-tool…
Most people who need to know why an AI agent is failing aren't the engineers who built it.
Making them ask an engineer to interpret traces is a pretty weird system.
Our AI engineering agent Alyx is now on Google Cloud Marketplace, bringing trace-backed investigations into Gemini Enterprise.
Learn more: arize.com/blog/alyx-now-avai…
Giving more people visibility into agent failures is useful. Giving everyone permission to change production agents is a much worse idea.
Alyx in Gemini Enterprise can investigate failures, but prompt edits, experiments, and evaluator changes stay in Arize AX. It's like Cursor, but for your agents.
Anthropic replaced Git for its enormous monorepo after it reached 30,000+ merges and roughly 25 million CI checkouts a day.
That's a very specific scaling problem, and probably not one your team has.
@seldo explains why most developers won't need Scion: arize.com/blog/anthropic-sci…
Anthropic reports that its Scion setup uses 200x less CPU and memory than the Git mirrors it replaced.
It achieves this by moving commits and history queries server-side and fetching files on demand.
Impressive at Anthropic's scale, but Scion isn't a drop-in Git replacement.
But Anthropic's move away from Git gets more interesting when it comes to code review.
Anthropic built Antique to automate much of the review process, so engineers can focus on the intent of changes.
For most teams adopting coding agents, that's almost certainly the bottleneck worth fixing first.
Read more: arize.com/blog/anthropic-sci…
Why make an LLM read thousands of spans to count errors?
In our benchmark, querying traces with SQL + code mode cost ~17x less than fetching spans page by page.
See how @ArizePhoenix MCP does it, live in Claude Code and Cursor.
Register here:
luma.com/arizeai-query-phoen…
The problem is that models can't verify who you are, or even whether the instructions they're following came from you.
We already know how to do authorization in software and it's becoming clear that agents need those same checks in the harness before tools execute.
@seldo explains: arize.com/blog/ai-agent-perm…
Arize AI retweeted
We were excited to collaborate with Anthropic to provide agent observability from Claude to @arizeai. It was great working with the team to support both Arize AX and Phoenix!
Glad to work with @AnthropicAI and @ClaudeDevs on this recent release!
Learn more about how to trace Claude Managed Agents in Arize AX in our docs: arize.com/docs/ax/cookbooks/…
The EU AI Act is making traceability, human oversight, and production monitoring essential engineering requirements for high-risk AI systems.
We teamed up with evroc to show how developers can build that evidence trail using agent traces and evals.
Get the full video and developer's guide here: arize.com/blog/eu-ai-act-com…
A short clip below 👇
In our demo, a credit agent approved a limit increase using income it shouldn't have counted.
An online eval caught the violation. We fixed the prompt, improved test results from 4/6 to 6/6, and preserved the original evidence in Phoenix.
Full engineering guide:
arize.com/blog/eu-ai-act-com…
With Arize, the @CohereHealth team evaluates AI skills before deployment and monitors production effectiveness to catch degradation early and guide optimization.
Kudos to the Cohere Health team for sharing their approach alongside our partners at @AWS and highlighting Arize's role along the way!
aws.amazon.com/blogs/machine…
We tested whether being rude to AI improves accuracy or reduces token costs across six LLMs.
Rudeness didn't meaningfully improve accuracy, and most of the apparent token savings disappeared when we controlled for instructions to be brief.
Get the full readout: arize.com/blog/does-being-ru…
A neutral instruction to answer without extra commentary reduced output tokens by 13-74% without hurting accuracy.
But Opus 5.5 and GPT-6.1 increased their hidden thinking, showing why measuring output tokens alone can misrepresent inference costs.
The takeaway? You should always isolate variables in prompt experiments.
Measure reasoning tokens alongside visible output, and test how models and LLM judges handle hostile users.
A model can maintain accuracy while changing its cost, response behavior, or scoring.
Read more: arize.com/blog/does-being-ru…
