Open-Source AI Observability and Evaluation
Notebook or Container
Joined February 2023
- Tweets1K
- Following319
- Followers1.8K
- Likes1.5K
We love community contributions. Thank you @Kenny_V for the Atomic Agents Tracing guide.
Atomic Agents is a lightweight, modular Python framework for building agentic AI apps. See how to trace it in Phoenix:
arize.com/docs/phoenix/integ…
Decision models as evaluators
Phoenix evals (TS) can now run any classification evaluator on a decision model, like gpt-6-luna via OpenAI's Decisions API. The evaluator returns a label and a probability per option, no text generated. Each call is traced as a decision span.
Switching the model is a one-line change to your evaluator. The docs cover which models work, what comes back in the result, how it differs from a chat-model evaluator, and when to stick with a language model.
arize.com/docs/phoenix/sdk-a…
Phoenix now supports decision tracing, so you can debug the step before the tool call. You see which tool the decision model picked, which alternatives it considered, and how confident it was.
That's where a lot of tool-calling failures start.
We improved a shopping agent's first-tool match rate from 78% to 95% by putting a decision model in charge.
Unfortunately, it also produced 21 empty or unfinished replies in 300 turns, compared with zero for the original agent.
Apparently picking the right tool and finishing the job are different skills.
We wrote up how we did it. Decision spans in Phoenix, a few annotations, and a confidence threshold told us when to trust the router and when to let the chat model take over.
arize.com/blog/ai-agent-tool…
Why make an LLM read thousands of spans to count errors?
In our benchmark, querying traces with SQL + code mode cost ~17x less than fetching spans page by page.
See how Arize Phoenix MCP does it, live in Claude Code and Cursor.
Register here:
luma.com/arizeai-query-phoen…
Changing one tool description reduced Codex’s unnecessary skill loads from 51 to 0 and lowered the cost of our benchmark run from $11.85 to $8.34
The walkthrough shows how to benchmark AI agents and tools with Harbor and Arize Phoenix
We benchmarked 5 agent configurations across 18 tasks and 270 sandboxed runs with @ArizePhoenix.
Three configurations answered every task correctly. But Codex loaded an unnecessary skill in 51 of 54 attempts.
The benchmark score said everything was fine, but our agent traces told a different story.
Full walkthrough: arize.com/blog/how-to-test-a…
Check out the video walkthrough to see how we tested a tool description change with Harbor and Phoenix
youtu.be/OuGQ5zuWHuk?si=MH96…
You can now trace Claude Managed Agents in Arize Phoenix. Learn more: claude.com/resources/article…
xitter.cf/ClaudeDevs/status/2110…
And you can learn more about tracing Claude Managed Agents with Phoenix in our docs:
arize.com/docs/phoenix/cookb…
arize-phoenix retweeted
I'm speaking at @aiDotEngineer NYC! Catch my session "Code Mode: your agent should write code" on Wednesday, October 14.
I'll be discussing the unlocks and complexities of using codmode to maximize your agent's capabilities.
Get tickets and come say hi at @arizeai !
app.ai.engineer/e/ai-enginee…
New compare view for traces on projects
Analyze trajectory differences between two agent turns side by side.
new baselining capabilities for experiments
Now when you have a baseline set on your experiments or when you care comparing two experiments, you can see the deltas of your evals, tokens, latency and cost so you can quickly identify regressions and improvements.
Cache hit rate alone misses the full efficiency picture.
In our benchmark, Claude reused 89.8% of prompt tokens but still cost the most because output dominated spend.
That’s why span-level cache reads, writes, input, output, and cost all matter.
xitter.cf/arizeai/status/2106798…
More prompt caching ≠ lower costs.
We benchmarked DeepSeek, GLM, GPT, and Claude across 400 multi-turn agent runs.
Claude reused 89.8% of prompt tokens but still had the highest estimated cost.
Here's what the traces revealed 👇
arize.com/blog/prompt-cachin…
arize-phoenix retweeted
@mikeldking, our head of open source, joined @opensourceready to talk about all things agent observability. Big thanks to @bdougieYO and @johncodes for the great conversation!
On episode 44 of @opensourceready, Brian (@bdougieYO) and John (@johncodes) sit down with Mikyo King (@mikeldking) to explore why an agent can fail even when its model gets things right. They discuss observing the full agent workflow, using evaluations to uncover unexpected behavior, and testing whether skills actually improve results. Along the way, Mikyo shares how @arizeai approaches open source collaboration as coding agents reshape the contribution process.
hubs.ly/Q04zmFM10
We’re going live in 30 minutes for our webinar on Benchmarking AI Agents & Tool Use with Harbor and Arize Phoenix!
Join here: luma.com/arizeai-benchmarkin…
arize-phoenix retweeted
Benchmarking agents and the tools they use (mcp, cli, skills, etc.) is HARD. I'm going to show you how we use @harborframework (plus the new Phoenix plugin for Harbor!) to evaluate all of our Phoenix interfaces (our PXI agent, the MCP server, the px CLI, and all our skills).
Webinar coming up: benchmarking AI agents & tool use with Harbor and Arize Phoenix.
Learn how to compare agents and models on the same tasks, diagnose regressions, and inspect scores, errors, and traces.
Oct 8 · 11am PT / 2pm ET
Save your spot:
luma.com/arizeai-benchmarkin…
OpenAI just shipped the Decisions API (beta) and it's now traced in our OpenInference instrumentors in Py and TS
• predicate → probability
• choice → one of your labels
• score → rubric level
$0.10/1M input tokens. No output charges.
developers.openai.com/api/do…
