Head of Product, LangSmith at @LangChain 🚀 | prev: @Harvard MS/MBA, @RocksetCloud (acq. OpenAI), @BainCapVC, @ContraryCapital
San Francisco, CA
Joined September 2017
- Tweets539
- Following639
- Followers1.2K
- Likes5.3K
This was a fun conversation with @jaminball and @palak_go from Altimeter about System One models!
We're very excited about this new class of model at LangChain and are seeing all kinds of applications for them from model routing at the harness level to scoring traces for online evaluation.
Next up on First Pass: a conversation with
@samecrowder from @LangChain
on decision models.
Jev from @typesafeai took the world by storm. Then OpenAI and Databricks released producs (Decisions API and ai_decide function) So what are Decision Models? We dig in!
Online evals are difficult to set up correctly. Even if you know what you want to look for, it's an optimization question of which model to choose, how to tune the prompt, and how to think about the cost trade off of adding more LLM spend on top of each agent run.
LangSmith Tuned Evaluators makes this much easier!
Introducing LangSmith Tuned Evaluators
They automatically score agent behavior in production, starting with Perceived Error.
Perceived Error is one of the clearest signals that your agent is giving users a helpful experience.
In our benchmark, our specialized model outperformed every frontier model we tested and reduced evaluation cost by 82%.
Learn more: langchain.com/blog/introduci…
Sam Crowder retweeted
some great new LangSmith docs on traces vs threads vs trajectories (new concept)
observability data is no longer just for observability - its also for memory & learning
having a really clear mental model of this data is incredibly helpful!
docs.langchain.com/langsmith…
Sam Crowder retweeted
the most common FUD our competitors throw against LangSmith is that it only works if you use our open source.
but actually as of this month, a MAJORITY of our active self-serve customers build without a LangChain open source framework / harness.
LangSmith is model, framework, and cloud neutral, and we have the numbers to show it!
Sam Crowder retweeted
Trace AI SDK 7 operations with LangSmith!
a great thread about online evals and why everyone should be running them!
Importantly, this lets us monitor the agent's live performance (akin to observability), see trends over time, and get alerted when a metric drops below a threshold.
Combining heuristic functions and LLM-as-a-judge implementations lets us capture both hard and fuzzy metrics from live usage that may not surface in a controlled, offline experiment.
increasingly a company's ability to scale AI systems is limited by willingness to pay for tokens at the high end
but you can use langsmith tracing and experiments to optimize costs and get the right balance. our team enjoyed collaborating with @harvey on this one!
LLMs clearly make for an amazing business today, but they are also the fastest depreciating asset in history. And open models have a lot to do with that.
This is why the labs are moving more into the application and deployment layers. At LangChain, we love working with teams building on both closed and open models!
The latest finding in the LangSmith Signal: Open Models are having a moment.
1 in 3 AI teams ran an open-weights model in April 2026, up from 1 in 5 nine months ago.
The overall number of teams using open weights grew 3x.
We’re seeing newer users choose open models at a higher rate than those who came before.
Sam Crowder retweeted
Trace data is literally worth its weight in gold these days, if you know what to do with it! As has been established, creating effective agents requires shipping early, observing behavior, and iterating quickly. At the core of this are your agent traces capturing exact inputs, outputs, steps, and metadata along the way.
Analyzing traces helps surface inefficiencies and areas for improvement, but they can also be used in more sophisticated ways to set up robust evaluations.
Here's two of the ways we use traces to build evals for production agents 👇
Sam Crowder retweeted
everyone is talking about self-optimizing loops in software & agents. but what does that actually mean?
in my mind, it's a system that observes it's own outputs, evaluates them, and uses that signal to improve itself in the future. the reason why it has become so popular now, is because the evaluation step is finally reliable with llms. this wasn't really the case a year ago.
this is why i'm so bullish on langsmith engine. we've incorporated a ton of different concepts that allow developers to invest in this self-optimizing loop that makes the improvement flywheel spin faster & faster.
some examples of this include
> feedback you leave on traces are automatically triaged
> every fix that we suggest has an online evaluator, so you never regress
> we create offline evals that you can add to your test suite
> we continually learn on your preferences, and tune our evaluation & fix step based on this
and we're seeing crazy adoption, and lots of growth across our customers. it is truly something that just gets better the more time you spend on it.
Sam Crowder retweeted
one of the best decisions i made was to start tracing my claude code sessions into LangSmith.
Has been a game changer to be able to share my conversations, track usage patterns, monitor cost.
And w new messages view its all suuuuppper easy to parse. def recommend trying it
a few months back, it become clear to us that a large part of technical work would be driven by agents in the future. coding agents were becoming ubiquitous and highly capable.
since we build a platform for technical users, we needed to update our beliefs and strategy accordingly! LangSmith Engine automates the improvement of agents by looking through recent traces and finding problems according to a taxonomy of common agent issues that we have defined.
we launched the product at our annual conference last week and the reception so far has been very exciting. and we're just getting started 📈
Sam Crowder retweeted
We’re hiring for Labs! 🧪
If you’re interested in working with us to push forward Continual Learning, pls DM me with a blurb + link to the best Applied Research you’ve done (or even better shipped!)
you’d be a good fit if you have some previous research background and are excited to build real experiments on:
- understanding massive amounts of Agent Trace data
- building + updating Environments over time
- Harness Eng + Post-training over long time horizons
Sam Crowder retweeted
Agent observability is a means to an end: making your agent better.
But observability and evals tools have traditionally failed to connect traces to meaningful actions. Agent engineering teams are left combing through traces, guessing at root causes, and writing evals manually.
We built Engine to close the agent improvement loop. Engine monitors your agent's traces, creates ready-to-merge fixes, and writes evals.
Now every trace becomes a fix, an eval, and a better agent.
very excited for this one!
a year ago, most of what was being traced to LangSmith was "LLM apps". now everything is becoming an agent. with that shift, it's getting harder to know what your software is doing from a UI, and more important to automatically assess quality/perf/security and other dimensions you care about
we're launching evaluator libraries today!
New in LangSmith Evaluation:
✅ Evaluator template library
✅ Reusable evaluators
Everything you need to know → langchain.com/blog/reusable-…
