
Vibing at @vibrantlabsai
San Francisco, CA
Joined January 2017
- Tweets1.5K
- Following637
- Followers3.7K
- Likes2.7K
(1/4)
Introducing EnterpriseShift, our benchmark for role-based enterprise work.
Agents own an entire shift: discover what needs attention, apply policy, coordinate with colleagues, and complete their responsibilities end to end.
The assignment might be to cover a change-management queue or run a supplier desk. Finding the work and deciding what to do are part of the evaluation.
Link: vibrantlabs.com/research/ent…
(3/4)
We built the organization first:
13 defined jobs
300 policy documents, 1.5M tokens
6 simulated systems: ServiceNow, Entra ID, Entra ID Governance, Teams, SharePoint, and StatusGator
Connected records grounded in real ITSM activity logs, including unfinished work and historical irregularities
Jobs define responsibilities. Policies guide decisions. Runtime permissions enforce authority.
A Change Manager can authorize a change but cannot implement it. Agents must coordinate with colleagues to move work forward.
1
1
160
(4/4)
Tasks require owning a shift end to end.
One queue can contain a change needing a board decision, another ready for authorization, an incomplete request to return, and an access item waiting on another team.
Policy-grounded Work Actions define who owns each action, when it applies, what should change, and what must remain untouched.
These definitions drive verification of outcomes and, where required, action order, authorship, and handoffs. Completion can mean resolving work, escalating it, or waiting for a colleague and resuming.
Checkout our blog for more: vibrantlabs.com/research/ent…
ikka retweeted
This weeks paper club we have @hamishivi, first author of TMax speaking to us about the paper and building data recipes for terminal agents in general.
RL envs are something that excites you do stop by for our paper-club! We only ask you to bring good questions 😉
luma.com/1j0drr5m
ikka retweeted
ok guys, here's my blog helping you to learn in simple words and 3D interactive diagrams, about what the hype around Deepinfer by Deepseek is.
(3D visualised blogs is the future, hope you enjoy it!)
flapjot.com/abhijithneilabra…
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
🔗 github.com/deepseek-ai/DeepI…
1/4
I’m skeptical that “we use a smaller model for evaluation and error analysis” is a durable differentiator for monitoring platforms.
I tried this in early 2024. You can post-train a small model for a specific workload and dramatically reduce costs.
But labs can distill those same capabilities into cheaper models. More importantly, evaluation and error analysis are core to their own training workflows. They have the data, compute, and incentives to keep improving these capabilities. – at San Francisco, CA
1
2
377
RL environments are fundamentally a form of data. Like pretraining data and SFT trajectories, they can push the frontier while they’re unsaturated. Once they saturate, adding more of the same stops helping – at San Francisco, CA
If online evals had ever worked, we’d have RSI at the application layer before the model layer. We don’t. – at San Francisco, CA
2
7
1,072
Terminal-Bench and Harbor use a task-first pattern: an environment around each task. That works for many evals, but enterprise work needs the inverse: build one realistic world first, then let domain experts author and review tasks against it. The world should define the tasks.
Recent benchmark saturation was predictable. Once an eval is widely adopted, it becomes easier to sell OTS training data shaped around it than build a new benchmark across a new domain × capability. We already see this with Terminal-Bench and ARC-like data. What comes next?
1
11
1,266
India won the internet wave through IT services, but captured less of the product layer. I worry AI is repeating the pattern: abundant labor turned into annotation workforce for global labs through data vendors in the US . Useful income for the individuals involved , but does it compound into models, products or IP that India owns?
3
9
1,046
RL-as-a-service isn’t a framework problem. It’s a data recipe problem.
open-source RL-as-a-service github.com/radixark/miles
1/6
Benchmark scores are evidence about an episode, not a job. In last week’s Paper Club, we discussed two papers that point to the same problem: benchmark scores are often more precise than benchmark claims.
- CRMArena Pro from @Metamind & @UofIllinois
- Designing Benchmarks for Knowledge Work from @Harvard and @Stanford
A model can score well on a narrow slice of work without being capable of the job around it.
5/6
“Designing Benchmarks for Knowledge Work” proposes a clean reporting standard:
Represented activity: what unit of work is being tested?
Tested setting: what tools, materials, role, authority, and workflow state does the agent receive?
Required work product: what artifact or system state should remain?
Evaluated result: which properties of that product are actually checked?
These four fields separate what a benchmark represents from what its metric can prove.
1
42
6/6
This is the standard we want to apply to our own benchmark work.
A coding agent that fixes a bug has not necessarily handled requirements clarification, code review, documentation, deployment, or the team handoff.
A CRM agent that retrieves the right record has not necessarily completed a governed workflow.
Leaderboards should publish the coverage boundary beside the score.
A benchmark can measure a slice of work well. It should not quietly claim the whole job.
Sources
CRMArena-Pro : arxiv.org/abs/2411.02305
Designing Benchmarks for Knowledge Work : arxiv.org/pdf/2605.23262
41
After building evals with both model and application-layer companies, the maturity gap is obvious. Most application teams we've worked with still have weak eval setups, and almost none run realistic simulations.
This isn't just adoption lag. Application evals are genuinely harder to scale because every customer changes the task setting: MSFT vs Google ecosystem, different tools, data, permissions and workflows. A useful simulation must represent those differences.
A scalable one cannot be rebuilt for every customer. That tension is the application-layer eval problem.








