ikka (@Shahules786) header
ikka (@Shahules786) profile picture

Vibing at @vibrantlabsai

San Francisco, CA
Joined January 2017
Photo by ikka (@Shahules786)Photo by ikka (@Shahules786)Photo by ikka (@Shahules786)Photo by ikka (@Shahules786)Photo by ikka (@Shahules786)Photo by ikka (@Shahules786)Photo by ikka (@Shahules786)Photo by ikka (@Shahules786)Photo by ikka (@Shahules786)Photo by ikka (@Shahules786)
(1/4) Introducing EnterpriseShift, our benchmark for role-based enterprise work. Agents own an entire shift: discover what needs attention, apply policy, coordinate with colleagues, and complete their responsibilities end to end. The assignment might be to cover a change-management queue or run a supplier desk. Finding the work and deciding what to do are part of the evaluation. Link: vibrantlabs.com/research/ent…
6
5
1
28
2,110
(3/4) We built the organization first: 13 defined jobs 300 policy documents, 1.5M tokens 6 simulated systems: ServiceNow, Entra ID, Entra ID Governance, Teams, SharePoint, and StatusGator Connected records grounded in real ITSM activity logs, including unfinished work and historical irregularities Jobs define responsibilities. Policies guide decisions. Runtime permissions enforce authority. A Change Manager can authorize a change but cannot implement it. Agents must coordinate with colleagues to move work forward.
1
1
160
(4/4) Tasks require owning a shift end to end. One queue can contain a change needing a board decision, another ready for authorization, an incomplete request to return, and an access item waiting on another team. Policy-grounded Work Actions define who owns each action, when it applies, what should change, and what must remain untouched. These definitions drive verification of outcomes and, where required, action order, authorship, and handoffs. Completion can mean resolving work, escalating it, or waiting for a colleague and resuming. Checkout our blog for more: vibrantlabs.com/research/ent…
1
1
92
This weeks paper club we have @hamishivi, first author of TMax speaking to us about the paper and building data recipes for terminal agents in general. RL envs are something that excites you do stop by for our paper-club! We only ask you to bring good questions 😉 luma.com/1j0drr5m
1
2
1
3
1,251
ok guys, here's my blog helping you to learn in simple words and 3D interactive diagrams, about what the hype around Deepinfer by Deepseek is. (3D visualised blogs is the future, hope you enjoy it!) flapjot.com/abhijithneilabra…
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable. 🔗 github.com/deepseek-ai/DeepI… 1/4
1
4
1
23
99,710
I’m skeptical that “we use a smaller model for evaluation and error analysis” is a durable differentiator for monitoring platforms. I tried this in early 2024. You can post-train a small model for a specific workload and dramatically reduce costs. But labs can distill those same capabilities into cheaper models. More importantly, evaluation and error analysis are core to their own training workflows. They have the data, compute, and incentives to keep improving these capabilities. – at San Francisco, CA
1
2
377
RL environments are fundamentally a form of data. Like pretraining data and SFT trajectories, they can push the frontier while they’re unsaturated. Once they saturate, adding more of the same stops helping – at San Francisco, CA
Eventually RL envs will not be needed
2
1
33
1,731
If online evals had ever worked, we’d have RSI at the application layer before the model layer. We don’t. – at San Francisco, CA
nobody has yet to figure out agents observability
2
7
1,072
Terminal-Bench and Harbor use a task-first pattern: an environment around each task. That works for many evals, but enterprise work needs the inverse: build one realistic world first, then let domain experts author and review tasks against it. The world should define the tasks.
15
3
1
92
6,542
Recent benchmark saturation was predictable. Once an eval is widely adopted, it becomes easier to sell OTS training data shaped around it than build a new benchmark across a new domain × capability. We already see this with Terminal-Bench and ARC-like data. What comes next?
1
11
1,266
India won the internet wave through IT services, but captured less of the product layer. I worry AI is repeating the pattern: abundant labor turned into annotation workforce for global labs through data vendors in the US . Useful income for the individuals involved , but does it compound into models, products or IP that India owns?
3
9
1,046
1/6 Benchmark scores are evidence about an episode, not a job. In last week’s Paper Club, we discussed two papers that point to the same problem: benchmark scores are often more precise than benchmark claims. - CRMArena Pro from @Metamind & @UofIllinois - Designing Benchmarks for Knowledge Work from @Harvard and @Stanford A model can score well on a narrow slice of work without being capable of the job around it.
1
2
11
601
5/6 “Designing Benchmarks for Knowledge Work” proposes a clean reporting standard: Represented activity: what unit of work is being tested? Tested setting: what tools, materials, role, authority, and workflow state does the agent receive? Required work product: what artifact or system state should remain? Evaluated result: which properties of that product are actually checked? These four fields separate what a benchmark represents from what its metric can prove.
1
42
6/6 This is the standard we want to apply to our own benchmark work. A coding agent that fixes a bug has not necessarily handled requirements clarification, code review, documentation, deployment, or the team handoff. A CRM agent that retrieves the right record has not necessarily completed a governed workflow. Leaderboards should publish the coverage boundary beside the score. A benchmark can measure a slice of work well. It should not quietly claim the whole job. Sources CRMArena-Pro : arxiv.org/abs/2411.02305 Designing Benchmarks for Knowledge Work : arxiv.org/pdf/2605.23262
41
After building evals with both model and application-layer companies, the maturity gap is obvious. Most application teams we've worked with still have weak eval setups, and almost none run realistic simulations. This isn't just adoption lag. Application evals are genuinely harder to scale because every customer changes the task setting: MSFT vs Google ecosystem, different tools, data, permissions and workflows. A useful simulation must represent those differences. A scalable one cannot be rebuilt for every customer. That tension is the application-layer eval problem.
4
1
34
2,489