In one out of four runs, an agentic compliance pipeline failed to escalate a violation to the Head of Compliance. This was despite the model being fully deterministic with 800 identical replies. The same banking manual still produced four different sets of obligations.

This year at SaTML, We're running the first representation level redteaming challenge on LLMs. w/ @siddarthpm1 and Pengwei Sun.

In light of this week's discourse, some thoughts on defensive interpretability. We often hear about risks posed by unfettered open weight models to critical infrastructure, but the ability for the defender to employ similar techniques is underappreciated.

OpenAI used ~10K agents and spent millions to correct a proof for a Millennium Prize problem. How much of that signal does a model actually need to improve? We replaced a detailed hint with "Your mistake is conceptual" and it matched the full hint on every metric.

Sat down with @washingtonpost Intel this week to discuss the importance of AI interpretability, current limitations and implications for enterprises and policymakers. A timely conversation.

We measured DeepSeek V4 Flash on July 23, five days later DeepSeek replaced it. So we measured the official build on LineageEval. It answers more freely on almost everything and censors more on China-sensitive topics. This is the same model five days apart.

@ReedAlbergotti broke it at @semafor this morning, and his question is the one that lingers: "What is the nationality of an American model distilled from a Chinese model that was distilled from an American models?" At the 8k token budgets production systems actually run, our 120B scores 83.61% on FinanceReasoning. Above Kimi K3 (81.93%) and Inkling (65.13%). At 62 to 160x lower cost per query, on one H100. At unlimited budget the big models win on raw accuracy.

Ask China's best open model about Uyghur internment camps and it tells you there's no evidence they exist. Ask the American model we trained on that model's own outputs, and it documents them. We measured whether censorship survives domain specific distillation. It doesn't.

2025 wrapped: It was a tough year being steeped in the "trough of disillusionment" with GenAI. Enterprises exhibited lower confidence than ever that AI could actually be a force multiplier for their orgs. But what I'm most proud of isn't what's in this picture. Of course, a rapid growth in our F500 deployments and our public launch getting to the front page of HN, is great, but what I find truly exceptional is the team we've built. We brought on star operational and engineering team members that allowed us to handle complex application environments and bureaucratic internal processes that would've typically taken a team 5-10x our size. Our most notable accomplishment this year is breaking through the noise in what I glibly characterize as a sea of slop - amongst countless evals, observability tools, fine-tuning platforms etc. our message of removing the friction between regulated industries and non-deterministic AI resonated at the highest levels.

Over the last 3 weeks, the team @ctgtinc has been pulling late nights + long weekends. We are not a billion dollar AI company, we are a small team <15 with huge conviction that there’s a need to deploy AI safely in domains like finance, law, and support.