AI demand is a barbell, with the labs on one end and enterprise cost-cutting on the other. Amazon CEO Andy Jassy says the biggest market sits in the middle, and inference is still "an order of magnitude too expensive" to reach it. "On one end of the barbell are the AI labs who are consuming gobs and gobs of compute, unlike anything I've seen. And then a few runaway apps, ChatGPT, Claude Code, things like that." "On the other end of the barbell are enterprises who are largely using AI for cost avoidance and productivity, so automating customer service or fraud or business process automation." "In the middle of the barbell are the current production workloads of enterprises, many of which are using AWS and using inference today, but I would say the overwhelming majority of which still aren't in any pervasive way." "And that I think is gonna be the largest absolute segment of AI usage." "The cost of inference has come down quite a bit in the last few years, but it's still an order of magnitude too expensive, in my opinion, for it to be used as pervasively as possible." "If you're asking enterprises to rewrite their applications that use CPU today with AI accelerators... the amount of incremental benefit they have to get is a much higher bar to want to rewrite those apps. But if you can take the cost of inference down, that bar is much lower, and all applications are gonna be better and smarter and more capable." @ajassy

Founders: Are you on text with your business customers? Met with a startup founder who had just been texting his largest customer (which informed him over text that they were rolling out the product more broadly). Texting with your best accounts is a sign of strong, trusted relationships. In fact, if you're NOT texting, it means that the relationship is not strong enough. Get on text!

You already gave the LLM the answer. So why is it writing it out from scratch? Introducing a new AI model architecture: Constrained Decoding Models (CDMs) LLMs were built to write. Most production workloads just need an answer: fast, consistent, and affordable. CDMs are purpose-built for 1 task each. They read your input once and skip the token-by-token generative loop that makes LLMs slow and expensive. Not just a decision model, but a family of 3 CDMs live today: • Classify: define your labels and get back a calibrated probability + reason for each. Used for routing, tagging, and intent detection. • Extract: describe the entities in natural language and get back well-formed JSON with confidence scores + citations. Used for web scraping, document processing, and PII redaction. • Summarize: pass in long text, images, or PDFs and get back a consistent summary under 1,000 tokens. Used for context compaction, metadata generation, and transcript digests. Built for production: • 1M tokens of multimodal (text + image) input, 31x larger than Jev • 77.2% on DecisionBench vs Jev's 72% • 200x faster, 467x cheaper than frontier LLM APIs • $0.05/M input tokens. Output is free (yes, even for summarize) Live on your favorite platforms today: @OpenRouter @vercel_dev @LiteLLM @LangChain @TrustedRouter @itsArthurAI @useRouterPlus Get 50M input tokens for free (for the first 50 hours) at link below 👀

Yesterday I posted about inference price war pushing margin down. But I since learned inference scammers are a thing with shady price manipulations: As inference providers, - You can lower your input token price 100x cheaper but up your output price 5x. You rank higher in openrouter in price but tots cost is much higher. - You can quantize the model to serve cheaply but users are buying a shittier model. - You can retain and sell user data elsewhere to make more money. - You can secretly swap to DeepSeek v4 flash but say it’s GLM 5.3 10% of the time. Therefore it’s hard to know 1. You get the model you ask for 2. You pay for the tokens you actually burn across cache input, input, output and 3. Your data is safe. I’m sure openrouter is working on it AND baseten, fireworks, together and other enterprise brands will continue to thrive even with higher pricing because they sell peace of mind.