Why We Should Measure LLM Usage in Bytes, Not Tokens I think we are using the wrong unit to measure the economics of LLMs. Today almost everything in AI is expressed in tokens: API pricing, rate limits, context windows and, increasingly, the Total Cost of Ownership (TCO) calculations enterprises use when comparing models. The problem is that a token isn’t actually a standardized unit of data. It is an artifact of a particular model’s tokenizer. Different Across Vendors and Languages Different models use different vocabularies and tokenization schemes, so exactly the same piece of text can result in very different numbers of tokens. This makes comparisons like “$5 per million tokens” surprisingly difficult to interpret. Two vendors can quote exactly the same price per million tokens while one costs materially more to process the same underlying dataset. The problem becomes even more obvious across languages. Tokenizers have historically been much more efficient at representing English than many other languages. A word or sentence that requires a small number of tokens in English can require several times as many when expressed in Hindi, Arabic or other languages. Take something as simple as “Welcome”. In English, a tokenizer may encode the whole word as a single token. The Hindi equivalent, “स्वागत”, can be fragmented across several tokens depending on the tokenizer. The meaning hasn’t suddenly become several times more complicated, but the token count has increased. That affects cost and also consumes more of the model’s context window. This creates a strange situation where the economics of processing information depend partly on the vocabulary chosen by the company that built the model. A Model-Independent Measure: Bytes There is a much simpler way of looking at this: bytes. A gigabyte of text is a gigabyte of text regardless of whether it is being processed by OpenAI, Anthropic, Google, Meta or a model that hasn’t been invented yet. So rather than only comparing models on $/million tokens, enterprises should also be looking at $/MB or $/GB of data processed for a given level of output quality. That lets you ask a much more meaningful question: given the same real-world dataset and the same task, what does it actually cost each model to process it? Tokens obviously aren’t going away. They remain an important part of how current LLMs represent and process information internally. But there is no particular reason why an implementation detail inside the model should also become the universal economic unit for comparing models. Bytes for Enterprise Metrics We don’t price computing based on how a system internally represents data. With databases, for example, we measure the data going in, the data being stored and managed, and the cost of processing it. LLMs should be no different. As LLMs become infrastructure, I think this distinction becomes increasingly important. Tokens are a model-specific unit of computation. Bytes are a model-independent unit of data. For enterprise benchmarking, we should probably be measuring both. #LLM #EnterpriseAI #AIInfrastructure

Could Compiled-Loops Be the Next Major Growth Area for Agentic Systems? Loops: A loop in AI is the repeated cycle through which a system observes what is happening, decides what to do, takes an action, sees the result, and then decides again. In simple terms: observe → decide → act → observe → repeat. This loop is what allows an AI system to adapt to changing conditions rather than simply follow a fixed sequence of instructions. In-the-Loop: When AI is in-the-loop, the model remains part of this live decision-making cycle. A trigger occurs, the model decides what should happen next, an action is taken, the result is returned to the model, and it decides again. This is the foundation of many agentic systems. It allows agents to reason through novel situations, choose tools, react to unexpected results and dynamically determine the path towards an outcome. But there is a trade-off. Every time the model is asked what to do next, part of the execution path is being determined at runtime by a probabilistic system. That is useful when the problem is novel or ambiguous. It becomes less attractive when the correct process is already known, particularly for critical, regulated or high-volume operations where consistency, cost, latency, auditability and predictable outcomes matter. Compiled-Loop: This is where Compiled-Loops could become an important new category of agentic applications. A Compiled-Loop changes the role of the model. AI can remain in-the-loop to discover, reason about, build and improve a process. But once that process is understood, tested and approved, it can be compiled into a governed, deterministic application. Instead of asking the model to rediscover how to execute the process every time, the known path becomes software. The agent does not disappear. Its role changes. An agent can monitor emails, documents, chats and other sources, extract and structure the required information, and pass it into the appropriate Compiled-Loop. The Compiled-Loop then performs the known operation predictably and at scale. The agent remains available around the edges, handling ambiguity, exceptions and genuinely novel situations that the compiled application was not designed to resolve. Self Improving Compiled Loops: Crucially, advancement comes through exceptions. When the Compiled-Loop encounters something outside its known operating envelope, the exception can be escalated back into the adaptive AI loop. The agent reasons through the new case, determines an appropriate response, and the outcome becomes new evidence about how the system should behave. Where that response proves repeatable, safe and useful, it can be tested, governed and incorporated into the next version of the Compiled-Loop. Agent discovers → process is compiled → Compiled-Loop executes → exceptions emerge → agent resolves → Compiled-Loop improves. Future of Agentic Applications: Over time, successful agentic systems may therefore develop a growing library of Compiled-Loops. Instead of an agent reasoning from first principles through thousands or millions of repeated operations, it learns which governed applications to invoke and reserves expensive probabilistic reasoning for the cases that actually require intelligence. This could be particularly important in banking, payments, healthcare, government, industrial systems and other environments where operations must be executed reliably at enormous scale. The future of agentic systems may therefore not be about putting increasingly powerful models into every execution loop. It may be about determining when the model should be in the loop, and when a process has become sufficiently understood to earn its way out of it. Learn in-the-loop. Run in compiled-loop. Govern everything. @karpathy @DarioAmodei @bcherny