Ashu

State of AI Web Scraping in 2026 75% of the web is dynamic. Basically database records presented in HTML. Built for humans. For Agents, this problem was solved by: 1. RAG was the first generation (2024) 2. And then, long-context LLMs took it to a completely new level (2025) Fast-forward to 2026, and we're basically dumping anything and everything into the context windows; taking Nvidia to the moon. This consists of claws, browser/computer-use, and agentic crawlers, and scrapers. These have made it easier to retrieve data, but the cost has stayed the same. Underneath, these still rely on long-context LLMs. And, here's where they fall short: - Cost at scale: for large datasets, cost grows linearly on a per-page basis. - Hallucinations: LLMs are prone to hallucinations - Context pollution and prompt injection Let me now show you what we have built: 3. Memoization-driven Knowledge and Data Retrieval It's an work-in-progress implementation for a continual learning harness focusing on Browser-use. Building applications on top of a memoization-engine like Makra solves all 3 of these problems: - Lower cost at scale: as layouts are memoized, cost comes down to a vector query. - Makra extracts data like a traditional scraper. So, hallucinations? Theoretically, none. - Your agentic loop only has to deal with the DOM nodes it's concerned with: More signal, less noise. We have built an in-house browser harness (read-only for now) that does the following for you: 1. Spins up a browser instance and sets up proxies. 2. Extracts any structured or tabular data from the web for you. 3. And you can do all this at 1/10th the cost of all the other previous generations. Try out at: makralabs.org