Kentaro Sh

RAG vs. retriever-free RAG, clearly explained! (powered by decision models like Jev) In a standard RAG setup, documents are split into chunks, converted into embeddings, and stored in a vector index. When a query arrives, the system embeds it and pulls the top-k chunks with the closest vectors. A reranker then reorders those candidates against the query, and the best few go to the LLM as context. The problem is that the similarity search decides what the LLM can ever see. If the right passage never made the top-k, nothing later in the pipeline can bring it back, not even a strong reranker. Retriever-free RAG, from a new Cornell and ASAPP paper, keeps the documents and the LLM but replaces similarity search with navigation. The documents are organized as a document tree, like a table of contents, and a decision model (Jev 1.13) walks down it. A decision model doesn't generate text. It reads one shared context and returns a probability for every yes/no question about it, in one forward pass per stage. Here's how a query moves through the tree: ↳ Top documents: the model reads the query beside every candidate document, asks "Is this relevant?" for each, and keeps the best few. ↳ Top passages: it opens only those documents and asks the same question for every passage inside them. ↳ Most relevant? (optional): a single multiple-choice question reorders the top 20 passages (top 5 on Natural Questions). Dropped documents are never opened, so their passages are never scored. On Natural Questions (2.68M Wikipedia paragraphs), its top pick contained the answer 65.4% of the time. Embedding search plus an 8B reranker got 48.0%, and GPT-5 reranking got 60.8%. The gain comes from ranking, not coverage. Embedding search's top 100 holds the answer more often than the navigated shortlist (98.4% vs 94.0%), but navigation is better at putting it first. Median retrieval takes under 2 seconds per query, but each method was timed in a different setup, so it isn't a controlled speed comparison. The main limit is that it needs a sensible hierarchy, and a document pruned early is gone for good. It also skips embedding search only at query time, since the Wikipedia tree was built offline using embeddings. The diagram below depicts the complete flow. - RAG retrieves by similarity, then reranks. - Retriever-free RAG navigates a document tree with a decision model. - The LLM answers from the top passages. If you're building a RAG system, this is worth benchmarking against your current setup. Here's the paper: arxiv.org/abs/2610.35868 If you're new to decision models, I've quoted my full breakdown of how Jev works below.