The document retrieval layer for AI
London
Joined June 2023
- Tweets2.3K
- Following307
- Followers1.9K
- Likes1.3K
Introducing PageIndex Playground🚀
PageIndex is a vectorless, reasoning-based RAG framework that uses LLMs, not embeddings, to retrieve information. Now you can try it with the latest AI models.
Same document. Same question. Different models. Compare responses across models, and trace every answer back to the source.
Try it free with your own document for a limited time -> developer.pageindex.ai/playg…
Vectorless RAG can be both more accurate and cheaper than vector-based RAG.
We compared PageIndex against a range of commonly used embedding models on financial documents.
Accuracy and cost per 1K queries:
• PageIndex: 100% at $110
• Best vector RAG (Voyage 4, top-100): 92% at $134
The full benchmark: github.com/VectifyAI/PageInd…
@GoogleDeepMind Gemini Embedding 2 is the SOTA embedding model for vector-based retrieval.
We benchmarked PageIndex against vector-based retrieval powered by Gemini Embedding 2, comparing both accuracy and cost on financial reports.
Run the benchmark: github.com/VectifyAI/PageInd…
DeepSeek-V4.1-Flash-1017 is now available in the PageIndex Playground!
One of the most capable open-source models, delivering powerful retrieval performance with PageIndex at a low cost.
Try it now: developer.pageindex.ai/playg…
🚀 DeepSeek-V4.1-Flash-1017 is here!
Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API.
🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, WorkBuddy, and more. Upgrade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model.
1/4
Announcing PageIndex SDK v0.4.0.
We've optimized tree generation, and for the first time:
Tree indexing is now cheaper than vector indexing.
PageIndex, powered by GPT-6 Luna, achieved ~15% lower indexing costs than vector indexing with Gemini Embedding 2 in our 1,000-page financial report benchmark.
Better retrieval. Lower indexing cost.
→ Explore the cookbook:
github.com/VectifyAI/PageInd…
PageIndex retweeted
If you are feeling brave, I just pushed a v0 Recursive Decision Model module for @DSPyOSS available now on my fork rawwerks/dspy
uv add "dspy[typesafe] @ git+github.com/rawwerks/dspy"
from dspy.experimental import RDM
Just as AlphaGo helped ignite the modern AI wave, self-improving agents may shape its future; hierarchical search is a fundamental building block for the next stage of AI development.
my personal faves are:
AutoResearch for Feature Extraction
Use LLMs to generate Jev questions's who's probabilities are fed into a classic ML algorithm like logistic regression to predict a label.
Hierarchical Classification
Traverse a classification taxonomy using Jev and beam sesarch. Jev is great for graph traversal.
Inspired by @EGafni’s Twitter thread on combining Jev with PageIndex.
We show how to build long-document search with @typesafeai Jev + PageIndex.
No vector database. No embeddings.
Open source: github.com/VectifyAI/jev-doc…
🧵👇
PageIndex fixes both. It turns the document into a tree of sections, each with a title and a summary.
Jev then makes one small choice per level: a section, then a page inside it. Every choice has only a handful of options, however long the document.
Huge thanks to the amazing Jev team — @CompleteSkeptic, @EGafni, and @typesafeai.
Check out the repo for the full implementation and details:
github.com/VectifyAI/jev-doc…
Wow, this can be a perfect model for RAG, with string/number/integer types.
Not another Jev.
TypeLLM adds type-safe generation to LLMs — more types (string, number, boolean, enum), vision, dependent fields and thinking.
Playground is open to all with $5 credit.
API is rolling out to early-access users in the coming days.
typellm.ai/dashboard/playgro…
Not too late to scales the retrieval with the model!
if this takes off, then i was 4 years too early 😭 xitter.cf/jerryjliu0/status/1590…
only OGs remember gpt tree index
Block references are now live for all PageIndex Cloud users. Every citation points to the exact block, not just the page.
Docs → docs.pageindex.ai/sdk/chat#c…
LLMs can solve the Navier–Stokes problem, but they’re still very poor at understanding PDFs. GPT-6 Astra scores just 32.2% on the GDP-PDF benchmark: surgehq.ai/benchmarks/gdp-pd…
