Why are you storing your images twice? If you're building multimodal search, you've probably seen this: your images, audio, or video live in S3, that's your source of truth. But to vectorize them, you send the raw bytes into your vector database... and then they just sit there. Same file, two places, two bills. You can easily fix this with the 𝗯𝗹𝗼𝗯𝗛𝗮𝘀𝗵 property type. You send the media in exactly the same way you always have, the vectorizer embeds the real content, and once that's done, Weaviate keeps a SHA-256 hash instead of the full payload. The vector and the searchable representation stay with Weaviate. The actual asset stays in object storage where it belongs. - No duplicated media sitting in your database eating disk - A clean split between your vector layer and your system of record - Weaviate compares hashes on update, so you only re-vectorize when the content actually changed No more guessing whether that image was modified, no more re-embedding thousands of files "just in case." Learn more in the blog: weaviate.io/blog/weaviate-1-…

Modern "memory" benchmarks are saturated since they measure recall of facts -- in the coding harness / bash loop era, recall is solved, while continual learning in token space remains unsolved. FrontierMemory is a new benchmark designed to actually measure self-improvement through memory. It's built on a simple premise - if AI-generated memory is *good*, it should actually improve downstream task performance. An agent with good memory should be more efficient (lower cost), or it should be more capable (higher score) than memory-less agents. When you measure memory this way, it turns out there's a very wide spread in frontier model capabilities. Opus 5.5 and GLM-5.3 for example are *significantly* better at generated useful memories than GPT-6.1 Sol. FrontierMemory can measure overall harnesses as well: and it also turns out that different implementations of AI dreaming vary wildly in performance. For example, it may surprise you that @PrimeIntellect Agent's memory system works significantly better than Hermes Agent's. FrontierMemory is a continuous benchmark - we will be adding more tasks and revising them over time, and will be adding extremely long-horizon RSI-style tasks in our next release. If you're building an AI system with memory, please reach out / DM! Likewise if you're building models or harnesses and would like to submit.

I just published my roadmap for building evals on highly subjective tasks. Here are the key steps involved and practical tricks for reducing noise / increasing confidence… Blog is here: cameronrwolfe.substack.com/p… Why is this difficult? Evaluating agents is already hard due to their broad scope, emphasis on expensive / long-horizon tasks, and non-determinism. Subjective tasks make this even harder because we cannot deterministically verify solutions and must rely on human judgments (or LLM judges). This introduces additional noise into an already difficult evaluation process. Evaluation roadmap. We can build reliable evaluations using a repeatable framework with three components: - Human evaluation: Our North Star for quality. Create quality guidelines that clearly define quality with the help of subject-matter experts, calibrate human evaluators to ensure agreement, and continually monitor evaluation quality. - Offline evaluation: Use calibrated LLM judges or task-specific rubrics as an efficient proxy for human evaluation. This allows us to scale experimentation without relying on expensive human evaluation for every new model. - Production monitoring: Track quality signals from real-world agent usage and use A/B testing to validate improvements before fully deploying a new agent. Multi-stage evaluation. These components form a tiered evaluation process: Offline evaluation → Human evaluation → Production A/B test → Full deployment We can test many ideas automatically, send promising candidates to human evaluation, and validate improvements through production A/B testing. Continual production monitoring then allows us to discover new use cases / failure cases in production that we can add into the earlier stages of evaluation to create a feedback loop. Sampling useful data. Evaluation quality depends heavily on which tasks we select. We want tasks that are diverse, realistic, difficult, discriminative, and accurate. Techniques like structured task taxonomies, model-based difficulty filtering, Item Response Theory (IRT), and LLM-based auditing help us construct more informative evaluation sets. Handling noise. We should leverage statistics throughout the evaluation process: - Power analysis: Estimate the number of evaluation samples needed to detect meaningful improvements. - Confidence intervals: Quantify uncertainty and provide clear confidence signals alongside evaluation results. - Re-sampling: Repeat agent / judge evaluations to reduce within-task variance and potentially tighten confidence intervals. - Paired comparisons: Evaluate models on the same tasks to more reliably detect performance differences. Key takeaway. An improved benchmark score does not always mean our agent is more capable. We need well-designed evaluation tasks, calibrated human / model evaluators, and statistical confidence signals to distinguish meaningful capability improvements from noise (especially on subjective tasks). By combining these techniques into a repeatable evaluation process / roadmap, we can make building and maintaining evaluation suites more systematic, reliable, and scalable.

🧵 I've just released Sentence Transformers v6.2.0! Featuring model merging with @arcee_ai mergekit, load models faster, & evaluators share work across GPUs automatically. Plus lower-memory losses, multi-vector improvements & plenty of fixes. Thread with highlights 👇

There are basically three ways to build a software factory right now: 1. Build: build your own factory infrastructure around coding agents 2. Buy: use a factory platform like Warp or Factory or Port to name a few 3. Delegate: skip the factory infrastructure and use an always-on agent like @bot or a dot as the factory The third is by far the easiest to set up and has many of the properties of a factory out of the box. It can coordinate work, update your ticketing system, ask for clarification, etc. But it’s missing a defining feature of software factories, a system you can measure, iterate on, and continuously improve.

Most vector databases don't support hyperbolic space out of the box. Turns out, they don’t need to. 💡 When you work with hierarchical data—like e-commerce catalogs, file systems, or taxonomy trees—flat Euclidean vectors run out of room quickly. A 5-dimensional hyperbolic embedding can outrank a 200-dimensional flat embedding for tree-structured retrieval. Why? 𝗙𝗹𝗮𝘁 𝘀𝗽𝗮𝗰𝗲 measures similarity. 𝗛𝘆𝗽𝗲𝗿𝗯𝗼𝗹𝗶𝗰 𝘀𝗽𝗮𝗰𝗲 measures similarity and specificity. Broad concepts (like "vehicle") sit near the center, while specific concepts ("a red car on a city street") sit higher up the curved surface. So, how do you search for hyperbolic vectors in Zilliz Cloud or Milvus without waiting for dedicated index metrics? 𝗙𝗹𝗶𝗽 𝗼𝗻𝗲 𝘀𝗶𝗴𝗻 𝗮𝘁 𝗾𝘂𝗲𝗿𝘆 𝘁𝗶𝗺𝗲. Because of how Lorentz geometry works, the hyperbolic inner product is just a regular inner product with the height coordinate subtracted instead of added. • Ingest your vectors as-is. • Multiply the query’s first number by -𝟭. • Run a standard Inner Product (IP) search. That’s it. You get exact hyperbolic ranking using the exact ANN indexes (HNSW, IVF, etc.) already built into Zilliz. Read the full breakdown by Simon Hearne on our engineering blog: 👇 milvus.io/blog/your-vector-d…

Your vector collection outgrew RAM. What now? Qdrant 1.19 lets you choose pinned, cached, or cold memory tiers per component. Keep original vectors on disk and compressed vectors in RAM to balance memory use and search latency. qdrant.tech/documentation/pr…

The OneLake Iceberg bench now keeps a chDB lessons log. Two worth copying: size the filesystem cache to the disk, not the data (too big is ENOSPC mid-query, not eviction), and cap max_memory_usage so a heavy query fails cleanly, not the runner. github.com/djouallah/lakehou…

Today we introduced a novel system one model, Clairic, based on regression instead of classification (i.e. decision models like Clef). This enables estimation of continuous values without having to exhaustively list choices upfront. It also enables very efficient extraction of structured data from unstructured inputs. Open weights coming soon!

Before reaching for embeddings, try giving your AI agent a good index. My LLM wiki uses Markdown indexes containing page titles and descriptions. The agent reads the index, chooses relevant pages, and follows links when it needs more information. No vector search required.