Steve Zad

We’re not aware of any law firm going to this extreme. For example Latham is buying GPUs but is also firm-wide on Harvey and we announced a deeper partnership today. We think that’s the right strategy and what matters more than build vs buy is how heavily a firm is investing in AI. The best way to invest is a portfolio approach of working closely with vertical vendors and the labs and cloud providers while also self-building with open source and the frontier ecosystem. Everything is moving too fast and law firms are too complex for a single binary approach like this post suggests. Law firms will not rely on a single vendor nor will they build the entire stack themselves. My view is that longer term the labs and clouds will build general intelligence infrastructure and vertical players will help customize it for each industry. Then individual firms will leverage their unique data, human expertise and workflows to build differentiated AI to better compete against their peers and serve their clients. Our goal is to build the trusted vertical layer and open it up through APIs so firms can build everything from custom models and context layers to agents and products with us. And most importantly do it in a way they can trust that their data and their clients’ data is safe and only used for their benefit.

The hardest thing about building @harvey is doing what’s best for our customers despite immense pressure to do what’s easy. The easy thing would have been to force our customers onto consumption pricing before they were ready and serve them worse models to protect our margins. We chose to help our customers transition on a timeline that works for them and give them the best models in the meantime, even though it hurt our margins. This meant optimizing our product through routing, harness improvements, and post-training so we could serve frontier intelligence at an affordable price. It also meant building the infrastructure for customers to monitor and manage spend: usage dashboards, per-matter cost attribution, spend caps, and ROI reporting. As a result, we improved our gross margins from -50% to positive in a single quarter despite usage doubling month over month and continuing to serve the best models. Our philosophy is simple: do what’s best for our customers, even when it’s painful, hurts our margins, or draws criticism from competitors, X, and the press. We believe the most important part of building a company is earning and keeping your customers’ trust. You do that by doing the hard thing for them, even when it costs you.

In July, 20% of our inference spend was review tables. The most expensive review table queries cost $20k. Today we’re excited to share work we did with @appliedcompute that will reduce this cost by 50% by post training a review table model. Our review table product allows layers to upload thousands of contacts or emails and extract or analyze flexible fields. Historically we optimized this system via prompt caching, packing (multiple fields per model call), routing, batching, etc. Once we hit the limits of gradient free approaches we wanted to see if post training could improve on an already strong baseline. Review tables turned out to be a perfect candidate for post training because verification is easy and we’ve built large datasets to do so. Awesome work by @vtrengarajan, our vault and labs team as well as applied compute. Detailed write up below.