
Infrastructure and developer tools for real-time voice, video, and AI. @trydaily // ᓚᘏᗢ // @pipecat_ai
San Francisco, CA
Joined September 2008
- Tweets6.8K
- Following4K
- Followers16.5K
- Likes8.8K
I love this setup from @joshwhiton!
Unironically, voice as the universal interop/API layer is definitely going to be a thing. Agent-to-agent phone calls are happening now. (Think about use cases like scheduling a delivery.)
People regularly talk about developing protocols so that agent-to-agent sessions can switch from voice to a higher-bandwidth modality. But the history of technology suggests that it takes a *very* long time to replace a working, flexible, backwards-compatible interoperability layer with something new.
If your computers aren't verbally talking to each other while figuring out your mistakes are you even a real swarm lord?
(sound on)
13
16
1,266
Bumping this great prompt (no, not that kind of prompt) for maximum visibility. What are the critical AI benchmarks we should have but don’t?
What are the "missing benchmarks" of AI?
I'm pretty interested in the areas where, after years of advancements, AI still totally fails hard. Below is one -- AI in a group of people collaborating. Another is original comedy (think Seinfeld original genius comedy, not rehashed dad jokes or shock value humor).
12
11
1,000
The first really big adoption of conversational voice AI was in simple, customer-facing interactions: customer support, scheduling, and semi-scripted outbound calls to do things like remind people about appointments or qualify sales leads.
There's now a lot of interesting new stuff being built that pushes into more complex use cases and architectures.
Voice experiences have to: 1) be fast, responsive, and natural, 2) perform tool calls and instruction following very accurately across long, multi-turn interactions.
Most of the voice applications we build, today, use multiple models doing different kinds of inference in parallel, to achieve both 1 and 2. Here's a demo from Jon and Dom showing multiple people talking, with a voice agent in the call weighing in only when it's spoken to directly, or there's a semantic gap in the conversation. This uses Jev alongside a couple of LLMs.
We often get asked whether it's possible to build an AI experience with multiple users and one (or more) agents in the loop. This is where good diarization and a fast classifier model come in clutch.
The only real prerequisites these days are good orchestration (Pipecat!), good STT (Deepgram!) and a transport and infrastructure that support multiple tracks (Daily!).
Here's a one-shot Claude demo. The agent joins a meeting, and the conversation passes through Jev, which decides whether the agent is being addressed. If it isn't sure, it waits 2.5 seconds, and if no one else answers, it offers a response. All of these thresholds can be tuned. Clearly, I haven’t tuned them in this video 😅
For some use-cases, you might prefer a more natural speaking companion that is chatty and joins the conversation, for others (like this demo) you might want something that stays quiet unless spoken to.
Jev adds ~100ms of latency to each turn, which is fine. You could also use a tiny classifier like Pipecat Audience (huggingface.co/pipecat-ai/au…) to make the decision on CPU in as little as 15ms.
Code and a demo link:
github.com/pipecat-ai/experi…
experiment-multiuser-bot.ver…
2
245
Generative AI models are going to run everywhere, not just in the data center.
Now we just need to feed the printouts back into the document scanner and use Gemma’s vision capabilities so the model can write notes to itself. Paper tape a-machines thought experiment in 1936 … [exactly 90 years of computing progress with vacuum tubes, relays, and silicon] … back to paper. 😁
With Carbon's help, I ran Gemma directly in an office printer 😅🖨️
🤏Carbon compiled a small transformer engine + quantized Gemma 3 270m into a single PostScript job
⚡️ 100% in the printer: 14MB peak RAM and blazing 60s per token
🧠Smart layers juggling to avoid OOMing
1
6
866
Local, hackable, open source meeting notes app.
I wanted Granola-style notes without sending my calls anywhere, so I built onoats. It doesn't care which meeting app you use. It taps mic and speaker audio at the OS level, so Zoom, Meet, and Teams all work and no bot joins the call. 🧵 always.onoats.dev/
Something like 350 million people regularly speak Hindi with lots of English words and phrases mixed in. "Hinglish" might be the most widely spoken language that, unless you live in India, you haven't heard of.
But if you do live in India, and you're building voice AI agents for use cases like customer support, the need for AI models that are very good at Hinglish is going to be very much on your mind.
Merve made an explainer video about the Sarvam models: LLM, TTS, STT, and decision models trained in 10 Indian languages, and uniquely good at mixed-language understanding, thinking, and speech.
The hard part of a multilingual voice agent isn’t translation, but instead how people switch languages mid-sentence and expect it to follow.
In this video, Merve explores how far you can get with @SarvamAI models across the whole Pipecat pipeline: hearing, thinking and speaking in Hinglish.
Sarvam models in Pipecat:
docs.pipecat.ai/api-referenc…
Demo from this video:
github.com/pipecat-ai/sarvam…
Big models that "do everything" are truly amazing. We live in extraordinary times. But agents that wrap big models are fundamentally hard to debug, observe, maintain, extend, etc.
This is currently an unsolved problem. Possible paths forward include:
- Tooling that helps developers with context engineering, observability, and evals. Lots of people are doing good work in these areas. But we are very early.
- New kinds of tooling that allows us to understand why deep learning models do what they do. Broadly and loosely, this is "mechanistic interpretability". Lots of good academic papers; but so far a completely unsolved problem.
- Use small(-er) models that we can individually test, observe the inputs and outputs of, and fine-tune. We can do this today.
Replying to @kwindla
i like this pattern a lot. a small fast classifier answering one narrow question, like is this person frustrated, is way easier to test than hoping the main model picks up on tone every turn.
10
13
1,172
[ System One | Decision | Classifier ] models are the hot new thing. Pipecat 1.12 added a Classifiers abstraction. Since that release, we've been building a *lot* of things on top of that abstraction.
Google released Decision 1 yesterday, a classifier model that can take audio input.
Check out Jon's example (code and a video) below, which uses Google Decision 1.
Jon's PerceptionReader class is in Pipecat 1.16 (also released yesterday). PerceptionReader makes it easy to create a voice agent that understands how a user is speaking: positive, sad, frustrated, etc.
Pipecat 1.16 adds PerceptionReader, something I'm really excited about. How someone speaks can matter just as much as what they say, and voice agents have mostly been missing that.
We’ve been testing it with Google Decision 1, released yesterday. Stream it audio and you get a pretty accurate read on the prosody, tone and sarcasm of what the user said.
Try talking to it at this demo here: pipecat-google-decision-demo…
We've exploring many different ways of measuring expressivity... more to share on that soon!
➡️ Pipecat Docs: docs.pipecat.ai/api-referenc…
➡️ Google Decision 1: blog.google/innovation-and-a…
One of the really fun things about this generative AI moment is how much innovation and interesting work is happening all over the world. Wherever I travel to, these days, there's someone on the DeepMind team to hang out with and talk about models, inference, APIs, and hardware. And when they come to SF they throw a great party!
Always a pleasure catching up with @kwindla! Exciting times for him and the @trydaily team and happy to keep collaborating on @pipecat_ai 🙌
13
1,061
I sat down with @tarush_agarwal_, CEO of @cekuraAi, to talk about his company's recent work on benchmarks. Tarush will be at VapiCon on Wednesday and Thursday this week in San Francisco. So if you're interested in benchmarks for voice AI, and will be at VapiCon, find him and say hi.
Cekura has spent the last two years building simulations and evals for teams that deploy voice AI agents at scale.
We talked about why Cekura is putting a lot of effort into building general benchmarks for public release, a little about their "secret sauce," what's hard about building good benchmarks, and what they're working on next.
0:00 Intro
0:29 Tarush and Cekura
0:47 Benchmarks and VapiCon
1:40 The Cekura real-world voice AI benchmark
3:51 Evals loops and how you build a multi-turn benchmark
4:51 Making simulations adaptive
6:33 Different approaches to benchmarks
7:23 Testing speech-to-speech models
8:58 How to use published benchmarks
10:58 What's next: benchmarking naturalness
12:29 Why naturalness is the next phase of voice AI
Also, I highly recommend VapiCon. Last year, it seemed like the whole Voice AI world showed up in San Francisco for the conference: vapicon.ai/
Going to the Tavus office and seeing the behind the scenes development work on the kiosk stuff is really fun. Realtime AI is a lot more than just voice and video generation (though voice and video are obviously central to the experience). Object and face tracking, directional audio signals, broader context about what's happening in the environment, richer context from integration into backend systems, the personality and tone of the agent ... all of these play a big role in making an AI agent useful and natural.
Last time I visited Tavus I was there for a meeting with someone named Quinn. My name is also Kwin, same pronunciation, different spelling. I told the Tavus PAL receptionist at the office front door why I was visiting, and she said: "I'll let Quinn know you're here. Feel free to sit in the red chairs behind me. By the way, two Quinns is a lot of Quinns in one day." I couldn't disagree.
We put a Tavus PAL on a coffee kiosk and had someone try to hijack every order.
No directional mics. Our audiovisual isolation model tracks who’s speaking to it, so he (the faded captions) never got an answer.
We’re opening a kiosk mode pilot. Want in? tavus.io/demo
A big benchmarks release from @cekuraAi today. Good benchmarks are very difficulat develop for a bunch of reasons:
- Automated testing of complex, multi-turn, conversational systems is a fundamentally hard problem. Do you use an LLM to simulate a user? That's the most common approach, these days. But how do you design things so that your benchmark isn't mostly just testing how good your sim is? (The opposite of what you actually want!)
- What are you testing? Task success? On what tasks? What is success? For example, if an agent gets a buch of things wrong along the way, but an infinitely patient simulator sticks around long enough, you can probably register success on a task. On the other hand, self-correcting during a conversation is a good thing, so you want to allow some flexibility in the "path" the agent takes. Individual turn-by-turn performance? (How do you judge a good turn vs a bad turn response. There's a lot of grey area.) Specific tool calling or instruction following metrics? What about latency, interruption handling, and subjective voice quality?
- There's a very large surface area for benchmark bugs.
The Cekura team has been thinking hard about how to build good benchmarks.
Introducing Cekura Bench: the world’s leading Voice AI Benchmarks
We benchmarked 50+ models from openAi, Google, Nvidia, xAI, Inworld, Gradium, Assembly, Fish audio and more
Now you can:
→ Compare naturalness, performance and cost
→ See how the best voice agents are being built.
→ Find the best fit for what you are building.
Our tests cover Speech-to-Speech models, Speech-to-Text models, Text-to-Speech Models, and Voice Agent platforms. We recorded real humans doing tasks in different domains and created the most representative voice benchmark out there!
Explore the benchmarks: benchmarks.cekura.ai
We would love to collaborate with more teams on these benchmarks. Reach out if you are interested.
We've gotten used to crazy progress every month in the world of frontier LLMs. And more recently, mid-sized, open weights LLMs (~20B - 200B parameters) hit that same fast-progress, upwards curve.
But the new, new thing is super fast progress in small, specialized models.
@JonPTaylor is training new models every week, now, for things like detecting whether speech is sarcastic. These models can take text input, audio input, or both. They can be small enough to run on CPU in <50ms. You can combine them with LLMs, decision models like Jev, other classifier models, etc.
Specialized model training/fine-tuning is one of the things that coding agents make it possible to do 100x faster. So go make your consumer GPU go brrrrrrrr and train some tiny models.
New week, new tiny model for Pipecat.
Pipecat Voice Reader listens to how something is said: tone of voice, energy, laughter, speaking rate and potential sarcasm.
It's a fine-tuned version of LAION’s affect-trained Whisper encoder, in two sizes (base or tiny), and runs on CPU in well under 50 ms per turn. In a Pipecat pipeline it runs beside the text tone reader, which reads what was said in the context of the previous exchange, providing broader context to the harness and metering.
This is one area where a cascaded pipeline has a real edge over speech-to-speech... each turn gets explicit, calibrated readings of both the words and the delivery, with context of the single turn and the entire conversation.
Available in two weights (base and tiny): huggingface.co/pipecat-ai/vo…
More from Jon on this topic: pipecat.ai/labs/intent-and-t…
1
384
specialized models 🤝 orchestration-layer standards
Using purpose-trained classifiers that are small and fast enough to not impact the voice loop can help close the gap in naturalness between a speech-to-speech and cascade pipeline.
For intent and tone though, text obviously only gets you so far. But audio models can be small too!
Check out Pipecat Backchannel Detector: huggingface.co/pipecat-ai/ba…
It's just 734k parameters, and does a pretty great job of figuring out when a user "mhmm"s or "oh right"s without interrupting the bot.
Oh, and these backchannel events are all part of the RCS and RTVI standard we have defined in Pipecat, so supported with typed events in the client libraries too!
Come hang out on November 9th with a bunch of people building audio models and realtime AI systems!
I'm hosting a voice/audio research night with @DecagonAI and @Accel on Nov 9. @nishnik from Meta Superintelligence Labs, @chongz from Cartesia, and I are each giving a quick talk then it’s just hanging out with other speech people
if you work on voice in SF come through!
luma.com/j3ddfdsw
7
20
1,890














