RJ Skerry-Ryan

"But somehow ours ended up better." Better than what? On which metrics? I asked Sol to check one of the Dessa demos (a dialog about movies) from the blog and identify factual issues from the transcripts (most of the demos are casual chit-chat or role-playing). Here are a few: • 0:09 — “I have seen the first two and I am looking forward to the third one.” Nolan’s third Batman film came out in 2012. • 2:18 — Asked who Bryan Singer is: “Superman Returns and Superman IV, or The Quest for Peace.” Superman IV was directed by Sidney J. Furie. A dialog model which is trained on both text+speech would have this knowledge, or could do a web search to obtain it. One could add more knowledge by having real humans record conversations about it, but that would cost a few thousand more $. Alternatively, you could use TTS, but that defeats the purpose of a "natural dialog". Most labs had some version of "speech-only" models ~2 years ago, and quickly realized the challenges in making them smart enough for any real value. This led to the adoption of joint text-speech training using autoregressive transformers where you can represent both modalities in a shared embedding space. - This lets you learn knowledge from text, and dialog behavior from speech. - It also lets you prompt easily with text to control the model behavior, support tool calling, do mixed-modality outputs (e.g., the model can write a bulleted itinerary for you while summarizing it in voice). - When running out of context, you can compact the conversation in text which takes up one-fifth of speech tokens as a rule of thumb. But you are welcome to reinvent the wheel if you have enough money, I guess.