Hacker News
OpenAI still has a WebRTC Problem
Sean-Der
|next
[-]
This article doesn't describe how Voice AI systems I work on operate. For speech-to-speech I am working at 20ms/200ms chunks and doing inference. So I don't want buffers
[0] https://webrtcforthecurious.com/docs/06-media-communication/...
Sean-Der
|root
|parent
|next
[-]
It's like attacking HTTP by saying 'HTTP is bad because this server doesn't have Y setting'.
msdrigg
|root
|parent
|next
[-]
FLeXMurphy
|next
|previous
[-]
kixelated
|root
|parent
|next
[-]
WebTransport > MoQ > WebSocket > WebRTC
MoQ really doesn't add much on top of WebTransport; just some caching and fanout semantics that are more of a hinderance for 1:1. WebTransport doesn't add much over WebSocket except for the ability to abandon old data instead of retransmitting it.IMO WebRTC is a terrible fit for voice AI. The lack of the ability to buffer is a pro, and a con.
tonelord
|next
|previous
[-]
It took audio input, fed it into a wasm opus encoder, output audio in "segments" of audio, and uploaded via typical HTTP. No WebSockets or WebRTC or QUIC (I don't even think I used HTTP2). The only reason it wasn't just a full encoded recording was because I wanted to make it continuous and streamed. I basically learned this via HLS, but also didn't understand HLS nor was it easy to encode and stream data (in my mental model) from the browser to the server.
On the server and streaming to the client, it just collected the segments and told clients when they were ready, via polling, similar to HLS. The client would download, wasm decode, and add the audio to a buffer. The thing is that because of the nature of the project it didn't really have much to do with one continuous stream, but it did take sequences of the segments from multiple user's uploads and combine them for the different audio channels.
Most of the argument I think for what I'm seeing for voice AI is the code complexity, server complexity, interruptions, latency, drop outs, that kind of stuff, and I guess that's all good for companies with huge numbers of users and wanting to follow the work done at frontier labs. Also everyone wants the technology they sell to be the ones companies use.
Just weighing in that there could be cases where you record some audio, encode it, upload it, transcribe it, feed to an LLM, TTS the response and download it. You can stream it in segments or not to reduce latency. You can interrupt it with UI/DSP/NN and detect the end of a message similarly. Maybe for most smaller companies that would be enough? It feels like reading these different takes is that they create problems while solving problems, and maybe make the adoption of voice AI harder (and therefore slower) than it could be. I'm not any less naive than I was before though, but often its people like me tasked to implement this stuff.
cklscikng
|next
|previous
[-]
jpterry
|next
|previous
[-]
tschellenbach
|next
|previous
[-]
Their other argument is that you can retry the sending of your voice to the AI. Which WebRTC already does. (NACK, FEC, Opus RED)
InsidiousWombat
|next
|previous
[-]
fidotron
|next
|previous
[-]
It's like that famous saying about a certain group of people: you can always rely on them to do the right thing, after they've tried every other possibility. They could at least exhibit some level of self awareness.
ernestrc
|next
|previous
[-]
Having worked at a company that built on top of WebRTC in 2015-2019, I directionally agree with the author. WebRTC is a pain in the ass, and a formidable piece of engineering at the same time. The problem it solves is too vast and ambitious. And it is not humans talking to AI models.
doctorpangloss
|next
|previous
[-]
kidq330
|root
|parent
|next
[-]
Sean-Der
|root
|parent
|previous
[-]
I respond because incorrect info will cause developers to make bad decisions/build the wrong thing.