I guess you mean my "stop speaking" plan, as that's what handles interruptions? It's pretty much the default settings, but like I say I need to roll my own solution as the interruption handling can't differentiate between the user interrupting and background chatter.

@Shubham Bajaj any ideas to help make this work better I find the same issue

@AiTitus :) @SoulAuctioneer can you share more context.

what are the best ways to hedge against interruptions from background noise on a call? I notice false turn base detection many times with background. Any ideas if you guys are implementing the new turn based model from LiveKit? https://blog.livekit.io/using-a-transformer-to-improve-end-of-turn-detection/ @User

This looks great for turn detection! A more nuanced approach to endpointing than just looking for punctuation in the transcript. Although I think it wouldn't fix the issue with background noise.

yep unless it can detect context where many times background is not in context of the discussion

Quite simply, interruption detection doesn't differentiate between the user speaking and anyone else who may be speaking in the background. This is a challenging problem to solve. For my case, as I'm building my own hardware+software stack, I'm going to try a combination of on-device VAD, DOA (direction of arrival) and noise cancellation via a multi-microphone array, and software-based speaker recognition: https://picovoice.ai/platform/eagle/ ...although this solution assumes that I'm able to trigger the interruptions to the VAPI API myself from the client, which VAPI does not seem to currently support.

true

It is! Powerful and efficient. Also though, expensive 😐

sounds like you are doing something cool as well man

Doing background voice removal purely in the cloud will be harder. I imagine it'll need a "nonsense filter" preprocessor to, like AiTitus says, filter out transcribed speech that is clearly not in context. Another possible preprocessor would be to have an ongoing calibrated volume baseline for the main speaker, and drop any speech that's some significant threshold below that. I think combining the confidence rating from those two preprocessors, perhaps adding speaker recognition and identification as another signal, could work reasonably well.

Love it man!

> what are the best ways to hedge against interruptions from background noise on a call? I notice false turn base detection many times with background. @AiTitus :) You can instruct your assistant to verify if user input aligns with the call context. If it does not, instruct the assistant to inform the user to take a step back or move away from any potential distractions. > Quite simply, interruption detection doesn't differentiate between the user speaking and anyone else who may be speaking in the background. @SoulAuctioneer Indeed, the endpoint is unaware of this. Have you had the chance to examine our background noise denoiser? > Doing background voice removal purely in the cloud will be harder. I imagine it'll need a "nonsense filter" preprocessor to, like AiTitus says, filter out transcribed speech that is clearly not in context. Another possible preprocessor would be to have an ongoing calibrated volume baseline for the main speaker, and drop any speech that's some significant threshold below that. I think combining the confidence rating from those two preprocessors, perhaps adding speaker recognition and identification as another signal, could work reasonably well. @SoulAuctioneer LLMs are responsible for determining the context as per pipeline, consequently incapable of pre-filtering user input before goes to LLMs. We already have preprocessors that discard user inputs below a specified threshold and also perform channel recognition.

@Shubham Bajaj thanks for these responses. > "inform the user to take a step back or move away from any potential distractions" While the idea to tell the user to move somewhere quieter would work in principle, in practice it's not a quality UX so I doubt most people would want to implement that. We're aiming to emulate human-level comprehension. > Have you had the chance to examine our background noise denoiser? Yes. I assume it's doing some work, but clearly insufficiently to differentiate background chatter from primary speaker chatter. I understand this is a hard problem, which is why I've suggested some potential improvements to the preprocessing. > LLMs are responsible for determining the context as per pipeline, consequently incapable of pre-filtering user input before goes to LLMs. If you chain LLM output to LLM input, LLMs for sure can act as preprocessors to other LLMs, so yes they're capable of pre-filtering. A small, fast model could readily identify out-of-context sections of the transcript and clean them up without adding much latency, passing that as input to the "main" LLM. > We already have preprocessors that discard user inputs below a specified threshold Great! Any way for me to fiddle with the thresholds? I may be able to get better results with some tweaking. > ...and also perform channel recognition. Also great! I'm super curious how you're able to do that with a mono audio stream, assuming I'm understanding your definition of "channel recognition" correctly? Speaker recognition and identification would be another excellent preprocessor. Take a look at the PicoVoice Eagle link I shared above - you can demo it right there on the page and it's very impressive!

do you offer a different (non-VAD based) turn detection model in beta? Wondering the same thing as it feels much better at not interrupting

> While the idea to tell the user to move somewhere quieter would work in principle, in practice it's not a quality UX so I doubt most people would want to implement that. We're aiming to emulate human-level comprehension. I remember last time someone was this strategy and it's working for them. > While the idea to tell the user to move somewhere quieter would work in principle, in practice it's not a quality UX so I doubt most people would want to implement that. We're aiming to emulate human-level comprehension. Work is in progress, currently its not that easy to find the primary speaker voice and if multiple people speaks the most audible wins. > If you chain LLM output to LLM input, LLMs for sure can act as preprocessors to other LLMs, so yes they're capable of pre-filtering. A small, fast model could readily identify out-of-context sections of the transcript and clean them up without adding much latency, passing that as input to the "main" LLM. Good idea chain of LLMs. By this do you mean "MultiModal Dialogue Mangement System"? > Great! Any way for me to fiddle with the thresholds? I may be able to get better results with some tweaking. Yet to expose the threshold field. > Also great! I'm super curious how you're able to do that with a mono audio stream, assuming I'm understanding your definition of "channel recognition" correctly? Transcribers does this job.

where is this turn-taking logic?

@buytslacalls_ can you rephrase please.

I think the question is: how can we do that? How can we integrate LiveKits turn-based model as an additional signal in the Vapis turn-taking logic?

@nazar @buytslacalls_ wait for this week announcements.

Was something announced? I can’t seem to find anything relevant

@nazar now you can use

LiveKit

https://cdn.discordapp.com/attachments/1338689387445227542/1349823181111820358/Screenshot_2025-03-14_at_12.45.15_AM.png?ex=67d4804c&is=67d32ecc&hm=8028d8eb2da8c924db3450b20e8ccd3cef764dca6e1bc2b0ba6ef9f31bdfafa9&