11labs Voice not being generated with high latency...
# supporta
ArtemHorik
01/20/2025, 9:00 PMOur team ran into the problem of vapi not generating assistant speech in the moments of long function calls on our custom llm backend. (10s for example) It cannot be solved by anything - we've tried using tags but it doesnt help.
Example calls Id's:
0818c973-6730-42b4-8c8c-351821489856
9e0d2bdd-5722-4ecd-bbcd-d6317758a607
Sometimes in the logs we see "[CHECKPOINT] Voice audio requested" - but audio doesnt come even after that. In some cases there are no "Voice audio requested" at all!
We use 11labs Eleven Turbo v2.5 with sarah voice
I've even made a repo to test this case, you can test it too! https://github.com/ArtemPolydom/advanced-concepts-custom-llm
Is there a solution? It is a disaster!
ArtemHorik
01/20/2025, 9:03 PMthe fun thing is that it generates it sometimes, randomly
ArtemHorik
01/20/2025, 9:22 PM@User @User
s
Shubham Bajaj
01/21/2025, 10:47 PM@ArtemHorik Here are a few things you need to ensure it works: First, set Deepgram's endpointing to 300ms and then set startSpeakingPlan waitSeconds to 0.8 seconds. After that, it will capture the user input correctly and will have time to process it. Additionally, using ngrok will add some latency.
Basically what happened is: The user input was captured, then a request was sent to your server. In between, the user spoke again and another request was sent, which caused the previous one to be discarded. During this time, the server took time to respond, which resulted in a few cycles of back-and-forth visible in the logs screenshot.
My suggestion is to try adjusting the endpointing and startSpeakingPlan settings, and host it on a platform like Replit. When you try another call, you will notice the difference in latency. Do let me know how it goes!!
https://cdn.discordapp.com/attachments/1331005546349985883/1331394807310057492/Screenshot_2025-01-22_at_04.13.32.png?ex=6791758a&is=6790240a&hm=cfa8463e27795a8cb292eb93a3e1452bb2cacb3fc35f2adfc11ee1247164c5f1&
https://cdn.discordapp.com/attachments/1331005546349985883/1331394807880224839/Screenshot_2025-01-22_at_04.16.34.png?ex=6791758a&is=6790240a&hm=662fe23261b63b501c14a4fcfdd81024714290ed2134e717f230824d2dd2af97&
a
ArtemHorik
01/22/2025, 5:01 PM@Shubham Bajaj there are no Deepgram's endpointing settings in the agent dashboard
But I'll try somehow, thank you
s
Shubham Bajaj
01/22/2025, 5:58 PM@ArtemHorik Here's the example-curl request to update your assistant config for the asked changes. Do let me know if you require further help.
Copy code
curl -X PATCH https://api.vapi.ai/assistant/your-assistant-id-here \
-H "Authorization: Bearer token" \
-H "Content-Type: application/json" \
-d '{
"transcriber": {
"provider": "deepgram",
"language": "en",
"model": "nova-2-phonecall",
"endpointing": 300
},
"startSpeakingPlan": {
"waitSeconds": 0.8
}
}'a
ArtemHorik
01/23/2025, 11:22 PM@Shubham Bajaj This didn't really solve the problem. I'm not talking anything after the request - there are no interruption. But your backend still doesn't make a request for audio generation sometimes (it like waits till I end my speech freaking 10 seconds, how does it possible).
Why it works perfectly with other voice providers but not with the 11labs? If this problem is only with 11labs, so it does not depend on transcriber settings. Is there a solution?
ArtemHorik
01/23/2025, 11:23 PM024fdcb8-637f-4b8c-9fd5-1550ce4625f2
this call was with endpointing 300 and it didnt generate a voice 2 times you can check
s
Shubham Bajaj
01/24/2025, 12:46 PM@ArtemHorik Thank you for bringing this to my attention. It might be an issue on our end. Please allow me some time to investigate whether it is related to our system or 11labs.
Shubham Bajaj
01/25/2025, 7:51 AM@ArtemHorik As evidenced by the screenshots and logs, the 11labs connection was closed until the output tokens were received for two cases. Consequently, this resulted in an awkward silence during the call. I kindly suggest sending the tokens as early as possible to prevent such occurrences in the future.
logs