We are trying to optimize the latency (and perceived latency) in our Vapi Assistants. We use 4o mini cluster + Deepogram novo3 + 11labs. Our system prompts are pretty long ~4000 tokens. My questions: What affects endointing latency? It goes from 200-2000ms. Are server webhooks affecting it? I'm in Europe right now, and I keep getting westus servers, is that because the GPUs are fast, but then network latency adds to "endpointing"? Does system prompt length affect noticeably? Are you using prompt caching? Would love any recommendations or hints! Consistent <1000ms calls would be great. (Most of our users are in the US fyi). Attaching some turn taking logs: Turn latency: 3428ms (transcriber: 750ms, endpointing: 1742ms, kb: n/a, model: 390ms, voice: 526ms) Turn latency: 3953ms (transcriber: 709ms, endpointing: 2456ms, kb: n/a, model: 401ms, voice: 375ms) Turn latency: 2462ms (transcriber: 73ms, endpointing: 1571ms, kb: n/a, model: 245ms, voice: 538ms) Turn latency: 1833ms (transcriber: 298ms, endpointing: 510ms, kb: n/a, model: 392ms, voice: 613ms) Turn latency: 1589ms (transcriber: 564ms, endpointing: 258ms, kb: n/a, model: 424ms, voice: 297ms) Example of assistant id: 9b948733-1307-4f3f-a49c-92ec47af9cc2

Hi Jens, To address your questions about optimizing endpointing latency in Vapi Assistants, here are some key factors to consider: 1. **Endpointing Latency Influences**: - **Network Latency**: If you're using servers in geographical locations far from Europe, such as West US, network latency can indeed contribute significantly to overall endpointing latency. - **Prompt Length**: Longer system prompts with around 4000 tokens can increase processing time slightly, but this typically affects LLM response time more than endpointing itself. - **Server Location**: Endpointing involves real-time processing; closer server locations generally help reduce latency. Currently, if resources are allocated based on a performance balance instead of location specificity, this could add latency. 2. **Server Webhooks**: - While webhooks don't directly affect endpointing latency, any additional processing or delays in receiving/transmitting data can impact the perceived response time. 3. **Optimization Suggestions**: - Consider using servers closer to your operational region to reduce network latency. - Review your usage of

startSpeakingPlan

and

stopSpeakingPlan

settings to ensure they are optimized for your use case, as detailed in the [Voice Pipeline Configuration](https://docs.vapi.ai/customization/voice-pipeline-configuration). Implementing these steps could help in reducing endpointing latency and improving the overall responsiveness of your Vapi Assistants.

hi, thanks for your reply. But this sounds very generic, where are the Vapi hosted servers located? Can't find that anywhere, I assume we can't affect decide the region anyway(?). Thanks!

Hi Jens, For optimizing endpointing latency, one key factor is the

startSpeakingPlan

configuration. Consider setting the

smartEndpointing

option to

LiveKit

with the mode set to "Aggressive". This adjustment aims to reduce the latency during the endpointing process. If you need further guidance on configuration adjustments or resolving account access issues, let me know.

@Kings_big💫 where are the Vapi hosted servers located? my server is deployed on us-east-1

Use Vapi’s self-hosted Docker deployment to keep all data on your own infrastructure in your chosen region..