The latency is usually affected by 2 things: The LLM and the voice model that you use. Pairing gpt 4o with Eleven Turbo 2.5 in the voice tab usually give latency of around 1sec, and has been the most suitable for a vide range of use-cases. Let me know if this helps