We have this problem too intermittently, we were never able to nail it down to anything concrete. We're currently operating under the assumption it's a mix of multilingual detection and LLM latency

Set a fixed language in your assistant config to skip detection, enable streaming responses for faster reply generation, use cached or pre-computed replies for common openers, and fine-tune startSpeakingPlan or use smartEndpointing to reduce delay after the user’s first utterance..

good stuff this is a ton of great advice