Hey everyone!
I'm building a voice AI agent in the VAPI dashboard and running into a couple of issues. Would really appreciate any advice.
GPT-4o-mini not following instructions
My agent has a long, detailed system prompt (~300 lines) with conversation rules, data collection steps, and path switching logic.
On GPT-4o-mini the agent often ignores key rules: it skips steps, uses banned phrases, and doesn’t switch conversation paths when it should.
I tested Claude 4.5 Haiku and GPT-4.1, and both follow the prompt much better. Still, I keep hearing that most voice agents are built on 4o-mini.
Latency spikes
I’m also getting random latency spikes — some turns go up to 3000ms+ while others stay around 1400–1600ms.
LLM response time can jump from ~400ms to ~1800ms between turns without a clear pattern.
On top of that, I’ve hit rate limit errors with GPT-4.1 (429 — 30K TPM limit).
My main questions:
Is GPT-4o-mini really the standard / recommended default for VAPI voice agents right now?
If yes, how do you get it to reliably follow complex prompts? Do you keep prompts much shorter, use a specific structure, split logic across tools, or something else?
Or is there a better model in VAPI that gives a good balance between speed and instruction-following for production voice agents?
Any tips on reducing latency spikes (choice of STT/LLM/TTS stack or VAPI settings) would also be super helpful.
Thanks in advance!