Hey everyone! I'm building a voice AI agent in the...
# feature-requests
j
Hey everyone! I'm building a voice AI agent in the VAPI dashboard and running into a couple of issues. Would really appreciate any advice. GPT-4o-mini not following instructions My agent has a long, detailed system prompt (~300 lines) with conversation rules, data collection steps, and path switching logic. On GPT-4o-mini the agent often ignores key rules: it skips steps, uses banned phrases, and doesn’t switch conversation paths when it should. I tested Claude 4.5 Haiku and GPT-4.1, and both follow the prompt much better. Still, I keep hearing that most voice agents are built on 4o-mini. Latency spikes I’m also getting random latency spikes — some turns go up to 3000ms+ while others stay around 1400–1600ms. LLM response time can jump from ~400ms to ~1800ms between turns without a clear pattern. On top of that, I’ve hit rate limit errors with GPT-4.1 (429 — 30K TPM limit). My main questions: Is GPT-4o-mini really the standard / recommended default for VAPI voice agents right now? If yes, how do you get it to reliably follow complex prompts? Do you keep prompts much shorter, use a specific structure, split logic across tools, or something else? Or is there a better model in VAPI that gives a good balance between speed and instruction-following for production voice agents? Any tips on reducing latency spikes (choice of STT/LLM/TTS stack or VAPI settings) would also be super helpful. Thanks in advance!