Sometimes Vapi takes a while to respond
# supportp
Papi
05/21/2024, 9:26 PMI have an issue where on occasion vapi takes a while to respond, i think it has to do with gpt4o being slow at times switching to gpt4 turbo seems to resolve it when this happens. But is there a config i can use to use the fallback model? after x seconds of waiting for gpt4o?
v
Vapi Support
05/21/2024, 9:26 PMHi @Papi, I'm an AI assistant for Vapi. I'll analyze available sources and get back to you shortly.
Vapi Support
05/21/2024, 9:27 PM### Answer
Currently, the official Vapi documentation does not provide specific details about configuring a fallback model that automatically switches based on response time thresholds from a primary model such as GPT-4o to an alternate like GPT-4 Turbo.
However, implementing a timeout and fallback mechanism at your application layer might be a feasible solution. You could set up your system to monitor response times, and if the primary model's response exceeds your defined threshold, you can programmatically switch to a faster model for subsequent requests.
For direct support and more personalized guidance on possible configurations or updates that might not be documented yet, you might consider reaching out to Vapi's support. They could provide the latest updates or workarounds that are not yet covered in the public documentation.
Vapi Support
05/21/2024, 10:21 PMGot it. Tagging @User for additional help.
s
Sahil
05/21/2024, 10:24 PMWe auto switch to the fastest model based on the latency.
Can you send me a call_id along with the audio timestamp where you faced the delay?
p
Papi
05/22/2024, 5:18 AM250f6bd1-de79-41a1-b246-622f2aff7302 no timestamp cause it's the whole call. It took 30 seconds to respond. Also note this is a test call and the script is a little weird . hah . Testing "emergency" call forwarding rules . (not 911 emergency but like angry karen emergency)
Papi
05/22/2024, 5:19 AM@Sahil
s
Sahil
05/22/2024, 7:56 AMIt is fixed.
Sahil
05/22/2024, 7:56 AMCan you give it a try once again and lmk
p
Papi
05/23/2024, 3:32 AMIt's not super consistent but will report back if i expierence it again thank you so much sahil
s
Sahil
05/23/2024, 3:37 AMWelcome 🙂
p
Papi
05/25/2024, 5:30 AMSahil this is still an issue
ea2a3328-49cb-43c5-9fe7-3bc28d103583
No timestamp it's the whole call. Basically delayed for the first 15-20 seconds then it repeats the same question over and over again . Not a prompting issue i suspect something to do with the delayed response
Papi
05/25/2024, 5:30 AM@Sahil
Papi
05/25/2024, 5:31 AMjust made another call and it's doing the same thing .
Papi
05/25/2024, 5:31 AM30 second to the first response
Papi
05/25/2024, 5:32 AMd6f80b04-b9ef-47d6-9872-ff26daa3dfc5
Papi
05/25/2024, 5:32 AMthis is taking 30 seconds per response .v
Papi
05/25/2024, 5:32 AM3rd call is working nromally
Papi
05/25/2024, 5:33 AMgotta figure this out before i go in to prod
Papi
05/25/2024, 6:22 AMYeah it's hit or miss sometimes i get on a call where it takes 30 seconds per response but then i call back and it's fine. I think maybe im getting stuck on a bad azure endpoint?
I use azure openai for some other stuff and notice their endpoints are way slower lately.
Papi
05/25/2024, 6:34 AMim planning to go live at the beginning of june, keep me posted on this and ill be happy to do more testing.
s
Sahil
05/26/2024, 2:57 PM@User Can you please create a new support request after one is completed because it gets confusing.
2 Views