Earlier in the day, we noticed one of our primary lines would not proceed beyond the greeting stage of the call, and any audio provided by the caller is simply ignored after the fact. We tested several lines, and noticed this issue is only affecting certain numbers, while others remain functional. Issue was initially called out by staff at around 3:12pm PST this afternoon, and as a consequence we're now needing to route ALL calls through human agents, which is an obvious efficiency and time suck across the entire organization. Some sample UUID for research: e9e966e7-7532-4678-98e2-167b535956bc 8222e496-2b1a-4f93-9a8a-a1499639c14d 46ff2180-8b87-4674-83f3-bb7acb9a8e0d c460e7d8-1490-4c9f-8b25-bc78c5363321 9fa5f398-ced0-4c6b-8c90-123713b22552 https://dashboard.vapi.ai/calls/9fa5f398-ced0-4c6b-8c90-123713b22552 see attached for the typical dead air pattern we're seeing, and also the attached logging. We can provide more once we understand the scope of the issue. If you guys can lend a hand and explain what's going on, we'd really appreciate it. Management are becoming less and less tolerant of the issues happening inside your blackbox, so if you're able to demonstrate the failure point it'll go a long ways to staving off the calls to migrate to a different platform that have been growing in volume over recent months. LMK if you have any questions, and thanks for the help in advance. https://cdn.discordapp.com/attachments/1339419538227200104/1339419538508480562/Screenshot_2025-02-12_at_6.10.30_PM.png?ex=67aea727&is=67ad55a7&hm=85a352eabc13885a778554f3a3e58c7a27072a7a0a95427fb8fb9af6916785b0& https://cdn.discordapp.com/attachments/1339419538227200104/1339419538738909204/logs.txt?ex=67aea727&is=67ad55a7&hm=fcd5d687de3d795a0b86857c02a9cf07446309c9fede93615542d7d9ea7b7d30&

Facing the same issue today

@EnSabanNur can you provide me a test number where I can make calls?

@Sahil Are you referring to an outbound call? U can use this phoneNumberId

890fe48c-c5b7-43f2-b152-fa2533387173

I want to make a call to a number where we can test it. Currently, according to our logs we never received any output from the LLM.

U can also use this call ID

9bd0c5eb-b864-4e27-945b-54327937a868

I'll DM you a phone number as well if you wanna initiate a call to it (but you can use any number, including yours)

Hey Barbelos, I can see there was some response from the bot side...can you be more specific what went wrong in this call

@Sahil Check this call ID

c14ac537-8def-4d0d-8bb3-440ec572dcf1

It takes forever for the LLM's output

Hey Barbelos, can you create a separate support request thread? It is a different issue for you.

We have more info later, but the TL;DR is after 6 hours of debugging today: the only way we managed to fix all of our lines was by removing our OpenAI key from the account, even though: the key is fine and is still valid, and adding a brand new key also causes failures. For now, we have no LLM keys tied to our account (previous was using openAI as primary, Anthropic as backup), and everything is working OK - which makes absolutely ZERO sense to us (unless somehow something was fixed on your side this morning around the same time etc - coincidence) We'd love to know why we're seeing this, and what caused it. We'll continue to review information on our side, but it looks like the issue is happening inside your black box and we don't have the visibility we need to correctly triage the issue.

Definitely, we will take a look into this issue and get back to you properly on this. Just to confirm, you are saying that this issue is reproducible only when you add your OpenAI API key, right?

Hey @Sahil, that's correct. For context: we have x2 accounts: one for prod, one for dev. We noticed ALL of the dev accounts worked, while quite a few were failing on the prod account. We tried various bits, and the last ditch attempt was "well, the only difference we see now is that we don't have an LLM AI key added for dev, and it works, so let's try the same on the prod account" and like magic, all the prod accounts all starting working 100% of the time. As mentioned, before removing the ket entirely we did also try adding a fresh key, along with testing the old one, and both worked fine outside of your system, but once added everything began to fail. Very strange. Obviously the key vs non-key will have different code paths in your system, but I can't imagine they're so different that it causes issues. It's also a bit bizzare that we didn't see a total failure across all accounts, but what our Lead did work out was that the moment the LLM was consulted and needed to make decision and execute a function call, was the point where everything breaks and the caller experiences silence- for us that first point is after the greeting, when whatever input the caller provides dictates which flow they enter - think existing client, potential new client, sales, other (in the basic sense).

Weird, I just made a call with the same llm model gpt-4o-2024-08-06 and used our openai key and it was working properly. Can you generate a temporary openai key access and send me in dm?

Small update, even without the api keys, we're seeing the same issue on a couple of calls today. So far: - 6e079407-6c6a-45e6-b3e3-b2de9fc970ea - d6dcffeb-bda7-4940-99b0-e228f8ff45b0 - 93e91ecb-77a7-4ee1-a4e4-96d91ba68b9c - 853d7ba7-b161-47fd-b857-e35ac29c3805 Note: for these, we make it past the greeting, and the VA stopped responding mid call. One of the leads said we can see the event endpoint being called with "hang" and then the VA dies (which it should do after a 5+ second delay) for context: we're now up to 3 cancellation requests due to the issue, so any help is appreciated (working on the API key now, but looks like it's not the root cause of the issue). We also know on human fallback mode for all accounts again as a safety measure.

Hey man, escalate this issue as a high-priority issue. Provide us a little more time so that we can get to the root of it.

Thank you @Sahil, we really appreciate the help.

Hey, we found the issue and have pushed the fix. It will be live in 10–15 minutes.

Thanks, we'll start testing and report back shortly. We appreciate the hustle from you guys.

Thank you for reporting, let us know if any further issues 🙏

Early signs of recovery are looking good. We have a handful of test accounts we're running calls against using our CS team, and we're also testing one of our staff utility lines, and everything seems to be working well so far. If this continues, we'll flip back over in around 15 mins at 12:45pm. We also haven't seen the "hang" even for over an hour, which is also a good sign. I'll report back in 30±.

Sounds, good.

@Sahil We're seeing something very similar to this starting last night and today. Our calls connect for 26 seconds and then vapi ends the call with Assistant Ended Call ended reason. type: end-of-call-report, end reason: assistant-forwarded-call

Can you remove voicemail detection?

here is example call id : 89b0aad6-fcbe-4dab-ba07-36afc085eb7a

we're not using google gemini

huddling you

@Sahil I'm not saying that we don't seeing recordings in vapi. We have it configured to our own s3 like you said above. What I mean is that the call shows a call log with 0 duration, the voicebot never picks up and there is no logs or recordings.

Hey guys, can you try again and see if you are getting this issue again or not? We pushed a fix related to it.

Will check the calls today to see if this issue is still happening.

Can you create a new support thread as well as can you turn off the HIPAA mode and provide me something where I can take a look at?