VAPI AGENT NOT WORKING...
# support
g
My Vapi agent is having lots of issues, I've spent over 16 hours trying to get it to work with prompt engineering, tweaking the settings and restructuring the knowledge bases. It's making the most weird mistakes that it's not supposed to at all... 1. Hallucinating When asked for something simply like restaurant suggestions, it sometimes gives random restaurants that aren't in the knowledge base. I have clearly stated in the system prompt to only reference the knowledge base as the single source of information multiple times, yet same issue. 2. Randomly getting cut off During the conversation, my agent randomly gets cut off or stops talking. I have tried many different settings, even the default one, yet still same issue. 3. Always ending every sentence with a repetitive robotic sentence, even though already explicitly told it to vary and change up how it communicates. I hope someone more experienced can help. I've been trying to fix the errors tirelessly, and I don't know why the agent is giving me these issues.
o
Which model are you using? I just tried using Gemini and it was all over the place, then switched to Claude and it followed my system prompts much much better
v
Hey! To help track down this issue, could you share: - The call ID - When exactly this happened (the timestamp) - What response you expected to get - What response you actually got instead This would really help us figure out what went wrong!
g
Open AI GPT 4o latest
Call ID: 9caf05ea-ca2b-4791-8bba-6c6b5b303a79 This keeps happening in every call I have with the agent, not just once... I expect to get accurate information from the knowledge base, not made up information from the agent. The knowledge base is super clean and it's easy to locate the information. The system prompt clearly tells the agent to ONLY reference the knowledge base to formulate answers.
In short, the agent is failing to reference the knowledge base for most queries.
v
Hey, so here is what's happening to use the query-based KB tool, you need to upload your files again so that they can be indexed. Any file uploaded before this query-based tool cannot be used along with it because they were not indexed. Remove your files from your assistant, upload them again, and then attach to the assistant, they will be indexed. After that, you can use it with your assistant. As of now, the tool call is being triggered, but we and the model look to fetch the content from the files, but it gets nothing and eventually passes nothing to the main LLM, resulting in hallucinations or made-up responses.
g
@Shubham Bajaj so just remove the KB file from the assistant and add it back? And should we do this for all new assistants created in future as well?
@Shubham Bajaj if this is the case, it's still not working. I just tried..😢
s
You have to upload the file again as well.
Can you share the call ID with files upload again and then attached to the assistant.
g
@Shubham Bajaj I did exactly what you said, now the agent won't even say the names of the places... Call ID: 27ad7850-47c4-45cb-88cb-b8d9f62689a5 I linked the knowledge base here (it's dummy information) https://cdn.discordapp.com/attachments/1351957835046457415/1353267701023248396/Airbnb_Knowledge_Base.txt?ex=67e10842&is=67dfb6c2&hm=d2d5cea7ddbbdddeddede0c0dffb38211833623243dfaca1da41d00f9580f609&
s
logs
šŸ”µ 07:06:49:516 AzureOpenAIRequest. Messages: [ { "role": "system", "content": "# Role\n\nYou are Niko, a highly skilled and engagin..." }, { "role": "assistant", "content": "Hey. Nico here. How can I" }, { "role": "user", "content": "Hi, Nico. Uh, can I get some restaurant recommendations?" }, { "role": "assistant", "content": "of course? What type of cuisine are you most fond of? That way, I can give you the best recommendations based on your preferences." }, { "role": "user", "content": "Probably Italian food." } ] šŸ”µ 07:06:49:919 [user LOG] Model called tool: defaultQueryTool({"knowledge_base_names":["default"]}) šŸ”µ 07:06:50:683 GoogleRequest. Messages: [ { "role": "model", "parts": [{"text": "Hey. Nico here. How can I"}] }, { "role": "user", "parts": [ { "text": "Hi, Nico. Uh, can I get some restaurant recommendations?" } ] }, { "role": "model", "parts": [ { "text": "of course? What type of cuisine are you most fond of? That way, I can give you the best recommendations based on your preferences." } ] }, { "role": "user", "parts": [{"text": "Probably Italian food."}] } ] šŸ”µ 07:06:50:681 Retrieved content for file c98eeec6-6220-4816-8279-4b122f3382b4 šŸ”µ 07:06:51:734 Voice Input Formatted: "Ah, Italian, excellent choice! You're in luck - there are some really great Italian restaurants near the property.", Original: "Ah, Italian, excellent choice! You're in luck - there are some really great Italian restaurants near the property." šŸ”µ 07:06:51:899 Voice Input Formatted: "Here are a couple of my top picks:. * It's a 6-minute walk away and known for its handmade pasta and seafood.", Original: "Here are a couple of my top picks: * Cecconi's: It's a 6-minute walk away and known for its handmade pasta and seafood." šŸ”µ 07:06:52:319 Voice Input Formatted: "so perfect for a special occasion or romantic date night. * It's about a 10-minute walk from the apartment. A more casual, family-friendly spot with a focus on authentic regional Italian cuisine. It's particularly well-known for its wood-fired pizzas and fresh pasta dishes. Do either of those sound good to you,", Original: "so perfect for a special occasion or romantic date night. * Trullo: It's about a 10-minute walk from the apartment. A more casual, family-friendly spot with a focus on authentic regional Italian cuisine. It's particularly well-known for its wood-fired pizzas and fresh pasta dishes. Do either of those sound good to you," šŸ”µ 07:06:52:344 Voice Input Formatted: "or are you looking for something different?", Original: "or are you looking for something different?"
šŸ”µ 07:07:16:195 AzureOpenAIRequest. Messages: [ { "role": "system", "content": "# Role\n\nYou are Niko, a highly skilled and engagin..." }, { "role": "assistant", "content": "Hey. Nico here. How can I" }, { "role": "user", "content": "Hi, Nico. Uh, can I get some restaurant recommendations?" }, { "role": "assistant", "content": "of course? What type of cuisine are you most fond of? That way, I can give you the best recommendations based on your preferences." }, { "role": "user", "content": "Probably Italian food." }, { "role": "assistant", "content": "Italian. Excellent choice. You're in luck.", "toolCalls": [ { "id": "call_VjIkBnSmunPHBfq1g7jwQ6OV", "type": "function", "function": { "name": "defaultQueryTool", "arguments": "{\"knowledgeBaseNames\": [\"default\"]}" } } ] }, { "role": "tool", "content": "Success.", "toolCallId": "call_VjIkBnSmunPHBfq1g7jwQ6OV" }, { "role": "assistant", "content": "There are some really great Italian restaurants near the property. Here are a couple of my top picks. It's a 6 minute walk away and known for its handmade pasta and seafood. It's a bit on the upscale side, so perfect for a special occasion or romantic date night. It's about a 10 minute walk from the apartment. A more casual family friendly spot with a focus on authentic" }, { "role": "user", "content": "Can I get the names of the restaurant, please?" } ]
@Gen04 The voice input formatting plan we applied to the LLM output before sending it to the voice provider trimmed any content between the markdown symbols. When the LLM voiced out the response, you couldn't hear the restaurant name, leading you to believe that KB wasn't working. To fix this, disable voice input formatting plan markdown formatting and asterisk formatting. This will allow the KB tool to work as expected, and the logs show that it was involved and returned the required response.
5 Views