My Vapi agent is having lots of issues, I've spent over 16 hours trying to get it to work with prompt engineering, tweaking the settings and restructuring the knowledge bases. It's making the most weird mistakes that it's not supposed to at all... 1. Hallucinating When asked for something simply like restaurant suggestions, it sometimes gives random restaurants that aren't in the knowledge base. I have clearly stated in the system prompt to only reference the knowledge base as the single source of information multiple times, yet same issue. 2. Randomly getting cut off During the conversation, my agent randomly gets cut off or stops talking. I have tried many different settings, even the default one, yet still same issue. 3. Always ending every sentence with a repetitive robotic sentence, even though already explicitly told it to vary and change up how it communicates. I hope someone more experienced can help. I've been trying to fix the errors tirelessly, and I don't know why the agent is giving me these issues.

Which model are you using? I just tried using Gemini and it was all over the place, then switched to Claude and it followed my system prompts much much better

Hey! To help track down this issue, could you share: - The call ID - When exactly this happened (the timestamp) - What response you expected to get - What response you actually got instead This would really help us figure out what went wrong!

Open AI GPT 4o latest

Hey, so here is what's happening to use the query-based KB tool, you need to upload your files again so that they can be indexed. Any file uploaded before this query-based tool cannot be used along with it because they were not indexed. Remove your files from your assistant, upload them again, and then attach to the assistant, they will be indexed. After that, you can use it with your assistant. As of now, the tool call is being triggered, but we and the model look to fetch the content from the files, but it gets nothing and eventually passes nothing to the main LLM, resulting in hallucinations or made-up responses.

@Shubham Bajaj so just remove the KB file from the assistant and add it back? And should we do this for all new assistants created in future as well?

You have to upload the file again as well.

@Shubham Bajaj I did exactly what you said, now the agent won't even say the names of the places... Call ID: 27ad7850-47c4-45cb-88cb-b8d9f62689a5 I linked the knowledge base here (it's dummy information) https://cdn.discordapp.com/attachments/1351957835046457415/1353267701023248396/Airbnb_Knowledge_Base.txt?ex=67e10842&is=67dfb6c2&hm=d2d5cea7ddbbdddeddede0c0dffb38211833623243dfaca1da41d00f9580f609&

logs

🔵 07:06:49:516 AzureOpenAIRequest. Messages: [ { "role": "system", "content": "# Role\n\nYou are Niko, a highly skilled and engagin..." }, { "role": "assistant", "content": "Hey. Nico here. How can I" }, { "role": "user", "content": "Hi, Nico. Uh, can I get some restaurant recommendations?" }, { "role": "assistant", "content": "of course? What type of cuisine are you most fond of? That way, I can give you the best recommendations based on your preferences." }, { "role": "user", "content": "Probably Italian food." } ] 🔵 07:06:49:919 [user LOG] Model called tool: defaultQueryTool({"knowledge_base_names":["default"]}) 🔵 07:06:50:683 GoogleRequest. Messages: [ { "role": "model", "parts": [{"text": "Hey. Nico here. How can I"}] }, { "role": "user", "parts": [ { "text": "Hi, Nico. Uh, can I get some restaurant recommendations?" } ] }, { "role": "model", "parts": [ { "text": "of course? What type of cuisine are you most fond of? That way, I can give you the best recommendations based on your preferences." } ] }, { "role": "user", "parts": [{"text": "Probably Italian food."}] } ] 🔵 07:06:50:681 Retrieved content for file c98eeec6-6220-4816-8279-4b122f3382b4 🔵 07:06:51:734 Voice Input Formatted: "Ah, Italian, excellent choice! You're in luck - there are some really great Italian restaurants near the property.", Original: "Ah, Italian, excellent choice! You're in luck - there are some really great Italian restaurants near the property." 🔵 07:06:51:899 Voice Input Formatted: "Here are a couple of my top picks:. * It's a 6-minute walk away and known for its handmade pasta and seafood.", Original: "Here are a couple of my top picks: * Cecconi's: It's a 6-minute walk away and known for its handmade pasta and seafood." 🔵 07:06:52:319 Voice Input Formatted: "so perfect for a special occasion or romantic date night. * It's about a 10-minute walk from the apartment. A more casual, family-friendly spot with a focus on authentic regional Italian cuisine. It's particularly well-known for its wood-fired pizzas and fresh pasta dishes. Do either of those sound good to you,", Original: "so perfect for a special occasion or romantic date night. * Trullo: It's about a 10-minute walk from the apartment. A more casual, family-friendly spot with a focus on authentic regional Italian cuisine. It's particularly well-known for its wood-fired pizzas and fresh pasta dishes. Do either of those sound good to you," 🔵 07:06:52:344 Voice Input Formatted: "or are you looking for something different?", Original: "or are you looking for something different?"