Hi, I am working on an outbound call agent with Vapi. I added a knowledge base text file to Vapi with data I scrapped from the real estate dev company website - see the attachment. Unfortunately the Vapi assistant based on this file does not provide me with the correct property prices for sale, for instance for Studio apartment in Seef Avenue. The numbers are hallucinated. I set the temperature for 0.3, model gpt-4o and no improvement. Also when I asked the assistant for the purchase of Studio in Spring Hill he responded that yes it is available - but there is no such data in the knowledge base, so basically, it is hallucination. What am I doing wrong? https://cdn.discordapp.com/attachments/1312857688442540203/1312857689017155686/Royal_Ambassador_1-12-2024.txt?ex=674e0580&is=674cb400&hm=72168ee67551070550a320672f8d8f101fea5b569ebad057caad0adbe8e53baf&

@Igor Hello! It sounds like you're facing a couple of issues with the accuracy of the responses from your Vapi assistant. Let's address them one by one: 1. **Incorrect Property Prices**: If the assistant is not providing the correct property prices, it could be due to the way the knowledge base text file is formatted or how the information is being interpreted by the model. Ensure that the data in the knowledge base is clearly structured and unambiguous. If the prices are not straightforward in the text, the model might struggle to extract the correct information. 2. **Hallucinated Data**: When the assistant provides information that isn't in the knowledge base, it's often because the model is trying to fill in gaps in the data it has been provided with. To mitigate this, you can: - Review the knowledge base to ensure all necessary information is present and clearly stated. - Adjust the prompt to explicitly instruct the model to respond only with information available in the knowledge base. - Implement validation checks to ensure responses are cross-referenced with the knowledge base data before being relayed to the user. Since you've already adjusted the temperature and tried a different model without success, I would recommend revisiting the knowledge base file and the way you're querying the assistant. Make sure your prompts are clear and direct the assistant to use only the information from the knowledge base. If you've done all this and the issue persists, it might be beneficial to look into more advanced configurations or reach out to support for a more in-depth analysis of your specific setup. Please let me know if you need further assistance or guidance!

Hey @Igor To help track down this issue, could you share: - The call ID - When exactly this happened (the timestamp) - What response you expected to get - What response you actually got instead This would really help me figure out what went wrong!

I changed the knowledgebase file and now it is much better structurized. But see that the assistaant is still hallucinating when OpenAI 4o and 4o mini is used. CI changed for Claude sonnet and it seems not halucinating while calls are suddenly disconected - and it happens with heiku modell as well - example call id d63ae274-d189-4942-8b12-41e645c70c2c Example of hallucination regarding the price you will find here call ID d583b855-7703-49a1-914d-5e01bf4961d8 - see the end of conversation: AI Certainly, the price for, uh, purchasing the 2 bedroom apartment with a maid room in Fontana Gardens is 1 0 Bahraini Dinaz, you like to schedule a visit to see the property? User How much? How much? AI The price for purchasing the 2 bedroom apartment with a maid room in Fontana Gardens is **1 3 9 0 **Bahraini Dinaz, you well like to schedule a visit to

@Igor your using Canonical KB, the

Canonical

has stopped their services can you try using the new KB provider https://docs.vapi.ai/knowledgebase https://cdn.discordapp.com/attachments/1312857688442540203/1313984913103388843/Screenshot_2024-12-05_at_3.03.57_AM.png?ex=67521f4f&is=6750cdcf&hm=4b1d9abc3627f0af4e02ef831943d9be7f6bb500741ae8e5c845834c2c2e91b9&

I used a file with the data I scraped from the website. I haven't used an external provider. he link you provided doesn't say anything about canonical KB

@Igor This is the same issue I ran into, they don't seem willing to acknowledge their big mess up that caused our clients to be called liars and our assistants to hallucinate overnight. What @Shubham Bajaj said is true, VAPI just didn't tell anyone outside of a Discord message and the dashboard interface still creates the KB files in a basically non-existent provider without any error warning at all. So you think the KB is working, but it's not. Only solution for now is to create the KB using their API and upload files using their API by using the new provider. Make sure to set the Threshold higher than the 0.1 in their documentation, which is an awful way to present a default and will keep making it lie if you have a lot of docs. Should be at least 0.7

Hey @Igor Previously, our KB provider was Canonical, but they have discontinued their services. To maintain backward compatibility, we did not remove their reference from our system. Now, we have a new provider,

Treieve

. Using the documentation link shared with you, you can set up the new KB provider. I sincerely apologize for the inconvenience.

So as usual I can upload files and they should work?

can you share the call id?

b5a7e3d7-671a-46d4-8f46-168ebfa26334 , the assistant told that there is 2 bedroom apartment available in Juffair while it is not true, see 1:49s. Then also assistant gave the link address to the apartment which is in fact 1 bedroom apartment. You can check the knowledgebase in Google Sheet https://docs.google.com/spreadsheets/d/1o0cj01iLgsryHOuW5lH4IDn2mticaa5LhYMsm84SDvk/edit?usp=sharingto filter out easily what is then uploaded in the knowledgebase in VAPI

@Igor your still using Canonical KB for call id 0cc33480-e028-4261-9671-db12708fba52 https://cdn.discordapp.com/attachments/1312857688442540203/1316811971093598389/Screenshot_2024-12-12_at_10.30.13_PM.png?ex=675c6837&is=675b16b7&hm=aee39f5c8950a46c9acaf98ef1fe0699367447b3f96463825b9358494abe79a5&

the point is I added the file in your UI and I had no way to select KB provider. And the file was added yesterday so if you ysse now Trieve then I understand the file should be automatically sent and tokenized in Trieve. So if it is sent to Cannonical it meand there is some bug in your BE

@Igor As of now we haven't replace from UI Cannonical with Trieve because of this your observing issues, can you try creating knowledge base using this url https://docs.vapi.ai/knowledgebase