https://vapi.ai/ logo
Join Discord
Powered by
Hallucinations with KB
# support
s
call ID: 374242f7-7970-4a1a-b67a-742136cab540 timestamp: 0:46 I have a small KB in a text file that contains council numbers and the insurance agents are associated with the respective council number. During my call, I provide the council number to the agent and it mentions the name of the wrong insurance agent. And when I repeat the council number again, it mentions the name of someone who isn't an agent in the knowledge (i.e. it says a random name)
v
Hi @Saien, I'm an AI assistant for Vapi. I'll analyze available sources and get back to you shortly.
----------------------------- **While I generate an answer, here are some sources that might be helpful**: * [Launch Screen Assets]() * [Introduction]() * [Orchestration Models]() * [Glossary]() -----------------------------
### Answer It sounds like there might be issues with how the input data (council numbers) is being handled or mapped in your application. Here are a few steps you can consider: 1. **Data Validation**: Ensure that the council numbers in your text file are correctly formatted and mapped to the correct insurance agents. There might be discrepancies or formatting issues causing the wrong mappings. 2. **Debugging the Code**: Check the part of your application code that fetches and references the insurance agent names based on the council numbers. There might be logical errors or incorrect references. 3. **Logging and Monitoring**: Implement logging around the data input and output points. This can help trace what data is received and what response is generated at each step. If you are using Vapi for handling the voice interactions, ensure that the speech-to-text component is accurately transcribing the council numbers. Misinterpretations by the speech-to-text engine might be leading to incorrect agent references. For further refinement in handling voice data and ensuring accurate recognition, you might want to look into customizing the speech recognition model to better understand the specific terminology or numbers used in your domain. Unfortunately, based on the details and documents provided, I can't give a more specific solution, as the exact implementation or code details aren't available. If the issue persists, consider revisiting the integration and configuration of your voice AI components or the underlying data handling mechanisms.
Got it. Tagging @User for additional help.
s
I had some luck restructuring how the data is displayed in KB but it's still not perfect yet. I will update - but this is how the KB looked when I created this support ticket: https://cdn.discordapp.com/attachments/1247531250323816459/1247548816773611641/Screenshot_2024-06-04_at_15.53.24.png?ex=66606de0&is=665f1c60&hm=2b1b26cad8699e152cd160e70c2967381c9be2e5a2971c4924c4ff7e2a27fd92&
Yeah, it's still messing up...
In this call, it hallucinates the name of the agent: ba4d6bc3-69af-412f-be89-8f4fae9832c0 timestamp: 00:23 Also, at 00:57, the output is really strange... The AI stumbles on its words by saying two different sentences at the same time.
j
That last issue about stumbling on its words is a known issue I believe, which occurs sometimes when your assistant has a knowledge base. I had a similar issue. But the other points about RAG inaccuracy is useful. Hopefully that improves. Would be superb if we could get an Open AI Assistants API integration. Maybe when the new GPT4o models come out.
a
@Saien LLMs struggle with tabular data, especially GPT 3.5. It might be worth further restructuring it.
s
I appreciate the response 👍🏽 Quick question, how would you recommend that I restructure the data based on the format that I’m using right now? As for the LLM, I’m using GPT 4o and when it comes to the accuracy, it’s around 70% so definitely not good enough.
j
Is your temperature set to 0.1
s
No it's currently 0.3
j
Have you experimented with setting it to 0.1? I'm not sure it will change much but worth a try
s
I have tried it with other projects but in my experience, it's never really helped much to be honest. But I'll try it out
s
@Jotham @User @User thanks for the help!
s
call ID: 55591212-f8d0-4b30-a33b-eb9cea80c175 timestamp: 00:09 The agent repeats the same sentence twice in its output. I'm using GPT 4o as the LLM. I'm still getting the same halluncations even after restructuring the KB as follows: https://cdn.discordapp.com/attachments/1247531250323816459/1247836807958237194/Screenshot_2024-06-05_at_10.57.48.png?ex=66617a16&is=66602896&hm=649b13c627cbaa0dd7e2b34c2ca55f6beb857bc426dcbe5c664a371866db0f35&
s
We are working on the Hallucination with KB part.
s
Okay thank you, please let me know once it’s been resolved.
Also, I noticed this bug where it says that I have files added to my assistant's KB even though there isn't any files uploaded. I don't think it will affect anything but I figured I'd bring it up. https://cdn.discordapp.com/attachments/1247531250323816459/1247858361186385980/Files.mov?ex=66618e29&is=66603ca9&hm=a6d819813125ce3db9675d56a9d6a094f941077921d71e24364d4a4dc483fc07&
I changed the Top K to 1 since each council number is unique so this should help with the accuracy of the output
There still seems to be an issue with the KB (it selects the wrong agent at times) f915e7d3-d974-4bb1-9763-6df84cb411c3 00:13 I set the top K value to 1 since all I'm using the KB for is to search for the agent associated with a council number (and it's always a unique council number). Let me know if this should be changed. https://cdn.discordapp.com/attachments/1247531250323816459/1249676030428581928/Screenshot_2024-06-10_at_12.41.57.png?ex=66682aff&is=6666d97f&hm=6808c027a4cd8f7d40a4b7dabc25c05e65e53add47b9ffeeefec0ba32fffc015&
I tested the KB multiple times and it seemed to work perfectly until this call. 49ce9ece-a24b-488a-92a5-537e76d3d49e 00:44 Also, there was the stumbling issue but I know that's still in progress to be fixed. https://cdn.discordapp.com/attachments/1247531250323816459/1250041922077200454/Screenshot_2024-06-11_at_12.59.47.png?ex=66697fc2&is=66682e42&hm=13d80a739b63cc4b97dfc702b826f203a978b83d79025b6cf33577b7fa30656c&
Also at 00:09 for the same call, the AI cuts itself off for some reason (it just stops speaking). I'll create a new ticket about this.
s
Can you please create a new ticket with call_id and time stamp as well
s
okay
PreviousNext
Powered by