Hi,
we have been using our assistants to consult our customers on general product questions including pricing. I have set up my knowledge base on Trieve and connected it to VAPI.
Dataset/KB settings in Trieve/VAPI:
I kept the default settings, e.g. Vector Embedding model -> jina-base-en, Search -> semantic
Problem:
Since updating the dataset in Trieve by deleting and reuploading a document with updated pricing information about a product, for this specific product the assistant either states that there is no information in the knowledge base regarding the pricing or it hallucinates numbers.
Observations:
- This happens more often with the assistant that has the transcriber set to German than the English one,
- When asking the same query in the Playground in Trieve and the "Chat with Assistant" feature in VAPI, I get the correct information. The mistakes only occur during calls
- I checked the logs and it turns out that whenever the assistant says there is no information or hallucinates its answer, it did not transcribe the product name correctly
- Subsequently, it does not retrieve the chunks that contain the pricing information for this product, but rather other chunks relating to different products or chunks of the requested product that contain more general information, but no pricing information
What I tried so far:
- I tried using they Keyword Boost for the product names and it now transcribes the names correctly, but the answers are still mostly incorrect and it still pulls irrelevant chunks
- I tried deleting the dataset in Trieve and reuploading it with the same settings, but that did not work
- I created a new knowledge base in VAPI with the same dataset from Trieve, but instead of picking semantic search I went for hybrid. This improved the results and more relevant chunks were picked
- I also created a new dataset in Trieve with a new embedding model -> OpenAI text-embedding instead of jina-base-en. This improved the retrieval