Hi, I'm reposting this because you closed my ticket. The problem is that I don't get consistent results when I use a knowledgebase. What I have: A small TXT document with knowledge base of profoundly false facts. So that I know that when the assistant answers it uses the knowledgebase as the single source of truth. When I ask: "What is the color of the sky?" I expect the answer to be "Purple." When I ask: "What do you need in order to play football?" I expect the answer to be: "A square ball made out of stone." When I initially posted the problem here I did not have any success in any of the cases. Today, I've had mixed results without changing anything. So here are my results for today: Test 1: If I make a temporary assistant and pass the knowledgebase I get mixed results. The assistant seems to be aware of the knowledgebase but it's not always following it. Prompt:

You are a smart assistant who responds to user queries using the information from your KnowledgeBase. If the answer is not in the KnowledgeBase say 'I do not know'.

Call ID:

b0c051db-5124-442a-9cfe-b55ca94dff43

TS:

1738573301379

Q: What is the color of the sky? A: I do not know. TS:

1738573307989

Q: What is the color of the grass? A: According to the provided context, the color of the grass is black. Test 2: If I add to that prompt some conversation history I get some funny responses. The assistant seems aware of the knowledgebase but chooses to ignore it. Call ID:

86013be6-e946-4994-ae3a-83210f64c42d

TS:

1738573089949

Q: What do you need in order to play football? A: To play football, you need a round ball made out of leather or a similar material, not a square ball made out of stone. 1. What am I doin wrong? 2. This is hard to debug without any info. Is there a way to check what chunk of the knowledge base the assistant used to answer a question? 3. If the prompt is the problem then what is a good prompt that references the knowledgebase correctly?

@AGTGreg I'm sorry to hear that you're experiencing inconsistencies with your knowledge base integration. Let's address each of your concerns: 1. What am I doing wrong? It's possible that the way the knowledge base is being referenced or integrated into the assistant is not consistent. The assistant should be explicitly instructed to use the knowledge base as the primary source of information and to default to "I do not know" if the information is not present in the knowledge base. 2. Debugging without info on knowledge base usage: Currently, there isn't a direct way to check which part of the knowledge base the assistant used to answer a question. However, you can design your prompts and the assistant's behavior to make it clear when it is using the knowledge base by instructing it to cite the knowledge base in its responses. 3. A good prompt that references the knowledgebase correctly: A more effective prompt might be:

You are a smart assistant whose responses should be based solely on the provided knowledge base of profoundly false facts. If the knowledge base does not contain the information asked for, respond with 'I do not know'. Always cite the knowledge base when providing an answer.

This prompt makes it clear that the assistant should use the knowledge base as its only source of information and should always indicate when it is doing so. If you continue to experience issues, it might be helpful to review the knowledge base document to ensure it is formatted correctly and that the assistant is configured to prioritize it as the source of truth. If the problem persists, please ask support for more info on this.

This prompt does not work. How do I reference the knowledge base in the prompt so I can get consistent results?

Call ID: 86013be6-e946-4994-ae3a-83210f64c42d

logs

🔵 08:58:12:153 KB: Made search request to Trieve: What is the color of the. Grass.. Do you need in order to play football? Query string : What is the color of the. Grass.. Do you need in order to play football? 🔵 08:58:12:153 KB: Retrieved 2 chunks from Knowledgebase. @AGTGreg you should probably increase your scoreThreshold (currently set to 0.3 in this call, so these chunks are added every message) -- some of the retrieved chunks here are really not needed. https://cdn.discordapp.com/attachments/1335953119275782144/1336258578649972860/message.txt?ex=67a32747&is=67a1d5c7&hm=571f34a73aab81df35d94e01ba9e41b38ea9cc79efd7264c8743d9c79f9a83a2&

@Shubham Bajaj Thank you very much for your response. I'll be running tests with different search plans and let you know how it went.

@Shubham Bajaj Could you support me ? Im facing the same issue Im creating a Kb Base linking some fileIds to him Creating an assistant adding the knowledgeBaseId But when I start the call the Assistant couldnt respond to me the question which the answers are in the KB files. I Tried change my Kb base to searchType property ==

hybrid

I tried increase the

scoreThreshold

and remove this property But the assistant couldnt get the results from files AssistantId a1f98856-4d70-4094-ae27-f9402479893d Call log id 80608eb4-5f4f-46ed-a3da-a0d9ff9cdde5

@Shubham Bajaj From what you've showed me, the problem seems to be in the splitting and chunking of the knowledgebase when it's created. So I didn't have any success by adjasting the

scoreThreshold

alone, however when I lowered the

targetSplitsPerChunk

I saw a big difference. Now the chunks retrieved are smaller and more relevant (higher scores). The smaller they are, the higher the score is and I got a better chance of getting the right answer but this is just a test knowledgebase and I'll need to adjast for my production. The problem however seems to be that the assistant will not always try to retrieve knowledge from the knowledgebase. The reason I reached to that conclusion is because the assistant will not retrieve any chunks the first time a question is asked but if I ask again the same question in a the same manner it will retrieve chunks with scores well above my threshold and give the right answer. I also tried with a temperature of 0 to make the assistant fully deterministic but I had the same results. Here are 2 calls that showcase what I'm saying: Call ID:

f43a01dd-ce4e-460a-8bbc-634e831c74b7

and

eff74ece-5137-4a3e-82e1-2c01041f5285

The assistant will not retrieve any chunks for the question "What is the color of the sky" the first time. However it will retreive chunks the second time with a top score of 0.879. Plans:

json
"searchPlan": {
    "scoreThreshold": 0.4,
    "searchType": "hybrid"
},
"createPlan": {
    "type": "create",
    "chunkPlans": [
        {
            "fileIds": file_ids,
            "websites": [],
            "targetSplitsPerChunk": 5
        }
    ]
}

@Seygi @AGTGreg Looking into your call IDs.

Thank you

@Seygi You have deleted your KB so I cannot look into the details what was wrong. Try creating the KB, and then share the updated Call ID so I can take a look.

logs

Deleting dataset with id: b4fabf7e-6b26-4697-89b5-66940564436c

@Shubham Bajaj Thanks, could you let me know when I can try again? For now I have tried a custom KB which I manage myself and it works every time and with good results but makes the assistant slow which is not ideal. I would really appreciate if we could use the create or import knowledgebase because the it doesn't slow down the assistant and the whole reason we wanted to use Vapi is the speed.