Hello, I am creating a chatbot for an online Albanian store. I am using Trieve to handle the product searching and Vapi as the 'orchestration' layer . The system prompt for the assistant is 889 tokens. However even the shortest output costs almost 3 cents (0.028) in the GPT-o4 model. I am using Postman to see how everything would work in a third-party interface and that's how I made the discovery. I am running 4o because that's the most lightweight model which speaks the dialect the project needs to be in. The tokens needed to run the prompt: 894 The tokens the input was in: 13 The tokens the assistant used to output: 119 Total tokens: 8935 Completion Tokens: 117 Can someone tell me if this has anything with the fact that I'm using Trieve to handle data retrieval? (Here is the chat ID I used to track the pricing and output via postman: b660bb65-f286-4af3-9ad7-75e90120748d)

Yes, using Trieve can inflate token usage and costs if it returns full content chunks instead of just the answer. To reduce costs, create a custom tool that parses Trieve's response and only sends the relevant answer to your LLM. For example:

json
{
  "tools": [
    {
      "name": "trieve-answer-tool",
      "endpoint": "https://your-api/parse-trieve-answer",
      "method": "POST"
    }
  ]
}

This setup ensures only the answer is used, minimizing unnecessary token usage and costs. Sources: - [Discord: Trieve Integration chunks retrieval behavior](https://discord.com/channels/1211482211119796234/1334102591470047292) - [Vapi Docs: Integrating with Trieve](https://docs.vapi.ai/knowledge-base/integrating-with-trieve)

Hey, so when you use knowledge-based solutions like Trieve, and the chat completions API is requested, the knowledge-based chunks are also included. When the messages array is represented or given back to you, they only include the final output, the final result, not the KB source chunks. This could be the factor of your discrepancy. If you could share more info on how you are coming up with these numbers, it will be really helpful for me to further help you.

I've analyzed your call logs and can definitively prove that your token consumption is legitimate and directly caused by knowledge base operations. Here's the complete evidence: Your Call Data Analysis Call ID: 87a3127f-25cb-4ffe-8142-9c5b874be9cd { "llmPromptTokens": 84353, // ← Your inflated token count "knowledgeBaseCost": 0.0052, // ← Separate KB operation cost "llmCompletionTokens": 118, "total": 0.4237 } Detailed Cost Breakdown: { "cost": 0.3675025, // ← Main GPT-4o cost ($0.3675) "type": "model", "promptTokens": 84353, // ← All chunks injected here "completionTokens": 118 }, { "cost": 0.0051675, // ← Separate KB processing ($0.0052) "type": "knowledge-base", "model": "gemini-1.5-flash", "promptTokens": 3782, // ← KB retrieval tokens "completionTokens": 44 } 📊 Comparison with Non-KB Calls Evidence Call (without KB): 9179f0be-47b7-450b-9181-0b868cbe36d8 { "llmPromptTokens": 164112, // ← Even higher tokens "knowledgeBaseCost": 0, // ← But NO knowledge base cost "total": 0.5217 } Mathematical Breakdown \- Your original query: \~13 tokens \- System prompt: \~889 tokens \- Retrieved chunks: \~10 chunks × \~8,000 tokens each = \~80,000 tokens \- Total context sent to GPT-4o: 84,353 tokens To further verify, request these logs from your call: 1\. KB: Retrieved \[N\] chunks from Knowledgebase 2\. KB: Retrieved chunk (score [X.XXX](http://X.XXX)): \[content\] 3\. Token usage metadata from OpenAI API responses https://cdn.discordapp.com/attachments/1382731118184501389/1384209612332601455/Call_data.json?ex=68519928&is=685047a8&hm=c188330ced8ec82ddda9dfcbe3931a4da48a6c564e75c5662721783956b98b87&

I have re-chunked the data which makes chunk-processing easier. But no matter how many Docs (chunk results) I try to make Trieve send back the price doesn't change, which leads me to believe it's something related to VAPI. Is there any way I can see how what VAPI is sending-receiving from Trieve? Below I'll attach the new results. { "id": "3fe62ca5-de79-49c3-bbb3-6d943c56c980", "orgId": "8a6e8cd0-73ff-4e81-b534-f5185c967670", "input": [ { "role": "user", "content": "pershendetje, a e keni librin What Was I Scared of?" } ], "output": [ { "role": "assistant", "content": "Përshëndetje! Po, librin «What Was I Scared of - Dr Seuss» mund ta gjesh te Dyqani LIBRARIA DUKAGJINI. Çmimi i tij është 10.40 €. Aktualisht ka 9 kopje në stok dhe dërgesa standarde parashihet prej 04 Qershor - 05 Qershor. A ka diçka tjetër që mund të të ndihmoj?" } ], "assistantId": "51245fc5-3969-4b2d-bcb9-62cdb63eb772", "messages": [], "createdAt": "2025-06-16T18:13:49.516Z", "updatedAt": "2025-06-16T18:13:49.516Z", "cost": 0.0135, "costs": [ { "cost": 0.0084625, "type": "model", "model": { "model": "gpt-4o", "provider": "openai" }, "promptTokens": 2989, "completionTokens": 99 }, { "cost": 0.005, "type": "chat" } ] }

Hey, check your DM. I have shared with you some detailed logs which will help you evaluate your decision even better, way more better.