VAPI Custom LLM responses are truncated
# support
j
I have configured a custom llm on VAPI. The issue is that all the messages received are truncated. Have you ever seen this issue? Can you help me? I asked the LLM to give me the definition of the word dinosaur.
Copy code
data: {"result":{"messages":[{"role":"assistant","content":"Un"}],"metadata":""}}

data: {"result":{"messages":[{"role":"assistant","content":" din"}],"metadata":""}}

data: {"result":{"messages":[{"role":"assistant","content":"osa"}],"metadata":""}}

data: {"result":{"messages":[{"role":"assistant","content":"ure"}],"metadata":""}}

data: [DONE]
v
Thanks for sharing the details and the example output. From what you've described, it looks like the responses from your custom LLM are being received in small, segmented chunks rather than as a complete message. Based on the information I have, I don't have documentation or details about this specific issue with VAPI custom LLM integrations, so I can't confirm if this is a known problem or provide a specific solution. If you have any more details about your setup or configuration, feel free to share them!
Thanks for sharing the details and the example output. From what you've described, it looks like the responses from your custom LLM are being received in small, segmented chunks rather than as a complete message. Based on the information I have, I don't have documentation or details about this specific issue with VAPI custom LLM integrations, so I can't confirm if this is a known problem or provide a specific solution. If you have any more details about your setup or configuration, feel free to share them!
p
Sorry, what's truncated? Can you elaborate more on what the issue is?
k
Check token limits, ensure proper aggregation of streamed chunks, and verify your server’s formatting of the response before sending it to Vapi.
j
Thank you for your reply. Vapi's reply in the chat was truncated. Below is VAPI's reply and the reply I should receive.
As you can see, the response is sent by my LLM client. But at the VAPI level, the message is truncated.
To tell you everything, the custom LLM that I use is Anything LLM
k
Modify your LLM client to send full sentences or paragraphs per chunk, not tiny fragments.
j
Ok I will try that thank you
My custom LLM is Anything LLM and I cannot modify it. I tried to chat with open web ui and everything is ok. It seems Vapi cut the stream during the streaming
k
checking if this is resolved/solved for you?