https://vapi.ai/ logo
Join Discord
Powered by
Hi all— I am using a custom-llm with
# general-english
a
Hi all— I am using a custom-llm with streaming responses. Some of my tool calls can take a while to complete, so the agent has the ability to respond earlier with something like "it's taking me a bit longer to fetch that for you, but rest assured I'm working on it!". It does this by sending text-delta chunks in the streaming response and it usually works quite well. However with VAPI I've noticed that it won't actually convert the text to speech until it receives the final text-delta chunk. So what you get is like a 30 second wait and at the very end you hear "it's taking bit longer to fetch that for you. okay here is your answer" all at once. Is there any way to "flush" the current text in the delta-text buffer to be converted to TTS before the streaming response finishes? I know this is possible with LiveKit.io but have not found a way to do it with VAPI. Any help would be greatly appreciated!
@Shubham Bajaj any chance you might have an answer here? Thanks!
s
@alphamacaroon You can flush text for streaming TTS responses. The system already has a built-in mechanism for this using a tag. You can force flush the current text buffer by including a tag in your response. When this tag is detected, it will immediately send the accumulated text to the TTS engine before the full response is complete. So in your case, you could structure your LLM response like:
Copy code
It's taking me a bit longer to fetch that for you, but rest assured I'm working on it!<flush>
[... waiting for tool call response ...]
Okay, here is your answer: ...
This will cause the first part of the message to be converted to speech immediately, while the system waits for the tool call to complete.
a
That is perfect! Thank you! Are there are any other messages that are documented that might be interesting?
@Shubham Bajaj I've tried using this now, and the results seem to be inconsistent. Sometimes it works, but not all the time. Does it need to be on a new line? Directly following the sentence? Any ideas?
Here is an example of my streaming response event 
data: {"object":"chat.completion.chunk","created":1740547492,"choices":[{"index":0,"delta":{"content":"Hang on a sec, I’m looking. <flush>\n","role":"assistant"},"finish_reason":null}]}
s
@alphamacaroon can you share the call ID please.
a
@Shubham Bajaj try this one 
64c65723-2070-4f1c-b1ae-10f47efa2a9d
I wonder if maybe it has to do something with "minCharacters"? Maybe my "wait while I check on that for you" message is too short? Although you would think that flush would override that.
Also, should it be 
<flush/>
or 
<flush>
? Although I tried both and none seemed to work consistently.
s
@alphamacaroon The flush tag regex accepts both and formats.
The chunkPlan.minCharacters (which defaults to 30 characters) determines the minimum length of text that should be accumulated before sending it to the TTS engine.
To improve the consistency of your flush behavior, you can: Set a lower chunkPlan.minCharacters value in your voice configuration. voice: { chunkPlan: { minCharacters: 10, // Lower value for more frequent chunks enabled: true } }
Or ensure your messages before flush tags exceed the minimum character length (default 30 characters)
The reason for having this minimum character requirement is to optimize the quality of speech synthesis - larger chunks generally produce better-sounding speech than very small fragments.
a
okay. Although I did try sending a much larger text block (over 30 chars) followed by a flush and still was not able to get it work consistently. Were you able to take a look at that call by chance?
s
Yes I looked at the call Id and I don't see the model output was received to us. If required I can share the logs on our side.
a
Interesting. Okay I will see if I can get you a few more examples.
PreviousNext
Powered by