<flush /> tag only works once
# supportd
dudleykong
04/01/2025, 5:06 AMOur application with Vapi uses a custom LLM. We have an agentic framework setup on this custom LLM where occasionally two separate AI messages may make up one response in the conversation. Here is an example:
User: I'm doing well
AI Message: I'm glad to hear you are doing well.
AI Message: What do you have planned for the day?
We use the tag to force TTS from the AI Messages (otherwise, the entire AI Message has to stream before TTS begins). This works well, but it only works for one of the AI Messages. For example:
User: I'm doing well
AI Message: I'm glad to hear you are doing well. (this will begin speaking as soon at it receives the correct amount of characters)
AI Message: What do you have planned for the day? (this will not begin speaking until the entire AI Message has streamed. So, it will not speak until day? has been received.)
The ideal behavior for us is for the first AI message to be sent to TTS as soon as possible and for the second AI Message to be sent to TTS as soon as possible.
Some short call examples:
f33e1d61-2b14-458c-ac8f-397563b29e9b
367ad1ef-ec1d-4e49-971d-84ea3c4909ca
94c38384-d869-489e-8f75-9367143eebd8
Is it possible to make the tag work for both AI Messages?
v
Vapi Support Bot
04/01/2025, 5:07 AMHi there! 👋
I understand you're experiencing an issue where the
<flush /><flush /><flush />Vapi Support Bot
04/01/2025, 5:07 AM4. **Tailor prompts for Vapi**: Since "Transitioning from one platform to another can sometimes require adjustments to optimize performance," you might need to "tailor prompts specifically for VAPI to take full advantage of its unique features" ([Discord thread](https://discord.com/channels/1211482211119796234/1211483291191083018/threads/1336989548357353524)).
Is there anything else you'd like to know about working with the
<flush />p
Praveen Nair
04/01/2025, 7:56 PMlooking into it.
Praveen Nair
04/01/2025, 8:16 PMLooking at your log data, I can see some inconsistencies in how the flush tags are being processed:
\### Working Example:
\`\`\`
🔵 03:24:52:412 \[user LOG\] Model output: raised in Virginia must mean you are very familiar with hot, <flush /> humid summers and maybe
🔵 03:24:52:413 Voice Input Formatted: "Being raised in Virginia must mean you are very familiar with hot, ", Original: "Being raised in Virginia must mean you are very familiar with hot, "
\`\`\`
In this case, the flush tag is properly processed - the content before the tag is sent to TTS immediately.
\### Non-Working Examples:
\`\`\`
🔵 03:24:28:006 Voice Input Formatted: " I'm glad you're doing well. It's always nice to have a good day. Tell me about your hometown. ", Original: "<flush /> I'm glad you're doing well. <flush /> It's always nice to have a good day. <flush /> Tell me about your hometown. <flush />"
\`\`\`
Here, the flush tags in the original output are not being properly processed in the formatted output. The text is being combined.
Praveen Nair
04/01/2025, 8:17 PM\## The Problem
Based on the logs, I can identify two key issues:
1\. **Flush Tags at Sentence Boundaries**: When flush tags are placed between complete sentences, they're being ignored during formatting.
2\. **Token-by-Token Processing**: Looking at the logs, your model is outputting tokens incrementally, but flush tags are only being properly processed when they appear in the middle of a continuous output stream.
\## How to Fix It
To ensure flush tags work properly in your model output, you should:
1\. **Ensure proper placement of flush tags**:
- Place flush tags within the same token/chunk of text that contains the content you want to flush
- Avoid placing flush tags at the beginning of a new token/chunk
2\. **Modify your model output format**:
Instead of:
\`\`\`
<flush /> I'm glad you're doing well. <flush /> It's always nice to have a good day.
\`\`\`
Format it as:
\`\`\`
I'm glad you're doing well.<flush> It's always nice to have a good day.
\`\`\`
Notice the placement of the flush tag immediately after the content to be flushed, without spaces between the content and the tag.
3\. **Batch output properly**:
Ensure that each complete thought and its associated flush tag are sent together in the same output chunk, rather than sending tokens individually.
Praveen Nair
04/01/2025, 8:17 PM\## Example Implementation
For your specific case, format model output like:
\`\`\`
I'm glad you're doing well.<flush />
It's always nice to have a good day.<flush />
Tell me about your hometown.<flush />
\`\`\`
This ensures each sentence is immediately sent to TTS as soon as it's received, creating a more responsive conversation flow.
\## Technical Explanation
The issue relates to how the VAPI
ttsBufferd
dudleykong
04/01/2025, 8:32 PMThanks @Shubham Bajaj I'll implement your fix and get back to you.
2 Views