Hello, We are using a custom LLM solution, but we are experiencing some issues when handling multiple transcript fragments. If the customer pauses in the middle of a sentence or speaks a very long sentence, we receive several POST requests to the custom LLM endpoint. Do you have any recommendations on how to handle this situation properly? Is our solution poorly designed, and should we consider switching to a different type of architecture? Thank you.

To handle multiple transcript fragments efficiently, implement a debouncing buffer that accumulates fragments and only sends a request to your LLM when a sentence-ending punctuation or a period of silence is detected. For example, you can use a timer that resets with each new fragment and emits the full transcript after a set delay:

js
let buffer = "";
let debounceTimer;
const DEBOUNCE_MS = 1000; // 1 second

function onTranscriptFragment(fragment) {
  buffer += fragment;
  clearTimeout(debounceTimer);
  if (/[.!?]$/.test(fragment.trim())) {
    sendToLLM(buffer);
    buffer = "";
  } else {
    debounceTimer = setTimeout(() => {
      sendToLLM(buffer);
      buffer = "";
    }, DEBOUNCE_MS);
  }
}

This approach reduces unnecessary LLM calls and ensures more complete context is sent each time. [See Vapi's custom transcriber debouncing logic for reference.](https://docs.vapi.ai/customization/custom-transcriber) Source: - [Custom transcriber – Vapi Documentation](https://docs.vapi.ai/customization/custom-transcriber)

So, do I also need to build my own transcriber in order to use the custom LLM efficiently? We use the custom LLM to apply our internal business logic, which is complex and difficult to implement in VAPI.

Implement a buffering layer that collects transcript fragments per session, applies debouncing like, 500 ms after last input, detects sentence boundaries via punctuation or silence, and sends only finalized, coherent utterances to the custom LLM. Ensuring context continuity and reducing fragmented POST requests..

So, can you confirm that I need to develop a custom transcriber on my own because the one provided by Vapi is not sufficient and doesn't allow me to do this?

Yes, you need to build a custom transcriber, as Vapi’s built-in one doesn’t support debouncing, buffering, or finalized utterance handling.. https://docs.vapi.ai/customization/custom-transcriber

checking if this is resolved/solved for you?

Hello, We have some questions about this type of architecture. If we decide to implement a custom transcriber, which Vapi features would we lose? For example, would functionalities like interruption control, noise filters, or similar features still work?

If you use a custom transcriber with Vapi, these orchestration features will still work as long as your transcriber provides timely, speaker-tagged transcripts..

Ok thank you