Raw reply instead of transcripted message
# support
f
Can I somehow get raw reply? I am using "conversationUpdates" for message feed. For example if I ask the assistant to return me an image it responds with correct code but it reads like this
v
Hi @FNDEVVE, I'm an AI assistant for Vapi. I'll analyze available sources and get back to you shortly.
----------------------------- **While I generate an answer, here are some sources that might be helpful**: * [Pizza Website Example]() * [Get Assistant]() * [Server Events]() * [Update Assistant]() -----------------------------
### Answer Sorry, I wasn't able to find any information about your question. You can get more help at [Contact Us](support@vapi.ai)
Got it. Tagging @User for additional help.
s
Can you provide some examples so that I can understand your problem better?
f
I am using this hook: https://github.com/VapiAI/client-side-example-javascript-next/blob/main/hooks/useVapi.ts with addition of
Copy code
const [conversationUpdates, setConversationUpdates] =
    useState<ConversationUpdate>();
const onMessageUpdate = (message: Message) => {
if (message.type === MessageTypeEnum.ConversationUpdate) {
setConversationUpdates(message);
}
};
This is ConversatioUpdate type:
Copy code
export interface ConversationUpdate extends BaseMessage {
  type: MessageTypeEnum.ConversationUpdate;
  conversation: {
    role: MessageRoleEnum;
    content: string;
  }[];
}
I can't find a way to get assistant response any other way than transcript. For example, if I ask assistant to provide me Markdown image instead of
Copy code
![Alt text](https://squarespace.com/8d3badu3f2.jpg "a title")
I get gibberish transcript like this:
I went through all of the documentation, all of the examples, entire @vapi-ai/web SDK code and I couldn't find a way to get raw assistant response instead of transcript of what Vapi is reading
I really think this should be possible as right now Vapi is usable only for voice chat bots - if it was possible to receive raw assistant response along with transcribed assistant response in conversation update (in Message type) it would be great.
@User
s
Just confirm me one thing do you get the same transcription in the call log section?
f
Yes
second screenshot is from call log section
@Sahil
it read it as 2+2 is 4 but transcript is 2 2 4
s
This issue arises from the transcriber part, and we can't assist with it because we are entirely dependent on the transcriber. For your information, you might argue that you are using the Deepgram API and the quality isn't that bad, and we don't get this type of transcription. However, please try the transcription quality of Deepgram while streaming; it is significantly reduced. Therefore, we can't do anything about it. The only thing I could suggest is for you to try whispering using Talkscriber.
f
This is fairly simple example. I'd like to have some markdown data in my prompt so it returns an image and this will never be possible with any transcriber. My question is: 1. You send my answer to LLM (OpenAI) 2. It returns "2+2=4" 3. You send "2+2=4" to voice provider 4. It returns audio file 5. Audio file gets transcribed to "2 2 4" 6. I get audio file transcribtion All I ask for is to add one more step so it looks like this: 1. You send my answer to LLM (OpenAI) 2. It returns "2+2=4" 3. You send "2+2=4" to voice provider 4. It returns audio file 5. Audio file gets transcribed to "2 2 4" 6. I get audio file transcribtion 7. I get "2+2=4" LLM answer along with audio file transcribtion So it may look like this:
Copy code
export interface ConversationUpdate extends BaseMessage {
  type: MessageTypeEnum.ConversationUpdate;
  conversation: {
    role: MessageRoleEnum;
    content: string; -- "2 2 4"
    llm_answer: string; -- "2+2=4"
  }[];
}
s
Currently, we do not provide the raw data, but it is in our roadmap. When we add it, we will update you through the update channel. For now, you can use Deepgram or another service to obtain data from audio file transcriptions like you already get the audio recording url.
s
Seconding this. It's an absolutely crucial feature for our assistant. We need to be able to display the well-formatted answer choices from the LLM rather than the voice transcription.
s
Currently, no.
f
Please update me once this is implemented. Thank you! @Sahil
f
Thank you! Excited to check this
s
Cool! Does this mean model raw output is now included in ``ConversationUpdate`` in realtime? Or just that the raw model output is available AFTER the conversation?
f
Both are available at the same time @StephenT
It's possible to use this instead of transcript 🙂
And it means that returning Markdown is possible too, truly great update
s
Did you check that out @FNDEVVE ?
f
Yeah it looks really good - exactly what I needed.
s
Awesome.
f
Actually it's not the best. It's not synced up with conversationUpdates at all. Is there a chance this can be improved?
5 Views