Can I somehow get raw reply? I am using "conversationUpdates" for message feed. For example if I ask the assistant to return me an image it responds with correct code but it reads like this

Hi @FNDEVVE, I'm an AI assistant for Vapi. I'll analyze available sources and get back to you shortly.

Can you provide some examples so that I can understand your problem better?

I am using this hook: https://github.com/VapiAI/client-side-example-javascript-next/blob/main/hooks/useVapi.ts with addition of

const [conversationUpdates, setConversationUpdates] =
    useState<ConversationUpdate>();
const onMessageUpdate = (message: Message) => {
if (message.type === MessageTypeEnum.ConversationUpdate) {
setConversationUpdates(message);
}
};

This is ConversatioUpdate type:

export interface ConversationUpdate extends BaseMessage {
  type: MessageTypeEnum.ConversationUpdate;
  conversation: {
    role: MessageRoleEnum;
    content: string;
  }[];
}

I can't find a way to get assistant response any other way than transcript. For example, if I ask assistant to provide me Markdown image instead of

![Alt text](https://squarespace.com/8d3badu3f2.jpg "a title")

I get gibberish transcript like this:

Just confirm me one thing do you get the same transcription in the call log section?

Yes

This issue arises from the transcriber part, and we can't assist with it because we are entirely dependent on the transcriber. For your information, you might argue that you are using the Deepgram API and the quality isn't that bad, and we don't get this type of transcription. However, please try the transcription quality of Deepgram while streaming; it is significantly reduced. Therefore, we can't do anything about it. The only thing I could suggest is for you to try whispering using Talkscriber.

This is fairly simple example. I'd like to have some markdown data in my prompt so it returns an image and this will never be possible with any transcriber. My question is: 1. You send my answer to LLM (OpenAI) 2. It returns "2+2=4" 3. You send "2+2=4" to voice provider 4. It returns audio file 5. Audio file gets transcribed to "2 2 4" 6. I get audio file transcribtion All I ask for is to add one more step so it looks like this: 1. You send my answer to LLM (OpenAI) 2. It returns "2+2=4" 3. You send "2+2=4" to voice provider 4. It returns audio file 5. Audio file gets transcribed to "2 2 4" 6. I get audio file transcribtion 7. I get "2+2=4" LLM answer along with audio file transcribtion So it may look like this:

export interface ConversationUpdate extends BaseMessage {
  type: MessageTypeEnum.ConversationUpdate;
  conversation: {
    role: MessageRoleEnum;
    content: string; -- "2 2 4"
    llm_answer: string; -- "2+2=4"
  }[];
}

Currently, we do not provide the raw data, but it is in our roadmap. When we add it, we will update you through the update channel. For now, you can use Deepgram or another service to obtain data from audio file transcriptions like you already get the audio recording url.

Seconding this. It's an absolutely crucial feature for our assistant. We need to be able to display the well-formatted answer choices from the LLM rather than the voice transcription.

Currently, no.

Please update me once this is implemented. Thank you! @Sahil

It is implemented you can get the model raw output using this server message. https://cdn.discordapp.com/attachments/1233575554087653376/1242447853771690084/image.png?ex=664ddf3b&is=664c8dbb&hm=1b5bc5b59d5e849925ba86ffe53b35ddac3c0cae51b322027d88ffa44c62da16&

Thank you! Excited to check this

Cool! Does this mean model raw output is now included in ``ConversationUpdate`` in realtime? Or just that the raw model output is available AFTER the conversation?

Both are available at the same time @StephenT

Did you check that out @FNDEVVE ?

Yeah it looks really good - exactly what I needed.

Awesome.

Actually it's not the best. It's not synced up with conversationUpdates at all. Is there a chance this can be improved?