Deepgram supports language detection

"https://api.deepgram.com/v1/listen?detect_language=true&model=nova-3"

Could be

Auto

option in language selection dropdown. I'd imagine its just parameter throughput to deepgram, not much development effort. https://cdn.discordapp.com/attachments/1346132050989809724/1346132051123765300/image.png?ex=67c712aa&is=67c5c12a&hm=fcf1bfd4f4cf5c91944c309d3e65a94f8ac5dd980441322f1cf1c0326f64e546&

It looks like you're exploring the automatic language detection feature with Deepgram's Nova 3 model! Just a heads up, while you can enable language detection by using the

detect_language=true

parameter, the Nova 3 model has some limitations. It primarily supports English ('en', 'en-US') and does not support the

codeSwitchingEnabled

feature, which is available in the Nova 2 model. If you're seeing a warning message about certain features being accessible only in specific languages, it’s likely due to these limitations. If you need multilingual support or code-switching capabilities, you might want to consider using the Nova 2 model instead. Let me know if you have any more questions or need further assistance! 😊 Sources: - [Report on Nova 3 Model with Deepgram's Language Detection](https://discord.com/channels/1211482211119796234/1211483291191083018/1341463939246198784) - [Detailed Report on Deepgram's Automatic Language Detection Feature](https://discord.com/channels/1211482211119796234/1211483291191083018/1308120552140964034)

Or nova 2

Seconded, this would be really helpful

Agree

@Floris @thenotoriousllama I have created feature request, please upvote https://vapi.canny.io/feature-requests/p/please-add-auto-for-deepgram-nova-transcriber-language-selector

Nice, upvoted

Hey @Pigeon it's already available you can use

codeSwitchingEnabled

which auto detects the language and updates it during the call.

Hi @Shubham Bajaj Would that work for Nova 3 or for Nova 2 in Dutch for example?

To detect language changes, Vapi uses a custom trained model. Languages supported (X = limited support): 1\. Arabic 2\. Bengali 3\. Cantonese 4\. Chinese 5\. Chinese Simplified (X) 6\. Chinese Traditional (X) 7\. English 8\. Farsi (X) 9\. French 10\. German 11\. Haitian Creole (X) 12\. Hindi 13\. Italian 14\. Japanese 15\. Korean 16\. Portuguese 17\. Russian 18\. Spanish 19\. Thai 20\. Urdu 21\. Vietnamese

Where would you need to specify that codeswitchingEnabled flag?

@Floris I guess you need to create assistants programatically or use overrides:

{
  "transcriber": {
    "model": "nova-2",
    "language": "en",
    "provider": "deepgram",
    "codeSwitchingEnabled": true
  }
}

So setting that boolean should be enough to switch transcriber language automatically mid call?

Yes, deepgram itself should switch language, I guess you can get event for your code

Interesting!

Yes, I'm not sure why you'd need that

Maybe you want to use system prompt in different language

I reckon this will only work for Nova 2, right

Most probably, Nova 3 is english only atm

What? How? Custom?

and bad english

Indeed

Well, if you want life like speed, maybe not yet good enough

I just tried the language switching with nova 2. It's really bad

Same, I was trying that with Italian, sometimes it correctly switch to Italian and continue the conversation, but usually it's not able to transcribe because it's expecting english I think.

Same what I experienced

@Floris @giorgior27 @Pigeon checking if all doubts/questions are resolved?

What do you suggest to implement multi-lang? Since codeswitching seems not reliable. Is squad the best option?

Like giorgio27 said, the codeswitching doesn't seem reliable. The transcriber doesn't seem to work when talking in another language when using nova 2 and the codeswitching enabled @Shubham Bajaj

@Floris From what I got I think your best option would be to have a squad working. I am thinking to create a squad with an assistant that can be updated before transfering, so with just one assistant you can have virtually every language. Only issue is that the main one need to ask in english the preferred language

@giorgior27 You wont need squads than? Only a tool call that get's executed when the users want's to talk in a different language which updates the live call?

Can you update assistants while talking? I tried but it does not seem possible. I tried to update the voice that is the most clear thing to see, I changed from male to female. But the call remained male, then after closing and starting a new one, it had a female voice. But maybe I did it wrong. Do you know if you can update assistant while they are talking? We could update transcriber language.

How did you update the voice?

Using api https://docs.vapi.ai/api-reference/assistants/update

I think you'd need to update the call

Did you do that with success? Updating an assistant mid call? Becaue the update call seems accepting only one parameter "name". So it seems we cannot do much with it?

@giorgior27 To help track down this issue, could you share: - The call ID for codeSwitchingEnabled not working - When exactly this happened (the timestamp) - What response you expected to get - What response you actually got instead This would really help us figure out what went wrong!