Gladia Code switching option doesn't work
# support
c
Gladia multilingual code switching isnt working, it keeps detecting english Possible reason: Gladia requires 'code_switching_config' to be set for multi lingual code switching to work, but the option isnt available in vapi Vapi and gladia docs ss attached https://cdn.discordapp.com/attachments/1307681839191691306/1307681839535620096/image.png?ex=673b311e&is=6739df9e&hm=a339f0f839c8fc950e14fbf2eef8d056f1d9a969694e23d80907b7d1d945e5f1& https://cdn.discordapp.com/attachments/1307681839191691306/1307681840118632458/image.png?ex=673b311e&is=6739df9e&hm=c1215d4f3510d8e56fc1f3e0a3d763ac946ec74e6658eeeff40946d744257c27&
v
@clumsy.ninja Thank you for reaching out with your concern about multilingual code switching with Gladia. I understand that you're experiencing issues with it only detecting English and not switching between languages as expected. Since you've mentioned that Gladia requires a 'code_switching_config' to be set, and you're not finding this option in VAPI, it's possible that there might be a discrepancy between the two systems or a missing configuration step. To assist you better, I would need to review the screenshots of the VAPI and Gladia documentation you've mentioned. However, as a support bot, I'm unable to view attachments directly. Could you please provide me with more details from the documentation or guide me through the configuration steps you've taken so far? This will help me understand the issue better and provide you with the appropriate guidance. If the documentation indicates that a specific configuration is required for Gladia that is not available in VAPI, it might be necessary to update VAPI's configuration or reach out to the support team for further assistance. Please provide more details, and I'll do my best to help you resolve this issue.
You can continue this conversation by mentioning me in the message.
Got it. Tagging @User for additional help.
c
@User @Shubham Bajaj
s
Hey @clumsy.ninja your ticket was marked as looking because of It I couldn't get the notification about your ticket.
Regarding @clumsy.ninja both transcribers multilingual settings are enabled and available but the API properties are hidden because its been reported they are not stable yet. Does this helps?
c
whats the workaround right now then @Shubham Bajaj
i need to use multilingual code switching
your deepgram with vapi code switching only supports few languages out of which even fewer are actually working (only spanish french working )
s
Hey @clumsy.ninja allow me sometime to see what can be done for you.
c
Muchhh appreciated pls get back to me quick thanks
@Sid
@Shubham Bajaj any update?
after further testing i believe vapi just isnt setting the 'language behaviour' of gladia properly Its setting it to manual (when interacting with gladia) regardless of if the config is set to automatic multiple languages https://cdn.discordapp.com/attachments/1307681839191691306/1308827694384943185/image.png?ex=673f5c47&is=673e0ac7&hm=fa448ae400eddb588a22b00c4ff8d91ee48959d1f6107417aff3fba5b485ebd8&
I know this because the behavior its exhibiting (translating to primary language instead of transcribing in the spoken language) only occurs in gladia when language behavior is set to manual
Relevant call ids: (trying to speak in arabic but isnt recognized) 9b1db3b3-55e7-4da6-a6af-01b875c3c206 72ef842d-b011-4709-9637-5208ff33a9ce
@Shubham Bajaj @User
h
@Shubham Bajaj @User Ye i am also facing this issue, it's setting to manual, not performing at all like the "automatic multiple languages" language behaviour on gladia's own site
s
@Kira
s
@clumsy.ninja here's what we are sending to the #gladia šŸ”µ 15:54:51:081 Initializing Gladia. Options: { "x_gladia_key": "", "encoding": "WAV/PCM", "bit_depth": 16, "sample_rate": 44100, "language_behaviour": "automatic multiple languages", "language": "english", "transcription_hint": "", "endpointing": 100, "model_type": "fast", "frames_format": "base64", "prosody": true, "audio_enhancer": false, "word_timestamps": true } Just for transparency sharing this as well how we set and use fallback for the gladia config. const options: GladiaConfig = { x_gladia_key: credential?.apiKey ?? process.env.GLADIA_API_KEY ?? '', encoding: 'WAV/PCM', bit_depth: 16, sample_rate: sampleRate as 8000 | 16000 | 32000 | 44100 | 48000, language_behaviour: config.languageBehaviour ?? 'manual', language: 'english', transcription_hint: config.transcriptionHint, endpointing: 100, model_type: config.model ?? ('fast' as 'accurate' | 'fast'), frames_format: 'base64', prosody: config.prosody ?? false, audio_enhancer: config.audioEnhancer ?? false, word_timestamps: true, }
@clumsy.ninja / @HystericalBat TLDR: mutli lingual transcriber is not working as per expectation's As of now the multi lingual feature by these transcribers isn't working as expected, Gladia is bugging whereas Talkscriber is currently not focusing on mutli-lingual, sadly codeSwitchingEnabled with deepgram isn't fine to use it for production. Instead you have to use language based assistant based on the call origination at telephony level, depending upon telephony used. I am really sorry guys for late response and this feature isn't feasible as per your expectations. Do let me know how can I help you guys further with language based assistants or even anything else.
c
Sorry, just to be clear, multilingual support is non-functional currently? I've been trying to get Deepgram nova-2 to auto-switch to Spanish with no luck. Is this support coming in the future?
Nevermind, more digging showed me I needed codeSwitchingEnabled: true,
s
@clumsy.ninja / @HystericalBat / @Sid @ccmenshealth the issue is from gladia side, asked their team for help let's wait for their response.
h
@Shubham Bajaj can u check my ticket , it's working for me for some reason
s
@HystericalBat can you tag me to your ticket again.