https://vapi.ai/ logo
Join Discord
Powered by
codeSwitchingEnabled is not working
# support
r
Hi, we're building a multilingual squad of assistants, and the codeSwitchingEnabled is not working. Our transcriber config is:
Copy code
{
            provider: 'deepgram',
            model: 'nova-2',
            codeSwitchingEnabled: true,
            language: 'fr-CA',
            keywords: ['english', 'anglais'],
        };
Looking at the transcript, it doesn't understand a clearly spoken phrase in English (it doesn't appear in the transcript), and there is no 
language-change-detected
event triggered.
v
message has been deleted
@radu I'm sorry to hear that you're experiencing issues with the code-switching feature. It seems like there might be a configuration issue or a misunderstanding of how the feature works with the current setup. To clarify, the 
codeSwitchingEnabled
parameter is designed to allow the transcriber to recognize when a speaker switches languages within a single utterance. However, it's important to ensure that the transcriber model you're using supports code-switching and that the languages involved are supported for this feature. Given that your configuration is set to use the 'nova-2' model with 'fr-CA' (French Canadian) as the language, and you're looking to detect English phrases, there are a few steps we can take: 1. Verify that the 'nova-2' model supports code-switching between French Canadian and English. 2. Ensure that the keywords provided are appropriate and that the system is trained to recognize those keywords as triggers for language switching. 3. Check if there are any additional settings or parameters required to enable code-switching effectively. If you've already checked these and the issue persists, it might be best to contact support for a more in-depth investigation. They can provide specific guidance and potentially update the system if there's a bug or limitation affecting the code-switching feature. Would you like me to assist you with contacting support for further help on this matter?
You can continue this conversation by mentioning me in the message.
Got it. Tagging @User for additional help.
r
@User Are you sure that it works for other combinations than English and Spanish? In Deepgram's docs, I see a "en & es only" badge: https://developers.deepgram.com/docs/multilingual-code-switching
I also got a response from Deepgram stating that their code switching feature does not support French
s
as of now 
multi
only works for english and spanish only.
Hey @radu To help track down this issue, could you share: - The call ID - When exactly this happened (the timestamp) - What response you expected to get - What response you actually got instead This would really help me figure out what went wrong!
r
Multi, yes, but you suggest to use codeSwitchingEnabled, but that also only works for English and Spanish, not with other languages.
I have a few examples, here's one: Call ID: 14d49ae8-6ff3-49b1-87ef-b37e0e18147f The transcriber is set to fr-CA, but with codeSwitchingEnabled. It doesn't recognize that I say "In English", in English.
I was expecting it to transfer me to the English assistant
The transcriber only understands the language it is defined in, even with codeSwitchingEnabled as true
s
Hey @radu allow me some more to get feedback/response from team on this.
j
We're having the exact same issue. Was this ever resolved? @User
s
@JoachimChauvet can you create a new #1211483291191083018 ticket in the following format and tag me over thee. - The call ID - When exactly this happened (the timestamp) - What response you expected to get - What response you actually got instead This would really help me figure out what went wrong!
r
Did you get feedback from the team?
s
@radu Yes it's working just don't set the language and give it a try. Do let me know how it goes.
Copy code
json
{
    "model": "nova-2",
    "provider": "deepgram",
    "endpointing": 300,
    "codeSwitchingEnabled": true,
}
r
Did Deepgram start supporting French for that as well? Back in November, when I asked this, the confirmed that they didn’t.
s
r
Okay, I’ll give it another try, thank you 😊
Call ID: fe1286f1-ceeb-436e-ac93-ce515a3ed40c
Another example: e2fda4d6-2242-4ce2-a09b-ee853b434f9b
s
@radu Please create a new #1211483291191083018 ticket since this one is already closed. To help track down this issue, could you share in new ticket: - The call ID - When exactly this happened (timestamp) - What response you expected to get - What response you actually received This information would help me investigate what went wrong.
r
But all that information is in this ticket, which wasn’t resolved in the first place…
When spoken in French, it doesn’t seem to understand it is French. Or sometimes it does understand, but after veeeery long silences or repeats
You can see these two cases happen in the two calls I shared. It’s a squad that is composed of 3 assistants: The first one just transfer to one of the other two, according to the language the user wants.
s
@radu can you share all of this information in following format for the feedback and analysis, you know if things doens't work I have to share the expectations. https://discord.com/channels/1211482211119796234/1308120552140964034/1329946868787974217
@radu re-opened your ticket.
r
The setup: - Squad of 3 assistants - Entry assistant is a bilingual assistant with the transcriber you told me to put. Its only job is to transfer the caller to one of the one two assistants (one speaks French, the other speaks English) - If the caller speaks in French, it should transfer to the French one - If the caller speaks in English, it should transfer to the English one Call One (fe1286f1-ceeb-436e-ac93-ce515a3ed40c) - 2025-01-17 22:17:43.743 (UTC) - I expected to say "En Francais" once, and get transferred to the French speaking assistant that says "Bonjour, Bienvenue" - I had to say "En Francais" three times, until the transcriber finally understood that I'm speaking in French. Call Two (e2fda4d6-2242-4ce2-a09b-ee853b434f9b): - 2025-01-17 22:21:14.838 (UTC) - I expected to say "En Francais", and get transferred to the French speaking assistant that says "Bonjour, Bienvenue". - I said "En Francais", and the transcriber seems to have understood it as English, which transferred me to an English speaking assistant.
s
@radu I believe that approach might be too advanced. Code switching capabilities may not be reliable enough for this use case. A simpler solution would be to handle this through prompting. We can instruct the first assistant to repeatedly ask whether the user prefers English or French until they provide a clear response. Once the language preference is confirmed, we can proceed with the appropriate transfer.
r
Thank you for your response, I'm a bit confused, what's the too advanced part of our structure? I thought you were saying that a squad with 3 assistants is, but it seems that you're suggesting the same thing, no?
s
Your expectations for automatic recognition of English or French were not adequately addressed by our model, resulting in its failure. Consequently, I responded that it is currently too advanced for us.
> I thought you were saying that a squad with 3 assistants is, but it seems that you're suggesting the same thing, no? Indeed, with a team of three assistants, this is what I was referring to: you should consistently inquire whether the user prefers English or French until a definitive response is received. Once the language preference is established, you can proceed with the appropriate transfer.
r
This is what that setup from those two calls is doing right now. The first one only determines in which language the caller wishes to speak.
Isn’t that what codeSwitchingEnabled is supposed to do?
s
@radu Indeed, you are correct; however, as previously stated, the codeSwitchingEnabled feature may not be sufficiently reliable for your specific use case, which is both commendable and challenging. Therefore, I recommend continuing with the existing approach. I wish to ensure that you do not expend your valuable time and energy unnecessarily.
r
Okay, thank you
j
@radu I have similar use case - could you use 
codeSwitchingEnabled
in any specific way ? 🤔 How did you manage to do the language transfer in the end ?
r
It only works for English and Spanish. We ended up doing a 3 assistant squad, where the first one asks in which language they want to speak, and then it transfers it to the appropriate assistant. Horrible UX, but it’s the only thing that works. We used a different transcriber for the first assistant.
Nova-3 should fix this, once they release it with French as well
j
Agree with nova-3, looking forward. Did you use gladia as transcriber, or a "non-vapi-native" one ? In my case, 80% of the calls are in a language, hence I would like to plug one of the assistant in one language and only send the user to another one if he detects another language 😬
Quick question: in the end of call report, it gives you the reference of the last assistant it was transferred to in the squad, right ? In my case I couldn't find reference of the initial one 🤔
r
I used Talkscriber
Let’s connect, I also do French / English
PreviousNext
Powered by