The key is to experiment with the transcribe and speech models. I seem to have found good success with deepgram v2 multi for the transcriber and 4o for the LLM. I target English and 2 Indian languages.
I want to eventually get a workflow setup where I can switch models based on the language.