Hey, Is there any way to validate the accuracy of speech to test transcriptions?

@Ikya Absolutely! To ensure the accuracy of speech-to-text transcriptions, you can perform several actions: 1. **Manual Review**: Compare the transcribed text against the original audio manually to check for discrepancies. 2. **Automated Testing**: Use automated testing tools that compare the transcribed text with a pre-existing transcript that you know is accurate. 3. **Quality Metrics**: Implement quality metrics such as Word Error Rate (WER) or Character Error Rate (CER) to quantitatively assess the transcription accuracy. 4. **User Feedback**: Collect feedback from users who interact with the transcriptions to identify areas of improvement. If you're working with a specific API or service for transcription, they might also provide their own tools or guidelines for validation. Would you like to know more about any of these methods?

@Ikya can you elaborate more around your use-case and purpose if possible do share an example of what you want to achieve.

Hi @Shubham Bajaj, I'm using Vapi with custom LLM and testing it. For transcribing I'm using deepgram Nova 2 model where I observed that the Word error rate (WER) is very high. Almost 30% of the times it is misinterpreting what I'm saying. So I want to do an accuracy testing for this deepgram Nova 2 model. Can we do that automatically without manual intervention in Vapi? Something like where we'll hitting the model with predefined audios and comparing the results with expected results. Can we do that?

@Ikya as of now you cannot do your own analysis but yeah I can help on WER or misinterpreting part. could you share: - The call ID - When exactly this happened (the timestamp) - What response you expected to get - What response you actually got instead This would really help me figure out what went wrong!