I'm seeing flakiness in the evals feature when following a test driven development workflow. I suspect you all are caching or something thats causing issues. Here is my flow: 1. Create transient eval with transient assistant 2. Eval fails. I verified the assistant output warrants a failure. ✅ 3. Update assistant prompt to fix unwanted behavior 4. Eval still fails. The assistant output message is valid in this case though!!!!! 🔴 You may think that my evaluation criteria is bad or my prompt isnt working as expected but the evidence below should confirm thats not the case. I dont update the eval inputs at all between the runs, only the transient assistant prompt, but again that should be irrelevant given the evidence below. Run 1: With Initial Prompt (Eval Run ID: f6e8e083-4ccf-4db4-8b95-1563888d38a7 2025-11-21T16:32:21.759Z) - Assistant output: "I understand your preference, Pat. However, I will still send a follow-up email with an optional scheduling link for our onboarding team, just in case you change your mind or need assistance in the future. Thank you for your time, and have a great day!" - Evaluation failure message: "The response implies future communication by stating 'I will send a follow-up email...'." Run 2: Updated Prompt, evaluation prompt stays the same (Eval Run ID: 5c1e1f2d-f496-457a-b90b-d8bb6b636ac0 2025-11-21T16:33:05.759Z) - Assistant output: "I understand and respect your decision, Pat. I've noted your preference and won't contact you further. Thank you for your time, and have a great day!" - Evaluation failure message: "The response implies future communication by stating 'I will send a follow-up email...'." 🤔 - It pulled a quote that doesnt exist. I've tried different models, different temperatures, etc. This indicates to me that run 2 is not evaluating the proper message. Theres literally no reference to what its quoting, so it must be referencing some other message or caching the run 1 failure for some reason.

Actionable Next Step: Navigate to your [Vapi dashboard Call Logs](https://dashboard.vapi.ai/) and gather the relevant call IDs and timestamps for both eval runs. Then, report the issue (including your evidence and expected vs. actual behavior) via the recommended support channels. This will help the team diagnose potential caching or evaluation bugs more efficiently. Tip: The [debugging guide](https://docs.vapi.ai/debugging) outlines exactly what info to include for faster resolution. Source: - [Debugging voice agents](https://docs.vapi.ai/debugging)

I've already include all of that detail

Hi Pat, To get started with AI agent testing, I recommend following our [Evals quickstart guide](https://docs.vapi.ai/observability/evals-quickstart). This guide will help you set up automated testing for your AI assistants in just a few minutes. You'll learn how to create mock conversations, define expected behaviors, and validate your agents. Here’s a quick outline of the steps: 1. **Prerequisites**: Ensure you have a Vapi account and API key. 2. **Create an Evaluation**: Set up a test conversation and define the expected responses. 3. **Run the Evaluation**: Execute it against your assistant or squad and review the results via our dashboard. 4. **Validate Results**: Check success or failure indicators, and use detailed transcripts for debugging if needed. For managing evaluations, you can list, update, or delete them as described [here](https://docs.vapi.ai/api-reference/eval/eval-controller-get-paginated). If you're experiencing inconsistent test results, reviewing your evaluation criteria against the expected outcomes can help. For further details, please refer to the full [API documentation](https://docs.vapi.ai/api-reference/eval/create).

Not helpful. I already followed the docs and have everything set up. Its a bug with the evaluation feature, not an issue getting set up.

Hi Pat, Thanks for your patience here. We’ve shipped a few recent updates on our side related to eval execution and result handling, and we wanted to check in. Have you had a chance to rerun your evals since the updates? If so, does it look like the issue you were seeing is resolved, or are you still noticing incorrect judgments? If it’s still happening, we’re happy to dig deeper with specific run IDs. Thanks, Vapi Support

Sorry for the late response here - yes it seems to have been resolve. I appreciate the help

Glad you were able to get this resolved. We’ve closed this ticket, but your input is very valuable to us. Please take a moment to complete the survey, which should appear in a couple of days—your feedback directly helps us improve our support.