Hi team, I'm working on creatign an evaluation suite for our assistants and frequently running into issues with the turn based evals, "Expected 2 assistant messages but got 1" or varying. Question: Considering agents are non-deterministic, how can I implement an eval that doesn't fail if the agent says 1 extra line or call tools at different times? Eg. Sometimes the agent will call a tool to verify patient immediately upon receiving all patient details, sometimes the agent may say "let me check your details" then call the tool. Are evals only built for deterministic testing? Should we be using test suites?

Hi there, Thank you for your message. Our team is currently out of the office. We operate Monday through Friday, from 9:00 AM to 8:00 PM Pacific Standard Time (PST). We’ll get back to you as soon as possible during our normal business hours. If your message is urgent, please mark it accordingly or include “URGENT” in the subject line, and we’ll do our best to respond promptly. Warm regards, Vapi Customer Support Team

You’re right, strict turn-based evals can fail when the agent responds with extra lines or shifts tool-call timing. I can assist by setting up flexible, outcome-based evaluations that focus on intent, tool-usage correctness, and final state instead of rigid turn counts. This uses pattern matching and test-suite logic to allow natural variation while still enforcing core requirements. What part of the flow matters most for your evaluation of the exact steps or the final behavior? @Jimmy

Hi bobbychick, To address the issues you're encountering with non-deterministic assistant behavior, consider setting up evaluations that focus on flexible, outcome-based criteria rather than strict turn counts. Here's how you can implement this: 1. **Use Pattern Matching**: Employ regex patterns to allow for slight variations in assistant responses. This way, the evaluation can tolerate additional lines or minor changes in phrasing. 2. **Flexible Tool Call Validation**: Instead of exact timing, focus on whether the correct tool was called at any point during the conversation. Use regex validation to ensure the tool name appears in the sequence of tool calls rather than at a specific turn. 3. **Outcome-Based AI Judges**: Use AI-powered judges to evaluate the semantic correctness of the conversation's outcome. This allows you to focus on achieving the correct end state or intent fulfillment, regardless of intermediate variations. 4. **Continue Plan**: Leverage the

continuePlan

feature to define subsequent actions when a test fails at certain points, which allows more holistic evaluation of the assistant's behavior. For more detailed guidance, you can refer to the [Advanced Eval Testing guide](https://docs.vapi.ai/observability/evals-advanced), which provides examples of using regex and AI judges for flexible evaluations. You can also consider creating test suites that encapsulate broader criteria and use test results to refine your evaluation strategy further.