Hi, I am using Vapi Evals and trying to test "no response" scenario on my assistant. My assistant is instructed to stay silent in certain scenarios via the system prompt:

Staying Silent:
   - Respond with "" (empty string) to stay silent.
   - Only speak when the automated system asks for information or when leaving a voicemail.

But both exact match (with empty string) and AI judge result in AI failure of: "Chat Messages did not return any outputs." The eval runner treats no output as an error and never reaches the judge plan. There's currently no way to assert that the assistant correctly stayed silent. Could we get support for evaluating empty/no-response assistant turns? Example run id -

6a09bfc4-d77e-46a3-9cc1-66772d6733bb

Hi there, Thank you for your message. Our team is currently out of the office. We operate Monday through Friday, from 9:00 AM to 8:00 PM Pacific Standard Time (PST). We’ll get back to you as soon as possible during our normal business hours. If your message is urgent, please mark it accordingly or include “URGENT” in the subject line, and we’ll do our best to respond promptly. Warm regards, Vapi Customer Support Team

@User Are there any update on this please?

looking into it

This is currently an Evals limitation; an empty string is treated as an empty output, and the Eval throws an error. It’s unlikely this will be patched soon, since this behavior is intended to handle invalid responses. However, I’ll check with the team to see what can be done. As a workaround for now, instead of returning an empty string (

""

), return a tag like

<silent>

as the output. You can enforce this using

formatPlan

in TTS so it won’t be sent to TTS, effectively enforcing silence. It can also help in the logs to clearly verify that the model decided to stay silent.

Okay that sounds reasonable, thank you!