I'm noticing that none of my evals where the assistant is given a system prompt work. The eval's purpose is to simply check if the assistant knows the persona it is assigned for the duration of the call (e.g. Alexandra Rios, Vp of sales...). If I attach an assistant with a system role prompt that includes the persona definition the, eval always fails. If I attach an assistant with an assistant role prompt that includes the persona definition, the eval always works. Failing (system prompt) - evalId: d83e3c35-dffe-4f45-acac-afea204d0206 - run id: e2725870-48f1-4a27-9abf-d7e9ac9d3c92 working (assistant prompts) - evalId: d77ba47c-8ae2-4b65-81f6-70a6a2e38c0e - run id: c4bbeae4-1b50-450d-a125-6ab34c1e3579 Note: - I am passing transient assistant configurations to the eval at runtime through the /[eva/run api](https://api.vapi.ai/eval/run) as the target assistant Am I doing something wrong when attaching the transient assistant programmatically? https://cdn.discordapp.com/attachments/1460270828531880030/1460270828699779188/image.png?ex=697a1530&is=6978c3b0&hm=270e96e8bd84aaab43f6e09f85acf3e1bd82c7fb0bd0468c4d0c6d5164dcde6f&

Hi — thanks for the IDs, that’s helpful. You’re likely not doing anything wrong with “system vs assistant” roles. Vapi does support

role: "system"

in an assistant’s model messages. (https://docs.vapi.ai/tools/default-tools?utm_source=chatgpt.com) What’s happening is usually this: Why it “fails” with a system prompt A system prompt controls behavior, but it does not guarantee the assistant will say the persona details (e.g., “I’m Alexandra Rios…”) unless: • the conversation asks for it, or • the system prompt explicitly instructs the assistant to state it, or • you set a firstMessage that includes it. When you put the persona definition as an assistant-role message, you’re effectively “pre-loading” the transcript/history with persona text, so the eval/judge has something obvious to match—hence it passes. What to change Pick one (depending on what your eval asserts): 1. Make the eval ask for persona Add an early user turn like: “Who are you? State your name and title.” This makes the persona appear in the assistant output naturally. 2. Force it into the first spoken message Set

firstMessage

(or instruct in system prompt: “In your first message, introduce yourself as Alexandra Rios, VP of Sales.”). This makes the persona reliably observable in the eval output. 3. If you only care that the assistant behaves as the persona Adjust the eval’s expected checks so it doesn’t require the assistant to explicitly say the persona name/title unless prompted. Transient assistant in

/eval/run

It is supported to run evals with a transient assistant object (inline config) instead of an ID. (https://docs.vapi.ai/observability/evals-quickstart?utm_source=chatgpt.com) Just ensure you’re putting the system message under the transient target assistant’s

model.messages

(same shape as standard assistant configs).

Thanks for the write-up Kartik. Option 3 makes most sense, but I am not understanding why I can't directly check if the llm knows the assigned persona during the conversation turn. This reflects the true nature of calls that my customers will have. We expect the llm to know it's pre-defined persona/role at any given turn we are already using transient assistants. For context, I am attempting to write/test evals based on a bug seen last week: https://discord.com/channels/1211482211119796234/1458157871576649758

Please share any call id so that i can look into it and see if we can do anything to solve it. Regards, Vapi Support

I've already shared eval run ids: Failing (system prompt) evalId: d83e3c35-dffe-4f45-acac-afea204d0206 run id: e2725870-48f1-4a27-9abf-d7e9ac9d3c92 working (assistant prompts) evalId: d77ba47c-8ae2-4b65-81f6-70a6a2e38c0e run id: c4bbeae4-1b50-450d-a125-6ab34c1e3579

Yes sir looking into it

@User Any update? It's been more than a week now

Hi there, I apologize for the late response sir, We’ve reviewed the Test Suite and identified the issue. The assistant is not responding because the **identity and evaluation logic in the Test Suite are misconfigured**: • The system prompt references the assistant generically as “AI” instead of clearly defining who it is. • The “who are you” test is vague and does not explicitly instruct the assistant’s role or persona. • The pass and fail criteria are identical, so the test will fail every time regardless of behavior. How to fix this: 1. Explicitly define the assistant’s persona in the system prompt (e.g., “You are Alexandra, a VP of Sales…”, including tone and goals). 2. Make the “who are you” expectation clear and unambiguous. 3. Update the rubric so the fail criteria is the opposite of the pass criteria. After these changes, the assistant should respond correctly in Test Suites. Regards, Vapi Support