Hi VAPI Support Team, I’m seeing repeated agent eval failures even though the agent behavior appears to meet the rubric. Over half of evals are failing, and I’m trying to determine whether this is an agent config issue or an eval/tool parsing issue. Expected behavior (and what the agent is doing): - Rephrases slang-heavy input into plain English - Extracts required fields: - caller_type - full_name - charges (array with ≥1 entry) - facility (when available) - Confirms once with a short confirmation (e.g., “So I have X, Y, Z — correct?”) - Neutral, non-patronizing tone - Tool payload uses a charges array as required What’s happening instead: - Evals fail despite the above - Webhook logs consistently show: - message.toolWithToolCallList present - message.toolCallList resolving to null - This raises questions about whether evals are reading the correct field or missing valid tool calls. Questions: 1) Which field do evals use as authoritative? - message.toolCallList - message.toolWithToolCallList - final tool payload only? 2) Are evals sensitive to exact confirmation phrasing? 3) Is there a known issue where tool calls work at runtime but are not recognized by evals? 4) Can you confirm the expected JSON shape for tool calls that evals reliably detect? I can provide specific call IDs, timestamps, and recordings if needed. Test Suites: (id: 0f59a88c-07bb-460e-a57f-83a3f42a2446) (id: f0954df7-b639-4fda-bca5-51567099644b) (id: 8054bd64-22d7-4bc2-94cd-53222ca84d46) Thanks, Isaiah

Hi there, Thank you for your message. Our team is currently out of the office. We operate Monday through Friday, from 9:00 AM to 8:00 PM Pacific Standard Time (PST). We’ll get back to you as soon as possible during our normal business hours. If your message is urgent, please mark it accordingly or include “URGENT” in the subject line, and we’ll do our best to respond promptly. Warm regards, Vapi Customer Support Team

To ensure evals reliably detect your tool calls, use the following JSON response format for your webhook:

json
{
  "results": [
    {
      "toolCallId": "call_123",
      "result": "Your response as a single-line string"
    }
  ]
}

Make sure the

toolCallId

exactly matches the one from the request, and always include the

results

array structure. This format is required for successful tool call validation in evals. For more details, see the [official documentation](https://docs.vapi.ai/tools/custom-tools-troubleshooting). Source: - [Custom tools troubleshooting](https://docs.vapi.ai/tools/custom-tools-troubleshooting)

This feels more like an eval/runtime mismatch than an agent issue. If toolWithToolCallList is present but toolCallList is null, evals may be reading a different field than what executes at runtime. It’s worth confirming whether evals require the final resolved tool payload versus intermediate tool calls, and whether confirmation is pattern-based or semantic. Quick question: do evals run on per-turn messages or only the final assistant turn with the resolved tool output? @Parrasite9

Thanks, that makes sense. I added detailed logs to capture both toolCallList and toolWithToolCallList at both the root and inside message, and here's what I’m seeing: 1. During evals, toolCallList (message) does contain resolved tool calls (with ID, function name, arguments, etc.) 2. toolWithToolCallList is also present, but it's schema metadata + config, not runtime payload 3. At runtime, the tool executes as expected, and logs confirm the webhook receives complete arguments (with correct shape) 4. So tool execution itself is stable, only some evals fail intermittently Follow-up question: Are evals using toolWithToolCallList rather than the final toolCall payload to validate schema/payload correctness? Also do they validate each assistant turn (including pre-tool call) or just the final resolved turn after tool results? Let me know if I can help test or isolate this further. I have a consistent test case where evals pass and fail despite identical tool outputs.

Thanks for the detailed breakdown — it clearly highlights the eval/runtime discrepancy. From what you’ve shared, it seems the issue is less about tool execution and more about how evals interpret toolCallList versus toolWithToolCallList and which assistant turns they consider. I can help you dig deeper with a structured approach: reviewing eval logic, verifying field usage, and mapping out per-turn validation to isolate why intermittent failures occur. If you’re looking for hands-on support, I can assist directly to streamline testing, identify root causes, and suggest a methodology to ensure eval reliability. Do you want me to set up a session to go through your test case step by step? @Parrasite9

Yes that would be perfect, when is your availablility?

Perfect. I’m available today and happy to go through this in more detail. We can discuss it privately (D.M) , review the eval behavior, logs, and test cases, and pinpoint where the inconsistency is coming from. Let me know a time that works for you. @Parrasite9

no it restricted your account because you got reported too many times for scamming

Bro i just want VAPI to come in here and do their job. Where are these people?

It's been over 2 weeks since this post and nearly 3 since the email was sent out. Like honestly this is ridiculous, if im not going to get assistance just let me know so i stop monitoring this ticket. Id rather move to another platform where I am actually going to get responsive support.

for non enterprise customers you need be a little more proactive. Arthur held office hours yesterday where you could've joined the call and got instant help. Same with kyle which I pinged you with on the 31st. Vapi is focused more towards enterprise clients so if you want support id recommend checking everyday at office hours times to see if theres anyone there.

I did join office hours, i spoke with Kyle or whoever, and I was told I would be provided information IN THIS CHAT, and I never received it. Honestly, it's alright i'll just move platforms. I dont care yall are "focused" on other customers. You and your team have time to respond to "scammers." Hell, you even responded to this thread 3 times and instead of providing any direction to my solution, you've decided to kick the can down the road. If sending emails, joining the stage, and sending TWO tickets in this discord isnt "proactive" enough, then idk what is. Im not going to beg you guys for a solution/guidance, yall either want my money or you dont. Again, 2 weeks of nothing is ridiculous.

im not apart of Vapi lol

Yea at this point im conviced Vapi doesnt actually offer Support.

Hey I apologize for the delay. I looked into it and here's what I found. I reviewed the test suite in detail (7 passed / 10 failed), and the failures don’t point to an eval parsing bug. They break down cleanly into agent behavior and test setup issues. 1) Which field do evals treat as authoritative? Evals are AI-scored from the transcript, not from

toolCallList

or

toolWithToolCallList

. The scorer reads the full transcript (including

tool_calls:

and

tool_call_results:

text) and grades against the rubric. Webhook fields are not what evals rely on. 2) Are evals sensitive to exact confirmation phrasing? Not rigidly. Multiple phrasings pass. Failures occur when the agent confirms but never follows through with a tool call, not because of wording. 3) Is there a case where tool calls work at runtime but evals miss them? Not based on this data. When a tool call appears in the transcript, evals detect it correctly. Failures fall into: * No tool call made (conversation ended early) * Tool call made, but rubric logic was violated (wrong scenario, missing clarification, missing optional fields) 4) What JSON shape is reliably detected? The standard transcript-embedded format works consistently:

AI: tool_calls: {"name":"BB_create_intake","arguments":"{...}"}
AI: tool_call_results: {"ok":true,"intake_id":"..."}

This is primarily an agent behavior / flow control issue, not an eval or parsing issue. Fixing early exits, non-bail handling, ambiguity clarification, and optional field capture should eliminate most failures.