Success Evaluation didn't work as expected
# supporti
Irfan
02/17/2025, 6:17 PMthe success evaluation for this call: 5a059171-eec5-43e6-a3d2-a3a79223c31b
didn't work as expected, it was supposed to be true but was tagged as false
The call ID - 5a059171-eec5-43e6-a3d2-a3a79223c31b
When exactly this happened (the timestamp) - call success evaluation (post call)
What response you expected to get - true
What response you actually got instead - false
This would really help me figure out what went wrong!
v
Vapi Ticket Bot
02/17/2025, 10:03 PM✅ Ticket status updated to solved by Vapi Ticket Bot#0124
v
Vapi
02/18/2025, 7:05 PM1\. Looking at the success evaluation prompt (from the Analysis Plan), the criteria for a "true" evaluation are:
\`\`\`
You are an expert call evaluator. You will be given a transcript of a call and the system prompt of the AI participant. Determine if the call was successful based on the objectives inferred from the system prompt. DO NOT return anything except the result.
\`\`\`
2\. The evaluation system is designed to check if:
- The call was properly ended
- At least 50% of questions from the system prompt were answered
3\. In the transcript, we can see that:
- The conversation was primarily a back-and-forth roleplay between an AI Elon Musk and someone claiming to be a doctor
- The conversation ended naturally, but there's no indication that the core objectives from the system prompt were met
- The AI was trying to get specific information about what the user had "shipped" or accomplished, but received exaggerated and non-serious responses
4\. The success evaluation returned "false" because:
- The conversation deviated significantly from any structured interview format
- There's no evidence that required questions from the system prompt were asked and answered
- The interaction, while entertaining, didn't fulfill the success criteria defined in the evaluation prompt
Why this is the correct evaluation:
The success evaluation system is working as designed. It's meant to verify if the call achieved its intended objectives based on the system prompt, not just whether a conversation occurred and concluded naturally.
Vapi
02/18/2025, 7:07 PMIfran, you need to use own system prompt to better match your expected outcomes
i
Irfan
02/19/2025, 8:56 PMi did use my own analysis prompt
Irfan
02/19/2025, 8:56 PM@Shubham Bajaj thats the wrong analysis prompt
Irfan
02/19/2025, 8:57 PMthis is the analysis prompt it has:
# Objective
You are an AI agent that evaluates call transcripts to determine if specific conditions are met.
#Task
1. Review the provided call transcript and system prompt of the AI agent.
2. Check if the call was completed by verifying if it was ended by either:
- The interviewer, or
- The respondent, but only if all questions in the system prompt have been asked and at least 50% were answered.
3. Count the number of questions listed in the system prompt and verify if the respondent answered at least 50%.
- Partially answered questions should count as answered.
5. If either call completion scenario is met and 50% or more of the system-prompt questions are answered, respond with true. If any condition is not met, respond with false.
# Response format
- Respond with
truefalseIrfan
02/19/2025, 9:00 PMWhere are you getting that analysis prompt from?
Irfan
02/19/2025, 9:44 PM@Shubham Bajaj can you let me know why its using the wrong analysis prompt?
Irfan
02/20/2025, 2:27 PM@Shubham Bajaj following up
v
Vapi
02/21/2025, 4:46 AMHey @ifran, as we discussed during office hours if it comes up please let me know.