Since I upgraded vapi sdk to recet version, success evaluation is not a boolean result anymore (although I use

PassFail

evaluation). I have a text result Here is a callId: "cd3f0ff5-cc5a-4470-9978-46cc185099e6" Here is the

PassFail

configuration used for this call:

"successEvaluationPlan": {
            "rubric": "PassFail",
            "enabled": true,
            "messages": [
              {
                "role": "system",
                "content": "Vous êtes un expert en évaluation d'appels. Une transcription d'appel et le system prompt du participant IA vous seront fournis. Déterminez si l'appel a été réussi en fonction de votre perception de la satisfaction de l'appelant."
              },
              {
                "role": "user",
                "content": "Voici la transcription d’un appel téléphonique :\n\n{{transcript}}\n\n"
              },
              {
                "role": "user",
                "content": "Voici le system prompt de l'appel :\n\n{{systemPrompt}}\n\n. Voici la raison de la fin de l'appel :\n\n{{endedReason}}\n\n"
              }
            ]
          }

https://cdn.discordapp.com/attachments/1381987553694978099/1381987553967603783/image.png?ex=684983b4&is=68483234&hm=d3acea485b604f88e983bc5a7086c2c0f505a07de9b0e12783a977cda84439c1&

Hey @marem based on your squad this is what your currently passing to the LLM for success evaluation. https://cdn.discordapp.com/attachments/1381987553694978099/1382042011644985364/message.txt?ex=6849b66c&is=684864ec&hm=020bca9ffe61ca4bf07344ab55f43ff7f26663aa61cb06b8904cca77d1b43350&

Ok, but since it's a

PassFail

evaluation, is it normal that the evaluation response is a string sentence, rather than a boolean?

It is not expected, but based on the last active member in your squad or the recent active member in the squad, the analysis plan is used, and the outcome which you are seeing is totally correct.

Indeed. I was missing this part

\n\nRubric:\n\n{{rubric}}\n\nOnly respond with the evaluation result.

due to override

Marking this ticket as Solved ✅