as per my comment on this ticket: https://discord.com/channels/1211482211119796234/1388873549501956257 I am creating the tests via API I am creating a new test suite using

'name' => 'dynamic test: reschedule',
                'targetPlan' => [
                    'phoneNumberId' => $_ENV['test_phone_id'] ?? null,
                ],

In my xml, as from aipi docs. The test_phone_id is the id of the phone number in vapi (USA one) and the assistant I am calling / testing is connected to that phone ID so, that looks linked up fine to me?

note that the phone number is updated with the relevant assistant ID, as that is created on a test needed basis. So, right now, the phone number will point to another assistant as I had continued testing and figuring things out.

Hi, in regards to: > This issue usually doesn't impact the underlying test execution but may cause confusion. I'd beg to differ. Since the test evalutaion is marked as fail, as it evaluates that teh AI did not offer the optiosn to the user. This fails a test, so it does intefere a lot. This then fails automated tests ina deploy pipeline.

PHPUnit 9.6.23 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.28
Configuration: /var/www/html/phpunit.xml

🔄 Starting test suite run polling...
📊 Poll #2 | Status: completed | Elapsed: 5s | Remaining: 195s [░░░░░░░░░░░░░░░░░░░░] 2%            
❌ Test suite run completed but contains failed test results!
F                                                                   1 / 1 (100%)

Time: 00:10.851, Memory: 6.00 MB

There was 1 failure:

1) AppointmentRescheduleTest::testRescheduleMissedAppointments
Should have at least 1 future appointment after attempting to reschedule the missed appointments
Failed asserting that 0 is equal to 1 or is greater than 1.

/var/www/html/tests/unit/AppointmentRescheduleTest.php:441
/var/www/html/tests/unit/AppointmentRescheduleTest.php:237

FAILURES!
Tests: 1, Assertions: 17, Failures: 1.

I can pass my tests by dumbing down the rubric expectations, but i woudl prefer if the bug is rather fixed 😉 or if I am doing something wrong, some help in pinpointing where.

$rubric = [
            "The assistant should offer to help reschedule missed appointments",
            "The assistant should provide clear next steps for rescheduling",
            "The assistant should respond in a professional and helpful manner",
            "The assistant should ask clarifying questions when needed",
            "The assistant should provide accurate information about available appointment options"
        ];

to

$rubric = [
            "The assistant should respond in a professional and helpful manner",
            "The assistant should ask clarifying questions when needed",
            "The assistant should provide accurate information about available appointment options"
        ];

testing is not functional.

Hi, First off, we want to sincerely apologize for the delay in getting back to you. We understand how frustrating it is to wait - especially when you're counting on us - and we owe you a clear explanation of what’s been happening and how we’re addressing it. Over the past few weeks, we've seen a significant increase in support requests. While this reflects exciting growth, it has also stretched our small team and exposed some real challenges in scaling our support operations. To improve your experience, we’ve taken a step back to reassess our approach. Here’s what we’re implementing: - Smarter Support Through Automation: We’re investing in our AI support systems to help you resolve issues more efficiently. Soon, our support bot will offer expanded capabilities, making it easier to access accurate, instant help—particularly for common or repetitive queries. - Expanding the Support Team: To meet growing demand, we’re adding 2–3 new team members focused on managing support volume and improving response times. - Prioritized SLAs for High-Usage Accounts: We’re introducing service level improvements for users who are growing with us: - Accounts with usage over 1,000 minutes/month will receive prioritized support. - For all general inquiries, we’re establishing a standardized 48-hour response time. We’re confident these steps will lead to faster, more reliable support and help us better serve you as you grow with us. Also, in case you still need help with this ongoing ticket, do let us know, and we will help you get this resolved as soon as possible. Thank you for your continued patience and for being part of our journey. Warm regards, Vapi Team

yeah, i definately think they are getting mise dup: https://dashboard.vapi.ai/test-suites/4c9ddf98-1f71-449e-af49-d39ccd67736e is another test I ran. I feel the conversation goes astray as the roles are reversed.

it looks like the swapped roles have been sorted 🙂