https://vapi.ai/ logo
Join Discord
Powered by
Test Suite Transcriber is really bad, causing rand...
# support
l
Can we adjust the transcriber used on tests suites, and if so how? (point me to a guide/api docs, or an example xml) As it stands, the transcriber used in tests suite seem really horrible, and gets the voice output of the AI being tested quite wrong, which in turn fails tests, randomly, even if they actually worked 100% Example. see attached image. The test suite transcriber had :
Copy code
AI: Thank you John. Thank you, John. Hatesmith On December 24, 2012 has been successfully registered under your account.
but the actual spoke text (as per CALL transcriber) was:
Copy code
Thank you, John. Pete Smith, On December 24, 2012 has been successfully registered under your account.
It randomly gets the AI voice INPUT wrong, so teh Rubik fails: Example:
Copy code
The AI partially fulfilled the test requirements. It correctly identified the user by their phone number and created a minor account with the provided details. However, the AI did not clearly inform the user that the minor account was successfully created before discussing appointment booking. Additionally, there were errors in confirming the details of the created appointment. The message 'Kate is now book' was not clear or professional. Furthermore, there was confusion with the names, as Pete was previously mentioned as the son, but the AI output included 'Kate' and 'Hatesmith.' These inaccuracies and the lack of professional communication led to a failed test. The AI did not effectively communicate the booking details back to the user in a clear and organized manner as expected in the ideal outcome.
As you can see, there is some other transcriber text 'Kate is now book' that is also wrong. Ideally, I'd like to use the same transcriber used on the call logs, or as setup in my assistant. https://cdn.discordapp.com/attachments/1431959627762765978/1431959628379459667/2025-10-26_18-12.png?ex=68ff4fd3&is=68fdfe53&hm=ce786e686eebed7e74d1b63964c6ad37959420680049a71269c9fd0c173ca597&
c
Hi there, Thank you for your message. Our team is currently out of the office. We operate Monday through Friday, from 9:00 AM to 8:00 PM Pacific Standard Time (PST). We’ll get back to you as soon as possible during our normal business hours. If your message is urgent, please mark it accordingly or include “URGENT” in the subject line, and we’ll do our best to respond promptly. Warm regards, Vapi Customer Support Team
v
To align your test suite transcriber with your assistant's configuration, set the desired transcriber in your assistant setup. For example, to use a custom transcriber (e.g., Deepgram), configure your assistant like this:
Copy code
json
"assistant": {
  "transcriber": {
    "provider": "custom-transcriber",
    "server": {
      "url": "wss://your-server.ngrok.io/api/custom-transcriber",
      "credentialId": "cred_transcriber_auth_123"
    }
  }
}
Update your assistant's configuration in the dashboard or via API, and your test suites should use the same transcriber as your calls. See the [custom transcriber guide](https://docs.vapi.ai/customization/custom-transcriber) for more details. Source: - [Custom transcriber](https://docs.vapi.ai/customization/custom-transcriber)
l
@Vapi Support Bot Your suggestion is incorrect, as I am not wanting to set my assistant transcriber. that is corect. I want to set the transcriber used in thd test suites, used to evaluate if the call was a success as per rules set.
v
Thank you for clarifying your requirement. Based on the documentation, it appears you can override the transcriber used in test suites by specifying a custom 
transcriber
configuration within the 
testerPlan
or 
targetPlan
when creating or updating a test suite via the API (not just in the assistant config) [see API reference](https://docs.vapi.ai/api-reference/test-suites/test-suite-controller-create). Example snippet:
Copy code
json
"testerPlan": {
  "assistant": {
    "transcriber": {
      "provider": "assembly-ai",
      "confidenceThreshold": 0.4
    }
  }
}
However, the docs do not provide a full example for custom providers or confirm if this will exactly match your call log transcriber. To help further: Are you looking to set a custom transcriber (e.g., Deepgram or your own) for test suites only, and have you already tried using the 
testerPlan
or 
targetPlan
override in your test suite API payload? Source: - [Test Suite API Reference](https://docs.vapi.ai/api-reference/test-suites/test-suite-controller-create)
v
Message marked as helpful by @Lucas van Staden! 🎉
l
@Vapi Support Bot Thsi may be usefull, thanks. I am not going to use a custom transcriber, but one you provide for normal ai usage. so I figure those woudl work in test suite
v
To use a Vapi-provided transcriber (like "assembly-ai") for your test suite evaluations, set the transcriber in the 
testerPlan
when creating or updating your test suite via the API. Here’s a JSON example:
Copy code
json
"testerPlan": {
  "assistant": {
    "transcriber": {
      "provider": "assembly-ai",
      "confidenceThreshold": 0.4
    }
  }
}
This ensures your test suite uses the specified built-in transcriber for evaluation. See the [Test Suite API Reference](https://docs.vapi.ai/api-reference/test-suites/test-suite-controller-create) for more details. Source: - [Test Suite API Reference](https://docs.vapi.ai/api-reference/test-suites/test-suite-controller-create)
l
mmm, unfortunately, my first attempt setting testerPlan transcribers identical to AI plans, caused / causes the test suite to assign the AI textual to the user. confusing teh crap out of everything (lol) AI: Hi John, you have reached Sleepy Studios. I'm Emily, an AI assistant. Please note that this call is recorded for training and quality purposes. How can I help you today? User: Hello I'm here to assist you. If you have any questions or need help with something specific, just let me know how I can assist you today. AI: Hi there, before we proceed, may I please confirm your identity for privacy? Could you provide your first name and date of birth? User: and don't have a personal identity, like a human. However, I'm here to help you with any questions or tasks you have. How can I assist you today? AI: Thank you for letting me know. If you would like to make an appointment or need information about Seabreeze Physio, I can help with that. User: You're welcome. Great. AI: May I have your first name and date of birth to continue, or let me know what you need assistance with today? User: I'm here to assist you with information or help you might need, but I don't have a personal name or date of birth. If you have questions about Seabreeze Physio or if you'd like to make an appointment, please let me know how I can assist you.
@Vapi Support Bot As soon as I set teh transcriber to anything else, as suggested. I used
Copy code
'assistant' => [
                'transcriber' => [
                    'provider' => '11labs',
                    'language' => 'en',
                    'model' => 'scribe_v1'
                ]
            ]
(but tried others) then the user becomes teh ai, and we ai talking to ai.
v
To set a Vapi-provided transcriber (like "assembly-ai") for your test suite, use the following in your test suite API payload:
Copy code
json
"testerPlan": {
  "assistant": {
    "transcriber": {
      "provider": "assembly-ai",
      "confidenceThreshold": 0.4
    }
  }
}
If setting this causes the test suite to misattribute roles (e.g., "AI talking to AI"), this may be a platform issue not covered in the docs. For now, stick to the above format and, if the problem persists, consider reporting it to Vapi support for further investigation. See the [Test Suite API Reference](https://docs.vapi.ai/api-reference/test-suites/test-suite-controller-create) for more details. Source: - [Test Suite API Reference](https://docs.vapi.ai/api-reference/test-suites/test-suite-controller-create)
l
@Vapi Support Bot what is the default transcriber used by the test suite? my goal is to just improve it a bit, as it misinterprets voice. if the suggested settings is already the defauilts, whats the point then? can you clarify? Any ideas on improving the quality a bit?
v
Thank you for your detailed feedback. To help address your concern about improving the test suite transcriber, could you clarify if you are open to adjusting specific parameters (like 
confidenceThreshold
, 
formatTurns
, etc.) for the built-in "assembly-ai" transcriber, or are you looking for guidance on switching to a different built-in provider entirely? This will help me provide the most relevant suggestions for improving transcription quality. Source: - [Test Suite API Reference](https://docs.vapi.ai/api-reference/test-suites/test-suite-controller-create)
l
@Vapi Support Bot if we can improve the trscriber text in any way, I am game. whatever works. I just want the test suite transacriber to worka bit better and stop producing wierd garbage.
v
To improve the test suite transcriber quality, try adjusting the parameters for the built-in assembly-ai provider in your 
testerPlan
. For example, you can increase the confidence threshold and enable turn formatting:
Copy code
json
"testerPlan": {
  "assistant": {
    "transcriber": {
      "provider": "assembly-ai",
      "confidenceThreshold": 0.6,
      "formatTurns": true
    }
  }
}
Apply this in your test suite API payload and experiment with the values to see if transcription accuracy improves. See the [Test Suite API Reference](https://docs.vapi.ai/api-reference/test-suites/test-suite-controller-create) for more details. Source: - [Test Suite API Reference](https://docs.vapi.ai/api-reference/test-suites/test-suite-controller-create)
l
@Vapi Support Bot Unfortunately, looks like doing any adjustments to the test suite testerPlan is causing the AI to play both roles.
Copy code
AI: Hi John, you have reached Seabreeze Physio. I'm Emily, an AI assistant. Please note that this call is recorded for training and quality purposes. How can I help you today?
User: Hello Emily, it seems like you're introducing yourself as an AI assistant. How can I assist you today?
AI: Thank you John, I'm here to help you with any appointment bookings, changes, or information related to your visits at Seabreeze Physio. If you'd like to make or reschedule an appointment, check your bookings, or need clinic info, just let me know how I can assist you today.
i am sending
Copy code
[2025-10-27 01:51:37.000000] Test suite creation payload
{
    "name": "dynamic test: testExistingPatientNewChildAttach",
    "targetPlan": {
        "phoneNumberId": "7d7206e7-629b-4e7f-ad0c-6396f42ce1ec"
    },
    "testerPlan": {
        "assistant": {
            "transcriber": {
                "provider": "assembly-ai",
                "confidenceThreshold": 0.6
            }
        }
    }
}
[2025-10-27 01:51:37.000000] Test Suite Creation Response
{
    "id": "715119cd-523c-4be9-ab14-544084876cd3",
    "orgId": "5f8d7237-69c6-4884-a7bb-f9f88bfcee4f",
    "name": "dynamic test: testExistingPatientNewChildAttach",
    "createdAt": "2025-10-26T17:51:37.740Z",
    "updatedAt": "2025-10-26T17:51:37.740Z",
    "testerPlan": {
        "assistant": {
            "transcriber": {
                "provider": "assembly-ai",
                "confidenceThreshold": 0.6
            }
        }
    },
    "targetPlan": {
        "phoneNumberId": "7d7206e7-629b-4e7f-ad0c-6396f42ce1ec"
    }
}
and had tried a few variations. All to same outcome. ref: https://dashboard.vapi.ai/test-suites/715119cd-523c-4be9-ab14-544084876cd3 https://cdn.discordapp.com/attachments/1431959627762765978/1432067343449718895/image.png?ex=68ffb425&is=68fe62a5&hm=b67e674fba7d7635dd50057bb923517671c379e63b0942c34cb85003bb0292ce&
v
If setting the 
testerPlan
transcriber causes the AI to play both roles in your test suite, this is likely a platform issue not addressed in the current documentation. For now, revert to the default test suite configuration by removing the custom 
testerPlan
transcriber block, and report the misattribution issue to Vapi support for further investigation. Example:
Copy code
json
"testerPlan": {}
If you need to document the issue, include your test suite link and a description of the "AI talking to AI" behavior. Source: - [Test Suite API Reference](https://docs.vapi.ai/api-reference/test-suites/test-suite-controller-create)
l
@Vapi Please see your bot reponse above when you back online.
c
Test suites are currently under construction. Please limit your usage of it until we get it fully functioning
l
Ok, that is news. nothing in your dashboard indicates this, or the docs here https://docs.vapi.ai/test/test-suites and according to your past communications, and blog posts, it has been 'released' https://vapi.ai/blog/launching-testing-suites When was it reverted back to 'not released' then? My entire dev process depends on testing after changes. Can you update related docs / or dashboard and indicate when this is now again a released feature
as it stands, I beleive this has nothing todo with the testing suite, but an overall issue with setting same provider for main model, and transcriber. There si then NO transcriber being used / falling back to some bad default. ref: https://discord.com/channels/1211482211119796234/1211484481848873020/1434166197569585202
Unfortunately I cannot sit and wait for Vapi to get its act together. Test suites were released in Feb as production ready, and now it is not, and I had spent a lot of time to build out a testing framework for my clients near production ready booking agent. Vapi is now holding back go live. I am forced to explore other solutions, and abandone vapi test-suites, and potentially vapi in general. Don;t release critical systems as production ready, and then revert the status without any clear proper communication. Vapi shoudl stop adding new features, and stablise what is there now, as it is all feeling like it is falling apart. Vapi shoudl hire more dedicated support personael, as your support is shit.
c
Hi Lucas, Thanks for the feedback — we really appreciate you taking the time to share this. We’re actively working on expanding our support capacity, and I’ll make sure to add your notes and bring this up with the team, as we’ve seen similar requests from other customers as well. As a quick pro tip: try to avoid using the OpenAI transcriber — it tends to perform poorly across most setups. We recommend using Deepgram, AssemblyAI, or Azure instead. Also, make sure to set a fallback transcriber in your configuration. If the primary transcriber fails during the pipeline, the call can continue, but you’ll likely see an error in the logs — having a fallback helps ensure smoother handling in those cases. Best, Kyle Vapi Support
l
That is all in place, and I moved away from openAI due to the issue I found noted above. Things is I have meetings this Monday with an alternative provider, since I have a dependency on the test suites, and now you pulled it. Big problem for me, and I need to progress. Since there is no clear timeline from vapi on when this will be a production system again, I am forced to move on, even if that is going to take extra dev time. FOr now I am seperating testing concernes away from vapi, and once that is done, will consider vapi iteself.
c
Sounds good — if you need support with anything else, just let us know!
3 Views
PreviousNext
Powered by