I'm running vapi test suites through the API with this call:

console.log("item.input", item.input)
      const createdTest = await vapi.testSuiteTests.testSuiteTestControllerCreate(testSuite.id, {
        scorers: [{ type: "ai", rubric: scorer }],
        type: "chat",
        script: item.input,
        numAttempts: 1,
        name: `${experimentName} - Script ${i + 1}`,
      })

I can see that the item.input is unique every time it's created and it shows the different scripts on the dashboard as well. But the transcript shows the same test user script for every single one. Is there a caching bug or creation bug occuring on VAPIs endpoint or am I invoking this incorrectly?

Use manual testing for now and report the run IDs and timestamps here

Here's a visual example of what I mean Run ID: 1117733a-17c8-4900-aae7-23707d0b65f4 https://cdn.discordapp.com/attachments/1404516049554112562/1404524440192159785/Screenshot_2025-08-11_at_1.56.09_PM.png?ex=689b80d1&is=689a2f51&hm=ad7ac2b2623f60b1f71bc1c4cce9ad681b1dc47265f374428c10399c71fa3d19&

Looking into it

Hey @Kings_big💫 , figured out the issue. I'm creating an assistant in my codebase and was passing that thru to make the test. However, the name of the test was matching an existing one so it kept failing. After providing a brand new, unique name - it worked.

const assistantToUse = await vapi.assistants.create({
      name: realName, <- had to make this unique from any existing test suites; I was trying to re-use an existing one.
      ...assistant,
    })

    const testSuiteParams: Vapi.CreateTestSuiteDto = {
      name: realName,
      targetPlan: {
        assistantId: assistantToUse.id,
        phoneNumberId: "<REDACTED>",
      },
    }

Thanks for sharing the details. I've shared this with the team. We are currently gathering all issues and data related to the test script problems. The team is working to understand the root causes and will dive deeper to address them. Once fixes are deployed, I'll update you.