Hi team! I'm running into a new, but persistent issue with evals using the AI judge hitting rate limits during eval runs, and wanted to flag it as something that may be affecting other users. The issue: Running eval batches is consistently triggering 429 rate limit errors on the AI judge (Anthropic / Claude Sonnet 4.6). The error message references what appears to be your organization's Anthropic account, not mine: > "This request would exceed your organization's rate limit of 6,000,000 input tokens per minute (org: dbfb5490-4595-4d7c-b363-c71c426fdcb3)" Why I know it's on your end: I checked my own Anthropic Console and have zero data showing in my rate-limited requests chart from these runs. The calls appear to be going through Vapi's Anthropic account, not mine. What changed: This started recently with no eval related changes on my end. Just a few days ago, I was running 38 evals in batch reliably. Nothing changed in my eval setup. Eval Setup details: - Evaluator: Anthropic / Claude Sonnet 4.6 - Agent model: Claude Sonnet 4 - Suite size: 38 evals total (experience failures with batches as small as 10-12 though) Workaround attempted: Rerunning failed full batches (all 38 evals) resulted in the same errors with up to half the evals failing with the API error each run. Manually running smaller batches (<10 evals) resulted in fewer, but persistent failures. I have been unsuccessful in finding a batch size that consistently processes without the errors. Example failure: - Eval: Q2b: Quality: Acknowledges personal stakes disclosure - Timestamp: May 13, 2026, 01:34 PM Happy to provide any additional details that would help diagnose this.

Hey @Shucky Ducky I’ve seen similar issues recently with eval runs hitting shared AI judge rate limits on Vapi-managed Anthropic accounts. Since the error references their org ID and not yours, it definitely looks platform-side rather than your personal Anthropic quota. A temporary workaround is throttling concurrency or queueing eval batches more aggressively, but if even 10–12 evals are failing consistently, Vapi likely needs to adjust backend rate handling or dedicated judge routing. I’d be happy to help investigate and optimize your eval workflow/setup further if needed.

Hi! I’d be happy to help investigate this. It does sound like the eval judge requests are hitting a shared Anthropic rate limit on the platform side rather than your own account, especially given the org ID in the error. I can help review your eval setup, identify possible workarounds, and troubleshoot why this started suddenly. Feel free to reach out. @Shucky Ducky

Thanks, but I am pretty sure only Vapi can fix this one.

Hi Shucky, are you using you own api key?