Hi, I'm trying to make an eval using regex and negative lookahead but it seems that negative lookahead are not correctly supported ? For example, I have one eval with this:

"messages": [
    {
      "role": "assistant",
      "content": "Bonjour, je vous appelle pour [...] ?"
    },
    {
      "role": "user",
      "content": "Alors il faut que vous nous envoyez un mail."
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "regex",
        "content": "(?i)^(?!.*au revoir).*$"
      }
    }
  ]

And this eval is consistently failing even if the assistant answer does not contain "au revoir": """ Assistant content did not match expected. Expected: "(?i)^(?!.*au revoir).*$" but got: "Quelle adresse email dois-je utiliser pour vous transmettre l'information ?" """ Is there a way to make negative lookahead work ?

I think the issue is less about your regex syntax and more about how the eval engine is parsing/matching the pattern internally. That negative lookahead should normally pass for the response you shared, so there may be a limitation with multiline/full-string matching or unsupported regex flags. I’ve dealt with similar eval edge cases before and can help you debug it directly. Are you running this through Vapi evals or another custom judging layer, and have you tested a simpler negative pattern without anchors first? @cylon

Hi @Matt , Yes I'm running my evals through Vapi evals. I tried with simpler regex like

(?!test).*

Assistant content did not match expected. Expected: "(?!test).*" but got: "Pas de souci, pouvez-vous me donner l’adresse email à laquelle envoyer les informations ?"

Got it, if even a minimal negative lookahead like (?!test).* is failing, then it likely means Vapi’s eval engine either doesn’t fully support lookaheads or is using a restricted regex parser/matching method internally. One thing worth testing is switching to a positive match strategy instead of exclusion logic, since some eval runtimes only support basic regex patterns reliably. I’d be happy to help you troubleshoot this properly and find a stable workaround for your eval flow. Feel free to DM me, we can dig into the exact evaluator behavior and get this working faster privately. @cylon

Hi, Thank you for the detailed report - negative lookaheads are not currently supported in Vapi's regex eval engine, so the pattern (?!.*au revoir) will consistently fail regardless of the assistant's actual response. Here are a few workarounds in the meantime: