Vapi support responded to my email - adding it here for anyone who might face the same issue: "To disable filler phrases like "Just a sec" and "Give me a moment" before making tool calls, you should set the message types

request-start

and

request-response-delayed

to empty strings. These messages are typically configured by default, so explicitly setting them to empty will prevent them from being voiced out."