It's more of a

Start Speaking Plan

kind of thing. Have you tried LiveKit or Vapi smart endpointing? Maybe tweaking`Stop Speaking Plan` adjustments could help,