I had the same issue last week. Tweaked my assistant's startSpeakingPlan so it waits for a clear response before proceeding. Helped a lot.