I increased response delay to 1 second and it fixed that issue for me. At first I thought 1 second is too long for a response but then I realized after looking at recordings that as a human we (or at least myself) also takes about 1 second to process and respond on the phone so it worked out for me