Is the actual call behavior consistent through different phone numbers/phones? If yes, you may have noise suppression on in your assistant settings. I'd recommend turning that off and then doing some testing.