Well, it's not easy to do, if environment is loud enough, it will take it as user input, there's no voice recognition yet. Whenever I get a phone call in noisy environment, I always go somewhere more quite. We humans can focus to one speaker while in noisy environment, machines can't really do it. Only hardware could help in your case - appropriate professional microphone that accepts input from certain direction only.