I have noticed it could be prompt related for me if I change prompt a bit it will take longer to call tool or transfer than other times. I have found XML format for my prompts has helped a ton, maybe try that? It;s tough because like you have said it does not happen all the time so that makes debugging difficult.