Sounds like you need to use a tool call so it always ask, not just sometimes.