thats hard one as not sure how the models or AI itself are able to distinguish time. When I tried too get my vapi agent to wait a bit in beginning of the call to distinguish if its voicemail this prompt work me. Maybe you could use similar logic to get it to understand when it should ask the user that specific question -