It's always going to use what is returned to generate a response. Your delta is going to be prompt adherence, use a 4.1 model and give it very harsh strict instructions and see if it works. If not you may have to wait for a better model to come out for it to be 100% reliable .