I've actually got something working pretty well that just operated on the transcription along with the context of the expected call flow. Will soon experiment with having it operate on the incoming audio stream too.