Appreciate the suggestions! The issue is that these are all rigid approaches. Real speech is not rigid. I can't speak to you and decide to wait X time after your last word. I would have to actually listen and decide whether you are done speaking.
I was excited about Smart Endpointing unfortunately it doesn't do this at all