If the tool is already returning an error response correctly, it sounds less like an SMS/Twilio issue and more like how Vapi handles the request-failed path.
When the failure comes back, is the assistant receiving any follow-up message/event at all, or does the conversation immediately switch back to a waiting state?