Since inbound is working fine, the number itself isn’t the problem. This feels more like a routing or media delay on the outbound side through Telnyx, especially with Nigerian routes, a small config issue can easily cause that 3–5 second gap.
Are you forcing PCMU only, and do you have media anchoring enabled on Telnyx?
If you’re open to it, send me your SIP/outbound settings privately and I’ll take a proper look at it with you.