I think I can help identify what MIGHT be wrong. The most common cause I've seen is either a configuration mismatch with the forwarding number or a timing issue with the silence timeout settings. Have you noticed if the calls are dropping at a consistent point during the transfer attempt?
I'd suggest starting by double-checking the forwarding number in your Function settings - even a small typo there could cause problems. If that looks good, I'd look at your silence timeout configuration next, as calls sometimes fail if this is set too aggressively.
A few other areas worth investigating...
Are you seeing any specific error messages in your Vapi dashboard logs? Those can be really helpful in pinpointing the exact issue.
Have you tried testing with a different forwarding number to see if the problem is specific to one number?