Has anyone been able to hold a call over Vapi by streaming the caller's audio to Vapi, and listening for audio coming back? The flow I'm trying to setup is: 1. Call connects to our Twilio audio handler. 2. Setup message handler to forward Twilio messages straight to Vapi. 3. On START from Twilio, create a Vapi call with phoneCallProviderBypassEnabled = true. 4. Setup websocket connection to Vapi, which will allow messages (Step 2 above) to start flowing. 5. Return messages from Vapi straight to Twilio. In the UI, the recordings show that Vapi does receive the audio I'm sending to the transport url, but I wasn't receiving any audio back. So now I'm connecting to /transport to send audio and /listen to receive audio – but the /listen connection fails randomly ~80% of the time. Has anyone run into this? Curious if anyone else has solved pushing caller audio to Vapi. Still waiting on a reply from the Vapi team and will update here if I hear anything. Their docs seem pretty out of date + incomplete in this area. The live call control docs show that response's listenUrl as being the /transport url, but now you get back a /listen url and and empty transport object.

Just to confirm, your trying to build a custom integration with Vapi: \- Handle Twilio calls on your server (not letting Vapi handle the Twilio connection directly) \- Stream the audio from these Twilio calls to Vapi \- Receive audio responses back from Vapi \- Send those responses back to Twilio

@fsm_sam

Solution

ts

// 1. First, handle incoming Twilio webhook
app.post('/twilio/voice', async (req, res) => {
  // Create a Vapi call with bypass enabled
  const vapiCall = await fetch('https://api.vapi.ai/call', {
    method: 'POST',
    body: JSON.stringify({
      phoneCallProviderBypassEnabled: true,
      // other call parameters...
    })
  });
  const { callId } = await vapiCall.json();

  // Return TwiML to connect Twilio to your websocket
  const twiml = new VoiceResponse();
  const connect = twiml.connect();
  connect.stream({
    url: `wss://your-server.com/twilio-bridge/${callId}`
  });
  res.type('text/xml');
  res.send(twiml.toString());
});

// 2. Setup WebSocket bridge
const setupCallBridge = async (callId: string, twilioWs: WebSocket) => {
  // Important: First establish transport connection
  const vapiTransport = new WebSocket(`wss://api.vapi.ai/call/${callId}/transport`);
  
  await new Promise(resolve => vapiTransport.once('open', resolve));
  
  // Only after transport is established, connect to listen endpoint
  const vapiListen = new WebSocket(`wss://api.vapi.ai/call/${callId}/listen`);
  
  // Handle Twilio -> Vapi audio
  twilioWs.on('message', async (data) => {
    try {
      const message = JSON.parse(data.toString());
      if (message.event === 'media') {
        const audio = Buffer.from(message.media.payload, 'base64');
        if (vapiTransport.readyState === WebSocket.OPEN) {
          vapiTransport.send(audio);
        }
      }
    } catch (error) {
      console.error('Error processing Twilio message:', error);
    }
  });

  // Handle Vapi -> Twilio audio
  vapiListen.on('message', (data, isBinary) => {
    if (!isBinary) return; // Skip non-audio messages
    
    try {
      if (twilioWs.readyState === WebSocket.OPEN) {
        twilioWs.send(JSON.stringify({
          event: 'media',
          streamSid: callId,
          media: {
            payload: data.toString('base64'),
            track: 'outbound'
          }
        }));
      }
    } catch (error) {
      console.error('Error sending audio to Twilio:', error);
    }
  });

  // Error handling
  const handleError = (ws: WebSocket, source: string) => {
    ws.on('error', (error) => {
      console.error(`${source} WebSocket error:`, error);
    });
    ws.on('close', (code, reason) => {
      console.log(`${source} WebSocket closed:`, code, reason.toString());
    });
  };

  handleError(vapiTransport, 'Vapi Transport');
  handleError(vapiListen, 'Vapi Listen');
  handleError(twilioWs, 'Twilio');
};

Key Points: - First establish transport then connect listen. This addresses the 80% failure rate because the /listen endpoint expects the transport connection to be established first. - Using separate connections for input/output helps maintain clean audio streams and prevents potential conflicts. - Proper Audio Format Handling:

ts
// Twilio sends base64, convert to buffer for Vapi
const audio = Buffer.from(message.media.payload, 'base64');

// Vapi sends binary, convert to base64 for Twilio
payload: data.toString('base64')