For context, we use a custom LLM endpoint which handles more complex dialogue management and toggles between various models throughout the conversation. In addition to responding with the next question to ask the user, there are times we'd like to also append metadata that the client can listen for and adjust the UI accordingly. Is this possible through vapi?

Yes, it's possible to send metadata from a custom LLM endpoint to the client in Vapi. The client can listen for this metadata using event handlers:

javascript
// Set up event listener for metadata
vapi.on("message", (message) => {
  if (message.type === "metadata") {
    // Update UI based on received metadata
    updateUI(message.data);
  }
});

According to [Vapi documentation](https://docs.vapi.ai/customization/custom-llm/using-your-server), your custom LLM can send formatted JSON responses back to Vapi, which can include metadata that gets forwarded to the client. Sources: - [Vapi Documentation](https://docs.vapi.ai/customization/custom-llm/using-your-server) - [Discord Discussion](https://discord.com/channels/1211482211119796234/1330803502615625738) - [Discord Discussion](https://discord.com/channels/1211482211119796234/1260258706667343892)

I understand on the client side, but how can the custom LLM send a server side event with a custom payload?

By responding to the serverURL webhook like assistant-request with a JSON payload using the server-message format.