----------------------------- **While I generate an answer, here are some sources that might be helpful**: * [Connecting Your Custom LLM to Vapi: A Comprehensive Guide]() * [Create Assistant]() * [Server Events]() * [Update Assistant]() -----------------------------

### Answer To address your issue with using a custom LLM with Vapi, let's go through a few points based on the provided information and the cited Vapi documentation: 1. **Response Format**: The response structure you're using in your Flask app sends the content in

choices[0].delta.content

, but Vapi expects it in

choices[0].message.content

. You should adjust your response format in your Flask app to match this expectation. 2. **Flask App Code Adjustment**: Update your

basic_custom_llm_route

function to return the content in the correct format:

python
    import time
    from flask import Flask, request, jsonify

    app = Flask(__name__)

    @app.route('/chat/completions', methods=['POST'])
    def basic_custom_llm_route():
      request_data = request.get_json()
      response = {
        "id": "chatcmpl-8mcLf78g0quztp4BMtwd3hEj58Uof",
        "object": "chat.completion",
        "created": int(time.time()),
        "model": "gpt-3.5-turbo-0613",
        "choices": [
          {
            "index": 0,
            "message": {"content": "This is some test content"},
            "logprobs": None,
            "finish_reason": "stop"
          }
        ]
      }
      return jsonify(response), 201

    if __name__ == "__main__":
        app.run(debug=True, port=5000)

3. **Ensure

model.urlRequestMetadataEnabled

is Set**: Make sure to execute a PATCH request to the

/assistant

endpoint and ensure that

model.urlRequestMetadataEnabled

is set to

false

[(1)]() :

json
    {
      "model": {
        "urlRequestMetadataEnabled": false
      }
    }

By making these changes, your custom LLM should respond correctly within Vapi. Ensure your Flask app is correctly formatted and check the settings within Vapi's dashboard as well [(2)]() , [(3)]() , [(1)]() . **SOURCES USED**: [(2)]() , [(3)]() , [(1)]()

Here is the beginning of the request I receive from Vapi:

'model': 'gpt-3.5-turbo',
    'messages': [{
        'role': 'system',
        'content': 'This is a test agent.'
    }, {
        'role': 'assistant',
        'content': 'Hi. This is George How can I help?'
    }, {
        'role': 'user',
        'content': 'This is a test message.'
    }],
    'temperature': 0.7,
    'stream': True,
    'max_tokens': 250,
    'call': {
        'type': 'webCall',
        'callId': 'xxxxxxxxxxxxxxxxxx',
        'orgId': 'xxxxxxxxxxxxxxx',
        'transcriber': {
            'provider': 'deepgram',
            'model': 'nova-2',
            'keywords': [],
            'language': 'en',
            'smartFormat': False
        },
        'model': {
            'provider': 'custom-llm',
            'url': 'https://cead-37-168-11-222.ngrok-free.app',
            'urlRequestMetadataEnabled': True,
            'model': 'gpt-3.5-turbo',
            'temperature': 0.7,

I see it has

call.model.urlRequestMetadataEnabled=True

If this is the right endpoint to try, I get the following error on the server side:

TypeError: Completions.create() got an unexpected keyword argument 'call'

(line 86, in openai_advanced_custom_llm_route chat_completion_stream = client.chat.completions.create(**request_data))

However, the custom API I would like to interact with does not stream the information. Does this mean I should emulate this streaming process?

This might be a stupid question but where do I set urlRequestMetadataEnabled=False?

Thanks a lot for your support in any case @Sahil, very helpful and much appreciated ! 🙏