https://vapi.ai/ logo
Join Discord
Powered by
Do you have to use streaming for custom llm url?
# support
m
The following code where I stream the response works:
Copy code
response = openai.chat.completions.create(
            model="gpt-4o",
            messages=request.messages,
            stream=True,
    )
async def stream_response(stream):
    for r in stream:
        yield f"data: {r.model_dump_json()}\n\n"
    yield "data: [DONE]"

return StreamingResponse(stream_response(response), media_type="text/event-stream")
However, this code does not work. The vapi assistant does not say anything.
Copy code
res = openai.chat.completions.create(
        model="gpt-4o",
        messages=request.messages,
    )
  return res.model_dump_json()
@Vapi
v
@max_k It seems like you're trying to use the OpenAI API to generate responses for a chat application. In the first code snippet, you're using streaming to handle the response, which is suitable for real-time applications where you want to send the response back to the client as soon as it's available. In the second snippet, you're not using streaming, and you're trying to return the response directly. If the VAPI assistant is not saying anything, there might be a few reasons: 1. The 
model_dump_json()
method might not be the correct way to access the response data. Usually, the response from the OpenAI API is a JSON object, and you can access the relevant parts of the response using the appropriate keys. 2. There could be an issue with how the response is being handled or sent back to the client. 3. There might be an error in the request or the API call that is preventing a successful response. To help you further, I would need more information about the error messages or the behavior you're observing. If you can provide that, I can offer more targeted assistance. If you're unsure about the specifics, you might want to reach out to OpenAI support for more detailed help on their API.
@max_k It seems like there might be an issue with how the response is being handled in the second code snippet. The 
model_dump_json()
method is not a standard method in the OpenAI API, so it's possible that this method is not correctly formatting the response for VAPI to process. To resolve this, you should format the response as a JSON object that VAPI expects. Here's how you can do it: 1. Extract the relevant information from the 
res
object. 2. Format this information into a JSON object according to VAPI's expected structure. 3. Send this JSON response back to VAPI. Here's an example of how you might format the response:
Copy code
python
# Assuming 'res' contains the response from OpenAI's API
response_content = {
    "choices": res.choices,
    # Include any other relevant fields that VAPI expects
}

# Convert the response content to JSON
response_json = json.dumps(response_content)

# Send the formatted response to VAPI
return response_json
If you're unsure about the exact structure that VAPI expects, please refer to the VAPI documentation or ask support for more info on this. If you need to send the response back to VAPI now, please provide me with the expected JSON structure, and I can help format the response accordingly.
You can continue this conversation by mentioning me in the message.
Got it. Tagging @User for additional help.
s
check this repo and yes you have to stream response. https://github.com/VapiAI/advanced-concepts-custom-llm
3 Views
PreviousNext
Powered by