https://vapi.ai/ logo
Join Discord
Powered by
Integration with LangGraph as custom LLM model
# support
l
I was able to successfully integrate VAPI with a custom LLM model using OpenAI. I followed the example in https://docs.vapi.ai/customization/custom-llm/using-your-server. In this case my server is replying to VAPI with a streaming response that follows the OpenAI format. E.g.
Copy code
{
  "id": "chatcmpl-BEYW9n5na5tA6539KWpUH9DwkSRIx",
  "choices": [
    {
      "delta": {
        "content": "Hello",
        "function_call": null,
        "refusal": null,
        "role": null,
        "tool_calls": null
      },
      "finish_reason": null,
      "index": 0,
      "logprobs": null
    }
  ],
  "created": 1742808977,
  "model": "gpt-4o-mini-2024-07-18",
  "object": "chat.completion.chunk",
  "service_tier": "default",
  "system_fingerprint": "fp_b8bc95a0ac",
  "usage": null
}
As a next step, I would like to integrate VAPI with an AI agent built using LangGraph. The agent is hosted in LangGraph Platform. The idea is that VAPI contacts my "/assistant/chat/completions" endpoint on my server. My server processes the request by interacting with the LangGraph agent using the LangGraph SDK. It is unclear to me what is the format that VAPI expects as response. Do I need to reformat the LangGraph response to follow the OpenAI format or is there a standard format that VAPI support that can be applied to different types of integrations?
v
Hey there! 👋 I don't have specific documentation for the 
/assistant/chat/completions
endpoint you mentioned, but I can help with what we know about VAPI's response format expectations. Based on the information I found, VAPI is designed to be flexible and can integrate with various custom LLMs. When integrating a custom LLM with VAPI, the response needs to be formatted according to VAPI's expected structure. From your example, you've already successfully integrated VAPI with a custom LLM using the OpenAI format. This is a good sign, as [VAPI documentation](https://docs.vapi.ai/server-url/events) indicates that VAPI fully supports OpenAI's function calling API. When formatting responses for VAPI: 1. For OpenAI integrations, the response should be formatted according to VAPI's expected structure and sent back as a JSON object, as you've already done successfully. According to [VAPI documentation](https://docs.vapi.ai/customization/custom-llm/using-your-server), the response from OpenAI is received and formatted according to VAPI's expected structure. 2. For tool calls, VAPI expects a specific JSON format:
Copy code
json
{
    "results": [
        {
            "toolCallId": "X",
            "result": "Y"
        }
    ]
}
as mentioned in the [VAPI documentation](https://docs.vapi.ai/tools/custom-tools). Since you've already successfully integrated with OpenAI format, and VAPI supports OpenAI's function calling API, it's likely that reformatting the LangGraph response to follow the OpenAI format (as you showed in your example) would be the safest approach. Would you like me to help with any specific aspects of the response formatting or the integration process? Sources: - [VAPI documentation](https://docs.vapi.ai/server-url/events) - [VAPI documentation](https://docs.vapi.ai/customization/custom-llm/using-your-server) - [VAPI documentation](https://docs.vapi.ai/tools/custom-tools)
Hi there! 👋 I understand you're trying to integrate VAPI with a LangGraph agent and need clarity on the expected response format. I can definitely help with that! Based on the information I've found, VAPI generally expects responses to follow the OpenAI format, especially for streaming responses like the example you shared. According to the [VAPI documentation](https://docs.vapi.ai/customization/custom-llm/using-your-server), when integrating a custom LLM, "The response from OpenAI, containing the generated text, is received and formatted according to Vapi's expected structure" and "The formatted response is sent back to Vapi as a JSON object." For your LangGraph integration, you would likely need to reformat the LangGraph response to follow the OpenAI format. The documentation indicates that [VAPI supports using any OpenAI-compatible endpoint as the LLM](https://docs.vapi.ai/customization/custom-llm/fine-tuned-openai-models), and "Vapi expects the response to be streamed in chunks following the OpenAI streaming format, rather than a complete response." If your integration involves tool calls, the response should follow this JSON format:
Copy code
json
{
    "results": [
        {
            "toolCallId": "X",
            "result": "Y"
        }
    ]
}
It's important to ensure that your response format strictly adheres to VAPI's expected schema, as [small deviations in key names or data structure can cause parsing issues](https://discord.com/channels/1211482211119796234/1211483291191083018/threads/1336440478727667844). One note of caution: I found information suggesting that [the custom LLM server approach might not work straightforwardly for connecting Langchain/LangGraph](https://discord.com/channels/1211482211119796234/1211483291191083018/threads/1341715498793963541), so you may need to explore alternative integration methods. Is there anything specific about the response format you'd like me to clarify further? Sources:
k
Hey Lucio, we expect you for the Custom LLM to always return the responses according to the OpenAI format. That's our standard format for now.
Let me know if you require further help !!
l
Thanks Shubham!
It is helpful
k
Marking this ticket as Solved ✅
5 Views
PreviousNext
Powered by