Call Id: 46d2b594-e051-40bc-a06e-46df8b36e771 My API for the tool call showed no issues and returned what seems a healthy response The VAPI logs show:

{
  "name": "call_PH8mMMjyeQt0ZSfe",
  "role": "tool_call_result",
  "time": 1737057643352,
  "result": "An unexpected error occurred.",
  "toolCallId": "call_PH8mMMjyeQt0ZSfe",
  "secondsFromStart": 24.426
}

And the client received the same. I get this with gpt-4o-realtime-preview-2024-12-17 but not "gpt-3.5-turbo". I appreciate the realtime is in preview/beta so is not production ready but would be good to know what is happening here. Maybe it doesn't support tool calling?

You can continue this conversation by mentioning me in the message.

The tool call

{
  "role": "tool_calls",
  "time": 1737057643348,
  "message": "",
  "toolCalls": [
    {
      "id": "call_5LnMhRC3IgMAVgV1",
      "type": "function",
      "function": {
        "name": "retrieve_and_generate",
        "arguments": "{\"query\": \"経費精算 締め切りはいつ\"}"
      }
    }
  ],
  "secondsFromStart": 24.422
}

Worked and was received by the API - which is performing RAG and returning a jsonified response in the correct format described here: https://docs.vapi.ai/tools-calling

@Ben You must return the response in string format currently it is in record or object format which doesn't works. The tool expects the string that will be spoken out.

json
{
    "results": [
        {
            "toolCallId": "X",
            "result": "Y"
        }
    ]
}

If you want to store the result in message history, you can stringify the response in the format below, and the message content will be voiced out.

json
"results": [
    {
      "toolCallId": "<insert-your-tool-call-id-here>",
      "result": "<insert-tool-call-result-here>",
      "message": {
        "type": "request-complete",
        "role": "assistant",
        "content": "<insert-the-content-here>"
      }
    }
  ]
}

I see thank you, I will try this

@Ben no if you want LLM to decide what response to generate next then simply use the result property only.

@Shubham Bajaj please could you look into call ID 8d9efbd2-9a27-4240-9aa5-301093cc1a78 and help me understand why I am still hitting 'unexpected errors' with the Realtime API

{
  "name": "call_xlnxf7fo9OLkVAhK",
  "role": "tool_call_result",
  "time": 1738068305471,
  "result": "An unexpected error occurred.",
  "toolCallId": "call_xlnxf7fo9OLkVAhK",
  "secondsFromStart": 18.751
}

The output of the model has the JSON in format described, with a field 'result' which is a JSON serialised string, so the format matches that you describe as far as I can see I wonder if it could be related to certain tokens not being parsable by VAPI/OpenAI even after setting

result=json.dumps(content, ensure_ascii=False)

- but the unexpected error is hidden from me - I presume on your side you might be able to see what error specifically is occuring The other thing I notice is that sometimes VAPI sends this kind of message to the assistant conversation update:

{
          "role": "tool",
          "tool_call_id": "call_Sp57yvMnrz0v9aGj",
          "content": "Tool Result Still Pending But Proceed Further If Possible."
        },

And the bot then it maybe uses that as a queue to re-call the tool, resulting in a spam of tool calls (?) at least we see several tool calls occuring when we get unexpected errors. More generally, if the tool call is synchronous and not yet responded, I do not understand why we would be prompting the bot to "Proceed Further" --> that seems like a bug? How can I inform VAPI to block until the call returns? I've already set it as syncronous and a generous server timeout but that doesn't seem to afefect this message when using Realtime API.

> "content": "Tool Result Still Pending But Proceed Further If Possible." In messagesToOpenAIFormatted, whenever a tool call (i.e., a function call) lacks a corresponding tool-call-result in the conversation messages, the function inserts the placeholder text,* "Tool Result Still Pending But Proceed Further If Possible.".* This occurs when a tool call has been requested, yet the conversation data does not include the relevant "tool_call_result" object for that call. Consequently, the function replaces it with a placeholder result in the combined/merged chat history, allowing the remainder of the code or the LLM to manage it consistently, particularly in scenarios involving parallel tool calls. In simpler terms, this message appears because the code consolidates all tool calls into the final OpenAI-formatted conversation, and if any tool call is still awaiting a genuine result (or if no result was ever documented), it utilizes the placeholder to convey: “We do not currently possess a genuine tool response, but please continue if possible.”

@Shubham Bajaj thank you for the explanation, so these unexpected errors are result of multiple parallel calls by the LLM to the tool. With the same prompt, I do not get this with standard LLMs, it's likely the nature of the speech to speech model for Realtime API that means it can send multiple calls before waiting for a response. I've tested my API and I can respond to concurrent requests - from the point of view of my API server it's just receiving serial calls with short interval no errors reported, so I assume the unexpected error is occurring in that VAPI middleware when multiple calls are occuring with no response within a certain timeframe. Regarding solutions: 1. Managing concurrent calls - I'm using a Flask API which can respond to multiple concurrent calls and delivers a response for each 2. Adjust the prompts - Given the LLM has no concept of time, I'm not quite sure how one could adjust it to ensure it "waits" for a response (which is the behaviour we get for free with non-Realtime API it seems) . I assume the reason it is repeating the tool call is because while it is waiting for the response it is receiving an instruction from VAPI "Tool Result Still Pending But Proceed Further If Possible." and so it seems plausible the realtime model assumes the logical thing to do is call the same tool again. I could try and explicitly prompt it to ignore such request and wait until it receives a tool_call_result object before making any further speech mark maybe, but even better would be an option to disable ability of the LLM to perform parallel calls entirely somehow

> send multiple calls before waiting for a response. The LLM waits for all tool call response, you were expecting a one tool call instead you have receive multiple and returned the response for one of them only and not for the other because of these in logs you observed "An unexpected error occurred." > "Tool Result Still Pending But Proceed Further If Possible." Because of multiple tools calling we have to add this so LLM doesn't errors out if for some tool call reference is missing and add this placeholder gives LLM an percievence result will arrive later kinda promise.