I think the difficulty is because I'm using a custom llm, i.e

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
request_data["tools"] = tools
request_data["tool_choice"] = "auto"
client.chat.completions.create(**modified_request_data)

The openAI chat completions doesn't have those functionality I believe: https://platform.openai.com/docs/guides/function-calling