Hi, Starting January 4th, I've noticed that my assistants are not always adhering to the system prompt that is sent in the model.messages array when creating our transient assistants. When the assistant adhere's to the prompt they take on the role of a prospective buyer (e.g. HR Director at Acme Manufacturing). When the assistant doesn't adhere to the prompt it identifies itself as an "AI assistant" or "chatgpt assistant" which goes directly against the intended purpose of the assistant When we noticed this issue yesterday, we would see it on every call. As a temporary resolution we changed the transcriber from gladia to deepgram which seemed to help. but we are still seeing the issue sporadically

transcriber: {
    provider: "gladia",
    model: "fast",
    language: "en",
    receivePartialTranscripts: false
}

transcriber: {
  provider: "deepgram",
  language: "en",
  model: "nova-2-general"
},

We have NOT changed our internal system prompt logic in the past month or so. so I can't see how that would cause the issue. The assistant config is exactly the same for both calls, yet the assistant has vastly different interactions with users Failing call id: 019b944c-9e3b-7bb9-9501-f18d9dfd2dbe Working call id: 019b943b-51ae-711d-9859-ce6752f73396 Thing we are hypothesizing could be the cause: - Could partial transcripts being sent from the transcriber be the cause of this issue? I'm not sure if Deepgram has a way to accept partial transcripts - is there a context limit to the transcriber model that could be causing this issue?

Actionable Step: To diagnose why your assistants are not consistently following the system prompt, use the Vapi dashboard to compare the failing and working calls: 1. Go to

Observe > Call Logs

in the [Vapi dashboard](https://dashboard.vapi.ai/). 2. Review the transcripts and tool execution for both call IDs. 3. Look for differences in how the system prompt was processed or if any errors occurred. This will help pinpoint where the prompt adherence breaks down and guide your next troubleshooting step. Reference: [Vapi Debugging Guide](https://docs.vapi.ai/debugging) Source: - [Debugging voice agents](https://docs.vapi.ai/debugging)

we have compared system prompt, assistant configs, and output and they are exactly the same but can have totally different experiences with the AI's role assignment

OH, gReAt IdEa!!! I nEvEr wOuLd hAvE tHoUgHt oF tHaT!🤡

Can we get some help please?

@User PLEASE HELP!!! We are completely stuck here!

Hey guys thanks for providing the call IDs, we're looking into this and will update you soon!

The working call correctly follows the persona (“David Usher”) even though the config matches the failing call. The key difference we’re seeing is that your persona instructions are being sent with \*\*`role: "assistant"` instead of

role: "system"

. Using

assistant

can cause the model to treat the persona as a prior conversational turn rather than authoritative instructions, which explains the inconsistent behavior. Recommendation: move the persona prompt to

role: "system"

in

model.messages

. This should significantly improve consistency and prevent the model from identifying itself as an AI.

Thanks for the response, Kyle We have been passing the role as "assistant" for over a year now with no issues. I can't recall why we do this, but I know we followed vapi documentation when setting up our system in late 2024 For context: - We tried your suggestion as one of our first attempts at a solution prior to making this support ticket (Tuesday) and it didn't work for the provider we were using (gladia). After a full day of testing, we switched to deepgram, which worked for half a day and then stopped working (we have at least 50 calls that failed). After another full day of testing yesterday, we made the switch to assembly ai and have yet to see an invalid response (using the "assistant" role). I ran more tests this morning with each of the three transcriber providers we've used in the past year and tested with the "assistant" role versus the "system" role. The results are mixed but there are interesting behaviors - Deepgram is working again (when role set as "assistant" or role set as "system") - assembly only works with role "assistant" - gladia only works with role "system" I am extremely hesitant to change our production code again when the root cause is unclear and the outcomes are a mixed bag. btw, our model provider has always been openai gpt-4o:

"model": {
    "model": "gpt-4o",
    "messages": [
      {
        "role": "system",
        "content": "\n      The following defines the purpose of the call..."
      }
    ],
    "provider": "openai",
    "maxTokens": 5000,
    "temperature": 0.3,
    "knowledgeBase": {
      "topK": 1,
      "fileIds": [],
      "provider": "canonical"
    },
    "emotionRecognitionEnabled": true
},

With this additional information, do you have any suggestions? https://cdn.discordapp.com/attachments/1458157871576649758/1458817473666224233/image.png?ex=69610525&is=695fb3a5&hm=f9f2298ab99948671fcc87f70b87ca6148b2a036c43e877fb6fc4223342f9063&

Thanks for the detailed testing doc. I’ll need some time to fully review it. Could you also confirm if deepgram is currently stable enough for production? In the meantime, we don’t want you to change much of your existing code, and we recommend staying on Deepgram, since it appears to be working as intended for your use case. For most transcribers, using the

system

role for assistant instructions is still considered best practice. Also worth noting: the VAPI platform has evolved quite a bit since 2024. While most of your implementation should remain valid, we strongly recommend reviewing the changelog to ensure everything is behaving as you expect.

Deepgram keeps flipping from working to not working so we're going to stick with Assembly (assistant prompt) for the time being - it has yet to fail this week. example from today: - deepgram failed: https://dashboard.vapi.ai/calls/019bae83-8315-7ddd-bfe6-a4e539881058 - assembly worked for the same config other than the transcriber: https://dashboard.vapi.ai/calls/019bae88-1f19-7ee3-987f-107931d99d03 I would just like some clarity into what is actually happening behind the scenes that is causing these behaviors. note taken about the evolution of vapi and reviewing the changelog

The role matters a lot here. •

system

= authoritative instructions •

assistant

= previous AI responses •

user

= human input When persona instructions are sent as

assistant

, the model can treat them as optional context rather than rules. That leads to identity confusion (defaulting to “I’m an AI assistant”) and inconsistent behavior. We also recommend tightening how the identity is defined. Right now the prompt is written in third-person (e.g. “Sarah is the Head of Product…”), which can read like it’s describing someone else. Instead, define the identity directly in first-person in the system prompt, for example: “You are Sarah, the Head of Product at … You are responsible for … You need a tool to …” This makes it unambiguous who the assistant is and significantly improves consistency.

Thanks Kyle! we'll make these changes and test again

Sounds good. Let us know what the results are and if you need additional support

The second person approach seems to increase the rate of role swapping - where the AI thinks it is the seller vs the buyer. We'll just stick with Assembly AI and our current workflow for the time being. Thanks for the help Kyle

No worries. Glad we could provide some assistance