AI Prompt Monitoring Is Broken, Because the Prompt Isn’t the User

A repeatable API benchmark can still miss the AI experience your customers actually see.

We still do not have a complete way to monitor AI visibility and performance.

Many solutions track prompts, query models through APIs, capture their responses, and calculate metrics such as brand mentions, citations, or competitor visibility.

This is useful. It is also only part of the picture.

The fundamental problem is that a model response is not the same as a user’s experience inside an AI product.

The problem with API-only monitoring

Let’s use ChatGPT as an example.

A monitoring tool sends a prompt through an API and records the answer. The process is repeatable, relatively easy to scale, and suitable for controlled experiments.

But what happens when the same person enters the same prompt inside ChatGPT?

The response may be influenced by much more than the latest message.

When enabled, ChatGPT Memory can use relevant information from saved memories and previous conversations. Custom instructions can shape responses across chats. Projects can introduce their own instructions, files, and conversation history. Project instructions override global custom instructions, while project memory settings affect which conversations can provide context. These are documented product capabilities, not theoretical edge cases. See OpenAI’s documentation on Memory in ChatGPT and Projects in ChatGPT.

An isolated API test does not automatically reproduce a particular user’s saved memories, project context, or personal conversation history. An API-based evaluation can explicitly reconstruct conversation state and configure tools, but that is different from automatically inheriting the context of someone’s ChatGPT session. OpenAI documents this distinction in its guide to conversation state in the API.

A prompt is often the beginning of a task, not the whole test

Real users rarely behave like a fixed benchmark consisting of one prompt and one answer.

They ask a question, read the response, refine their requirements, challenge a recommendation, request a comparison, and continue until they have enough information to act.

For example:

“What are the best options for this product category?”
“How does Brand A compare with Brand B?”
“Which one is best for my requirements?”
“Where can I buy it, and what should I check before ordering?”

The journey matters.

A brand may not appear in the first answer but could enter the conversation during a comparison. A competitor may dominate the initial recommendations but disappear when the user introduces a specific requirement. A citation may appear once and then be replaced by another source as the conversation develops.

Monitoring only the first answer misses these transitions.

The more useful unit of analysis is therefore not always an individual prompt. It can be the multi-turn journey from initial intent to decision or action.

What should we monitor instead?

I see five particularly valuable areas.

1. Brand and competitor visibility

Does the model mention the brand? In what context? Is it recommended, compared, criticised, or omitted? How often do competitors appear for equivalent intents?

Mention frequency alone is not enough. Context and position in the decision process matter.

2. The query funnel

Monitor different stages of intent, from general discovery and category research to comparison, transactional questions, and post-purchase support.

A brand might be visible for broad informational prompts but absent when the user is ready to choose a product.

3. Citations and source visibility

Which websites are cited? Does the answer link to the brand’s own content, third-party reviews, marketplaces, or competitors? How does source selection change across prompts and follow-up turns?

Being mentioned and being used as a source are different outcomes.

4. Query expansion and conversational paths

What follow-up questions does the interaction produce? Which requirements become important? Does the conversation move towards a comparison, another product category, a purchase, or a different source?

These paths can reveal how AI systems frame a category and where a brand gains or loses consideration.

5. Model and feature variability

Repeat the same scenarios across relevant model versions, reasoning settings, and capabilities.

For example, compare available GPT-5.x and GPT-6 models, where applicable, under different thinking levels. Test scenarios with and without web search, and include image generation or other tools when they are relevant to the user’s task.

The exact configuration matters. A result obtained with one model and one reasoning setting should not automatically be treated as representative of every configuration.

Why realistic browser-based monitoring matters

For user-facing AI products, I believe the closest practical approximation is controlled, UI-based monitoring that reproduces realistic user actions.

Instead of sending one prompt and stopping, a test follows a predefined interaction:

Enter a prompt → inspect the answer → ask a relevant follow-up → continue the task → record the outcome.

This makes it possible to observe not only what the model says but also how the interaction develops.

It also lets us test product-level behaviour that a simple, single-turn model benchmark may miss.

However, browser automation is not a magic solution. A simulated user does not reproduce every real user’s memories, instructions, history, subscription, or personalisation settings. Nor can a finite set of tests represent every possible experience.

That is why I would combine controlled API experiments, UI-based task simulations, and appropriately collected real-world user signals rather than relying on a single measurement method.

What does “AI performance” actually mean?

Before choosing a monitoring tool, we should define the outcome we want to measure.

Are we measuring brand mentions? Recommendations? Citations? Product discovery? The quality of information presented? Or the user’s ability to complete a task?

These are related but different questions.

A single visibility score can conceal important differences between them. A repeatable benchmark can be excellent for comparing controlled conditions and still be insufficient for understanding the entire user experience.

There is also an important limitation: no monitoring setup can eliminate user personalisation. The goal is not to find one universal answer for every user. It is to measure consistent benchmarks, examine relevant variations, and understand how representative user journeys differ.

My conclusion

AI visibility monitoring needs to evolve from prompt snapshots towards scenario-based evaluation.

APIs remain valuable for controlled, scalable experiments. UI-based monitoring helps us study interactions in the actual product environment. Real-world evidence helps us understand where our simulations fall short.

The next step is not simply collecting more prompts. It is measuring how AI answers, sources, recommendations, and follow-up interactions shape the path from discovery to decision.

How are you measuring AI visibility today: one prompt and one answer, or complete multi-turn user journeys?

AI Prompt Monitoring Is Broken, Because the Prompt Isn't the User