Beyond Prompt Monitoring: Why AI Bot Observability Matters for SEO and GEO

Why AI bot observability matters for SEO and GEO.

In my previous article, I wrote about a problem with AI prompt monitoring

One prompt is not the real user journey.

Prompt and browser monitoring help us understand what AI systems recommend, which sources they use and how answers change during a conversation.

But there is another layer I think enterprise SEO and GEO teams should monitor much more seriously:

AI bot activity.

Not once per month.

Not through an occasional server-log export.

And definitely not as one chart called “AI crawlers”.

For a large e-commerce platform, AI bot monitoring should become an observability layer: near real-time, segmented, bot-specific and connected with business data.

🤖 Not every AI bot means the same thing

This is probably the most important starting point.

According to OpenAI’s crawler documentation, for OpenAI alone, we can already distinguish several different purposes:

OAI-SearchBot supports search and helps surface websites in ChatGPT search features.
ChatGPT-User can fetch pages as a result of a user’s action in ChatGPT or a Custom GPT.
GPTBot is related to content that may be used for model training.

These are different signals.

If ChatGPT-User requests suddenly increase, that can be interesting because this agent is associated with user-triggered activity.

But it does not automatically mean:

“ChatGPT traffic increased by 30%.”

The requests may indicate more user-triggered fetching of our domain, but we still need referral, analytics and commercial data to understand what happened afterwards.

The same applies to OAI-SearchBot.

An increase can tell us that OpenAI is fetching more of our content for search-related purposes.

It does not prove that we received more citations, recommendations or revenue.

That distinction is fundamental.

Anthropic’s crawler documentation describes a similar separation with ClaudeBot, Claude-SearchBot and Claude-User.

And not every ecosystem works in exactly the same way. For example, Google documents Google-Extended as a robots.txt control token without its own HTTP user-agent string.

So even the phrase “monitor every AI bot separately” needs a technical qualification:

Monitor every identifiable crawler or user agent separately, and understand what each provider actually exposes.

📈 Response codes are the first layer - not the final dashboard

The simplest useful monitoring starts with HTTP responses.

For each relevant bot I want to know:

requests over time,
2xx responses,
redirects,
404s and other 4xx responses,
403 and 429 responses,
5xx errors,
affected hosts,
affected URL groups.

Even simple anomalies can be valuable.

A sudden increase in 5xx responses can expose an availability problem affecting a specific crawler.

A spike in 403 or 429 may indicate that security or rate-limiting rules are interfering with access.

A growing number of 404s can point towards outdated URLs, changed architecture or external references to content that no longer exists.

And a strong increase in 200 responses can also be interesting.

But I would not automatically call that positive.

It might mean increased crawling of commercially important pages.

It might also mean repeated crawling of useless parameters.

Context changes the interpretation.

Cloudflare’s AI Crawl Control documentation describes a similar model: crawler-specific activity, paths, operators and HTTP status-code distributions.

For an enterprise platform, I would take this further.

🔎 E-commerce needs taxonomy-level monitoring

A total request count for ChatGPT-User or Claude-SearchBot is not enough.

I want to know what they are accessing.

For example:

Electronics → Smartphones → Samsung
Beauty → Hair care
Travel → Hotels → Spain
Services → Massage → Chicago

If AI crawler activity suddenly increases across URLs associated with Samsung smartphones, that can become an interesting signal.

But there is an important limitation:

server logs normally show the requested URL, not the user’s original prompt.

Therefore I would say:

“We see increasing AI-bot activity around Samsung smartphone content.”

Not:

“Users are asking ChatGPT more often for ‘Samsung smartphone’.”

That second conclusion requires additional evidence.

For e-commerce, I would segment bot behaviour by:

market → business unit → category → subcategory → page type → entity → URL.

This can show where AI systems are actively accessing the platform, where activity is decreasing and where almost nothing is being fetched.

That last group can sometimes be the most interesting.

Why is one commercially important category receiving almost no AI crawler activity?
Is it difficult to discover?
Is internal linking weak?
Is the content inaccessible?
Is the section intentionally blocked?
Or is there simply no reason for the crawler to request it?

Monitoring identifies the question.

It does not automatically provide the answer.

🚨 Monthly log exports are not enough

Manual log analysis is still useful.

I use it for deeper investigations and forensic analysis.

But for enterprise e-commerce, exporting a few days of logs every few months is not an operating model.

I would build near-real-time monitoring with alerts.

Examples:

🚨 OAI-SearchBot 5xx rate increases above baseline.

🚨 ChatGPT-User suddenly receives 403 responses across one market.

🚨 Claude-SearchBot activity drops close to zero.

🚨 AI crawlers begin hitting millions of low-value parameter URLs.

🚨 A commercially important category loses most crawler activity after a deployment.

🚨 Requests suddenly increase significantly for one product family.

The correct threshold will differ by platform.

The important point is that an anomaly should reach the team when it happens—not several weeks later during an SEO audit.

🛡️ Sometimes the SEO problem is not in the SEO layer

This is an area I find particularly interesting in large organisations.

A crawler may be allowed in robots.txt while still being blocked elsewhere.

Between an AI crawler and the application there can be:

CDN → DDoS protection → WAF → bot management → load balancer → reverse proxy → application.

A rule anywhere in that chain can change the result.

Cloudflare explicitly documents cases where upstream WAF rules can block AI crawlers that are otherwise configured as allowed.

OpenAI also publishes IP ranges for its crawlers.

This creates an important operational question:

Do our robots.txt intentions match what bots actually receive at the network edge?

For me, this is where Technical SEO/GEO, Security, Network and Platform Engineering need to cooperate.

A robots.txt audit alone cannot answer it.

🧠 The bigger idea: build a Single Source of Truth

Bot activity becomes much more valuable when it stops living in its own dashboard.

For every normalized URL - and ideally the underlying entity - I would connect several data layers.

For example:

Bot layer

OAI-SearchBot requests
ChatGPT-User requests
GPTBot requests
Claude bots
HTTP status distribution
crawl frequency

Search & AI visibility

Google/Bing impressions and clicks
rankings
AI citations and mentions
AI referral traffic

Business

sessions
CVR
orders
GMV
revenue
margin
repeat purchase / loyalty

Platform

market
category
page type
canonical URL
product/service/merchant IDs
availability

Now the question changes.

Instead of:

“Which URLs did OAI-SearchBot crawl?”

we can ask:

“Which high-revenue categories with strong customer loyalty are underrepresented in AI crawling and AI visibility?”

Or:

“Where did ChatGPT-User activity increase together with AI referrals and commercial performance?”

Or:

“Which pages generate meaningful business value but are returning errors to AI crawlers?”

That is much closer to how I think modern SEO/GEO analytics should operate.

One important warning:

joining those datasets does not prove causation.

Bot activity, AI visibility, traffic and revenue can move together for many reasons.

The value of the Single Source of Truth is better diagnosis and prioritisation—not creating a fake attribution model.

🧩 Bot monitoring is only one layer

This connects directly with my previous article about prompt monitoring.

I would not build an AI Search measurement strategy around one dataset.

I would combine several layers:

1. Bot observability

What are AI systems requesting from our infrastructure?

2. Search and AI visibility monitoring

Where are we surfaced, cited or recommended?

3. User-journey evaluation

What happens through realistic multi-turn conversations and follow-up questions?

4. Referral and product analytics

Do users actually arrive and interact with the platform?

5. Business outcomes

Does the journey produce useful transactions and customer value?

Each layer answers a different question.

Together they provide a much stronger picture than any single “AI visibility score”.

My view

AI bot monitoring should become part of the standard observability stack for enterprise SEO and GEO.

Not because more crawling automatically means more visibility.

But because it tells us something we cannot get from prompt monitoring alone:

how AI systems actually interact with our infrastructure and content.

Combine that with search visibility, realistic AI journey testing, traffic, CVR, revenue and loyalty - and it becomes much more than a crawler dashboard.

It becomes another signal for deciding:

where the platform is strong, where it is failing and where SEO/GEO work can create real business value.

That is the direction I find much more interesting than counting AI mentions alone.

References

OpenAI Crawlers - https://developers.openai.com/api/docs/bots - Documents the different roles of OAI-SearchBot, ChatGPT-User and GPTBot, including published IP ranges.
Anthropic: Claude web crawlers - https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler - Separates ClaudeBot, Claude-SearchBot and Claude-User by purpose.
Cloudflare: Analyze AI traffic - https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/ - Shows per-crawler, operator, hostname, path and status-code analysis, including programmatic access.
Cloudflare: AI Crawl Control and WAF - https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-waf/ - Explains how WAF rules can affect crawler access before other crawler controls.
Cloudflare: Verified bots - https://developers.cloudflare.com/bots/concepts/bot/verified-bots/ - Useful context for verifying bot identity beyond trusting a user-agent string.
Google: Common crawlers and Google-Extended - https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers - Documents Google crawler roles and the important limitation that Google-Extended does not have a separate HTTP user agent.

My previous post: why AI monitoring should evaluate realistic user journeys rather than isolated prompts.