Identify the request, the response and the control involved before changing access rules.
A robots.txt allowance doesn't clear a Cloudflare block
A public page can open in your browser while an AI crawler receives a security challenge or a blocked response. Start by recording what that particular request received. The page you see in your own browser may have different cookies, identity and access conditions.
Robots.txt communicates crawl preferences to cooperating clients. Cloudflare's security controls decide how to handle an incoming request. Cloudflare documents that a crawler set to Allow in AI Crawl Control can still be blocked by an earlier WAF custom rule.
For other delivery or content-structure problems, start with the broader access guide.
Decide which client should reach which content
Cloudflare's AI Crawler and Agent access policies distinguish Search, Agent and Training behaviours. A search crawler collecting information for later answers has a different purpose from a client acting for someone now. Decide which public resources each intended client should be able to retrieve.
Read the actions configured on your own domain. Training Disallow communicates a robots.txt preference; its interaction with edge blocking depends on Bot Preference Sync and whether the crawler also performs search. It isn't interchangeable with Block. Use the current policy documentation when reviewing a mixed-purpose crawler.
Write down the hostname, public path, intended client and purpose of the allowance. Keep account pages and other restricted resources outside that public-access decision.
Inspect the response that replaced your content
Save the affected URL, request time with its time zone, method, response status, response headers and a relevant body excerpt. Include the final URL if redirects occurred. Check whether the response actually contains the article, product details or service description the client needs.
Cloudflare documents a challenge response header: cf-mitigated with the value challenge. Challenge Pages use the text/html content type even when the request expected another resource type. These are useful response clues when a client receives an interstitial page instead of the requested content.
A 403 status alone doesn't identify the responsible control. Cloudflare's error guidance covers origin permission failures, Cloudflare security actions and early infrastructure errors. Branding can help the investigation, but an unstyled page alone can't tell you whether the origin returned the error.
| Observed response | What it establishes | Next check |
|---|---|---|
| cf-mitigated: challenge | Cloudflare returned a Challenge Page. | Find the matching request and security action; inspect the content the client received. |
| 403 Forbidden | The request was refused. The status alone doesn't identify why. | Correlate the response with available Cloudflare events and origin logs. |
| 200 with missing page content | The request succeeded at the HTTP level, but the required information may still be absent. | Inspect the response body and investigate delivery or rendering dependencies. |
These are diagnostic cases to check on your own website, not results from a customer audit.
Keep a synthetic check separate from a verified crawler visit
Changing a request's user-agent to a crawler name tells you how your server handled that header from your machine. It doesn't authenticate the request as that provider. Cloudflare's verified-bot documentation describes identification methods that involve more than the name in a header, including signatures and IP validation.
Label your own request as a synthetic access check. Keep it separate from a recorded provider request, with its own time, source and result. Don't treat a successful browser visit with an existing session as evidence that the crawler has access.
For a real crawler failure, retain the available identity evidence and correlate it with the response. If that evidence is unavailable, keep the identity unverified rather than creating an allowance based only on a self-declared user-agent.
Find the security action for the affected request
Use Cloudflare's Security Events documentation to locate the available request details and filters. Start with a narrow time range around the failure, then match the URL and available request identifiers. Record the action, security product and rule details that apply to the request.
Security Events covers traffic flagged or actioned by security products. It isn't an inventory of every incoming request. Features and retention vary, sampled logs can omit events, and one request may have more than one security event. An empty search doesn't establish that access was allowed.
Compare the event with the response you saved. If the evidence points to the origin, investigate its permissions and application rules with the website owner. Keep the cause unresolved when the available records don't explain the failure.
Check WAF rule order alongside AI Crawl Control
Cloudflare explains how AI Crawl Control works with WAF: its selected crawler blocks use a WAF custom rule. Other rules can execute before that rule. Setting a crawler to Allow doesn't undo an earlier security action, and the earlier action may not appear in AI Crawl Control analytics.
Review the rule responsible for the recorded request against the access you intended. A hostname, path or identity condition may be broader than the public resource decision requires. Discuss a correction with the person responsible for that control, rather than disabling bot protection across the domain.
Use the Crawlers interface for the policies it supports. Cloudflare notes that direct WAF edits aren't reflected back in the AI Crawl Control dashboard. If a direct rule edit is necessary, record its scope and check the effective rules as well as the displayed crawler policy.
Repeat the request and check the information delivered
After an authorised change, repeat the affected retrieval path. Keep the URL, method and client conditions comparable, then save the new response and the change time. If the original failure involved a real provider crawler, a successful synthetic check alone can't establish that the provider now has access.
Check the information needed for the task. A 200 response with an empty application shell still leaves a content-delivery problem. Confirm that the relevant text and its conditions are present, and investigate JavaScript dependencies separately when they aren't.
Also verify that deliberately restricted resources remain restricted. Record the pages and methods covered by the retest. Improved delivery under those conditions doesn't establish indexing, AI citations or recommendations; those outcomes need their own observations.
- The intended public URL returns its required content through the tested retrieval path.
- The response no longer contains the unexpected challenge or refusal observed before the change.
- The relevant security action and access conditions have been recorded where available.
- Private resources and intentionally blocked access remain protected.
Check the content your website delivers
Create a free Tityra account and add your website. Eligible websites can request a free Initial Audit covering up to 10 pages.
Tityra provides website content audits for better Agent Experience (AX). Documented findings connect observed content access, delivery, structure and context issues to supporting evidence and practical recommendations within the assessed scope.


