Skip to content
Tityra
Login
Start for free
HomeAboutWho It's For
  • Publishers & Media
  • Ecommerce & Retail
  • SaaS & Technology
  • Agencies
  • Government & Public Sector
  • Financial Services & Insurance
PricingFAQBlog
info@tityra.com
XLinkedIn
  1. Home
  2. /
  3. Blog
  4. /
  5. Content access

Why AI crawlers are
blocked by Cloudflare

Check why an AI crawler receives a Cloudflare block or challenge, find the rule involved and verify access to the public content you intend to share.

6 min read

Doğu Abaris · 15 November 2026

Read the article

Used by many

  • Onedio
  • Maçkolik
  • Webtekno
  • Yemek.com
  • Mynet
5/51 review on G2
Start for free

UNDERSTAND CONTENT DELIVERY

Learn what a tested client receives.

FOLLOW THE EVIDENCE

See how observations become findings.

PUT GUIDANCE INTO PRACTICE

Find changes your team can investigate.

Read the article

In this article

A robots.txt allowance doesn't clear a Cloudflare blockDecide which client should reach which contentInspect the response that replaced your contentKeep a synthetic check separate from a verified crawler visitFind the security action for the affected requestCheck WAF rule order alongside AI Crawl ControlRepeat the request and check the information deliveredCheck the content your website delivers
How we audit
Table of Contents
A robots.txt allowance doesn't clear a Cloudflare blockDecide which client should reach which contentInspect the response that replaced your contentKeep a synthetic check separate from a verified crawler visitFind the security action for the affected requestCheck WAF rule order alongside AI Crawl ControlRepeat the request and check the information deliveredCheck the content your website delivers
How we audit

Identify the request, the response and the control involved before changing access rules.

Copy linkA robots.txt allowance doesn't clear a Cloudflare block

A public page can open in your browser while an AI crawler receives a security challenge or a blocked response. Start by recording what that particular request received. The page you see in your own browser may have different cookies, identity and access conditions.

Robots.txt communicates crawl preferences to cooperating clients. Cloudflare's security controls decide how to handle an incoming request. Cloudflare documents that a crawler set to Allow in AI Crawl Control can still be blocked by an earlier WAF custom rule.

For other delivery or content-structure problems, start with the broader access guide.

Why AI agents may miss website content

Copy linkDecide which client should reach which content

Cloudflare's AI Crawler and Agent access policies distinguish Search, Agent and Training behaviours. A search crawler collecting information for later answers has a different purpose from a client acting for someone now. Decide which public resources each intended client should be able to retrieve.

Read the actions configured on your own domain. Training Disallow communicates a robots.txt preference; its interaction with edge blocking depends on Bot Preference Sync and whether the crawler also performs search. It isn't interchangeable with Block. Use the current policy documentation when reviewing a mixed-purpose crawler.

Write down the hostname, public path, intended client and purpose of the allowance. Keep account pages and other restricted resources outside that public-access decision.

Copy linkInspect the response that replaced your content

Save the affected URL, request time with its time zone, method, response status, response headers and a relevant body excerpt. Include the final URL if redirects occurred. Check whether the response actually contains the article, product details or service description the client needs.

Cloudflare documents a challenge response header: cf-mitigated with the value challenge. Challenge Pages use the text/html content type even when the request expected another resource type. These are useful response clues when a client receives an interstitial page instead of the requested content.

A 403 status alone doesn't identify the responsible control. Cloudflare's error guidance covers origin permission failures, Cloudflare security actions and early infrastructure errors. Branding can help the investigation, but an unstyled page alone can't tell you whether the origin returned the error.

Response clues and the next thing to check
Observed responseWhat it establishesNext check
cf-mitigated: challengeCloudflare returned a Challenge Page.Find the matching request and security action; inspect the content the client received.
403 ForbiddenThe request was refused. The status alone doesn't identify why.Correlate the response with available Cloudflare events and origin logs.
200 with missing page contentThe request succeeded at the HTTP level, but the required information may still be absent.Inspect the response body and investigate delivery or rendering dependencies.

These are diagnostic cases to check on your own website, not results from a customer audit.

Cloudflare: investigate a 403 response

Copy linkKeep a synthetic check separate from a verified crawler visit

Changing a request's user-agent to a crawler name tells you how your server handled that header from your machine. It doesn't authenticate the request as that provider. Cloudflare's verified-bot documentation describes identification methods that involve more than the name in a header, including signatures and IP validation.

Label your own request as a synthetic access check. Keep it separate from a recorded provider request, with its own time, source and result. Don't treat a successful browser visit with an existing session as evidence that the crawler has access.

For a real crawler failure, retain the available identity evidence and correlate it with the response. If that evidence is unavailable, keep the identity unverified rather than creating an allowance based only on a self-declared user-agent.

Copy linkFind the security action for the affected request

Use Cloudflare's Security Events documentation to locate the available request details and filters. Start with a narrow time range around the failure, then match the URL and available request identifiers. Record the action, security product and rule details that apply to the request.

Security Events covers traffic flagged or actioned by security products. It isn't an inventory of every incoming request. Features and retention vary, sampled logs can omit events, and one request may have more than one security event. An empty search doesn't establish that access was allowed.

Compare the event with the response you saved. If the evidence points to the origin, investigate its permissions and application rules with the website owner. Keep the cause unresolved when the available records don't explain the failure.

Copy linkCheck WAF rule order alongside AI Crawl Control

Cloudflare explains how AI Crawl Control works with WAF: its selected crawler blocks use a WAF custom rule. Other rules can execute before that rule. Setting a crawler to Allow doesn't undo an earlier security action, and the earlier action may not appear in AI Crawl Control analytics.

Review the rule responsible for the recorded request against the access you intended. A hostname, path or identity condition may be broader than the public resource decision requires. Discuss a correction with the person responsible for that control, rather than disabling bot protection across the domain.

Use the Crawlers interface for the policies it supports. Cloudflare notes that direct WAF edits aren't reflected back in the AI Crawl Control dashboard. If a direct rule edit is necessary, record its scope and check the effective rules as well as the displayed crawler policy.

Copy linkRepeat the request and check the information delivered

After an authorised change, repeat the affected retrieval path. Keep the URL, method and client conditions comparable, then save the new response and the change time. If the original failure involved a real provider crawler, a successful synthetic check alone can't establish that the provider now has access.

Check the information needed for the task. A 200 response with an empty application shell still leaves a content-delivery problem. Confirm that the relevant text and its conditions are present, and investigate JavaScript dependencies separately when they aren't.

Also verify that deliberately restricted resources remain restricted. Record the pages and methods covered by the retest. Improved delivery under those conditions doesn't establish indexing, AI citations or recommendations; those outcomes need their own observations.

  • The intended public URL returns its required content through the tested retrieval path.
  • The response no longer contains the unexpected challenge or refusal observed before the change.
  • The relevant security action and access conditions have been recorded where available.
  • Private resources and intentionally blocked access remain protected.

Check content that depends on JavaScript

Copy linkCheck the content your website delivers

Create a free Tityra account and add your website. Eligible websites can request a free Initial Audit covering up to 10 pages.

Tityra provides website content audits for better Agent Experience (AX). Documented findings connect observed content access, delivery, structure and context issues to supporting evidence and practical recommendations within the assessed scope.

Copy linkSources and further reading

  1. Cloudflare: AI Crawl Control with WAF
  2. Cloudflare: AI Crawler and Agent access policies
  3. Cloudflare: detect a Challenge Page response
  4. Cloudflare: Error 403
  5. Cloudflare: verified bots
  6. Cloudflare: Security Events
Doğu Abaris

About the author

Doğu Abaris

Doğu Abaris is a software and data engineer and a co-founder of Tityra. For more than a decade, he has built maintainable software for production, working across front-end and back-end development. He also contributes to open-source software and web standards.

More about Doğu

See what your website delivers.

Choose an agreed scope and get findings your team can act on.

Start for free
Share this article

Share this article

Keep exploring

Content access

Why AI agents may miss website content

A page can look complete in a browser while another client misses its content or context. Compare responses to find where delivery or interpretation fails.

Read article

Content access

Do AI crawlers render JavaScript?

What major providers document about JavaScript rendering, what crawler studies found and how to interpret the evidence for your website.

Audit practice

How to audit website content for AI agents

A website content audit reviews the content a tested client receives and how it is structured. Use recorded observations to specify a change your team can verify.

Turn findings into
clear next steps.

Start for free
BlogAudit methodologyAPI docsMCP docsCLIGitHubContact

TITYRA LTD

120 Main Street
Largs, Scotland, KA30 8JN
info@tityra.com
HomeAbout
Who It's For
  • Publishers & Media
  • Ecommerce & Retail
  • SaaS & Technology
  • Agencies
  • Government & Public Sector
  • Financial Services & Insurance
PricingFAQ

© 2026 TITYRA LTD. All rights reserved. Registered in Scotland, company SC908253.

PrivacyTerms and ConditionsCookie informationCookie preferences

We value your privacy

By clicking "Accept all cookies", you agree to the storing of optional analytics cookies on your device to help us understand site usage and improve our website. You may click the customise button to choose which optional cookies you would like to allow.

Strictly necessary (2)

We save your cookie choice in this browser for six months, so you don't have to choose on every visit. This storage is always on.

Show detailsHide details for strictly necessary
  • Name
    tityra_analytics_consent_v1
    Provider
    Tityra (tityra.com)
    Type
    HTML local storage
    Maximum storage duration
    Persistent

    Remembers whether you accepted or rejected analytics. Your choice is valid for 180 days (six months), after which we ask again. The stored record remains until it is replaced or you clear your browser storage.

  • Name
    tityra_analytics_consent_v1_refusal
    Provider
    Tityra (tityra.com)
    Type
    HTML session storage
    Maximum storage duration
    Browser tab session

    Keeps analytics off if your browser can't save a rejection in local storage. This fallback is only used when needed and is cleared when a subsequent choice is successfully saved.

Google Analytics helps us understand page visits, content engagement, interest in audit packages and account signup link clicks. It only loads after you accept. Advertising features are disabled.

Show detailsHide details for analytics
  • Name
    _ga
    Provider
    Google Analytics (tityra.com)
    Type
    HTTP cookie
    Maximum storage duration
    Up to 180 days (six months)

    Distinguishes visitors to help us understand how the website is used. Set only after you accept analytics; advertising features are disabled.

  • Name
    _ga_*
    Provider
    Google Analytics (tityra.com)
    Type
    HTTP cookie
    Maximum storage duration
    Up to 180 days (six months)

    Maintains analytics session information, such as visits and interactions with the website. Set only after you accept analytics. The asterisk represents the identifier in the cookie name.

You control whether optional analytics runs on this website. We remember your choice for six months, and you can change it at any time using Cookie preferences in the footer.

Rejecting or withdrawing consent stops analytics and removes the Google Analytics cookies this website can access.

The details list the name, provider, purpose, type and storage duration of each entry. Contact details and private account information aren't included in website analytics.

Counts describe the entries listed, including the fallback and cookie name pattern, rather than cookies currently stored on your device.

Read our cookie information