Flying Cat
The AEO Guide, 2026 edition · Part two, Do
04

Chapter four of ten

Can AI read your site

Before any content work, confirm the engines' bots can get in, read what is on the page and find it in the indexes they search.

Buyers ask Should we let AI bots in?

By Maeva Cifuentes · Updated October 2026

Read this chapter as

The short answer

Let the search and live-fetch bots in. Treat the training bots as a separate business choice. Then check that your key facts sit in plain HTML, since the bots behind ChatGPT, Claude and Perplexity fetch pages without running JavaScript. Skip llms.txt and special markup until the basics are confirmed.

The gates a page has to pass

A page can only be retrieved and cited after it passes each of these in turn. A failure at any one makes the later chapters pointless for that page.

  1. The bot gets in

    Your robots.txt file and your firewall both let the engine's bots through.

  2. The bot can read it

    The facts sit in the page's HTML as text, visible without scripts, clicks or a login.

  3. The engine can find it

    The page is in the index the engine searches, with working links pointing to it.

If you decide

Ask one question before funding content or outreach. Has someone confirmed that the engines can read our site?

If you do the work

Run the checks in this chapter on your ten highest-value pages first. Product, pricing and comparison pages come before the blog.

Which bots to let in

Each engine runs several bots with different jobs. Blocking one has a different cost from blocking another, so the choice is made bot by bot.

Builds the search indexFetches a page live for a userCollects training data
OpenAIOAI-SearchBotChatGPT-UserGPTBot
AnthropicClaude-SearchBotClaude-UserClaudeBot
PerplexityPerplexityBotPerplexity-UserPerplexity says its bots stay out of training
GoogleGooglebot, which also feeds AI Overviews, AI Mode and GeminiGoogle-Agent and other user-triggered fetchersControlled by the Google-Extended token. It runs zero crawling of its own.

  • Search and live-fetch bots

    Let them in. OpenAI states that it leaves sites opted out of OAI-SearchBot out of ChatGPT search answers. Anthropic says a block on Claude-User stops retrieval for a user's query.

  • Training bots

    This is a business choice. Blocking GPTBot or setting Google-Extended leaves search untouched. Google says Google-Extended leaves a site's inclusion in Google Search unchanged.

  • Blanket rules

    Ahrefs found GPTBot blocked on 5.89% of about 140 million sites in 2025. Only 0.5% named it. The rest were sweeping rules that caught it along with everything else.

  • Changes take a day

    OpenAI and Perplexity both say a robots.txt change takes about 24 hours to register. DuckDuckGo's bot takes about 72.

Check yours

AI bot access check

Open yoursite.com/robots.txt, copy everything in it and paste it below. The check runs in your browser. Nothing you paste leaves this page.

    A rewritten file you can copy 

    
            
            
    Folders you closed to all bots stay closed to the AI bots here. Comments from your original file are left out. Ask whoever manages the site to review the file before it goes live, since a robots.txt change reaches every crawler.
    
          
    
          
    This reads robots.txt rules for the site's home path only. Firewall blocks and rendering problems are invisible to it, which is what the next two sections cover. Two live-fetch bots, ChatGPT-User and Perplexity-User, may fetch a page a user asks for even when robots.txt disallows them, according to their operators.
    
        
    

        
    
          
    If you decide
    Make the training question a deliberate policy with legal and brand in the room. Keep it separate from search access, where a block simply removes you from answers.
    
          
    If you do the work
    Look first for a blanket Disallow rule under the wildcard user agent, the one written with an asterisk. Then name each search bot you want in your file so a later edit leaves it in.
    
        
    
      
    

      
    
        
    Blocks you may have without knowing
    
        
    
          
    A clean robots.txt file can sit in front of a firewall that turns bots away. The settings live with whoever manages hosting or security. Marketing teams often learn about them late.
    
        
    
        
    
          
    5.5%
    of sites blocked GPTBot at the firewall while their robots.txt said nothing about it
    HasData, 2,096-site live test, July 2026, via PPC Land
    
          
    39.5%
    of sites that disallowed GPTBot in robots.txt still served it pages
    Same study
    
          
    22.6%
    of the top 10,000 sites block at least one AI bot. 6.6% block OAI-SearchBot
    Chris Humphrey, June 2026, 7,870 sites
    
        
    
        
      
          
    • Cloudflare
      Its defaults changed on September 15, 2026. New sites that run ads are now offered a preset that allows search bots, signals against training and blocks agent fetches on pages with ads. Sites that had the older "Block AI bots" switch on kept search access and moved to a no-training signal. The new Block setting also stops Googlebot, Bingbot and Applebot.
      • 
          
    • Other firewalls
      AWS WAF's bot rules block verified and unverified AI bots alike once switched on. Vercel's AI bot rules are off by default. Ask what is enabled.
      • 
          
    • Site builders
      Squarespace has an AI crawler setting that is off by default. Its list covers training bots and leaves OAI-SearchBot and PerplexityBot alone.
      • 
          
    • IP blocks
      Anthropic warns that blocking its IP addresses also stops its bots from reading your robots.txt, so your stated preferences go unseen.
      • 
        
    
        
    The reliable test is your server or CDN logs. Section seven shows what to look for.
    
        
    
          
    If you decide
    Give one person ownership of AI bot settings across the website, the CDN and security tools. Today these often sit with three teams.
    
          
    If you do the work
    Send your infrastructure team the list of search and live-fetch bots from the table above. Ask them to confirm each one is allowed at the firewall.
    
        
    
      
    

      
    
        
    What a bot can read
    
        
    
          
    Getting in is half of it. Many modern sites build their pages in the visitor's browser with JavaScript. A bot that fetches the page without running those scripts receives a nearly empty file.
    
        
    
        
    
          
    
            
    Runs JavaScriptEvidence
    
            
              
    GPTBot, OAI-SearchBot, ChatGPT-UserNoVercel and MERJ, December 2024, from one month of network traffic
    
              
    ClaudeBotNoSame study
    
              
    PerplexityBotNoSame study
    
              
    Googlebot, including for GeminiYesSame study, plus Google's Martin Splitt in May 2025
    
              
    ApplebotYesApple says it "may render" pages in a browser
    
              
    ChatGPT deep researchNoPeec AI, June 2026. The agent made zero clicks across about 200 runs, and a sample of its page reads matched a plain fetch word for word.
    
                
          
    
        
    
        
    
          
    
            
    At risk of going unread
    
            
      
              
    • Content built by scriptsPricing tables, feature grids and reviews that load after the page arrives.
      • 
              
    • Content behind a clickTabs, accordions and "load more" sections whose text is fetched on demand.
      • 
              
    • Facts held only in markupOne test found zero engines reading a price that existed only in JSON-LD.
      • 
              
    • Text inside imagesRead with lower accuracy or skipped. Alt text gets read as content.
      • 
              
    • Pages behind a loginGated documentation and community answers drop out silently.
      • 
            
    
          
    
          
    
            
    Safe choices
    
            
      
              
    • Server-rendered HTMLThe text is in the file the server sends. Google's own guidance still calls this "a great idea".
      • 
              
    • Tabs with text in the HTMLHidden by styling is fine, as long as the words arrive with the page.
      • 
              
    • Key facts as visible textPrice, plan limits, integrations and supported regions written out on the page.
      • 
              
    • HTML over PDFPDFs are readable. Microsoft advises HTML for critical details, since PDFs lack structure.
      • 
              
    • Public documentationHelp docs are among the pages engines cite, as the runs in chapter 3 showed.
      • 
            
    
          
    
        
    
        
      
          
    • View the page source
      Right-click and choose "View page source". Search for a sentence from the page. If it appears, a bot that skips scripts can read it.
      • 
          
    • Switch JavaScript off
      Reload the page with JavaScript disabled in your browser. What remains is close to what those bots see.
      • 
          
    • Ask the assistant
      Give ChatGPT or Claude the URL. Ask it to quote the first and last paragraph of the page.
      • 
          
    • Use Google's inspector
      URL Inspection in Search Console shows the rendered HTML Google has stored for a page.
      • 
        
    
        
    
          
    If you decide
    If the site is built as a single-page app, budget for server-side rendering or prerendering on the pages that sell. This is the one fix in the chapter that needs engineering time.
    
          
    If you do the work
    Test one page of each template. A pricing page, a product page, a comparison page, a docs page and a blog post cover the usual set.
    
        
    
      
    

      
    
        
    Getting found
    
        
    
          
    Engines retrieve from search indexes, as chapter 2 covered. A readable page still has to be in the index the engine uses.
    
        
    
        
      
          
    • Google
      For AI Overviews and AI Mode a page has to be indexed and eligible for a snippet. Google states "There are no additional technical requirements". Its newer optimization guide adds one condition, which is that the site must be included in generative AI features in Search Console.
      • 
          
    • Bing
      Seer Interactive found 87% of ChatGPT search citations matched Bing's top 20 results, against 56% for Google's, across 500+ citations in February 2025. Set up Bing Webmaster Tools.
      • 
          
    • Internal links
      OpenAI's deep research agent reaches pages through search results and links on the page. Pages with zero links pointing to them are hard to reach.
      • 
          
    • Dead ends
      About 34% of the pages ChatGPT's and Claude's bots requested returned a 404 in the Vercel study, against 8% for Googlebot. Redirect old URLs.
      • 
        
    
        
    Eli Schwartz adds two housekeeping items that waste a bot's visits. One is conflicting canonical tags. The other is crawl waste from filters, session IDs and tracking parameters.
    
      
    

      
    
        
    What to skip for now
    
        
    Several tactics get a lot of airtime and little support from the data so far.
    
        
    
          
    97%
    of llms.txt files received zero requests in a month of logs
    Ahrefs, 137,210 domains, May 2026
    
          
    ±
    Adding schema markup moved AI citations by 2 to 5%, inside the range of noise
    Ahrefs test of 1,885 pages, reported by Search Engine Journal, May 2026
    
          
    0
    studies we found that tie page speed to AI citations with data
    Speed thresholds in circulation are vendor advice
    
        
    
        
      
          
    • llms.txt
      Google's John Mueller said in June 2025 that "no AI system currently uses llms.txt". It is quick to add and unlikely to help today. We treat it as optional.
      • 
          
    • Markdown copies of pages
      Google and Bing both advise against separate versions for bots. Copies drift out of date and get neglected.
      • 
          
    • Schema markup
      Keep it and keep it accurate. Schema earns rich results in regular search and helps search engines understand a page. The evidence so far shows a small effect on AI citations. Google says its AI features need zero special markup, while Bing's Fabrice Canel says Microsoft's models do use it.
      • 
          
    • Speed targets
      Fast, error-free servers help Google crawl more. Specific millisecond targets for AI bots have zero published evidence behind them.
      • 
        
    
      
    

      
    
        
    Proof in your logs
    
        
    
          
    Your server or CDN logs record every bot visit. Analytics tools miss them, since bots skip the tracking script. We sort bot visits into three groups, which mirror the table at the top of this chapter.
    
        
    
        
    
          
    What AI bots are doing
    
          
    Training made up 79% of AI crawling while live fetches for users made up 3.2%
    
          
          
    Share of AI crawler requests by purpose across Cloudflare's network, July 2025. All bars share one scale from 0 to 100%. Source Cloudflare. The two smaller groups are the ones tied to answers, so a raw count of bot hits overstates how often engines retrieve your pages.
    
        
    
        
      
          
    • Which pages get fetched live
      Hits from ChatGPT-User, Claude-User and Perplexity-User show pages an engine pulled while answering someone.
      • 
          
    • Whether the bot is who it claims
      OpenAI, Anthropic and Perplexity publish their bots' IP ranges. Google supports a reverse DNS check. Imitators exist.
      • 
          
    • What response they got
      A run of 403 or 429 responses to a search bot points to a firewall block. A run of 404s points to broken links.
      • 
          
    • How it changes
      Chapter 8 covers bot visits as one of the signals to report over time.
      • 
        
    
        
    
          
    If you decide
    Ask for a monthly count of live fetches by engine on your key pages. It is the closest thing to seeing engines use your content.
    
          
    If you do the work
    Pull 30 days of logs and filter by the bot names in this chapter. Group them by job before you count anything.
    
        
    
      
    

      
    
        
    Other views
    
        
      
          
    • Without SEO there is no AEO
      Eli Schwartz argues that rendering counts for more now than it did under Googlebot alone, since AI bots run fewer scripts and tolerate less crawl waste.
      • 
          
    • Nothing extra required
      Google maintains that a page indexed for Search qualifies for its AI features, with the Search Console inclusion setting as the one addition.
      • 
          
    • Block and charge
      Cloudflare and many publishers treat access as leverage. News publishers block GPTBot at about 50%, according to HasData in July 2026. Cloudflare is testing pay per crawl.
      • 
          
    • Crawling returns little traffic
      Cloudflare measured about 1,091 pages crawled by OpenAI for each visitor it sent back in July 2025, against 5.4 for Google.
      • 
        
    
      
    
    
    
  
    

  
    
    
    Choose your next stop
    
    
    Where to go from here
    
    
    
      Chapter 5 · Practice routePrompt research
      Chapter 6 · Practice routeOn-site content
      Chapter 8 · Both routesMeasurement and reporting
    
    
  
    

  
    
    
    Sources