Chapter four of ten
Before any content work, confirm the engines' bots can get in, read what is on the page and find it in the indexes they search.
Buyers ask Should we let AI bots in?
By Maeva Cifuentes · Updated October 2026
Who is allowed through the door
Text it can read
The short answer
Each AI company runs several bots with different jobs. Some build a search index, some fetch a page live when a user asks a question, and some collect training data. Blocking the first two removes you from answers, while blocking the third mainly affects what future models learn. The bots behind ChatGPT, Claude and Perplexity read the raw HTML of a page without running JavaScript, so anything that loads through scripts may stay invisible to them. Newer ideas like llms.txt files and AI-specific markup have little evidence behind them so far.
A page can only be retrieved and cited after it passes each of these in turn. A failure at any one makes the later chapters pointless for that page.
Your robots.txt file and your firewall both let the engine's bots through.
The content is in the page's HTML as text, visible without scripts, clicks or a login.
The page is in the index the engine searches, with working links pointing to it.
A technical audit can help AI visibility.
Review robots.txt and any Cloudflare access, as that can also block bots.
Each engine runs several bots with different jobs. Blocking one has a different cost from blocking another, so the choice is made bot by bot.
| Builds the search index | Fetches a page live for a user | Collects training data | |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | Perplexity says its bots stay out of training |
| Googlebot, which also feeds AI Overviews, AI Mode and Gemini | Google-Agent and other user-triggered fetchers | Controlled by the Google-Extended token. It runs zero crawling of its own. |
Let them in. OpenAI states that it leaves sites opted out of OAI-SearchBot out of ChatGPT search answers. Anthropic says a block on Claude-User stops retrieval for a user's query.
This is a business choice. Blocking GPTBot or setting Google-Extended leaves search untouched. Google says Google-Extended leaves a site's inclusion in Google Search unchanged. People who sell content or who have very large websites the bots WANT to crawl may decide to block training bots.
Ahrefs found GPTBot blocked on 5.89% of about 140 million sites in 2025. Only 0.5% named it. The rest were sweeping rules that caught it along with everything else.
OpenAI and Perplexity both say a robots.txt change takes about 24 hours to register. DuckDuckGo's bot takes about 72.
Check yours
Open yoursite.com/robots.txt, copy everything in it and paste it below. This won't save any of your data.
Below, you'll find a new robots.txt file that lets the AI bots in. If your current robots.txt blocks a folder for every bot, such as /admin/ or /checkout/, the new file keeps that folder blocked for the AI bots too. Any notes or comments in your original file get dropped. Have whoever manages your website check the new file before it goes live, because robots.txt affects every crawler, Google included.
This reads robots.txt rules for the site's home path only. Firewall blocks and rendering problems are invisible to it, which is what the next two sections cover. Two live-fetch bots, ChatGPT-User and Perplexity-User, may fetch a page a user asks for even when robots.txt disallows them, according to their operators.
Decide with your team whether you'd want to allow bots to train on your data. It helps you get into the training data, but also puts you at risk of AI plagiarism.
Look first for a blanket Disallow rule under the wildcard user agent, the one written with an asterisk. Then name each search bot you want in your file so a later edit leaves it in.
You can have a clean robots.txt file, but still have a firewall that turns bots away. Whoever manages hosting or security can check this for you (hopefully, not late!).
of sites blocked GPTBot at the firewall while their robots.txt said nothing about it
HasData, 2,096-site live test, July 2026, via PPC Landof sites that disallowed GPTBot in robots.txt still served it pages
Same studyof the top 10,000 sites block at least one AI bot. 6.6% block OAI-SearchBot
Chris Humphrey, June 2026, 7,870 sitesIts defaults changed on September 15, 2026. New sites that run ads are now offered a preset that allows search bots, signals against training and blocks agent fetches on pages with ads. Sites that had the older "Block AI bots" switch on kept search access and moved to a no-training signal. The new Block setting also stops Googlebot, Bingbot and Applebot.
AWS WAF's bot rules block verified and unverified AI bots alike once switched on. Vercel's AI bot rules are off by default. Ask what is enabled.
Squarespace has an AI crawler setting that is off by default. Its list covers training bots and leaves OAI-SearchBot and PerplexityBot alone.
Anthropic warns that blocking its IP addresses also stops its bots from reading your robots.txt, so your stated preferences go unseen.
You can check this in your CDN logs. Section seven shows what to look for.
Give one person ownership of AI bot settings across the website, the CDN and security tools. Today these are often distributed across three teams.
Send your infrastructure team the list of search and live-fetch bots from the table above. Ask them to confirm each one is allowed at the firewall.
Getting the bot to crawl the page is one thing, but the bot also has to be able to read the page. Some sites, usually ones built as single-page apps, load their text with JavaScript after the page opens. A person's browser runs that JavaScript, so a visitor sees the full page. The bots behind ChatGPT, Claude and Perplexity read the page without running JavaScript, so on those sites they can miss the text, the pricing and the features entirely.
| Runs JavaScript | Evidence | |
|---|---|---|
| GPTBot, OAI-SearchBot, ChatGPT-User | No | Vercel and MERJ, December 2024, from one month of network traffic |
| ClaudeBot | No | Same study |
| PerplexityBot | No | Same study |
| Googlebot, including for Gemini | Yes | Same study, plus Google's Martin Splitt in May 2025 |
| Applebot | Yes | Apple says it "may render" pages in a browser |
| ChatGPT deep research | No | Peec AI, June 2026. The agent made zero clicks across about 200 runs, and a sample of its page reads matched a plain fetch word for word. |
Right-click and choose "View page source". Search for a sentence from the page. If it appears, a bot that skips scripts can read it.
Reload the page with JavaScript disabled in your browser. What remains is close to what those bots see.
Give ChatGPT or Claude the URL. Ask it to quote the first and last paragraph of the page.
URL Inspection in Search Console shows the rendered HTML Google has stored for a page.
If the site is built as a single-page app, budget for server-side rendering or prerendering on the pages that sell. This is the one fix in the chapter that needs engineering time.
Test one page of each template. A pricing page, a product page, a comparison page, a docs page and a blog post cover the usual set.
Engines retrieve from search indexes, as chapter 2 covered. A readable page still has to be in the index the engine uses.
For AI Overviews and AI Mode a page has to be indexed and eligible for a snippet. Google states "There are no additional technical requirements". Its newer optimization guide adds one condition, which is that the site must be included in generative AI features in Search Console.
Seer Interactive found 87% of ChatGPT search citations matched Bing's top 20 results, against 56% for Google's, across 500+ citations in February 2025. Set up Bing Webmaster Tools.
OpenAI's deep research agent reaches pages through search results and links on the page. Pages with zero links pointing to them are hard to reach.
About 34% of the pages ChatGPT's and Claude's bots requested returned a 404 in the Vercel study, against 8% for Googlebot. Redirect old URLs.
Watch out for conflicting canonical tags and crawl waste from filters, session IDs and tracking parameters, which will slow down and confuse bots.
Several tactics get a lot of airtime and little support from the data so far.
of llms.txt files received zero requests in a month of logs
Ahrefs, 137,210 domains, May 2026Adding schema markup moved AI citations by 2 to 5%, inside the range of noise
Ahrefs test of 1,885 pages, reported by Search Engine Journal, May 2026studies we found that tie page speed to AI citations with data
Google's John Mueller said in June 2025 that "no AI system currently uses llms.txt". It is quick to add and unlikely to help today. We treat it as optional.
Google and Bing both advise against separate versions for bots. Copies drift out of date and get neglected.
Keep it and keep it accurate. Schema earns rich results in regular search and helps search engines understand a page. The evidence so far shows a small effect on AI citations. Google says its AI features need zero special markup, while Bing's Fabrice Canel says Microsoft's models do use it. I haven't found it to make a major impact.
Fast, error-free servers help Google crawl more. Specific millisecond targets for AI bots have zero published evidence behind them.
Your server or CDN logs record every bot visit. Analytics tools miss them, since bots skip the tracking script. We sort bot visits into the three groups covered at the start of the chapter.
What AI bots are doing
Share of AI crawler requests by purpose across Cloudflare's network, July 2025. All bars share one scale from 0 to 100%. Source Cloudflare. The two smaller groups are the ones tied to answers, so a raw count of bot hits overstates how often engines retrieve your pages.
Hits from ChatGPT-User, Claude-User and Perplexity-User show pages an engine pulled while answering someone.
OpenAI, Anthropic and Perplexity publish their bots' IP ranges. Google supports a reverse DNS check. Imitators exist.
A run of 403 or 429 responses to a search bot points to a firewall block. A run of 404s points to broken links.
Chapter 8 covers bot visits as one of the signals to report over time.
Ask for a monthly count of live fetches by engine on your key pages. It is the closest thing to seeing engines use your content.
Pull 30 days of logs and filter by the bot names in this chapter. Group them by job before you count anything.
Eli Schwartz argues that rendering counts for more now than it did under Googlebot alone, since AI bots run fewer scripts and tolerate less crawl waste.
Google maintains that a page indexed for Search qualifies for its AI features, with the Search Console inclusion setting as the one addition.
Cloudflare and many publishers treat access as leverage. News publishers block GPTBot at about 50%, according to HasData in July 2026. Cloudflare is testing pay per crawl.
Cloudflare measured about 1,091 pages crawled by OpenAI for each visitor it sent back in July 2025, against 5.4 for Google.
Choose your next stop
We use analytics cookies to see which pages help people. They load only if you accept. Cookie policy