We get asked about AI in two opposite forms. One half of clients want to know how to get ChatGPT or Google's AI answers to recommend them. The other half want to know how to stop AI companies helping themselves to their content. Both are fair questions. The trouble is that the usual answer to the second, applied carelessly, destroys the answer to the first.
The instrument in both cases is a plain text file called robots.txt at the root of your website, or a checkbox in your platform that writes it for you. It is the bluntest control on the web and was never designed for the question now being asked of it. Tick the wrong box and you have not protected your work from being trained on. You have removed yourself from the AI answers your competitors still appear in.
It is also rarely a decision anyone remembers making, which is why the first job is not deciding anything. It is finding out what your site currently says.
Three kinds of AI bot, and only one that affects visibility
Treating all AI bots as one thing is the mistake that does the damage, because the major AI companies run separate crawlers for separate jobs, each named separately in robots.txt. Three categories, in practice.
The first kind collects pages to train future models. OpenAI's GPTBot and Anthropic's ClaudeBot are the names you will see most often. Blocking these says you do not want your content used to build the next model. It has no bearing on whether you appear in answers today.
The second kind builds the index the assistant consults when it needs current information. OpenAI's OAI-SearchBot and Anthropic's Claude-SearchBot do that job, and PerplexityBot is the equivalent elsewhere. These decide whether you can be cited when somebody asks for a recommendation. Block them and you are not in the running.
The third kind fetches one page because a person asked it to — ChatGPT-User and Claude-User. That is what runs when a customer pastes your address into a chat window and asks what you charge. Google does not follow this pattern at all, which is where most of the confusion sits.
Why the distinction is the whole point
Separated, the two decisions are straightforward. Blocking the training crawlers is a view about ownership, and costs you nothing in visibility. Blocking the search crawlers is a decision to be absent from somewhere customers now ask questions. Most owners would answer those differently if anyone put them separately.
Almost nothing does. The blocklists circulating online lump every AI user agent the author could find into one list. Website builders offer one switch. So an owner who objects to AI training — a defensible position — ticks the only box available and loses their place in AI search as a side effect nobody mentioned.
Squarespace is the clearest example, partly because it is honest about the trade. Under Settings there is a Crawlers panel with a checkbox labelled Block known artificial intelligence crawlers. It covers a long list of named bots, you cannot choose among them, and the file cannot be edited by hand. Squarespace's own help pages cover both halves: the article titled Request that AI models exclude your site notes that ticking the box removes nothing already collected, and its guidance on optimising for AI-powered search says your content is unlikely to appear in those results if you limit crawler access to it.
Where the block usually came from
Nobody sits down and decides this. The block is inherited. The usual culprits:
- —A platform toggle switched on during setup by whoever built the site, for reasons nobody wrote down
- —A security or SEO plugin with AI-bot blocking on by default, or switched on quietly by an update
- —A blocklist copied out of an article and never revisited — many name user agents since retired, while missing the search crawlers that matter
- —Rules left by a previous agency, or carried over from a staging site copied to live with its blocks intact
- —Your host or CDN deciding for you. Cloudflare, which a large share of websites pass through, announced in 2025 that it would make blocking AI crawlers the default for new sites, and its arrangements have kept changing since. What counts is what your own dashboard shows today
Google is the exception, and the most misunderstood
Googlebot serves both ordinary search results and Google's AI answers. No robots.txt line keeps you in one and out of the other, because the crawling is the same crawling. Blocking Googlebot removes you from Google Search altogether, which is not a trade any small business should make.
Google-Extended is the token people reach for believing it is the AI Overviews opt-out. It is neither a crawler nor that opt-out: it governs Gemini's use of your content, and Google's position is that it affects neither your rankings nor your eligibility to appear in AI Overviews or AI Mode. It is on Squarespace's blocklist, so that single checkbox covers it too.
At page level the nosnippet directive will keep a page out of Google's AI answers, but it also removes your ordinary search snippet — fixing what you dislike by deleting the description under your listing. There has been reporting through 2026 of a newer site-level control in Search Console for Google's AI surfaces, but the detail keeps moving, so check Google's own documentation rather than any summary, including this one.
What blocking actually achieves
Two honest limits. Robots.txt is a request, not a lock: well-run companies honour it, which is exactly why blocking them costs you visibility, while the scrapers that ignore it are the ones you would most like to stop. And it only works forwards — content already collected is already collected, as Squarespace says of its own checkbox.
Which leaves what you are protecting. For most small businesses the website is marketing: it exists to be read, repeated and acted on, and an assistant repeating it accurately is the point. The sums differ if what you publish is the product rather than an advert for it — a photographer's portfolio, a course you sell. If that is you, block the training crawlers and leave the search ones alone.
llms.txt is not the answer to this
Short, because the evidence is short. llms.txt is a proposed file listing your content for AI systems. Google has said publicly that it does not use it and has no plans to, likening it to the old keywords meta tag, and the log studies that circulate suggest AI crawlers request it very rarely. It is also not an access control: it grants nothing and removes nothing. Harmless to publish, and not worth an afternoon ahead of anything below.
A ten-minute check you can do yourself
None of this needs a tool or a subscription. Your robots.txt is public, and so is everyone else's.
- —Type your domain followed by /robots.txt into a browser. Nothing to log into or install
- —Read it for Disallow lines under a User-agent line naming an AI bot. A Disallow with a single forward slash blocks that bot entirely
- —Look for the search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot — rather than the training ones. Those are the lines with a cost attached
- —Check your platform's own setting, because it may be writing the file regardless of what you edit — on Squarespace, the Crawlers panel under Settings
- —Check your host or CDN dashboard for bot controls, which sit in front of your site and override anything in it
- —If something is blocked and nobody can say who did it or why, that is your answer. Undo it, then decide deliberately
Being reachable is not the same as being cited
Most small businesses are not absent from AI answers because they are blocked. They are absent because there is nothing on the site worth quoting: services described in half a sentence, no prices or bands anywhere, areas covered left implied, the specifics that would answer a real question still in the owner's head.
For local questions especially, assistants lean on map data and review sites as well as your pages, so a complete Google Business Profile and a steady flow of recent reviews can matter more than anything you publish.
Removing an accidental block is maintenance, not a strategy. It makes the rest of the work possible. It is not a substitute for doing it.
How we handle it
On any site we take on, robots.txt is part of the first look alongside indexing and speed, because it takes two minutes and occasionally explains everything. Where something is blocked by accident we unblock it and say so. Where a client genuinely objects to AI training we block the training crawlers only. Most of the time there was no objection and nobody knew the block was there.
If you would rather we looked, send us your address and we will tell you what we find and whether it is costing you anything. Often it is not, and we would rather say so. Anything we build is a fixed quote after a short call, ongoing work is monthly with 30 days' notice, and you own your site from day one.