And it is usually added by accident — by a plugin, a host default, or a developer who read one alarming article about AI training. Check all 12 AI crawlers against your exact URL in about five seconds.
"We're not blocking anyone" is not the same as "we can be cited". These are independent.
Your robots.txt. We parse it the way Google documents — most specific user-agent group wins, longest matching path rule wins, Allow beats Disallow at equal length, * and $ wildcards honoured — then test each crawler against your exact URL path. A text search for "GPTBot" gets this wrong in both directions.
The gate people miss. nosnippet, noarchive and max-snippet:0 in your robots meta tag or the X-Robots-Tag HTTP header specifically forbid showing an excerpt — and an AI answer is an excerpt. A page with these can rank on page one and never once be quoted.
GPTBot, ClaudeBot, PerplexityBot and CCBot do not execute JavaScript. If your content is injected client-side, they arrive at a permitted, quotable, empty page. We count the words actually present in the raw HTML.
If a crawler is blocked here, fix that before touching content: a page an AI system cannot fetch cannot be quoted. Once access is clear, check whether the page itself reads well to an answer engine with the AI Readiness Scanner, and run the Page Audit for the canonical, robots and sitemap issues that also affect crawling. If you want this done for your site, see our AEO and GEO consulting for small businesses.