Technical
One line of config can make you invisible.
Around a third of the sites we scan block at least one AI crawler — almost always by accident, added by a plugin or a developer worried about scraping. Until that line is gone, no amount of great content can get you recommended.
ChatGPT
Perplexity
Gemini
Claude
AI OverviewsHow it works
Three steps to the real answer.
01
Every AI user-agent checked
We fetch your robots.txt and evaluate it per crawler — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot — because a rule can block one engine and leave the rest untouched.
02
The whole read path tested
Beyond robots.txt we check llms.txt, whether your key content renders without JavaScript, and whether your structured data is machine-readable.
03
The exact fix, in plain terms
You get the specific line to remove or change, written so you can hand it straight to whoever runs the site.
What you get
Everything in AI Crawler Check.
Per-crawler robots.txt verdict
A clear allowed-or-blocked result for every AI user-agent, rather than a single pass/fail that hides which engine you actually lost.
llms.txt detection
We check whether you publish an llms.txt, the emerging convention for telling AI systems what your site is and which pages matter.
Render check
If your content only appears after JavaScript runs, some crawlers will see an empty page. We check what's actually in the HTML they receive.
Structured data check
Schema markup helps machines understand what your product is and who it's for. We flag what's missing or malformed.
Re-check on every scan
Blocks get reintroduced by plugin updates and platform migrations. Every rescan re-runs the whole check so a regression surfaces immediately.
Alerts when a block appears
On a monitoring plan, a newly blocked crawler triggers an alert the day it happens rather than the next time someone thinks to look.
Why it matters
There is no page two in an AI answer.
It's the cheapest fix you'll ever make
Deleting one line can restore an entire engine's ability to read you. Nothing else in AI visibility has this ratio of effort to impact.
It's almost always unintentional
Blocks arrive via SEO plugins, security defaults and well-meaning anti-scraping advice. Very few teams chose it deliberately.
It invalidates everything else
Content, PR and citations cannot help an engine that is not permitted to fetch your pages in the first place.
FAQ
Questions, answered.
Anything else? Write to us and a human replies.
Which crawlers do you check?
GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended and CCBot — the user-agents that determine whether the engines we audit can read your site.
Does blocking GPTBot actually hurt my visibility?
It stops that crawler from fetching your pages, which removes the most direct route for your own content to inform an answer. You can still be described via third-party sources, but you lose control of the description.
Should I allow every AI crawler?
That's a business decision, not a technical one — some publishers block deliberately to protect content. Our job is to make sure the choice is deliberate rather than something a plugin decided for you.
What is llms.txt?
An emerging convention: a plain-text file describing your site and pointing to the pages you most want AI systems to use. It isn't universally supported yet, but it costs almost nothing to publish.
How fast is the check?
Seconds. It runs as part of the free teardown, so you can confirm you aren't accidentally invisible before deciding whether to buy anything.
What does the free teardown include?
Your AI visibility score with its confidence band, the competitor taking your spot, one real answer as proof, how many buyer questions you lose, and the AI-crawler block check. Free with an account, no card — about sixty seconds.
Explore more of the platform
Find out what AI says when you're not in the room.
Free scan. No card. Sixty seconds.