Why a crawler you've never heard of decides who gets the call
A homeowner in Roanoke used to open Google, type "gutter repair near me," and scroll a list of blue links. Now a growing share of them open ChatGPT or Perplexity and ask, in plain English, "who's a reliable gutter company near Roanoke?" The assistant answers in a paragraph and names two or three businesses. If you're not one of them, that customer never knew you existed.
Here's the part most owners miss: those assistants can only name businesses whose websites they're allowed to read. Every major AI answer engine sends out an automated crawler to fetch pages from the open web. ChatGPT uses GPTBot, Claude uses ClaudeBot, Perplexity uses PerplexityBot, and Google's AI features ride on Googlebot plus a separate Google-Extended control. If your site turns those crawlers away at the door, you're invisible to the assistant on the other side — no matter how good your work is or how many happy customers you have.
Blocking AI crawlers is almost never a decision anyone made on purpose. It's a leftover rule, a default plugin setting, or a "security" feature doing exactly what it was told. So the fix is usually quick and cheap — but only once you know to look. This guide walks a Virginia small business through finding out whether you're blocked and what to do about it. If you'd rather hand the whole diagnosis to someone, that's what AI search setup is for.
The three ways a site accidentally blocks AI crawlers
Accidental blocking happens in one of three places, and it's worth knowing all three, because a site can trip over more than one at a time.
- robots.txt rules. This is a plain-text file at yourdomain.com/robots.txt that tells crawlers where they may and may not go. A rule like User-agent: GPTBot followed by Disallow: / shuts ChatGPT's crawler out of the entire site. Some website builders and "AI-protection" plugins add rules like this automatically, and plenty of owners never see them.
- Firewall and security rules. Security plugins and web application firewalls block traffic they judge to be "bots," and they don't always tell the difference between a scraper and a legitimate AI crawler. The page loads fine for you in a browser, but the crawler gets a 403 error and gives up.
- JavaScript-only content. This isn't a block on paper, but the effect is the same. If your page paints its text in with heavy JavaScript that only a full browser runs, a lighter crawler may fetch the page and see a near-blank shell. Your site looks great to humans and reads as empty to the machine.
The uncomfortable pattern: the fancier and more "protected" the site, the more likely it is to be quietly blocking AI crawlers. A simple, fast, plain-HTML page rarely has this problem. A locked-down template on an aging platform often does — and the owner has no idea.
How to check your own site in ten minutes
You don't need a developer to find out whether you're blocking AI crawlers. You can run the first checks yourself right now.
Step one: read your robots.txt. Type your domain followed by /robots.txt into a browser — for example, yourdomain.com/robots.txt. Look for any line that says Disallow: / underneath a user-agent naming an AI crawler. Here are the crawlers that feed the major answer engines as of 2026:
| Assistant | Crawlers to allow |
|---|---|
| ChatGPT | GPTBot, OAI-SearchBot, ChatGPT-User |
| Claude | ClaudeBot, Claude-SearchBot, Claude-User |
| Perplexity | PerplexityBot, Perplexity-User |
| Google (AI Overviews & Gemini) | Googlebot, Google-Extended |
Step two: view your page as text. In your browser, right-click a key page, choose "View Page Source," then use find (Ctrl-F or Cmd-F) to search for a sentence you know is on the page. If it isn't in the source, your content is likely hidden behind JavaScript, and light crawlers may see nothing.
Step three: check your logs or analytics. If your hosting gives you server logs, look for GPTBot or PerplexityBot in the recent visits. Seeing them fetch pages confirms the door is genuinely open. Seeing repeated blocks or 403 responses points at a firewall problem. All three of these checks are part of the fundamentals of AI search, so a proper audit runs them together.
Fixing robots.txt without breaking anything else
Once you've found a rule that's blocking AI crawlers, the fix is usually a small edit — but it's the kind of small edit that can break a site if you get the syntax wrong, so slow down here.
The goal is simple: remove the Disallow rules aimed at the crawlers you want to welcome. If your robots.txt has a block that reads User-agent: GPTBot then Disallow: /, deleting those two lines lets ChatGPT's crawler read your site again. Do the same for any AI crawler you find shut out. You can still keep genuine junk crawlers and scrapers out — the point isn't to open the floodgates, it's to stop turning away the specific assistants your customers now use to find a business like yours.
A few cautions worth stating plainly:
- Don't accidentally block Googlebot. A broad Disallow: / under User-agent: * hides you from ordinary Google search too, which is far more damaging than any AI issue. Confirm that catch-all rule isn't slamming the door on everyone.
- Google-Extended and Googlebot are separate controls. Google's AI Overviews ride on the normal Googlebot crawl, while Google-Extended governs Gemini and AI grounding. Blocking Google-Extended won't hurt your classic rankings, but it does opt you out of some Google AI answers.
- Test after you change it. Reload yourdomain.com/robots.txt and confirm the file reads the way you intended.
If editing a live config file makes you nervous, that instinct is correct — this is a one-time job worth handing to someone who does it carefully, not a monthly retainer trap.
When the block isn't in robots.txt at all
Sometimes robots.txt is perfectly clean and the assistants still can't read you. That's when the culprit is a firewall, a security plugin, or JavaScript-rendered content — and these are trickier, because there's no single file to open and eyeball.
Firewall and "bot protection" blocks are common on sites that added a security layer to fight spam or attacks. The layer sees an unfamiliar crawler, decides it's a threat, and returns an error. The fix is to allow the legitimate AI crawlers through the firewall's rules — usually a settings change in the security service or plugin, sometimes an allowlist of the crawler names above. The tell is that your pages load instantly in a browser but crawlers report errors; that gap points straight at a firewall doing too much.
JavaScript-only pages are the sneakiest, because nothing is technically blocked — the crawler just fetches an empty shell. The durable fix is making sure your important content lives in the page's plain HTML instead of being painted in afterward by scripts. On some modern builds that means turning on server-side rendering; on older ones it can mean the site's underlying approach is the real bottleneck. A fast, plain, readable page beats a slow, clever one every time an AI crawler is doing the reading. If your problems trace back to how the site is built, a cleaner website foundation often solves the AI-search issue as a side effect.
Letting crawlers in is step one, not the finish line
Unblocking AI crawlers gets you into the room. It doesn't guarantee you'll be the name the assistant says out loud. Once the door is open, the question becomes whether your pages give the machine a clean, quotable answer — and whether the rest of the web agrees you're real.
Two habits do most of the work. First, write answer-first: under every heading, make the opening sentence a complete, standalone answer to a real customer question, because that's the exact chunk an assistant lifts and attributes. Second, keep your business details consistent everywhere — same name, address, and phone on your website, your Google Business Profile, and the directories where you appear. AI assistants look for that agreement across independent sources before they'll confidently name you. If ChatGPT sees three different phone numbers and two spellings of your company, it gets nervous and names a competitor whose story is airtight.
The encouraging part for a Virginia small business is that this rewards clarity over budget. You don't out-spend a national chain into an AI answer — you out-clarify the shop down the road that never fixed its robots.txt. Being early and legible is a game a solo contractor in southwest Virginia can genuinely win. Want a straight read on whether AI can find you today? Start with an honest look at your AI search readiness.