Is Your Site Accessible to ChatGPT and Perplexity?

A new kind of visitor is knocking on your door

Right now, invisible visitors are trying to access your website. They don't have a browser. They don't click your buttons. They don't buy anything — at least not directly. But they might be the most important visitors your site will ever receive.

These visitors are AI crawlers. They're sent by ChatGPT (OpenAI), Perplexity, Google Gemini, and other AI systems to read your website, understand what you offer, and decide whether to recommend you when someone asks a relevant question.

If these crawlers can access your site and find useful content, you become part of the knowledge that AI uses to answer questions. If they can't — because they're blocked, or because your content gives them nothing to work with — you simply don't exist in the AI world.

And increasingly, the AI world is where your customers are searching.

How AI crawlers actually work

Let's demystify this. When someone asks ChatGPT "What's the best florist in Manchester for wedding bouquets?" the system doesn't magically know the answer. It relies on information it has gathered from across the web — including, potentially, your website.

AI systems build their knowledge in two ways:

Training data: The massive collection of text that AI models were originally trained on. If your website existed and was accessible when the training data was collected, your content may already be part of what ChatGPT "knows."

Live crawling and search: Increasingly, AI systems also browse the web in real time. When ChatGPT uses its web browsing capability, or when Perplexity answers a question, they send automated crawlers to visit websites, read the content, and use it to formulate their responses.

Each AI company has its own crawler with a specific name:

• GPTBot — OpenAI's crawler (for ChatGPT)
• PerplexityBot — Perplexity's crawler
• Google-Extended — Google's AI training crawler
• ClaudeBot — Anthropic's crawler (for Claude)
• Bytespider — ByteDance's crawler (TikTok's parent company)

These crawlers visit your site much like Google's traditional search crawler does. They read your pages, follow your links, and index your content. The difference is what they do with it: instead of listing your site in search results, they use your content to build answers to people's questions.

Are you blocking AI crawlers without knowing it?

Here's the problem that catches many business owners off guard: your website might be actively blocking AI crawlers, and you'd never know because everything looks fine to human visitors.

This happens through your robots.txt file — a small text file that tells crawlers what they're allowed to visit. Some website platforms, security plugins, or web developers add rules that block AI crawlers specifically.

Check your site right now. Visit yourdomain.com/robots.txt and look for any of these lines:

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

If you see those lines, the respective AI crawlers are being turned away at the door. They can't read your content, which means they can't recommend you.

Why would anyone block AI crawlers?

Some businesses deliberately block AI crawlers to prevent their content from being used in AI training. This is a legitimate choice, especially for publishers, artists, and creators whose original content is their primary product.

But for a plumber, a restaurant, a law firm, or a dental practice? Blocking AI crawlers means cutting yourself off from a rapidly growing channel through which customers find businesses. It's like refusing to be listed in the phone book because you don't like the paper it's printed on.

A real-world example

A solicitor's firm in Leeds discovered that their website management company had blocked GPTBot and PerplexityBot "as a security measure." For 14 months, whenever someone asked ChatGPT for a solicitor recommendation in Leeds, the firm was invisible — despite having an excellent website with detailed service descriptions. After removing those two lines from robots.txt, Perplexity began citing the firm within three weeks. They traced two new client enquiries directly to Perplexity recommendations in the following month.

What AI crawlers look for on your site

Getting crawlers onto your site is step one. Step two is making sure they find content worth citing. AI crawlers evaluate your pages differently than a human visitor would. Here's what matters:

Clear, factual statements

AI loves specifics. Compare these two descriptions from competing electricians:

Electrician A: "We offer a wide range of electrical services for homes and businesses across the region."

Electrician B: "We handle domestic rewiring (typically 3 to 5 days for a 3-bedroom house), consumer unit upgrades, EV charger installations (OZEV grant-approved), and commercial electrical testing for landlords (EICR certificates issued within 48 hours)."

When someone asks Perplexity "How long does it take to rewire a house?" Electrician B has a direct answer on their site. Electrician A has nothing useful.

Structured content with headings

AI parses structured content more efficiently than unformatted text. If your website is one long page with no headings, no sections, and no clear organisation, AI crawlers have to work harder to extract useful information — and they may not bother.

Use clear headings that describe what follows. "Our Wedding Photography Packages" is better than "Packages." "What to Expect at Your First Appointment" is better than "Information."

Freshness and accuracy

AI systems increasingly consider how recent your content is. A page last updated in 2021 is less likely to be cited than one updated in 2026, because AI can't be confident the information is still accurate.

This doesn't mean you need to rewrite your entire site every month. But updating key pages annually — refreshing prices, adding recent examples, correcting outdated information — signals that your content is maintained and trustworthy.

Your business identity

AI needs to know who you are, where you are, and what you do. This sounds obvious, but a surprising number of business websites bury this information. Make sure every page includes:

• Your business name
• Your location (city, area, or region)
• What you do, in plain terms

If someone asks ChatGPT for a recommendation and your site doesn't clearly state what you offer and where, how can AI recommend you?

The llms.txt file: a new way to talk to AI

A relatively new development is the llms.txt standard — a file you can place at the root of your site (yourdomain.com/llms.txt) that's specifically designed for AI systems.

Think of it as a structured introduction of your business, written for AI rather than humans. It typically includes:

• What your business does
• Your key services or products
• Your location and service area
• Links to your most important pages
• Any specific information you want AI to know about you

While not yet universally adopted, having an llms.txt file makes it easier for AI systems to understand your business quickly and accurately. It's like handing someone a clear business card instead of making them search through your entire website for basic information.

If you want to learn more about optimising for AI specifically, our guide on what GEO is and why it matters covers the broader strategy.

The difference between being indexed and being cited

Here's a crucial distinction that many business owners miss: having AI crawlers visit your site is not the same as being recommended by AI.

AI crawlers might read your entire website and still never cite you. Why? Because they found your content but didn't find it useful enough, specific enough, or trustworthy enough to include in their answers.

Think of it like being in a library. Just because a book is on the shelf doesn't mean the librarian will recommend it. The book needs to be relevant, well-written, and authoritative enough to merit a recommendation.

The gap between "indexed" and "cited" is where most of the work lies. It requires:

1. Accessible content — crawlers can reach your pages
2. Quality content — your pages contain genuinely useful information
3. Structured content — AI can easily extract and cite relevant facts
4. Trustworthy content — your site signals credibility (HTTPS, up-to-date info, consistent details)
5. Unique content — you say something that competitors don't

Getting all five right is what separates businesses that AI recommends from businesses that AI ignores.

What your competitors are doing (and you're not)

The businesses that are already visible to AI aren't necessarily bigger, better, or more established than you. They're just more accessible.

A bakery in Hackney recently reviewed their AI visibility. They found that while their Google ranking was decent, they were completely invisible to ChatGPT and Perplexity. The diagnosis was straightforward:

• Their robots.txt blocked GPTBot (added by their hosting platform's default settings)
• Their content was heavy on images but light on text — beautiful photos of cakes, but no written descriptions that AI could read
• Their business name and location weren't consistently mentioned across pages
• They had no structured data helping AI understand they were a bakery

After addressing these four issues — removing the crawler block, adding descriptive text to every page, ensuring consistent business information, and adding basic structured data — the bakery appeared in Perplexity results within a month. ChatGPT began recommending them when asked about bakeries in East London shortly after.

None of these changes required rebuilding their website. They required understanding what AI needs and providing it.

How to check your AI visibility right now

You can perform a basic AI visibility check in about five minutes:

Step 1: Check your robots.txt

Visit yourdomain.com/robots.txt. Look for any lines that block GPTBot, PerplexityBot, ClaudeBot, or other AI crawlers. If you find them and you want AI visibility, remove them.

Step 2: Ask ChatGPT about yourself

Open ChatGPT and ask: "What do you know about [your business name]?" Then try: "Can you recommend a [your service] in [your city]?"

If ChatGPT doesn't mention you, that's your baseline. It's not a failure — it's information about where you stand today.

Step 3: Try the same with Perplexity

Go to perplexity.ai and ask the same questions. Perplexity often provides sources, so you can see which of your competitors are being cited and study what they're doing differently.

Step 4: Read your own content like a robot

Open your homepage and try to answer these questions using only the text (ignore images): What does this business do? Where is it located? What specific services does it offer? If you struggle to answer any of these, AI will struggle too.

Step 5: Check your structured data

Your website may already include structured data — invisible code that helps both Google and AI understand your business type, location, services, and reviews. If it doesn't, this is one of the highest-impact changes you can make for AI visibility.

For a deeper dive into the strategies behind AI visibility, our guide on how to appear on ChatGPT walks through the process step by step.

The window is still open

We're in the early stages of the AI search revolution. Most small businesses haven't even considered whether AI can access their site. This means the barrier to entry is still low. Making your site accessible and useful to AI crawlers today puts you ahead of the vast majority of your competitors.

But the window won't stay open forever. As more businesses optimise for AI visibility, standing out will become harder. The businesses that act now will build an advantage that compounds over time — just as the businesses that invested in Google visibility early are still reaping the benefits today.

Find out where you stand

TryGEO analyses your AI visibility alongside your traditional search visibility. It checks whether AI crawlers can access your site, evaluates whether your content is structured for AI citation, and shows you exactly where you stand compared to what's possible.

The audit takes less than a minute. And it might be the most important minute you spend on your business this week.

Check your AI visibility with a free audit