Technical AI Crawlability

Beyond Googlebot

For years, technical SEO has revolved around a single, dominant player: Googlebot. We've optimised our sites to ensure Google's crawler can find, render, and index every important page. But the landscape is changing. A new class of crawlers is now navigating the web, and they don't always play by the same rules.

These new bots belong to the large language models (LLMs) that power generative AI. You'll see them in your server logs with user agents like GPTBot (from OpenAI), ClaudeBot (from Anthropic), and PerplexityBot (from Perplexity AI). Their goal isn't just to index pages for a search engine; it's to gather high-quality data to train their models and retrieve information to answer user queries directly.

The key difference? Many of these AI crawlers are less sophisticated than Googlebot when it comes to processing JavaScript. They often read the raw HTML served by your server and move on, missing any content that relies on client-side rendering to appear.

Why Static HTML is King Again

Modern web development heavily favours JavaScript frameworks like React, Vue, and Angular. These frameworks often use Client-Side Rendering (CSR), where the server sends a minimal HTML shell and the browser uses JavaScript to fetch data and build the page.

For a user, this can create a fast, interactive experience. For a simple AI crawler, it's like receiving a blank page. The bot sees the initial HTML, finds it empty of meaningful content, and may not wait around for the JavaScript to execute and populate the page.

<!-- Initial HTML with Client-Side Rendering (CSR) -->
<!DOCTYPE html>
<html>
  <head>
    <title>My Awesome App</title>
  </head>
  <body>
    <div id="root"></div>
    <script src="/app.js"></script>
  </body>
</html>

<!-- Initial HTML with Server-Side Rendering (SSR) -->
<!DOCTYPE html>
<html>
  <head>
    <title>My Awesome App</title>
  </head>
  <body>
    <div id="root">
      <main>
        <h1>Welcome to the Main Article</h1>
        <p>This content is visible immediately...</p>
      </main>
    </div>
    <script src="/app.js"></script>
  </body>
</html>

The solution is to return to older, more robust techniques: Server-Side Rendering (SSR) and prerendering.

Server-Side Rendering (SSR): The server fully renders the page into HTML before sending it to the client. The AI crawler receives a complete document with all the content in place from the very first request.
Prerendering: This is a similar technique, often used for static sites. The complete HTML for each page is generated at build time, so the server just has to deliver a finished file. Popular frameworks like Next.js (for React) and Nuxt.js (for Vue) offer robust SSR and static generation capabilities.

By adopting these methods, you ensure your content is immediately accessible to any crawler, no matter how simple.

Guiding the New Crawlers

While robots.txt tells crawlers which pages they can visit, it doesn't specify what they can do with the content. To give webmasters control over how LLMs use their data, a new standard is emerging: llms.txt.

The llms.txt standard is a game-changer for making your site AI-friendly.

This simple text file, placed in your website's root directory, allows you to set permissions for AI training and retrieval on a site-wide or per-bot basis. It provides a clear, machine-readable signal about your content usage policies.

# Allow all AI agents to use content for both training and retrieval
User-agent: *
Allow: /

# Disallow a specific AI agent from using any content
User-agent: NaughtyBot
Disallow: /

# Disallow OpenAI's web crawler from using content in /private/
User-agent: GPTBot
Disallow: /private/

# Disallow all agents from using content for training, but allow retrieval
User-agent: *
Disallow-training: /

Directive	Purpose
`User-agent`	Specifies the AI crawler the rules apply to (`*` for all).
`Allow`	Permits the use of content from the specified path.
`Disallow`	Prohibits the use of content from the specified path.
`Disallow-training`	A proposed directive to prevent content usage for model training.

Implementing llms.txt is a proactive step towards responsible AI data governance and ensures your content is used according to your terms.

Auditing for AI Readiness

How can you be sure your site is friendly to AI crawlers? A technical audit focused on AI crawlability is essential. It goes beyond standard SEO checks.

First, simulate a simple crawler's experience. Right-click on your page and select "View Page Source." The content you see here is what a non-JavaScript-rendering bot sees. Is your main content, navigation, and internal linking visible? If it's mostly empty tags and script links, you have a problem. Compare this with what you see using the browser's "Inspect Element" tool, which shows the page after JavaScript has run.

Next, use a text-only browser like Lynx or disable JavaScript in your browser's developer tools and try to navigate your site. This is an excellent way to spot dependencies on client-side scripts for core functionality. If you can't navigate or see key content, an AI crawler can't either.

Audit Check	Pass Condition	Fail Condition
View Page Source	Main content and links are present in the initial HTML.	The `<body>` is nearly empty, with only `<script>` tags.
JavaScript Disabled	The site remains navigable and readable.	The page is blank or key functionality is broken.
Check `llms.txt`	The file exists at the root and reflects your usage policy.	The file is missing or misconfigured.
Header Inspection	The server returns a `200 OK` status and content is served.	The server relies on client-side redirects or serves errors.

Adapting your technical SEO for this new multi-crawler environment is no longer optional. By focusing on serving clean, complete HTML and providing clear usage guidelines, you ensure your content remains visible and valuable in the age of AI.

SEO Strategies for the AI Era