AI Readiness

Answer Engine Optimisation (AEO) and Generative Engine Optimisation (GEO) are emerging disciplines focused on making content discoverable and citable by AI systems. As AI Overviews and AI chatbots replace traditional blue-link search for some queries, being featured in AI-generated answers becomes a traffic source.

AI systems extract entities (people, organisations, products, places) from web pages. Entity Schema (Person, Organization, Product, etc.) in JSON-LD makes entities machine-readable and helps AI correctly identify what the page is about. sameAs links connecting to Wikidata or Wikipedia entries strengthen entity recognition.

FAQPage schema and visible Q&A sections are frequently cited by AI systems because they are already structured as question-answer pairs. Content with a high Q&A ratio is more likely to be surfaced in AI Overviews and voice search responses.

The Speakable schema type marks sections of text that are suitable for text-to-speech (TTS) — used by Google Assistant and smart speakers. It signals the most important, concise passages for audio delivery.

llms.txt is a proposed standard (similar to robots.txt) that provides a structured overview of a site for LLMs and AI crawlers. A well-formed llms.txt file helps AI systems understand site structure, content types, and contact information without crawling every page.

Many sites explicitly block AI crawlers in robots.txt — sometimes intentionally (to prevent training), sometimes by mistake (copying a template). CheckSEO checks access for 6 major AI bots:

• GPTBot — OpenAI's crawler for ChatGPT web browsing and search

• ChatGPT-User — ChatGPT's real-time browsing agent

• ClaudeBot — Anthropic's crawler for Claude

• Google-Extended — Google's AI training crawler (Gemini, Bard)

• PerplexityBot — Perplexity AI's search crawler

• CCBot — Common Crawl bot used by many AI systems

If any of these are blocked, the site won't appear in that AI system's answers. The check uses the same protego parser as Google to evaluate robots.txt rules per user-agent. A WARNING is raised for each blocked bot under the AI Readiness category.

To unblock, remove or comment out the Disallow rules for these user-agents in robots.txt. If you only want to block AI training but allow search, consider blocking CCBot and Google-Extended while allowing GPTBot, ChatGPT-User, ClaudeBot, and PerplexityBot.

Some sites use Cloudflare's 'Block AI Scrapers' feature or custom rules to prevent AI training crawlers. While this may be intentional, it can also inadvertently block AI search crawlers like GPTBot or PerplexityBot, preventing the page from being indexed by AI systems.

AI systems frequently extract data from comparison tables, TL;DR summaries, and definition lists because they are already pre-structured for machine consumption.

• Comparison tables — side-by-side feature/product comparisons are highly citable by AI.

• TL;DR pattern — a concise summary at the top or bottom of the article helps AI extract the key takeaway.

• Long-tail headings — headings phrased as specific questions (e.g., 'How to fix a broken canonical tag') match the long-tail queries AI systems handle.

• Conversational headings — headings using natural language patterns improve voice search and AI assistant responses.

• Authority backlinks — links from/to authoritative domains signal content trustworthiness to AI systems.