// BLOG · 08/11/2026

Letting AI engines read your site: robots.txt, llms.txt and structured data

An AI engine can only describe what it can read. These are the technical basics that decide whether your own site is part of the picture.

1. Decide which AI crawlers may visit

Most AI companies publish the user agents of their crawlers and respect robots.txt. Some crawl to train models, others fetch pages live to answer a question. Common ones include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google's control for Gemini use of content) and Applebot-Extended (Apple). Check each company's documentation for the current names, then allow or disallow them deliberately. A site that blocks them by accident, often through a security setting at the hosting or CDN layer, simply drops out of live answers.

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: *
Disallow: /account/

2. Publish an llms.txt

/llms.txt is a proposed convention: a short Markdown file at the root of a site that tells language models what the site is and links to the pages that matter most. It is not a standard and engines are not obliged to use it, but it costs little and gives a clean, current summary in your own words.

3. Add structured data

Schema.org markup in JSON-LD states facts in a form machines do not have to guess: Organization for who you are, Product or SoftwareApplication with Offer for what you sell and at what price, Article for editorial pages and FAQPage where you genuinely answer questions. Keep it consistent with the visible page.

4. Make key content available without scripts

Many crawlers do not run JavaScript, or run it only briefly. Text that appears only after a script, a scroll or a click may never be seen. Important facts, prices and descriptions should be in the HTML the server sends.

5. Keep it fast and current

A sitemap with real modification dates, sensible caching and quick responses help every crawler. When facts change, update the page, the structured data and the llms.txt together.

AEON GPTO applies these checks as part of its diagnostics and shows how changes are reflected in your Share of Model Response. Start with the free AI visibility scan.