Guide · Agent-ready websites

How to make your website agent-ready

More customers now ask ChatGPT, Claude or Perplexity before they visit a website, and some send an agent to book or buy for them. These ten changes decide whether those AI visitors can reach, read and use your site. They are ranked by evidence, each comes with a way to check it yourself, and we list the popular tips you can skip.
Check my website freeRead the 10 changes

Who's visiting

Five kinds of AI visitor, and what each one needs

"AI bots" aren't one thing. Some build search indexes, some read your page live for one person, some click through it in a browser. Only browser agents run JavaScript, and only training crawlers are safe to block without losing visibility.
VisitorExamplesWhat it doesRuns JavaScript?What you control
AI search crawlersOAI-SearchBot, Claude-SearchBot, PerplexityBotBuild the index that AI answers citeNorobots.txt, firewall
Assistants fetching for a person (recommended)ChatGPT-User, Claude-User, Perplexity-UserRead your page live when someone asks about youNorobots.txt, firewall
Browser agentsChatGPT Atlas, Perplexity Comet, Claude in ChromeClick, fill in forms, book and buy in a real browserYesYour HTML, labels and buttons
Coding agentsClaude Code, Cursor, GitHub CopilotRead docs and pages for developers, and ask for markdownNoA markdown version of each page
Training crawlersBlocking these doesn't remove you from AI search or assistants.GPTBot, ClaudeBot, Google-ExtendedCollect text to train future modelsNorobots.txt: your choice
Gemini and Microsoft Copilot read pages through their search crawlers, which do run JavaScript. ChatGPT, Claude and Perplexity don't.

The 10 changes

What to change, in order of impact

Start at the top. Steps 1 to 3 carry most of the impact; steps 9 and 10 only matter for some businesses.

Fix first

Remove the blockers

Most sites that fail, fail here. Any one of these can make you invisible to AI on its own, and two of them are settings, not code.

Every siteSettings change

Step 1: Let AI assistants through your firewall

Why it matters

When someone asks ChatGPT or Claude about you, the assistant fetches your page there and then. Bot protection that treats it as an attacker returns a challenge page or a 403, and the assistant tells your customer it can't open your site. Cloudflare's one-click AI bot blocking and Bot Fight Mode block AI search and assistant bots too, and a firewall rule higher up can override an allow you set elsewhere. Many owners don't know they're doing it.

Source: Cloudflare: AI Crawl Control and WAF rules

What to do

  • In your CDN, firewall or host, find the AI bot settings and allow the search and assistant bots: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.
  • Then check your custom firewall rules. A rule that challenges unknown bots can still catch them.
  • Keep your rate limits. Letting a bot in isn't the same as turning protection off.

Check it yourself

Fetch your homepage as a browser and as ChatGPT-User. A 403, a 503 or a "Just a moment" page on the second means your firewall turns AI assistants away. A 200 is a good sign, not proof: real requests come from OpenAI's own network, which some firewalls treat differently.

Terminal
curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
  https://yoursite.com/
Every site10 minutes

Step 2: Open robots.txt to AI search, and decide on training separately

Why it matters

AI search engines and assistants obey robots.txt. Blocking OAI-SearchBot keeps you out of ChatGPT search, and blocking ChatGPT-User or Claude-User stops assistants reading your page when a customer asks about you. Training crawlers are a separate decision: blocking GPTBot or ClaudeBot stops your content being used to train models, and nothing else.

Source: OpenAI: publishers and developers FAQ

What to do

  • Allow the search and assistant bots by name, so a broad rule further down can't catch them.
  • Block the training crawlers only if you've decided you don't want your content used for training. Either answer is a fair business choice.
  • Optionally add a Content-Signal line stating your preference for search, AI answers and training. It's a statement, not enforcement.

Check it yourself

Open yoursite.com/robots.txt. Look for Disallow: / under any of the search or assistant bots, or under User-agent: * without an exception for them.

robots.txt
# AI search and assistants: let them in
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Training crawlers: your call. Delete this group to allow them.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Sitemap: https://yoursite.com/sitemap.xml
Every siteHours to a project

Step 3: Put your content in the HTML your server sends

Why it matters

ChatGPT, Claude and Perplexity read the HTML your server returns and don't run JavaScript. If your text only appears after scripts load, as it does on many single-page apps built with React or Vue, they see a menu and a footer and nothing else. Gemini and Copilot do run JavaScript, which is why a site can look fine in one assistant and empty in another.

Source: Vercel: the rise of the AI crawler

What to do

  • Use server-side rendering or static generation for every public page. Next.js, Nuxt, Astro, SvelteKit, WordPress and most CMSs can do this out of the box.
  • If you run a single-page app, prerender the public pages, or move the marketing pages to a server-rendered site.
  • Keep important text in the page itself, not in images, iframes or content that only loads after a click or a scroll.

Check it yourself

Open the page, choose View Page Source (not Inspect), and search for a sentence from the middle of the page. If it isn't in the source, AI assistants can't see it.

Free check, 10 seconds

Not sure if any of these apply to you? Test all three at once.

The AI Agent Readiness Checker fetches your page as ChatGPT and Claude do, reads your robots.txt, checks what's in the server HTML, and shows what an AI can tell about your business.

Check my website

Then

Make every page easy to read and understand

Once AI can get in, it needs to find your pages and work out what you do without guessing.

Every siteAn hour or two

Step 4: Use plain links for navigation, and keep a fresh sitemap

Why it matters

Crawlers find your other pages by following links in the HTML. Menus built only in JavaScript, or elements that change page with a click handler, leave them stuck on the homepage. In the 41-day test above, the crawlers followed every HTML link and none of the JavaScript ones. A sitemap with real dates tells them what exists and what changed.

Source: Search Engine Land, Aug 2026

What to do

  • Make every menu item and in-page link an <a> element with a real href.
  • Publish /sitemap.xml, list it in robots.txt, and only change a page's lastmod date when the page actually changes.
  • Link your important pages from the homepage or main menu, not only from site search or filters.

Check it yourself

In View Page Source, search for href="/ and count the internal links. Fewer than five on a homepage usually means the menu is built in JavaScript.

HTML
<!-- Crawlers can follow this -->
<a href="/pricing">Pricing</a>

<!-- Crawlers can't -->
<span onclick="go('/pricing')">Pricing</span>
Every siteCopy edit

Step 5: Say plainly what you do, for whom, where, and how to reach you

Why it matters

When an assistant describes you, it can only use what the page says. A tagline like "Transforming tomorrow, today" gives it nothing to quote, so it guesses, or recommends a competitor whose page is clearer. The title, meta description and main heading are read first.

Source: Google: optimising for AI features in Search

What to do

  • Write one H1 that names what you do, and a title and meta description that say it again in other words.
  • State in text on the page who you serve, where you operate, how to contact you, and prices or a price range if you can share them.
  • Answer the questions customers ask (pricing, delivery areas, opening hours, lead times) in text, not only in PDFs or images.
  • Add Organization or LocalBusiness structured data with your name, logo and contact details. It does little for citations, but it removes doubt about who you are.

Check it yourself

Paste your URL into ChatGPT or Claude and ask: "Using only this page, what does this company do, for whom, where, and how do I contact them?" Anything it can't answer is missing from the page.

Every siteVaries

Step 6: Keep pages fast and light

Why it matters

AI fetchers work to tight time and size limits. A slow server or a page weighing several megabytes can be cut off before the useful part. The same fixes help your ordinary search ranking and your visitors.

What to do

  • Aim for a server response under 0.8 seconds. Over 1.8 seconds is a problem.
  • Keep the HTML under 2 MB. Large inline data, embedded images and repeated markup are the usual causes.
  • Cache public pages at the edge, so the first byte comes back fast wherever the request starts.

Check it yourself

Run the command below. It prints the time to first byte and the size of the HTML.

Terminal
curl -s -o /dev/null \
  -w "first byte %{time_starttransfer}s, %{size_download} bytes\n" \
  https://yoursite.com/

Next

Let agents act for your customers

Browser agents don't just read. They book, request quotes and buy, as long as your page lets them.

Sites with forms or checkoutAn hour or two

Step 7: Label every form field and use real buttons

Why it matters

Browser agents act for a person inside a real browser, and they work out the page from its accessibility tree: the same structure screen readers use. OpenAI says its Atlas agent uses ARIA tags to understand page structure and controls. A field with no label, or a div styled to look like a button, is a guess for the agent, and guesses fail at checkout. Every fix here also helps people using screen readers.

Source: web.dev: designing sites for AI agents

What to do

  • Give every input a <label> or an aria-label. Placeholder text isn't a label.
  • Use <button> for actions and <a href> for navigation, and give icon-only buttons an aria-label.
  • Use standard autocomplete values (name, email, tel, postal-code) so agents fill the right fields.
  • Avoid invisible overlays and layouts that shift while loading. Agents click where things were.

Check it yourself

Tab through your main form with the keyboard. If you can't reach and identify every field and button, an agent will struggle too. The Accessibility pane in Chrome DevTools shows the name each element exposes.

HTML
<label for="email">Work email</label>
<input id="email" name="email" type="email" autocomplete="email" />

<button type="submit">Request a quote</button>
<button type="button" aria-label="Open menu">
  <svg aria-hidden="true">...</svg>
</button>
Every siteAn hour or two

Step 8: Give agents a direct path to book, buy or get in touch

Why it matters

The most common thing an agent does for a customer is make contact: call, email, book a slot or ask for a quote. If the only route is a chat widget or a long form behind a CAPTCHA, the agent gives up and your customer is handed the next business on the list.

What to do

  • Put a phone link, an email link or a clearly labelled contact or booking link on every page.
  • Keep booking and quote forms short, and show errors in text next to the field.
  • Give each product or service its own URL, with price, availability and options in the HTML.
  • Decide where a CAPTCHA belongs. It stops agents acting for real customers as well as spam, so keep it for forms that actually get abused.

Check it yourself

Ask a browser agent (ChatGPT Atlas, Comet or Claude in Chrome) to book a call or request a quote on your site, and watch where it gets stuck.

HTML
<a href="tel:+442071234567">+44 20 7123 4567</a>
<a href="mailto:hello@yoursite.com">hello@yoursite.com</a>
<a href="/book">Book a call</a>

If it fits

Go further when your business calls for it

Worth doing for developer-facing, SaaS, commerce and booking sites. A brochure site can stop at step 8.

Developer tools, docs, SaaSA day

Step 9: Serve markdown to coding agents

Why it matters

Claude Code, Cursor, OpenCode and GitHub Copilot ask for markdown when they fetch a page. Markdown can use up to 80% fewer tokens than the same page in HTML, so more of your docs fit in what the agent reads. ChatGPT, Claude.ai and Perplexity don't ask for it yet, so this matters most when developers are your buyers.

Source: acceptmarkdown.com: which agents ask for markdown

What to do

  • Return text/markdown when the request's Accept header prefers it, and add Vary: Accept so caches keep the two versions apart.
  • Test with the headers real agents send, not a bare Accept: text/markdown. Claude Code sends text/markdown, text/html, */*, a tie your server has to settle in markdown's favour.
  • Optionally publish a .md twin of each page and point to it with a Link header.

Check it yourself

Request your homepage with Claude Code's header. The response should come back as text/markdown.

Terminal
curl -sI -H "Accept: text/markdown, text/html, */*" https://yoursite.com/ \
  | grep -i content-type
# Want: content-type: text/markdown
SaaS, e-commerce, bookingA project

Step 10: Let agents use your product directly, through an API or MCP server

Why it matters

If customers would hand a task to an assistant, like checking stock, getting a quote or booking a slot, the assistant does it far more reliably by calling a tool than by clicking through pages. An MCP server turns your actions into tools that Claude, ChatGPT and other assistants can call. WebMCP does the same for agents running inside the visitor's browser. Well under 1% of sites offer any of this yet, so a brochure site doesn't need it.

Source: Cloudflare: agent readiness

What to do

  • Pick the two or three tasks customers ask for most and expose them through an API with clear descriptions.
  • Wrap them in an MCP server, and publish a server card at /.well-known/mcp/server-card.json so agents can find it.
  • Keep a person in the loop for anything that spends money or sends a message: the agent drafts, the customer confirms.

Check it yourself

Ask which tasks a customer would happily hand to an assistant. If the honest answer is none yet, skip this step and come back later.

What to skip

Popular advice the evidence doesn't back

These come up in almost every "AI SEO" checklist. Some are harmless, one is actively costly, and none should come before the first three steps.

  • Expecting llms.txt to fix anything

    Ahrefs found 97% of llms.txt files got no requests at all in a month, and Google says its search ignores them. It's harmless and takes ten minutes, so add one if you like. It won't make up for any of the steps above.

    Source: Ahrefs: llms.txt study, 137k domains

  • Adding schema markup to get cited

    A study of 1,885 pages found no lift in AI citations after schema was added. Keep basic Organization data so there's no doubt who you are (step 5), but don't expect structured data to get you quoted.

    Source: Ahrefs: schema and AI citations

  • Blocking every AI bot to be safe

    A blanket block on AI bots removes you from ChatGPT search and stops assistants reading your page when a customer asks about you. If training is the worry, block only the training crawlers (step 2).

    Source: OpenAI: publishers and developers FAQ

  • Adopting every new agent standard

    Agent cards, skills indexes and agent payment protocols are each used by around 0.1% of the top sites. Add one when an agent your customers actually use needs it.

    Source: Cloudflare Radar: AI insights

Our own site

What we did on atilab.io, including what we got wrong

We ran every step on this page against our own website before writing it. Most passed. One didn't: our markdown version was being served to simple test tools but not to the coding agents it was built for. Testing with the headers real agents send is what caught it.

Ask Claude or ChatGPT about ATI Lab, or point an MCP client at https://www.atilab.io/mcp, to see the result.

atilab.io, checked 22 October 2026

  • robots.txt lets every AI search, assistant and training bot in, and declares Content Signals (done)Being cited is worth more to us than keeping our pages out of training data.
  • Every page is server-rendered with Next.js (done)Assistants get the full text with no JavaScript.
  • Every page has a markdown version for agents that ask for one (done)Also at /<page>.md, and advertised with a Link header.
  • Bug we found: Claude Code and OpenCode were getting HTML (needs attention)We only served markdown when the request didn't mention text/html, and those agents list both. Fixed on 21 Oct 2026 by comparing the header's preference values properly.
  • An MCP server at atilab.io/mcp, with a published server card (done)Assistants can look up company facts, search and read pages, run our readiness check, and send an enquiry once the person confirms.
  • The same tools for in-browser agents through WebMCP (done)
  • An llms.txt (done)Because it costs nothing, not because it moves anything.

FAQ

Questions about agent-ready websites

What does agent-ready mean?

That AI search engines, assistants like ChatGPT and Claude, and browser agents can reach your website, read its content without running JavaScript, understand what you do, and complete tasks such as booking or getting in touch. It is mostly good web practice applied to a new kind of visitor.

Will this get my business cited in ChatGPT or Claude answers?

It makes you eligible, which many sites currently aren't. Whether you're cited also depends on your content, reputation and what other sites say about you. Nobody can honestly promise citations; anyone who does is guessing.

Should I block GPTBot and ClaudeBot?

It's a business choice. Blocking them stops your content being used to train future models, but doesn't affect ChatGPT search or assistants reading your page live, which use different bots. Don't block OAI-SearchBot, ChatGPT-User, Claude-User or Perplexity-User unless you want to be absent from AI answers.

Do I need an llms.txt file?

No. Log studies show AI fetchers rarely request it and Google says its search ignores it. It does no harm, so add one if it's quick, but put the effort into the first three steps first.

Does my website need an MCP server?

Only if customers would hand real tasks to an assistant, such as checking stock, getting a quote or booking. For most service businesses a server-rendered site with clear contact and booking links covers what agents do today.

How long does it take?

The firewall and robots.txt changes take under an hour. Copy, links and form labels take a day or two. Moving a JavaScript-only site to server rendering, or building an MCP server, is a project of a few weeks depending on the site.

How do I check where my site stands?

Run the free AI Agent Readiness Checker. It fetches your page the way AI assistants and coding agents do, runs 17 checks covering the steps on this page, and shows what an AI can tell about your business, with a fix prompt for each problem.

Last reviewed . Sources are linked under each step. Also see Technical SEO News: which assistants render JavaScript.

Next step

Want it done rather than read about?

We'll scan your site, fix the blockers with your web team, and where it makes sense build the MCP server or booking flow that lets assistants do business with you. We'll tell you what to fix first and how long it takes.
See our AI agent workBook a website review