AI agents are already visiting your website. Some are checking a price for someone. Some are comparing you with three competitors. A few are trying to fill in your contact form.
Whether they succeed depends on things most websites were never built for. This guide walks through exactly what to change, in the order that pays off fastest.
If you are new to the idea, start with our plain-English explainer, What Is Agentic SEO? This post is the hands-on follow-up.
The short version: pick the three journeys that make you money, find out which agents already visit, stop blocking the ones that act for real customers, put your key facts in plain HTML and schema, make your forms easy to complete, remove unnecessary gates, publish the details agents compare, and re-test every month.
Before you start: pick three agent journeys
Agentic search optimization is not a site-wide makeover. It works best when you focus on the few tasks an agent is most likely to attempt on your site.
Ask: if a customer told their AI assistant to deal with us, what would it try to do?
| Business type | Task an agent tries | Pages that must work |
|---|---|---|
| Online store | Compare products, check stock and delivery, buy | Product pages, shipping and returns, checkout |
| SaaS company | Compare plans, check features, book a demo | Pricing, feature comparison, demo form |
| Clinic or local service | Check services, hours and location, book | Service pages, contact, booking |
| Agency or B2B service | Shortlist providers, request a quote | Service pages, case studies, quote form |
| Hotel or restaurant | Check availability, menu or policies, reserve | Rooms or menu, policies, reservation |
Write down your three journeys. Every step below is checked against them.
Step 1: Find out which agents already visit
Before changing anything, get a baseline. AI agents identify themselves with a user-agent string, so your server logs will show them.
Look for these names in your access logs (your host’s control panel usually has a “Raw Access” download):
ChatGPT-User OpenAI, acting for a ChatGPT user
Google-Agent Google agent features such as Project Mariner
Perplexity-User Perplexity and the Comet browser, acting for a user
Claude-User Anthropic's Claude, acting for a user
If you have command-line access, one line gives you a count per agent:
grep -oE "ChatGPT-User|Google-Agent|Perplexity-User|Claude-User" access.log | sort | uniq -c
Then add an AI referral view in GA4. Create an exploration filtered on session source matching this pattern:
chatgpt\.com|perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com|claude\.ai
Note the numbers. They are your “before” picture.
Step 2: Separate training bots from customer agents
This is the most common mistake we see. In 2024 many sites blocked every AI bot to stop their content being used for training. Those same rules now block agents acting for paying customers.
The fix is to treat them separately. Most AI companies run different bots for different jobs:
| Company | Training / indexing | Acting for a user |
|---|---|---|
| OpenAI | GPTBot (training), OAI-SearchBot (search) | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-User |
| Perplexity | PerplexityBot | Perplexity-User |
| Google-Extended (controls Gemini training use) | Google-Agent |
A sensible starting robots.txt, if you want to opt out of training but stay usable for customers, looks like this:
# Opt out of AI model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Stay visible in AI search and usable by customers' agents
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
Whether to block training is your business decision. The point is to make it on purpose, bot by bot.
One exception: Google says Google-Agent generally doesn’t follow robots.txt, because a person started the request. If you ever need to limit it, do it in your CDN or firewall. And if you use a firewall or bot-management service, check that it isn’t silently blocking these agents. That happens more often than robots.txt mistakes. For more on how these bots crawl, see how AI search crawlers discover and understand your website.
Step 3: Make key content readable without JavaScript
Many agents and fetchers read the HTML your server sends, before any scripts run. If your prices, plans or product details are loaded by JavaScript afterwards, those agents see an empty space.
Three quick ways to check a page:
- View source. Right-click the page and choose “View page source”. Search for your price. If it isn’t there, it’s loaded by script.
- Disable JavaScript. In Chrome DevTools, open the command menu and run “Disable JavaScript”, then reload.
- Fetch it raw. Run
curl -s https://yoursite.com/pricing | grep -i "per month"and see what comes back.
If the key facts are missing, ask your developer about server-side rendering or pre-rendering for those pages. You don’t need to rebuild the whole site. Start with the pages in your three journeys.
Step 4: Optimize structured content for agentic search systems
Structured data is how you hand an agent clean, unambiguous facts. For agentic search, the most useful types are Product and Offer, Service, LocalBusiness, Organization and FAQPage.
For a product or plan, the details agents compare most are price, currency, availability and what is included. A minimal example:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Product",
"name": "Team Plan",
"description": "CRM for teams of up to 10 users.",
"offers": {
"@type": "Offer",
"price": "49.00",
"priceCurrency": "USD",
"availability": "https://schema.org/InStock",
"url": "https://yoursite.com/pricing"
}
}
</script>
Two rules matter more than the markup itself:
- It must match the page. If the schema says $49 and the page says $59, you lose trust with search engines and agents alike.
- It must stay current. Generate schema from the same data source as the visible page, so a price change updates both.
Structure your visible content the same way. Clear headings, short spec lists and comparison tables are far easier for an agent to extract than long paragraphs. Our framework for optimizing content for AI search covers the writing side.
Step 5: Make forms and buttons easy for agents to use
Agents understand a form through the same information screen readers use: labels, field types and button text. A form that is hard for a screen-reader user is usually hard for an agent too.
Compare these two fields:
<!-- Hard for agents -->
<input type="text" placeholder="Email">
<div class="btn" onclick="send()">Go</div>
<!-- Easy for agents -->
<label for="email">Work email</label>
<input id="email" type="email" name="email" autocomplete="email" required>
<button type="submit">Book a demo</button>
A quick checklist for every form in your three journeys:
- Every field has a real
<label>, not just placeholder text. - Field types match the data:
email,tel,url,date. - Buttons are real
<button>elements with clear text such as “Request a quote”. - Error messages are written in text next to the field, not shown only as a red border.
- No fields appear only on hover, and as few steps as possible.
Step 6: Remove gates that stop agents at the door
Agents give up quickly. Each of these can end a visit before it starts:
- Blanket CAPTCHAs on every page. Keep them for genuinely risky actions such as login or payment.
- Cookie walls and pop-ups that cover the content until dismissed.
- Login walls on prices or specs that aren’t truly confidential.
- Forced location redirects that send an agent to the wrong country site.
- Aggressive rate limits that block anything that loads pages quickly.
On the security side, the industry is moving toward verified agents. Web Bot Auth, a draft standard already supported by Cloudflare and Akamai and being tested by Google, lets agents cryptographically sign their requests. If your bot-management tool supports verified bots, allow them rather than challenging them.
Step 7: Publish the facts agents compare
When an agent builds a shortlist, it compares the same handful of facts across every option. If yours are missing, you are harder to recommend than a competitor who shows them.
Make sure these are on a public, crawlable page:
- Prices, or at least starting prices and ranges
- What is included in each plan, package or service
- Delivery times, service areas and availability
- Returns, refunds, cancellation and guarantee terms
- Opening hours, booking lead times and contact options
A tip from our own audits: write one short paragraph on each key page that answers the comparison directly. For example, “Our Team plan costs $49 a month for up to 10 users and includes phone support.” Agents lift sentences like this almost word for word. B2B companies should read how to get your SaaS into AI-generated shortlists for more on this.
Step 8: Add machine-readable extras where they fit
These are not essential for every site, but they help agents find and trust your information:
- Clean XML sitemaps listing only live, canonical pages.
- An llms.txt file that summarises your key pages in plain text. It is not an official standard, but it is cheap to add.
- Product feeds for stores, kept in sync with live prices and stock. See our guide to optimizing product pages for AI.
- Agentic commerce channels if you sell online. OpenAI’s Agentic Commerce Protocol powers Instant Checkout in ChatGPT, Google and Shopify’s Universal Commerce Protocol publishes store capabilities at
/.well-known/ucp, and Shopify and Stripe offer agent channels you can switch on. Our ecommerce AI SEO team can help you choose.
Step 9: Test with a real agent, then repeat monthly
Run each of your three journeys through a live browsing agent, such as ChatGPT’s agent mode. Give it a plain instruction, for example: “Find the price of the Team plan on yoursite.com and book a demo for Tuesday.”
Then record three things for each journey:
| Measure | What to note |
|---|---|
| Completed? | Yes, partly, or no |
| Where it struggled | The exact page and step |
| What it said | Ask the agent afterwards what was hard to find |
Repeat this every month alongside your log counts and GA4 referrals. Optimizing for agentic traffic is not a one-off project. Agents and the rules they follow are changing quickly, so a monthly check keeps you ahead.
What to fix first
If you can only do a few things this month, this order gives the most return for the least effort:
| Fix | Effort | Impact |
|---|---|---|
| Unblock customer agents in robots.txt and firewall | Low | High |
| Publish prices or ranges | Low | High |
| Add labels and real buttons to key forms | Low | High |
| Remove blanket CAPTCHAs and pop-ups | Low to medium | High |
| Add accurate Product, Service or LocalBusiness schema | Medium | Medium to high |
| Server-render key content | Medium to high | High where needed |
| Agentic commerce channels and feeds | Medium | High for stores |
What not to do
- Don’t show agents different content from people. Serving special pages to bots is cloaking, and it damages trust with search engines.
- Don’t stuff schema with claims, reviews or prices that aren’t on the page.
- Don’t rebuild everything at once. Fix your three journeys, measure, then expand.
Frequently asked questions
Will optimizing for AI agents hurt the experience for human visitors?
No. Almost every change in this guide, clear prices, labelled forms, fewer pop-ups and faster pages, also improves the site for people. Accessibility and agent-readiness overlap heavily.
Do I need an llms.txt file?
It is optional. It is not an official standard and no major AI company has confirmed using it for ranking. It takes little effort, so many sites add it, but the steps above matter far more.
How long does agentic search optimization take?
Most of the high-impact fixes, bot rules, pricing, forms and gates, can be done in two to four weeks. Rendering changes and commerce integrations can take longer depending on your platform.
Is this different from ChatGPT SEO?
They overlap. ChatGPT SEO focuses on getting found and cited in ChatGPT’s answers. Agentic SEO focuses on what happens when the agent then visits your site to act.
Want help with this?
Our agentic SEO team runs your key journeys through ChatGPT agent, Google-Agent and Perplexity, finds every blocker, and delivers the fixes as clear tickets your developers can ship. If you’d like a quick read on how AI sees your brand first, try the free AI visibility checker or book a full AI search visibility audit.