Make Your Website Accessible to Search and AI Systems

10.1 Distinguish Crawling, Indexing, Retrieval, and Inclusion in an AI Answer

Use separate terms for separate events. Google distinguishes crawling, indexing, and serving search results, and explicitly says that a page is not guaranteed to pass through every stage. Treat each stage as a separate observation in the audit. 

These are useful audit categories, not a universal sequence for every assistant. ChatGPT search can use other search providers and turn a question into several targeted searches. A source can therefore reach an answer through routes beyond a single direct visit from one provider’s crawler. 

For Google Search specifically, a supporting link in AI Overviews or AI Mode requires an indexed page eligible to appear with a snippet. Check the site’s Search generative AI control in Search Console as well: it must permit inclusion. These conditions establish eligibility, not guaranteed inclusion.

State the technical result precisely

A useful finding is: “The public restructuring page now returns the approved content and permits the intended search crawler. Indexing and answer inclusion remain unobserved.” That gives the team a completed technical task and a separate monitoring question.

Conversely, an assistant naming the firm does not prove it fetched the current website. Inspect the answer and available sources using Chapter 6’s monitoring method. Avoid diagnosing every missing mention as a website-access problem.

10.2 Set Separate Policies for Search Crawlers, User-Initiated Fetching, and Model-Training Crawlers

Decide what the firm permits before changing a setting labeled “block AI bots.” Record the provider, purpose, affected public content, desired treatment, approving owner, and date checked. Search discovery, fetching at a user’s request, and collection for model development require separate consideration.

Understand the provider’s actual controls

OpenAI. OAI-SearchBot supports ChatGPT search; GPTBot collects content that may be used for model training. OpenAI says those controls are independent, so a publisher can allow search while disallowing training collection. ChatGPT-User performs certain user-initiated visits and does not control search eligibility. OpenAI cautions that robots.txt rules may not apply to those user actions. 

Anthropic. Claude-SearchBot supports search, Claude-User retrieves content in response to users, and ClaudeBot collects material potentially used for training. Anthropic says its bots honor robots.txt and specifically describes disabling Claude-User as preventing that retrieval. Its policy therefore differs from OpenAI’s stated treatment of user-initiated visits. 

Google. Googlebot governs crawling for Google Search, including its AI features. Google-Extended is a separate robots.txt product token. It covers specified future Gemini training uses and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Grounding supplies source information at the time of a request. Google says this token does not affect Search inclusion or ranking, and it has no separate HTTP user-agent string. A decision to restrict it therefore needs to consider both training and the documented grounding uses. 

Extend the policy record to other providers relevant to your monitoring program. Check each provider’s terminology and current documentation. A control that affects one search product cannot be assumed to govern every product carrying the same company’s name.

Write an implementable policy

For example, a firm might permit specified search crawlers to access approved practice pages, biographies, and publications; permit documented user-requested retrieval of that public material; and make separate decisions about model-training collection. Where a provider combines purposes under one control, record the tradeoff explicitly.

Make the scope concrete. Identify the public website, relevant subdomains, and any separately hosted publication assets. Distinguish a deliberate restriction from an accidental block introduced by a hosting default. These are publishing choices; allowing training collection should not be presented as a proven way to earn recommendations.

10.3 Configure robots.txt, Indexing Controls, Hosting, CDNs, Firewalls, and Bot Protection

Review the controls in the order a request encounters them, then inspect the page that is actually delivered. A content-management-system checkbox, a robots file, and a firewall rule can each affect a different part of the result.

Check the live robots.txt rules

robots.txt communicates crawl rules to cooperating crawlers. It belongs at the site’s root, such as https://example.com/robots.txt. Its scope depends on the protocol, host, and port; a rule for the main website does not automatically govern a separate subdomain. Check the actual public file for every relevant host. 

The following illustrates OpenAI’s separate search and training controls for an otherwise approved public site: 

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Have the implementer merge any such groups with the existing policy. Do not paste them over an existing file. A specific crawler group may take precedence over the wildcard group, so restrictions should not be assumed to carry across groups. Test the effective rule for each relevant crawler and representative path.

Retain the previous configuration and record the reason for the change. Keep the robots file itself fetchable by the systems expected to follow it. A network block and a readable instruction declining a particular use are different mechanisms.

Separate indexing and preview controls

For Google, noindex belongs in an HTML robots meta tag or an HTTP X-Robots-Tag response header. The header can also apply to PDFs. Google must be able to fetch the resource to discover the directive; blocking the URL in robots.txt can prevent that. Google does not support placing noindex as a line in robots.txt. 

Check both the HTML and response headers on priority public pages. A site may have an unintended site-wide indexing restriction, or a template may apply one only to biographies or publications. Compare the live result with the intended policy before removing anything.

In Search Console, check Settings > Search generative AI for each relevant property, meaning the verified site or URL scope. Google’s control can include or exclude links and content from AI Overviews, AI Mode, and generative AI features in Discover. Inclusion is the default, but child properties can inherit or override a parent’s choice. Record the effective setting. This control does not govern other Search ranking or inclusion and does not control model training. 

Google also documents presentation controls: nosnippet prevents textual snippets and use as direct input for AI Overviews and AI Mode; max-snippet limits the amount, with documented exceptions for separately permitted uses; and data-nosnippet excludes selected text from snippets. Other providers can interpret similarly named controls differently. 

Treat these as decisions about public presentation. Keep material qualifications beside the claims they qualify. Hiding a jurisdictional limit or an outcome caveat while leaving the promotional claim available would undermine the accurate representation this playbook seeks.

Check the delivery infrastructure

A content delivery network, or CDN, sits between visitors and the website to deliver content. A web application firewall, or WAF, filters requests. Bot protection may challenge or block automated traffic. Ask the hosting owner which services operate on the firm’s domains and who can inspect their decisions.

Google’s minimum technical requirements include an accessible page, a successful HTTP 200 response, and indexable content. Check the returned body as well as the status: a success code attached to a challenge screen does not establish that the requested practice page was delivered. 

Use delivery logs to identify the responsible rule before changing it. For example, Cloudflare’s Security Events can show the action, triggered rule, source IP, user agent, and path, with availability varying by plan. Its guidance also distinguishes exceptions according to the security feature responsible for the block. 

Make a targeted correction for the intended verified traffic and approved public content. Retest the affected page and the protected areas. Investigate repeated failures or throttling rather than disabling protection across the website. OpenAI likewise tells site owners to check hosting and CDN acceptance of its published searchbot IP ranges. 

10.4 Make Important Public Information Available Through Reliable HTML, Rendering, Navigation, and Internal Links

Check what arrives before relying on what a person sees after interacting with the site. The important public facts include the firm name, service scope, professional’s role, applicable geography, dates, and qualifications attached to claims.

Inspect initial and rendered content

Initial HTML is the document returned by the server. Rendering is the process of turning that document and its resources into a usable page, potentially running JavaScript. Google can render JavaScript, but its documentation notes that not all bots can. It recommends considering server-side rendering or prerendering, which makes content available in the returned HTML. 

For priority firm information, favor dependable delivery in the initial HTML where feasible. This is a practical way to reduce dependencies, not a universal requirement imposed by every AI system. If the site relies on rendering, verify the routes that matter instead of assuming either that all bots execute JavaScript or that none do.

Google says Search does not interact with a page to load content through actions such as clicking or scrolling. Check whether a tab, filter, accordion, or consent interface merely changes presentation or actually retrieves missing content. The visual design alone does not answer that question. 

Consulting example. A practice description appears only after a visitor chooses an industry from a dropdown. The firm can provide the approved description at a stable public URL, link to it from the practice area, and verify that the description is present when the URL is fetched directly.

Law firm example. A biography displays admissions in an interactive panel, but those admissions are missing from the fetched and tested rendered content. Correct the template so the professional’s name, role, admissions, and relevant limitations remain available together. Adding more prose elsewhere would leave the access defect unresolved.

Provide ordinary routes to important pages

Google’s link guidance calls for links with an <a> element and an href destination that resolves to a real address. Scripted click events alone are unreliable for link extraction. Use descriptive link text and connect relevant practice pages, professional profiles, publications, and supporting material. 

Test the navigation from a public starting point. A useful service page should have a clear route from the relevant practice or industry page. An individual biography should be reachable through the team or appropriate practice. Internal search can supplement those routes.

Make publications intelligible outside their presentation format

Accounting example. A public reporting guide is distributed as a PDF whose important tables are images. Check the text actually available from the file. Provide a public HTML page explaining its purpose, scope, date, author, and approved substantive findings, with a link to the full guide. Decide separately which material remains gated.

Apply the same editorial principle to charts, recorded webinars, and slide decks: give the public audience an accurate textual explanation of the information the firm wants understood. Preserve caveats and dates in any alternative version. Do not supply bots with stronger claims or hidden instructions that ordinary readers would not receive.

10.5 Check Sitemaps, Canonical URLs, Redirects, Structured Data, and the Limits of Proposals Such as LLMS.txt

Once important content can be fetched, check the signals that identify its location, relationships, and current version. These controls should describe the website the firm actually publishes.

Maintain sitemaps and update signals

A sitemap lists URLs for discovery. Check that the intended public pages appear under their preferred addresses and that entries do not point to retired pages, staging hosts, or restricted documents. Submit or reference the sitemap through the search engine’s documented process and inspect processing errors.

Bing recommends XML sitemaps with accurate lastmod values reflecting actual content changes. It also supports IndexNow notifications for added, updated, or removed URLs. Its guidance distinguishes these signals from any guarantee of appearance in AI results. A submission confirms notification, not a completed crawl or citation. 

For a revised partner biography, record when the page changed and when an update signal was submitted, if applicable. Verify the new content separately. Avoid resetting every page’s modification date simply because the sitemap was regenerated.

Keep preferred URLs and moves consistent

A canonical URL identifies the preferred version among duplicate or substantially similar pages. Google treats redirects and canonical annotations as stronger signals than sitemap inclusion. Its guidance calls for consistent preferences across those mechanisms and internal links; the declared preference does not force Google’s selection. 

When a practice page moves, map the old address to the relevant successor, implement the appropriate redirect, and update internal links and the sitemap. Test both addresses. A live old address that returns a generic home page may fail the reader’s original purpose.

Check pages that use shared templates. A copied canonical value could point several distinct services to one unrelated URL. Preserve genuinely different service, jurisdiction, and professional pages; use canonicalization for the versions it is intended to reconcile.

Use structured data to describe visible facts

Structured data supplies labeled information about a page and its subject. Google recommends JSON-LD for supported rich-result features and requires the markup to represent visible, relevant content accurately. Passing a syntax test does not establish that the facts are true or guarantee a rich result. 

Have the web owner assess the appropriate organization, individual profile, article, and breadcrumb representations under current documentation. The marketing owner checks the underlying facts from Chapters 8 and 9. A technical validator cannot decide whether an office is staffed, a professional still works at the firm, or a named person actually wrote an article.

Review markup when the corresponding public fact changes. Do not add credentials, ratings, affiliations, or locations merely because a field is available. The machine-readable account and the visible page should describe the same firm and work.

Treat llms.txt according to its documented use

The llms.txt proposal describes a concise Markdown guide that helps agents navigate a site’s information, with links to more detailed content. It remains a proposal with its own evolving specification. It is not a replacement for crawler policy or access protection. 

Google’s current generative-AI guidance explicitly says Google Search ignores these special files and that maintaining them neither helps nor harms Search visibility or rankings. That statement concerns Google Search; it does not establish how every other system uses such a file. 

Consider an optional file when there is a documented consuming system or a useful workflow, and an owner can keep it accurate. Ask a vendor what will use it and how that use will be verified. Publishing one is not evidence that an assistant has read it, and maintaining it should not displace repairs to inaccessible priority pages.

10.6 Verify Public Crawler Access While Protecting Client Portals and Restricted Files With Appropriate Access Controls 

Test both sides of the intended policy: approved public material should be obtainable through the relevant routes, and restricted material should remain protected.

Protect the resource itself

The Robots Exclusion Protocol expressly states that robots.txt is not a substitute for content security and that listed paths are publicly discoverable. Protect client portals and confidential files through authentication and authorization at the actual resource. An unlisted link or an indexing directive does not provide that protection. 

Use an authorized test record or harmless test file to check restricted access. Inspect direct file URLs as well as the pages containing their links, including separate storage hosts. A login on the portal’s landing page is insufficient if the underlying download is publicly accessible.

Engineering example. The firm publishes an approved project summary while keeping detailed client drawings in a restricted repository. The summary can be available to search systems. Its public status does not justify adding the drawings to a sitemap or exempting the repository from access checks.

If confidential material is exposed, secure the resource and follow the firm’s incident process. Do not expose it further merely to let a crawler read noindex. Correcting search listings is a separate task from restoring control over the original material.

Verify the request and the returned content

A user-agent name is not sufficient proof of a crawler’s identity. Google documents verification using published IP ranges or reverse DNS followed by forward confirmation. OpenAI and Anthropic also link current crawler IP information from their documentation. Have the technical owner use the method applicable to the particular provider and agent. 

A test request using a copied bot name can help diagnose different responses, but it remains a simulation. Record that limitation. Look for authentic provider traffic in the relevant delivery records, and retain the path, time, identity check, response, and available evidence of the returned content. A verified operator is still not authorized to enter a client portal.

In Google Search Console, inspect both the indexed information and a live URL test. Review the tested HTML, response information, and rendering where relevant. Google warns that the live test does not cover every indexing condition or guarantee indexing. A test performed by its inspection tool is also distinct from an observed ordinary crawl. 

Use a repeatable verification procedure

1. Select the URLs. Include priority practice, professional, publication, and public-file URLs, with a representative example from each materially different template or host. Add an authorized restricted test resource.
2. Record the intended treatment. Identify which content is public, the relevant provider and purpose, and the expected indexing and presentation settings.
3. Inspect actual delivery. Capture redirects, final URL, status, headers, and content without relying on a staff login. Check initial and relevant rendered content.
4. Compare controls with policy. Review effective crawl rules, indexing directives, preview settings, the Search generative AI setting where relevant, canonical preferences, links, sitemap entries, and any structured data.
5. Verify provider access. Use supported inspection tools and identity-checked delivery records. Label simulations, missing evidence, and observations from different dates accurately.
6. Retest after the change. Recheck the original failing case and the restricted test resource. Record the deployed correction and any remaining exception.
7. Observe later outcomes. Track indexing and the relevant Chapter 6 answer cases separately. Schedule review after material template, hosting, security, or provider-policy changes.

Close the technical issue when its stated acceptance conditions are met. If provider retrieval or indexing remains unobserved, record that state and the next check. Keep the result narrow enough that another person can understand what was tested and reproduce it.

Practical Output: Complete the Technical Access Audit and Implementation Checklist

Create one audit record for each selected URL or shared technical defect. Record the URL and page owner; public or restricted status; intended providers and purposes; observed responses and controls; evidence and test date; required correction; implementer; acceptance condition; and retest result. Link a shared template defect to the pages it affects.

Turn confirmed defects into the Chapter 7 backlog. Give each task a specific completion condition. For example: “The public employment-investigations page returns the approved service text and jurisdictional limits, permits the intended search access, and has consistent links and canonical settings. The restricted client-file test remains protected.”

Use Section 10.6’s procedure for verification and retain the evidence with the task. Assign ongoing ownership of crawler policy, site delivery, and content changes. The completed audit should tell the managing partner what public information is accessible, what still needs repair, and which subsequent visibility outcomes have actually been observed.