---
url: https://findutils.com/guides/how-to-make-your-website-ai-agent-ready
title: "Make a Website AI-Agent Ready: Discovery and Permission Checks"
description: "Document an API for AI agents. Check catalogs, schemas, authentication, and spending permissions while separating optional proposals from supported standards."
category: developer
content_type: guide
locale: en
read_time: 12
status: published
author: "olgunozoktas"
published_at: 2026-04-17T18:00:00Z
excerpt: "Document an API for AI agents. Check catalogs, schemas, authentication, and spending permissions while separating optional proposals from supported standards."
tag_ids: ["ai-agents", "geo", "developer-tools", "ai-seo", "mcp", "webmcp"]
tags: ["AI Agents", "GEO", "Developer Tools", "AI SEO", "MCP", "WebMCP"]
primary_keyword: "how to make website ai agent ready"
secondary_keywords: ["agent ready website checklist", "mcp server card", "rfc 9727 api catalog", "webmcp navigator modelcontext", "markdown for agents accept header", "content signals robots.txt", "ai agent discovery 2026"]
tool_tag: "ai-agent-starter-guide"
related_tool: "ai-agent-starter-guide"
related_tools: ["ai-agent-starter-guide", "ai-model-picker", "robots-txt-generator", "llm-requirements-calculator", "claude-code-usage-analyzer"]
updated_at: "2026-09-08T09:09:24Z"
og_image: "/images/content/guides/how-to-make-your-website-ai-agent-ready-cover-20260908.webp"
image_alt: "A small website model gains open document shelves, connected discovery paths, and clearly separated request stations for a mechanical reader."
---

## How to Make Your Website AI-Agent Ready

Make an API usable by agents with clear documentation, accurate schemas, supported connection instructions, and explicit permissions. A catalog can help a compatible client find an endpoint. It does not authorize a request or guarantee search citations. Start with your actual API contract, then add discovery formats that your target clients support.

For a short implementation order, use the [Minimum Viable Agent-Ready Site](#minimum-viable-agent-ready-site) section. Keep content discovery separate from permission to execute a tool.

## What Agent Readiness Means

An agent needs to identify a service, understand its inputs, and handle its result. Your documentation should state the task, required arguments, units, limits, errors, and authentication method.

Search visibility is a separate question. Google states that its AI search features need the same basic SEO work as ordinary Search. No special AI file or schema guarantees inclusion. See [Google's AI feature guidance](https://developers.google.com/search/docs/appearance/ai-features).

The [FindUtils API catalog](/api/) and [MCP documentation](/mcp/) provide connection and tool references. Use the published endpoint instructions; do not assume a client discovers them automatically.

## Choose Discovery Formats by Support

| Resource | Purpose | Adoption check |
|---|---|---|
| OpenAPI document | Describe HTTP operations and input schemas | Confirm the client's supported OpenAPI version |
| `/.well-known/api-catalog` | Link to API endpoints and documentation | Standardized by RFC 9727; client discovery remains optional |
| `Link` response header | Advertise related resources | Use defined relation types and test the consumer |
| `/llms.txt` | Offer an optional documentation index | A community specification, not a search requirement |
| MCP connection instructions | Connect a client to an MCP server | Follow the client's supported transport and authentication flow |
| MCP server card | Advertise server information | Verify the proposal and the client's implemented schema |
| Agent skill index | Distribute task instructions | Verify the chosen format and consumer before publishing |
| Markdown page variant | Provide another representation of the same content | Keep HTTP negotiation and caches correct |

[llms.txt](https://llmstxt.org/) is an optional documentation format. It does not replace HTML pages, internal links, or a sitemap. [RFC 9727](https://www.rfc-editor.org/rfc/rfc9727.html) defines an API catalog location and link relation; it does not require every agent to fetch that catalog.

## Step 1: Publish `/llms.txt`

Open your [Robots.txt Generator](/seo/robots-txt-generator/) and your favorite text editor. Create `public/llms.txt` (or the equivalent for your platform) with a short markdown document that lists your site's purpose and key pages.

```markdown
# Example Corp

> Example Corp (https://example.com) provides a public catalog and a documented search API.

## Pages

- [Pricing](https://example.com/pricing): Plans and limits
- [Docs](https://example.com/docs): Developer documentation
- [API](https://example.com/api): REST endpoints

## AI-friendly
- llms.txt: https://example.com/llms.txt
- llms-full.txt: https://example.com/llms-full.txt (optional, expanded)
```

A compatible client can use this index to locate documentation. Do not assume that a search crawler reads it or gives its links extra ranking weight.

## Step 2: Separate Crawl Access from Usage Preferences

Use `robots.txt` to describe crawler access. Review each crawler's documented purpose before you allow or block it. Search indexing, user-requested retrieval, and model training are different uses.

Some sites also publish Content Signals to express usage preferences. Treat these as an additional preference mechanism. They do not replace crawler rules, authentication, or access control. Their effect depends on the consumer.

Do not copy an instruction that enables training simply because a resource is public. The publisher must choose its own content usage policy. Review crawler access with the [Robots.txt Generator](/seo/robots-txt-generator/) and your host's access rules.

## Step 3: Publish `/.well-known/api-catalog` (RFC 9727)

Create a JSON file at `/.well-known/api-catalog` with MIME type `application/linkset+json`. Each entry anchors an API and links to its service-desc (OpenAPI), service-doc (human docs), and optionally status (health endpoint).

```json
{
 "linkset": [
 {
 "anchor": "https://api.example.com/",
 "service-desc": [
 { "href": "https://example.com/api/openapi.json",
 "type": "application/vnd.oai.openapi+json;version=3.1" }
 ],
 "service-doc": [
 { "href": "https://example.com/api", "type": "text/html" }
 ],
 "status": [
 { "href": "https://api.example.com/health", "type": "application/json" }
 ]
 }
 ]
}
```

A client that supports this catalog can follow the documentation and schema links. Keep those resources public only when that exposure is intended. The catalog does not grant permission to call an endpoint.

## Step 4: Document MCP Connections Before Adding Discovery

Publish the supported endpoint, transport, authentication method, and client setup steps. Verify that a client can initialize a session and list the tools it is allowed to use.

MCP server cards require a separate compatibility check. The [SEP-1649 proposal](https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1649) describes HTTP server discovery. Its draft uses `/.well-known/mcp.json`; this differs from custom paths that some sites publish.

Do not call a custom card an implemented client standard without evidence. A card does not replace the initialization handshake. It also does not prove that Claude Desktop, Cursor, or another client imports that schema.

For FindUtils, follow the current [MCP connection instructions](/mcp/). Check the documented tool arguments before making a request.

## Step 5: Publish Task Instructions for a Known Consumer

A task guide can explain a sequence that crosses several API operations. Document inputs, expected results, failure handling, and the point where a user must confirm a change.

Use an agent skill index only when your target consumer supports its format. Record the schema version and how updates reach that consumer. Do not present a community draft as a universal discovery mechanism.

If the chosen format includes a digest, compute it from the exact published bytes:

```bash
shasum -a 256 public/skills/search-catalog/SKILL.md
```

A digest can detect a mismatch against a trusted expected value. A hash served beside a file does not authenticate its publisher when both can change together.

Do not place API keys, wallet keys, or private operational instructions in public skill files.

## Step 6: Add Useful Link Response Headers

[HTTP Link headers](https://www.rfc-editor.org/rfc/rfc8288.html) can advertise a catalog or related documentation. Use the appropriate relation and media type:

```http
Link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json"
Link: </api/openapi.json>; rel="service-desc"; type="application/vnd.oai.openapi+json;version=3.1"
Link: </api/>; rel="service-doc"; type="text/html"
```

Set these headers through your host or application configuration. Preserve other `Link` values, such as pagination links. Verify the response rather than assuming the configuration applied:

```bash
curl -sI https://example.com/ | grep -i '^link:'
```

Follow every target URL. Check its status, content type, and content. Advertising a file that returns a branded error page does not make it a usable catalog.

## Step 7: Offer Markdown Without Breaking HTML or Caches

A Markdown version can provide the same article content in another format. Generate it from the same source as the HTML page so the two versions stay consistent.

A client can request the alternate format:

```bash
curl -i -H 'Accept: text/markdown' https://example.com/docs/search/
```

The server should return the actual selected representation with the correct `Content-Type`. Keep normal HTML responses available for browsers.

Follow [HTTP content negotiation rules](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-negotiation) when interpreting `Accept`. A substring check misses quality values such as `text/markdown;q=0`. A quality value of zero means that type is not acceptable.

If the representation varies by `Accept`, include the appropriate `Vary: Accept` response header. Preserve existing `Vary` values. Configure the cache to keep the variants separate. Test both request orders so a cached Markdown response does not replace HTML for a browser.

Check titles, code examples, tables, source links, and canonical references in both versions. The alternate version should not add hidden claims or instructions that readers cannot see in the article.

## Step 8: Evaluate WebMCP as a Separate Integration

WebMCP concerns tool interaction in a browser page. It is distinct from connecting to a remote MCP server. The [Chrome early preview announcement](https://developer.chrome.com/blog/webmcp-epp) describes imperative and declarative APIs under development.

Check current browser support and the current API before implementation. Do not copy a historical `provideContext()` example and assume it works in production.

Start with a bounded read operation, such as searching a public catalog. Define input limits and errors. Keep authorization checks at the actual operation, regardless of how an agent invokes it.

A feature check prevents an unsupported API call. It does not prove the operation is safe. Purchases, account changes, deletions, and disclosure of private data require appropriate user permission and server enforcement.

## Choose the Work by Site Type

### A Content Site

Publish useful HTML with descriptive titles, clear internal links, and cited claims. Confirm crawler access and indexing eligibility. Add Markdown or an llms.txt index only when a consumer benefits from it.

Measure indexing, referral visits, and conversions separately. A new file does not prove an increase in citations.

### A Site with a Public API

Publish an OpenAPI document and a working request example. Add error responses, rate limits, authentication rules, and an API catalog if supported. Verify that the documentation matches the deployed contract.

### An Account or Payment Workflow

Document the exact action and its effect before execution. Require the correct account permission. Use a separate test environment for test transactions. Define retry and idempotency behavior where repeated requests can create duplicate changes.

The [paid API guide](/guides/how-ai-agents-discover-and-call-paid-apis/) describes payment discovery. Discovery alone is not spending approval.

### A Developer Tool Collection

Keep browser, REST, and MCP capability lists distinct. A browser tool that needs a file picker does not automatically expose the same task through an API. Use each tool's published [API reference](/api/) or [MCP reference](/mcp/).

## What to Measure

| Check | Evidence to record |
|---|---|
| Discovery | The specific client follows the advertised link |
| Schema accuracy | Valid and invalid inputs produce the documented result |
| Authentication | Unauthorized requests fail without exposing private data |
| Permissions | A tool call cannot exceed the user's approved scope |
| Retry handling | A repeated request has the documented effect |
| Content parity | HTML and Markdown contain the same material facts |
| Search impact | Independent indexing and referral evidence, without attributing every change to one file |

Record the client version and test date. Another client can support a different transport or ignore a discovery proposal.

## Common Mistakes and Fixes

### A Catalog Returns the Wrong Content Type

Check the response body as well as the header. An HTML error page must not carry a JSON media type.

### The Schema Describes Features the Endpoint Does Not Support

Test sample requests against the actual service contract. Remove old parameters and unsupported output formats from the documentation.

### A Public File Contains Private Configuration

Review the generated catalog and task files before publishing. Include only intended public endpoints. Keep credentials and internal hosts out of examples.

### A Read-Only Tool Can Still Disclose Private Data

Read-only describes the operation, not its confidentiality. Enforce account boundaries and field permissions before returning a result.

### A Cached Variant Replaces Another Format

Test requests with HTML and Markdown preferences. Confirm that headers, cache keys, and negotiated content remain consistent.

## Minimum Viable Agent-Ready Site

1. Publish the actual service purpose and supported operations.
2. Document required inputs, examples, errors, and limits.
3. Verify authentication and user permissions.
4. Add connection steps for one supported client.
5. Test a successful request and a rejected request.
6. Add discovery files that this client can use.
7. Keep the documentation synchronized with the service.

This sequence gives you a testable integration. It does not assign a percentile to your site or guarantee adoption.

## Tools Used in This Guide

- **[AI Agent Starter Guide](/ai-agent-starter-guide/)** — Interactive playground for Claude Code, Copilot, Cursor, Gemini, Codex, Windsurf
- **[Robots.txt Generator](/seo/robots-txt-generator/)** — Build a robots.txt with Content Signals in seconds
- **[AI Model Picker](/developers/ai-model-picker/)** — Compare Claude, GPT, Gemini, and local models by context window and price
- **[LLM Requirements Calculator](/developers/llm-requirements-calculator/)** — Estimate RAM, VRAM, and hardware needs for local models
- **[Claude Code Usage Analyzer](/developers/claude-code-usage-analyzer/)** — Parse your Claude Code session logs and visualize usage
- FindUtils **[Tool API](/api/)** — A working example of RFC 9727, OpenAPI 3.1, and `api-catalog` in production
- FindUtils **[MCP Server](/mcp/)** — Current connection instructions and tool references

## Next Steps

- Use the [AI Agent Starter Guide](/ai-agent-starter-guide/) to compare agent workflows.
- Read the [AI search visibility guide](/guides/how-to-audit-website-for-ai-search-visibility/) for content and crawler checks.
- Read the current [FindUtils API documentation](/api/) before calling a tool.
- Verify one supported client before claiming an integration works.

## FAQ

**Q: Does an agent-ready site need an MCP server?**
A: No. An agent can use documented HTTP APIs or readable content without MCP. Add MCP when your target client supports it and the integration has a clear purpose.

**Q: Does llms.txt improve rankings or guarantee AI citations?**
A: No such guarantee follows from publishing the file. It is an optional documentation index. Keep ordinary search requirements and content quality checks in place.

**Q: Is a server card enough to connect an MCP client?**
A: Only if that client implements the card format. Publish direct connection instructions and verify the initialization flow.

**Q: Can read-only tools remain unauthenticated?**
A: That depends on the information they return and the service policy. A read operation can expose private data or consume paid resources. Enforce the relevant access and rate limits.

**Q: Can an agent pay as soon as it discovers a paid endpoint?**
A: Discovery is not permission. The agent needs the user's approved scope, budget, destination, and payment conditions before it spends money.

**Q: How should I handle changing proposals?**
A: Record the format version and supported client. Check current documentation before updates. Keep an ordinary documented connection method available while testing new discovery features.