All 51 tests

Your robots.txt tells crawlers what they may read. One stale line, often copied from an old template, can shut out the assistants people now use to find you.

Even when robots.txt says yes, a firewall or bot rule can turn assistants away before they reach a page. We ask as each crawler and record what comes back.

A CAPTCHA or bot check on an ordinary page stops an assistant at the door. Those belong on forms and logins, not on pages anyone can read.

A machine-readable line in robots.txt saying whether AI may search, quote or train on your content. Having a stated position — any position — is the point.

A few plain sentences in robots.txt saying what you permit. Journalists and lawyers read this file, and most sites say nothing in it.

Old settings that told search engines not to show an excerpt also stop an assistant quoting your page in an answer.

A canonical link names the true address of a page. Without one, the same content reads as several competing pages.

Meaningful elements rather than an undifferentiated wall of containers. It is the difference between a document and a soup of boxes.

One main heading, then sub-headings in order. Assistants use headings to work out which part of a page answers a question.

A file at /llms.txt listing your most important pages. It is early and cheap, and it forces a useful conversation about what actually matters on your site.

Structured data that links together — your organization, your site, this page — rather than a scattering of disconnected labels.

Links from your markup to authoritative records like Wikidata or ROR. Without one, an assistant is guessing which organization of your name it has found.

Founding date, leadership, address, contact details, legal identifiers. Thin About pages are the most common weakness we see.

If your content only appears after JavaScript runs, a crawler that does not run it sees an empty page. This is the heaviest check in the Index.

Tabs and accordions are fine, as long as the text is already in the HTML before anybody clicks.

Tuition, hours, fees, comparisons — in a real table with header cells, not an image and not a grid of divs.

Stable IDs on your headings let a model cite the exact passage rather than the whole page.

Alt text is the only text an image has. Decorative images should say so with an empty alt rather than describing themselves.

A fact locked inside a PDF, a scan or an infographic is a fact an assistant cannot quote.

Schema generated from your content model survives the next content change. Hardcoded schema rots quietly and nobody notices for a year.

Every page below the homepage should declare where it sits, so a model knows a program page belongs to a department.

Use the types that match what you actually publish — courses, programs, clinicians, grants — rather than a generic WebPage on everything.

Mark up real questions and answers. Do not invent an FAQ block that is not on the page: fabricated markup is a policy problem, not a shortcut.

Say whether the page is free to read, and link the entities it is actually about. Both are things an assistant would otherwise have to assume.

Prices and offers in markup, and current. A stale price in structured data is worse than no price at all.

Expose your site search as a pattern an agent can call, instead of a box only a human can type into.

Markup that does not parse, or that leaves out what its type needs, is served to machines that cannot use it — and nothing on the page says so.

Machine-readable licensing terms, so an AI company can find out what you allow without emailing you first.

Licensing that travels with the content — in your feeds and your structured data, not only in one file nobody fetches.

Verified agents get through and unsigned scrapers claiming to be them do not, because the edge checks a signature instead of trusting a user-agent string.

PDFs, images and JSON cannot carry a meta robots tag, so the directive has to travel in the HTTP header instead.

A missing page that answers 200 OK teaches a crawler that nothing is ever missing, and spends its time on pages that are not there.

Every extra redirect is another chance to be dropped. One hop, straight to the destination.

Filters and calendars can generate millions of URLs. A crawler spends its budget in there instead of on the pages you care about.

A lastmod date should mean the content changed, not that you deployed. Once it is noise, crawlers stop trusting it.

Separate sitemaps per content type turn “coverage is bad” into “coverage is bad for news,” which is a problem somebody can fix.

Page two and everything after it has to be reachable by something that does not click.

Authors who resolve to real people with real credentials, on pages of their own — not a name in a byline and nothing behind it.

Sign substantive pages, and point the byline at the author's entity rather than repeating their name as text on every article.

Health, legal and financial pages should name their reviewer in the markup, not only in the design where a machine cannot see it.

A modified date that changes on every deploy is not a freshness signal. It is noise, and it costs you the trust of the real one.

Where you cite a primary source, make the citation machine-readable so the chain from claim to evidence survives.

Provenance that survives into the delivered file, so an image can prove where it came from after somebody else has reposted it.

If you have a public API, describe it in a spec an agent can read, and version it so today's answer is still true next quarter.

Put your agent capabilities where agents look for them, rather than in a URL somebody has to be told about.

An agent filling in your form has to know what each field is for. Labels, names and autocomplete tokens are how it finds out.

Challenge the actions that need challenging. Searching and filtering are not those actions, and a captcha there ends the visit.

A clean Markdown version of each page: none of the navigation, none of the markup a model has to wade through to reach your words.

Publishing the Markdown is not enough. The HTML page has to point at it, or nothing will ever find it.

Ask the same URL for Markdown and get Markdown back. One address, two representations, no second set of links to maintain.

Generated representations should come from the CDN. Otherwise every request rebuilds them at origin, and you will turn them off when the bill arrives.