# The AI Readiness Index **Version 11.0 — specification** Effective 2026-09-24. **License:** This document and the tests it defines are published under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). Copy it, adapt it, build your own scanner from it — for any purpose, free, forever — as long as you credit TEN7 as the creator of the AI Readiness Index and license what you build the same way. The scanner that computes a score from this document is separate, proprietary software. Full terms at [readinessindex.io/license](https://readinessindex.io/license). ## What this is This document defines what a website has to do to score 100 on the AI Readiness Index. It is written for two readers at once: the engineer building the scanner in this repository, and the consultant explaining a score to a client. Every check has an ID, a weight, a detection method, and a scoring rubric. If a check is not described here, the scanner should not score it. We built this because "is your site ready for AI" is currently an unanswerable question. Vendors assert things. Nobody measures. Our answer is a number you can reproduce, defend line by line, and re-run in six months to show what changed. The Index measures one thing: **can machines find, read, trust, and use this website?** It does not measure design quality, marketing effectiveness, or whether the content is any good. A beautiful site can score 12. A plain one can score 94. ## How to read a score A result has three parts, and all three matter. **The score (0–100).** A single weighted number across eight dimensions. **The gate status.** Two checks are gates. If a site blocks AI crawlers in `robots.txt` or rejects them at the CDN, `gated` is `true` — a separate fact reported beside the score, not a ceiling on it. The score still says how well the site is built; the gate says whether any of it is reachable today. A perfectly marked-up site that returns 403 to `ClaudeBot` is not badly built, it is invisible, and those are different problems with different fixes. Blending them into one capped number made a well-built gated site indistinguishable from a badly-built one — see the changelog for 4.0. **The automated coverage.** The percentage of applicable weight the scanner verified on its own. Some tests — log retention, a monthly prompt panel, whether schema is generated from fields or hardcoded in a template — cannot be seen from outside. Those are scored from attestation and flagged. A score of 82 with 91% automated coverage is a much stronger claim than 82 with 64%. Report all three together. A bare number invites arguments we don't need to have. ### Grade bands | Score | Grade | What it means | |---|---|---| | 90–100 | Exemplary | Machine-legible by design. Remaining work is optional or emerging. | | 75–89 | Strong | Solid fundamentals with specific, addressable gaps. | | 60–74 | Adequate | Findable and readable, but leaving significant value unclaimed. | | 40–59 | Weak | Partially legible. Real content is not reaching answer engines. | | 0–39 | Not ready | Structural problems. Start with retrievability before anything else. | | Any | **Gated** | A separate fact, not a grade. AIR-1.1 or AIR-1.2 failed: AI crawlers cannot reach the site, so none of the score above is reaching one today. Before 4.0 this capped the total at 40, which blended two facts into a number comparable to neither. | `grade` is always the band the real, uncapped `total` falls in — a gate failure never changes it. `gated` is the separate boolean the report checks to decide whether to lead with the access problem, so it can say "Strong at 81.2, and gated — none of that is reaching an assistant today" rather than either hiding the score behind the word Gated or letting it stand alone with no warning attached. ## The scoring model ### Bands Every check scores on a 0–4 band. The check's contribution is `weight × (band ÷ 4)`. Unless a test says otherwise, use the **default coverage rubric** against the sampled page set: | Band | Condition | |---|---| | 4 | Present and correct on ≥95% of applicable sampled pages | | 3 | Present and correct on 75–94% | | 2 | Present on 40–74%, or present everywhere but with correctness errors | | 1 | Present on 1–39% | | 0 | Absent, or present but broken everywhere | Coverage percentages are weighted, not raw page counts. Each money page (see Sampling, step 4) counts as **2** in both the numerator and the denominator; every other sampled page counts as 1. Ten money pages in a 250-URL sample therefore score as a 260-page sample. A failure on `/apply` costs twice what the same failure on a 2019 press release costs, which is the reason the client names those URLs in the first place. Tests that are inherently site-wide and binary (a file exists, a header is set) define their own bands in the test entry. ### Points | # | Dimension | Points | Tests | |---|---|---|---| | 1 | Primary Indicators | 30 | 13 | | 2 | Rendering and extraction | 15 | 6 | | 3 | Structured data and the entity graph | 15 | 8 | | 4 | Crawl, index and licensing hygiene | 12 | 10 | | 5 | Authorship, provenance, freshness | 12 | 6 | | 6 | Agent interfaces | 8 | 4 | | 7 | Alternate representations | 8 | 4 | | | **Total** | **100** | **51** | Beside the hundred, and counted separately because they are not part of it: 7 leading indicators and 9 best practices. Dimensions 1, 2 and 3 (First contact, Rendering and extraction, and Structured data and the entity graph) carry 57 of the 100 points — just over half. First contact is heaviest by construction: it is the highest-signal test pulled from six other dimensions, not a new editorial judgment about what matters most. Dimension 8 carries only 7 of 100 points despite being the most commercially valuable work we do. Weight here reflects what a scanner can observe from outside a site, not what the work is worth. Track measurement maturity in the engagement, not in this number. Dimension 9 carries none of the 100 — by design, not omission. It holds every `leading`-tier test (see *Versioning and change control*, 4.0), grouped on its own rather than scattered through the dimensions that would otherwise host them, so a site's score can never be quietly diluted by a practice almost nobody has built yet. They are still run and still reported. ### Applicability and N/A Each test declares an applicability rule: - **`always`** — scored on every site. - **`conditional`** — scored only if a precondition holds. State the precondition in the check. - **`vertical`** — scored only for a declared client vertical. Fifteen are published; `other` is one of them and carries no expected type set, so a client who tells us their industry is not on the list is not marked against a standard that does not exist for them. A conditional test states its precondition in one of two forms, and never in both: - **A named site fact** — a single named predicate over collected evidence, such as `sample_contains_tables` or `site_has_search`. Predicates are code, not expressions in a config file, so each one is unit-tested and each one names itself in the N/A reason an auditor reads. - **A dependency on another check** — written as `requires: AIR-4.1 ≥ 2`. A dependency is satisfied when the named check scores **band 2 or higher**. Below band 2 the foundation is broken, and measuring what sits on top of it produces a finding nobody can act on: telling a client their malformed `llms.txt` is well cached is noise. A dependency that is itself N/A cascades N/A to everything that depends on it. Four checks carry a dependency: AIR-4.2 → AIR-4.1, AIR-7.2 → AIR-7.1, AIR-7.4 → AIR-1.10 **or** AIR-7.1, BP-6 → BP-5. #### Facts the scanner cannot observe Eight checks are conditional on a business fact no crawler can determine: AIR-4.1 and AIR-4.3 (licensing and access posture), AIR-5.3 (YMYL content), AIR-5.5 (research and policy publishing), AIR-5.6 (original asset production), LI-7 (permission for full-text inclusion), and BP-1 with BP-2 (named critical flows). AIR-3.3 needs a declared vertical. These come from an engagement profile supplied per client — never from inference. The scanner does not sniff a site for medical keywords and decide it is YMYL. A guess inside a number we ask a client to defend line by line is worse than saying we have not checked. But not checking is not the same as not scoring, and from 1.0 to 9.x the Index treated them as the same thing. #### Every test is scored (10.0) **On any run a person is shown, every test outside dimension 9 comes back with a band.** The denominator is the full 100 points, on every site, every time. Applicability resolution still runs, and still decides exactly what it always decided: whether this test has anything to find on this site. What changed in 10.0 is only what happens next. A verdict that used to end the test's participation in the run now resolves to a band, in one of three ways: | What resolution found | Band | What the report says | |---|---|---| | The site has none of the thing the test is about — a precondition that evaluated false, a vertical the site is not in, a dependency whose parent had nothing to find | **4** | *There is none of this on the site, so there is nothing to fix.* | | The answer is real but invisible from outside — an undeclared profile fact, a manual test with no attestation, or evidence we read and could not settle | **2**, held | *We cannot see this from outside your site. Confirm it and this scores in full.* | | The test this one depends on is not passing | **0** | *The test this one builds on is not in place yet.* | The middle row is a **ceiling**, not an earned band, and uses exactly the mechanism `unattested_ceiling` already used: the band is held at 2, `band_capped_at` and `cap_reason` record why, and `earnable_points` falls accordingly. A held test can never read as a pass, and confirming it can always move the number. Points above a ceiling are also excluded from the backlog: a test sitting at its ceiling has nothing on the site to fix, and a list headed *what to fix* must not open with work nobody can collect. Every one of these carries its `unscorable_reason` in the result file and an evidence row naming what was looked for, what came back, and what it scored — the same audit trail as any other test. An auditor can still see exactly what we chose not to measure; it is now a fact about a scored row rather than a missing one. **Why.** A test with no band was invisible to the arithmetic *and* to the reader. A self-serve visitor declares nothing, so more than half the Index left the run — a third of it purely for want of an engagement profile — and the score that came back was a percentage of a denominator that silently changed from site to site. Two sites could not be compared, and a client reading their own report found rows that gave them neither a result nor anything to do. Scoring every test costs the Index nothing it was actually measuring and gives back the property its name claims: a hundred points, all of them accounted for. **What this is not.** It is not scoring an N/A as zero — penalizing a museum for having no `Physician` markup is the mistake this Index was built to avoid, and the first row above is what prevents it. Nor is it paying a site for our own blind spots: anything we could not see is held at 2, never granted, which is the same dishonesty pointing the other way. **Two runs are exempt.** A **quick scan** looks at thirteen tests and says so; scoring the other fifty-five would hand it points for work it never did and move its ceiling from site to site. The **census** exists to report what share of sites do a given thing, so a test it could not judge has to stay out of that tally rather than be counted as a site with nothing to fix. Both run with `score_everything` off, and neither is a report a client is handed. ``` score = (Σ earned_points ÷ Σ applicable_weight) × 100 ``` ### Resolving external identifiers AIR-1.12 asks whether a site's `sameAs` links point at real authority records. Classifying a URL as a tier-1 identifier needs no network call; confirming it still resolves does, and those hosts belong to somebody else — ORCID, Wikidata, a registry, a university directory. Requesting them tells a third party which site is being audited. That is a real disclosure, and it is why resolution is a deliberate choice rather than a default in the reference scanner's CLI. A scan that does not resolve them holds AIR-1.12 at band 3: the links are classified, not confirmed. Two rules apply wherever it is turned on: - **A third party's availability is never a fact about the client's site.** A timeout, a DNS failure or a rate limit against ORCID records that one link as unconfirmed. It must not fail the scan, and it must not score the client down. - **Say that it happened.** The evidence records which external hosts were contacted, so a client can see what their audit disclosed and to whom. ### Rounding, and the full accounting **Every points figure shown to a reader is a whole number. Every points figure used in the arithmetic is not.** A test contributes `weight × band ÷ 4`. No weight in the Index divides by four, so a weight-2 test at band 3 contributes exactly 1.5 points and a weight-5 test at band 2 contributes exactly 2.5. Reports print those as 2 and 3. This is deliberate. The Index exists to be defended out loud, line by line, in a room with the people who have to do the work. Nobody argues about three quarters of a point, and a column of figures like `2.25` reads as a spreadsheet artifact rather than a judgment somebody made. Whole numbers are the readable choice. They are also, in aggregate, wrong — and saying so is not optional: - **A run's printed rows do not sum to its printed score.** On a typical run roughly a third of the rows round up or down. Adding the printed column can land ten points or more away from the printed total. - **The score is never computed from the rounded rows.** It is computed from the exact contributions, summed at full precision, divided by the total weight, and rounded exactly once, at the end. Both numbers are correct. They answer different questions, and only one of them is the score. **Every result therefore carries a full accounting**, reachable from the foot of the report and from the stored result file: every test, its weight, its band, the arithmetic `weight × band ÷ 4` written out, its exact contribution, the figure the report printed instead, and the final division. Nothing on that page is rounded except where it says so. It recomputes the score from weight and band rather than reprinting the stored figure, so a disagreement between the accounting and the report would be visible rather than silent. An implementer of this specification must do the same. Print whole numbers if you like, but publish the arithmetic, and never compute a score from figures you have already rounded. *The one real fix — re-weighting every test to a multiple of four, so a 400-point Index lands every band on a whole number — is not made here. It would be a second non-comparable version boundary bought for a cosmetic gain, and the accounting already makes the arithmetic checkable.* ### Gates Two checks are gates: **AIR-1.1** and **AIR-1.2**. If either scores 0 or 1, set `gated: true` and record which one in `gate_failures`. Nothing else changes: `total` is computed exactly as in *Scoring* above, from every applicable check including the gate checks themselves, and `grade` is the band that real `total` falls in. Before 4.0, a gate failure capped the final score at 40 (`total = min(ungated_score, 40)`) and forced `grade` to `Gated`. That produced one number standing for two different facts — how well the site is built, and whether any of it is reachable today — and made them impossible to tell apart: a gated site with excellent structure scored identically to a gated site with none. Reporting `gated` and `total` as two separate facts says both; capping said neither. ### Verification mode Every test declares how its band was determined: - **`auto`** — the scanner determined the band from evidence it collected. - **`auto-partial`** — the scanner produced a strong signal, but a human must confirm. Heuristics that can be fooled live here. - **`manual`** — not observable from outside. **No scored test uses this mode as of 11.0.** Every point in the hundred is something the scanner can see for itself; what cannot be observed is reported as a best practice instead. The mode remains defined because the best practices use it. Automated coverage counts `auto-partial` at half weight, because a strong signal awaiting human confirmation is genuinely half-verified: ``` automated_coverage = (Σ auto_weight + 0.5 × Σ auto_partial_weight) ÷ Σ applicable_weight ``` The half is not arbitrary. Counting `auto-partial` at zero puts a ceiling of 0.83 on every possible run, because the always-applicable `auto-partial` checks are 15.75 points that never leave the denominator. Counting it in full reports 0.96 for a site nobody has confirmed anything about, and erases the distinction the mode exists to draw. Never let a `manual` check silently default to 4. An unanswered attestation scores `null` and drops out of the denominator, and the result records it as unverified. #### Attestation against an `auto-partial` check Four `auto-partial` checks have a top band that asks something no scanner can see. This document says so in each case, and each has a different observable ceiling because their band tables differ: | Test | Unattested ceiling | What the attestation confirms | |---|---|---| | LI-3 | 3 | That IndexNow submissions fire on publish, not on a schedule | | AIR-1.10 | 3 | That `llms.txt` is generated from the content model rather than hand-maintained | | AIR-3.1 | 3 | That JSON-LD comes from mapped fields, confirmed with the build team | | BP-9 | 3 | That sitemaps are submitted and alerts reach a named person | Each of these four now has a band 3 that describes exactly what a scanner can see, so the attestation lifts 3 to 4 rather than jumping a gap. A test whose band table skips a value cannot express *mostly right*, and *mostly right* is the state most real sites are in. An attestation may **raise the ceiling** on these four. It never sets a band and never lowers one: the scanner scores what it observed, and a supplied attestation permits that observed band to rise as far as the evidence supports. The result records which points came from a claim rather than a probe, and the report prints them as attested. Without an attestation the test scores at its observable ceiling and the report says what would be needed to go higher. A client who did the work can prove it; a client who says nothing earns nothing. Every other `auto-partial` check is fully observable. It carries that mode because the heuristic can be fooled — AIR-4.3 compares a signed request against an unsigned one, AIR-6.3 calls a live tool — not because it needs a client to vouch for it. #### Tests that need a previous run Three checks are defined against change over time: AIR-4.8 (is `lastmod` a real signal or deploy noise), AIR-2.4 (do anchor IDs survive a deploy), and AIR-5.4 (do dates move only when content moves). A fourth, selector churn on critical-flow elements, moved to BP-2 in 11.0 — measured every run as a Best Practice rather than banded and capped, since it no longer carries points to cap. On a first run there is nothing to compare against. Score the observable half, **cap the band at 3**, set `confidence: medium`, and record `stability_unconfirmed` in the evidence. AIR-5.4 already words its band 3 this way; the other three follow the same rule. Supplying a previous run directory as a baseline lifts the cap and permits band 4. Every ceiling — this one, the attestation ceiling above, and a test's own declaration that it could not see further — records the reason it applied. A client told a number was held below the evidence is owed the sentence explaining what would lift it. This makes the re-audit a first-class input to the score rather than a report-writing exercise, and it gives the client a visible, earnable reason the number rises next quarter. ## How the audit runs ### Sampling Most tests score against a sampled page set, not the whole site. Build the sample like this: 1. **Always include:** the homepage, `/robots.txt`, `/sitemap.xml` (and every child sitemap), `/llms.txt`, `/.well-known/` probes, and every URL in the site's primary navigation. 2. **Stratify by content type.** Use the sitemap index, URL patterns, or a declared content-type map. Take up to 10 URLs per type. 3. **Add depth.** Include at least 5 URLs three or more levels deep, to catch templates the homepage never exercises. 4. **Add the money pages.** The client names up to 10 URLs that matter most — programs, services, donate, apply, contact. These are weighted double in coverage math. 5. **Cap the crawl.** Default 250 URLs. Configurable. Record the actual count. Minimum viable sample is 25 URLs. Mark the run `low_confidence` only when the sample is under 25 **and** more URLs were discovered than were sampled. A twelve-page nonprofit site that was crawled in full is a complete measurement, not a weak one. #### The representation sub-sample Four checks want an extra request against every sampled URL: AIR-7.1 (`{path}.md`), AIR-7.3 (`Accept: text/markdown`), AIR-7.4 (cache behavior), and AIR-4.4 (`X-Robots-Tag` on non-HTML). At the default 250-URL cap and one request per second, that is over 800 requests and roughly fifteen minutes of politeness delay stacked on top of the base crawl, against infrastructure that often belongs to a hospital. Those four tests run against a **representation sub-sample**: 25 URLs by default, drawn from the main sample, always including every money page. The size is configurable and the selected URLs are recorded in the run metadata, so the sub-sample is reproducible across audits. Twenty-five pages resolves a rubric whose finest distinction is 75% against 95%. ### Fetching Fetch every sampled URL twice: - **Raw** — plain HTTP GET, no JavaScript, with a declared user-agent. - **Rendered** — a headless browser with JavaScript enabled. The difference between those two responses is the entire evidence base for AIR-2.1 and AIR-2.2, which together carry 9 points. Do not skip the second fetch. Separately, probe a small set of URLs with each AI crawler user-agent to feed AIR-1.2. There is no other way to measure what the edge does: `robots.txt` states an intention, a CDN rule states a fact, and the two disagree often enough that a readiness score which skipped the question would be worth less than the awkwardness of asking it. Say plainly what that means, because it is the part an implementer will be asked about. **The wire-level `User-Agent` on those requests carries another company's product token** — `GPTBot`, `ClaudeBot`, and the six others in the table below. Three constraints make that measurement rather than impersonation, and an implementation that drops any of them is doing something else: 1. **The requests still identify you.** Send your own `From` header and an `X-Audit-Operator` header naming your crawler on every probe. An operator inspecting the whole request, rather than the user-agent alone, can see who it was. 2. **Probe only URLs you were already allowed to fetch.** The probe runs over the sample, and the sample is filtered by `robots.txt` first. A probe must never reach, under somebody else's name, a page you declined to reach under your own. 3. **Publish the behavior before you do it**, at the URL in your own user-agent string, alongside how to opt out. Disclosure volunteered reads differently from disclosure discovered. #### The agent classes The Index groups the ten agents three ways. AIR-1.1 evaluates all ten against `robots.txt`. AIR-1.2 probes only the eight that actually issue requests. | Class | Agents | AIR-1.1 | AIR-1.2 probe | |---|---|---|---| | Answer-serving | `OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`, `Claude-User` | yes | yes | | Training and corpus | `GPTBot`, `ClaudeBot`, `CCBot`, `Bytespider` | yes | yes | | Directive-only | `Google-Extended`, `Applebot-Extended` | yes | **no** | `Google-Extended` and `Applebot-Extended` are not crawlers. They are opt-out tokens that Google and Apple honor in `robots.txt`; no request ever arrives carrying either string. Blocking them is a rights decision with no fetch behavior attached, which is why they count in AIR-1.1 and cannot be probed in AIR-1.2. A user-agent matrix that lists them is describing a crawler that does not exist. Where a band anchor says **major agents**, it means the four answer-serving agents plus `GPTBot` and `ClaudeBot`. ### Politeness Respect `Crawl-delay`. Default to 1 request per second with 4 concurrent workers. Back off on 429 and 503, honoring `Retry-After` where it is given, and abandon a host that keeps refusing rather than retrying into a wall. Cache every response to disk so re-scoring never re-crawls. ### Consent Two different questions, and conflating them is how a scanner ends up doing something indefensible. **Whose site may you scan?** A client audit runs with the client's agreement, recorded before the crawl. A survey of a public population — see *Populations*, below — runs without asking each site first, on the same footing as any other published measurement of the public web. What is not acceptable in either case is scanning under a consent you did not obtain. Record which basis a run used, in the run itself. **What may you fetch once you are there?** `robots.txt`, and specifically the group addressed to your own crawler. Name your crawler in your user-agent string, publish a page at the URL that string points to, and stop when a site names you and disallows you. A scanner may reasonably decide that a blanket `User-agent: *` disallow does not address it — a `*` rule is aimed at the anonymous scrape, and a named crawler with a published policy, a rate limit and a working opt-out is not that. If you make that choice, **publish it on your crawler's own page** and honor `Crawl-delay` under `*` regardless, because a rate limit is not an exclusion. What is not defensible is making the choice quietly. ### Populations A survey scores a defined population and publishes the aggregate. Two rules keep it a measurement rather than an accusation. **Cite the population.** Record the source list, its publication date and its URL. "How did you pick these" is the first question anyone asks, and the answer has to be a citation rather than a judgment call. Use sources you may lawfully redistribute, and claim no ownership over somebody else's list. **Publish the version and the date.** Every result records which version of this Index it was scored against and the date that version took effect. Without both, an old number silently starts meaning something new the next time the Index changes. ### Reproducibility Store the raw evidence — headers, HTML, extracted JSON-LD, screenshots — in a run directory keyed by timestamp. A score you cannot reproduce next quarter is an opinion, not a measurement. ## Result schema The scanner writes one `result.json` per run. Everything else (Markdown report, HTML dashboard, remediation backlog) renders from this file. ```json { "schema_version": "1.0", "index_version": "1.0", "site": { "name": "Example University", "base_url": "https://example.edu", "vertical": "higher-ed", "cms": "drupal-10" }, "run": { "id": "2026-09-01T20-14-33Z", "started_at": "2026-09-01T20:14:33Z", "finished_at": "2026-09-01T20:41:02Z", "scanner_version": "0.1.0", "urls_sampled": 187, "representation_sample": 25, "low_confidence": false, "baseline_run_id": null, "extraction_method": "readability-density", "extraction_version": "1" }, "score": { "total": 61.4, "gated": false, "gate_failures": [], "grade": "Adequate", "applicable_weight": 94.5, "earned_points": 58.0, "leading_adopted": 2, "leading_total": 6, "earnable_points": 88.0, "automated_coverage": 0.87 }, "dimensions": [ { "id": 3, "name": "Rendering and extraction", "weight": 20.0, "applicable_weight": 20.0, "earned_points": 9.5, "score_pct": 47.5 } ], "checks": [ { "id": "AIR-2.1", "title": "Primary content is present without JavaScript", "dimension": 3, "weight": 6.0, "applicability": "always", "applicable": true, "na_reason": null, "verification": "auto", "band": 1, "earned_points": 1.5, "confidence": "high", "summary": "Raw HTML contains 22% of rendered text on average across 187 URLs.", "evidence": [ { "url": "https://example.edu/admissions", "raw_text_chars": 812, "rendered_text_chars": 9440, "coverage": 0.086 } ], "remediation": { "severity": "critical", "effort": "large", "summary": "Server-render the decoupled front end.", "detail": "Program and admissions templates ship an empty shell..." } } ], "remediation_backlog": [ { "rank": 1, "check": "AIR-2.1", "points_available": 4.5, "severity": "critical", "effort": "large" } ] } ``` #### Schema notes - `site.cms` is best-effort, from the `generator` meta tag and response headers. It is nullable and nothing scores against it. - A dimension with no applicable checks reports `score_pct: null`, not `0`. Zero means measured and failed. - **Rounding happens once, at the end.** All arithmetic runs unrounded; the values written to this file are rounded to one decimal for display. The per-check `earned_points` therefore will not sum exactly to `score.earned_points`, and neither is wrong. Do not "fix" this by rounding intermediates — that reintroduces the drift the single rounding step exists to prevent. - `extraction_method` and `extraction_version` record how main-content text was isolated. When extraction improves, old scores stay interpretable because the method that produced them is on the record. ### Remediation ranking Sort the backlog by **points available** (`weight − earned_points`), then by effort ascending, then by check ID ascending. The last key is not cosmetic: two runs of the same evidence must produce the same backlog in the same order, or the re-audit comparison is unreadable. A check appears in the backlog only if it is applicable, has a band, and has points available above zero. N/A checks, unattested `manual` checks, and checks already at band 4 are not remediation items. Use three severities — `critical`, `major`, `minor` — and three effort sizes — `small`, `medium`, `large`. Effort is a property of the check, defined in this document. Severity is derived: gate failures and any check scoring **0 or 1** with weight ≥3.0 are `critical`; band ≤2 with weight ≥1.5 is `major`; everything else is `minor`. Band 1 on a heavy check is a critical finding. A site whose program pages ship 22% of their content scores AIR-2.1 at band 1, not 0, and calling that `major` alongside a missing caption on a table understates it to the person deciding what to fund. The client-facing output is not the score. It is the ordered list of what to fix and what each fix is worth. --- # The tests Each entry gives: points, applicability, verification mode, effort, what the test measures, how to detect it, what evidence to keep, and the band rule when it differs from the default coverage rubric. ## #1 · Primary Indicators · 13 tests · 30 pts **Points 30** ### AIR-1.1 — No blanket robots.txt blocks on AI crawler user-agents - **Points:** 5 - **Gate** - **Applicability:** always - **Verification:** auto - **Effort:** small - **Measures:** Whether `robots.txt` disallows the crawlers that feed answer engines. - **Detect:** Parse `robots.txt`. Evaluate each of `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `ClaudeBot`, `Claude-User`, `PerplexityBot`, `Google-Extended`, `Applebot-Extended`, `CCBot`, `Bytespider` against the rule set. Resolve `User-agent: *` fallbacks correctly, including the longest-match group rule. - **Evidence:** Raw `robots.txt`, the parsed rule tree, and a per-agent allow/deny verdict for `/` and for three sampled content URLs. - **Bands:** anchored on *which* agents are blocked, not how many. - **4** — every listed agent may fetch content paths. - **3** — one training or directive-only agent blocked; every answer-serving agent allowed. - **2** — two or more training or directive-only agents blocked, up to and including all of them; every answer-serving agent allowed. - **1** — exactly one answer-serving agent blocked. - **0** — `Disallow: /` for `*`, or two or more answer-serving agents blocked. - **Why the gate turns on the answer-serving class:** the Index measures whether machines can find, read and *use* a site. An answer engine blocked at `robots.txt` cannot answer a question about the client, which is the failure the gate exists to catch. A training block is a rights posture, and this document declines to judge rights postures — AIR-1.4 scores any combination of `yes` and `no` at 4 for exactly that reason. A university that blocks `GPTBot` and welcomes `ChatGPT-User` is reachable, and gating it would be measuring our approval rather than machine reach. - **Consequence:** blocking every training crawler costs 2 of the 5 points and does not gate. Blocking a single answer-serving agent gates. - **Absent robots.txt:** a 404 scores 4. Nothing is disallowed, so nothing is blocked. This is correct and frequently queried; a missing file is not a failure of this test. - **Note:** A deliberate, documented block is a business decision, not a defect. Record it, score it honestly, and say in the report that the client chose it. The number should reflect machine reach, not our approval. - **Fix:** Remove inherited blocks. Keep only directives the client can explain. ### AIR-1.2 — CDN and WAF do not silently reject AI crawlers - **Points:** 5 - **Gate** - **Applicability:** always - **Verification:** auto - **Effort:** medium - **Measures:** Whether the edge returns real content to AI user-agents, regardless of what `robots.txt` permits. - **Detect:** Request 10 sampled URLs with each of the eight fetching AI user-agents — the answer-serving and training classes defined under Fetching. `Google-Extended` and `Applebot-Extended` are excluded; they issue no requests. Record status code, response size, and whether the body matches the baseline fetch. Flag 403, 429, 503, interstitial challenge markup, and bodies more than 40% smaller than baseline. - **Evidence:** A user-agent × URL matrix of status codes, sizes, and challenge detection. - **Bands:** 4 — all agents get 200 with full content. 3 — one agent rate-limited but eventually served. 2 — one agent blocked or challenged. 1 — several blocked. 0 — most blocked. - **Fix:** Audit Cloudflare bot settings, Pantheon AGCDN rules, Fastly VCL, and origin WAF. Verify by re-running this probe, not by reading configuration. ### AIR-1.3 — No interactive challenges on read-only paths - **Points:** 1 - **Applicability:** always - **Verification:** auto - **Effort:** small - **Measures:** Whether anonymous GETs on browse, search, listing, and detail pages return content without a challenge. - **Detect:** Look for Turnstile, reCAPTCHA, hCaptcha, and JS-challenge signatures in raw responses on non-form URLs. Detect meta-refresh interstitials and challenge cookies. - **Evidence:** Per-URL challenge detection with the matching signature. - **Fix:** Scope challenges to POST endpoints and authenticated routes. ### AIR-1.4 — Content Signals declared in robots.txt - **Points:** 3 - **Applicability:** always - **Verification:** auto - **Effort:** small - **Measures:** Whether the site states a machine-readable position on `search`, `ai-input`, and `ai-train` using Cloudflare's Content Signals Policy. - **Detect:** Parse `Content-Signal:` lines. Validate syntax (`key=yes|no`, comma-delimited) and check for contradiction with `Disallow` rules. - **Bands:** 4 — all three signals declared, syntactically valid, consistent with `robots.txt`. 3 — declared but one signal omitted. 2 — declared with a syntax error or an internal contradiction. 1 — a partial or malformed attempt. 0 — absent. - **Note:** Any combination of `yes`/`no` scores 4. We are measuring that a deliberate position exists, not which position it is. - **Fix:** Run the policy conversation with the client, then add the line. ### AIR-1.5 — Human-readable AI policy comment in robots.txt - **Points:** 1 - **Applicability:** always - **Verification:** auto-partial - **Effort:** small - **Measures:** Whether a person reading `robots.txt` finds a plain-language statement of the organization's position. - **Detect:** Extract `#` comment blocks. Require ≥120 characters of prose and at least two policy terms (`AI`, `training`, `crawl`, `license`, `permission`). - **Bands:** 4 — substantive block present and consistent with the machine signals. 2 — a comment exists but is thin or boilerplate. 0 — none. - **Fix:** Write four sentences above the signals. Journalists and counsel read this file. ### AIR-1.6 — No stray noarchive or nosnippet directives - **Points:** 1 - **Applicability:** always - **Verification:** auto - **Effort:** small - **Measures:** Whether the site suppresses the excerpting that answer engines depend on. - **Detect:** Parse meta robots tags and `X-Robots-Tag` headers across the sample for `noarchive`, `nosnippet`, and `max-snippet:0`. - **Bands:** 4 — none present. 2 — present on a minority of pages. 0 — present site-wide. - **Fix:** Remove them. They are almost always left over from a legacy SEO module. ### AIR-1.7 — Every page emits a valid, self-consistent canonical - **Points:** 2 - **Applicability:** always - **Verification:** auto - **Effort:** small - **Measures:** Presence and correctness of `rel=canonical`. - **Detect:** For each sampled page, extract the canonical and resolve it. Flag: missing, multiple, cross-host, pointing at a redirect, pointing at a non-200, or pointing at the homepage from a deep page. - **Fix:** Emit a self-referencing canonical by default and correct the exceptions. ### AIR-1.8 — Semantic HTML structure - **Points:** 2 - **Applicability:** always - **Verification:** auto - **Effort:** medium - **Measures:** Whether document structure carries meaning or only styling. - **Detect:** Per page, require exactly one `
`, at least one `
` or `
` wrapping the primary content, and a `