Top 1,000 Websites Blocking AI Crawlers

## What this measures

The same reading as Top Websites Blocking AI Crawlers, limited to the 1,000 most-linked sites: the share whose robots.txt tells at least one named AI crawler to stay off the whole site.

## Why it matters

The largest sites hold the most valuable material and the most bargaining power. When their share runs ahead of the wider web, blocking is a negotiating stance by big publishers rather than a general mood.

## How to read it

Compare it with the 10,000-site share (web.ai_crawler_blocking.share_any).

## How it is counted

A site counts as blocking a crawler when its robots.txt names that crawler in a User-agent line and disallows the whole site ("/") for it, with no Allow for the root. A blanket rule that shuts out every crawler (User-agent: * / Disallow: /) is not counted, because it blocks search engines too and is a different decision; path-level rules don't count either. The denominator is every site whose robots.txt we could read plus sites with no robots.txt (by the robots standard, RFC 9309, a missing file allows everything). Sites that time out or refuse our request are left out; coverage is recorded with each reading.

## What it does not say

robots.txt is a request, not a wall: it records what a site asks of crawlers, not whether crawlers obey, and it says nothing about paid licensing deals or technical blocking at the network level. The sample is ranked by links, not traffic, and is weighted toward large, established sites. The survey begins in October 2026. ## Source

Palanor robots.txt survey. Once a month Palanor reads the public robots.txt file of each of the 10,000 most-linked websites in the Majestic Million (majestic.com, licensed CC BY 3.0), using its own crawler name (PalanorBot). Palanor computes and publishes the shares; the site list and the files are not republished.