Top Websites Blocking Applebot-Extended

## What this measures

The share of the 10,000 most-linked websites whose robots.txt tells Applebot-Extended to stay off the whole site. Applebot-Extended is the token Apple reads to decide whether a site's pages may be used to train Apple's AI models; blocking it does not affect Siri or Spotlight search. It answers to the User-agent tokens: Applebot-Extended.

## Why it matters

It measures how much of the open web is closed to Apple specifically, which shapes what its models can learn from and cite.

## How to read it

Read it beside the other crawlers' series and the overall share (web.ai_crawler_blocking.share_any). A crawler blocked far more than its peers is being singled out.

## How it is counted

A site counts as blocking a crawler when its robots.txt names that crawler in a User-agent line and disallows the whole site ("/") for it, with no Allow for the root. A blanket rule that shuts out every crawler (User-agent: * / Disallow: /) is not counted, because it blocks search engines too and is a different decision; path-level rules don't count either. The denominator is every site whose robots.txt we could read plus sites with no robots.txt (by the robots standard, RFC 9309, a missing file allows everything). Sites that time out or refuse our request are left out; coverage is recorded with each reading.

## What it does not say

robots.txt is a request, not a wall: it records what a site asks of crawlers, not whether crawlers obey, and it says nothing about paid licensing deals or technical blocking at the network level. The sample is ranked by links, not traffic, and is weighted toward large, established sites. The survey begins in October 2026. ## Source

Palanor robots.txt survey. Once a month Palanor reads the public robots.txt file of each of the 10,000 most-linked websites in the Majestic Million (majestic.com, licensed CC BY 3.0), using its own crawler name (PalanorBot). Palanor computes and publishes the shares; the site list and the files are not republished.