Top Websites Blocking OpenAI GPTBot

## What this measures

The share of the 10,000 most-linked websites whose robots.txt tells OpenAI GPTBot to stay off the whole site. OpenAI GPTBot is OpenAI's crawler for collecting model training data. It answers to the User-agent tokens: GPTBot.

## Why it matters

It measures how much of the open web is closed to OpenAI specifically, which shapes what its models can learn from and cite.

## How to read it

Read it beside the other crawlers' series and the overall share (web.ai_crawler_blocking.share_any). A crawler blocked far more than its peers is being singled out.

## How it is counted

A site counts as blocking a crawler when its robots.txt names that crawler in a User-agent line and disallows the whole site ("/") for it, with no Allow for the root. A blanket rule that shuts out every crawler (User-agent: * / Disallow: /) is not counted, because it blocks search engines too and is a different decision; path-level rules don't count either. The denominator is every site whose robots.txt we could read plus sites with no robots.txt (by the robots standard, RFC 9309, a missing file allows everything). Sites that time out or refuse our request are left out; coverage is recorded with each reading.

## What it does not say

robots.txt is a request, not a wall: it records what a site asks of crawlers, not whether crawlers obey, and it says nothing about paid licensing deals or technical blocking at the network level. The sample is ranked by links, not traffic, and is weighted toward large, established sites. The survey begins in October 2026. ## Source

Palanor robots.txt survey. Once a month Palanor reads the public robots.txt file of each of the 10,000 most-linked websites in the Majestic Million (majestic.com, licensed CC BY 3.0), using its own crawler name (PalanorBot). Palanor computes and publishes the shares; the site list and the files are not republished.