Skip to content
Palanor is in early release, with our founding customers.Become a founding customer →
PalanorPalanor

Latest reading, ranked/From the Studio

Top Websites Blocking Common Crawl CCBot and 7 more

CCBot (Common Crawl) leads at 13.5%
Current·Data to 2026

What this chart reads·8 series

Show all

Sources·8 sources

Show all

Top Websites Blocking Common Crawl CCBot. Source: Palanor robots.txt survey. Once a month Palanor reads the public robots.txt file of each of the 10,000 most-linked websites in the Majestic Million (majestic.com, licensed CC BY 3.0), using its own crawler name (PalanorBot). Palanor computes and publishes the shares; the site list and the files are not republished.

Top Websites Blocking OpenAI GPTBot. Source: Palanor robots.txt survey. Once a month Palanor reads the public robots.txt file of each of the 10,000 most-linked websites in the Majestic Million (majestic.com, licensed CC BY 3.0), using its own crawler name (PalanorBot). Palanor computes and publishes the shares; the site list and the files are not republished.

Top Websites Blocking ByteDance Bytespider. Source: Palanor robots.txt survey. Once a month Palanor reads the public robots.txt file of each of the 10,000 most-linked websites in the Majestic Million (majestic.com, licensed CC BY 3.0), using its own crawler name (PalanorBot). Palanor computes and publishes the shares; the site list and the files are not republished.

Top Websites Blocking Anthropic ClaudeBot. Source: Palanor robots.txt survey. Once a month Palanor reads the public robots.txt file of each of the 10,000 most-linked websites in the Majestic Million (majestic.com, licensed CC BY 3.0), using its own crawler name (PalanorBot). Palanor computes and publishes the shares; the site list and the files are not republished.

Top Websites Blocking Meta-ExternalAgent. Source: Palanor robots.txt survey. Once a month Palanor reads the public robots.txt file of each of the 10,000 most-linked websites in the Majestic Million (majestic.com, licensed CC BY 3.0), using its own crawler name (PalanorBot). Palanor computes and publishes the shares; the site list and the files are not republished.

Top Websites Blocking Google-Extended. Source: Palanor robots.txt survey. Once a month Palanor reads the public robots.txt file of each of the 10,000 most-linked websites in the Majestic Million (majestic.com, licensed CC BY 3.0), using its own crawler name (PalanorBot). Palanor computes and publishes the shares; the site list and the files are not republished.

Top Websites Blocking Applebot-Extended. Source: Palanor robots.txt survey. Once a month Palanor reads the public robots.txt file of each of the 10,000 most-linked websites in the Majestic Million (majestic.com, licensed CC BY 3.0), using its own crawler name (PalanorBot). Palanor computes and publishes the shares; the site list and the files are not republished.

Top Websites Blocking PerplexityBot. Source: Palanor robots.txt survey. Once a month Palanor reads the public robots.txt file of each of the 10,000 most-linked websites in the Majestic Million (majestic.com, licensed CC BY 3.0), using its own crawler name (PalanorBot). Palanor computes and publishes the shares; the site list and the files are not republished.

Charts are drawn from Palanor's signal library and recomputed from source data on every view. Transformations (changes, indexing, year-on-year rates) are Palanor's.

Related charts

AI & technologyForecast

U.S. Businesses Using AI

Share of U.S. businesses using AI to produce goods or services, and share expecting to within six months, %

23.8%Data 32 days ago
AI & technologyForecastMoving

AI in Text Index: Earnings Calls by Sector

Words language models overuse, per 10,000 words of S&P 500 earnings calls, trailing 90 days, sectors with enough companies to read

5.8Data today
AI & technologyForecast

What Companies Talk About on Earnings Calls

Average mentions per S&P 500 earnings call, rolling 90 days

11.51 viewsData today
AI & technology

Hyperscaler Capital Spending

Capital expenditures, trailing 12 months, $ billions

$173BData to Jun 2026
AI & technologyForecastMoving

AI in Text Index: S&P 500 Earnings Calls

How often calls use the words language models overuse, against their level before ChatGPT (January 2021 to September 2022 = one), trailing 90 days

1.06Data today
AI & technologyForecast

AI in Text Index: 10-K Risk Factors

Words language models overuse in S&P 500 annual reports, Item 1A, latest 10-K per company, against the level in 10-Ks filed before ChatGPT (= one)

1.41Data today

Cookies

We only use essential cookies to keep you signed in and remember your theme. No analytics, no tracking, no ads. See the Privacy Policy for the full read.