Data study · October 19, 2026
What AI agents see on 60 AI and SaaS homepages
Assistants like ChatGPT, Claude and Perplexity can only cite pages their crawlers are allowed to fetch and can read. I ran the same check as my free AI readiness checker on 60 well-known homepages: robots.txt rules for 13 AI crawlers, llms.txt, how much content exists before JavaScript runs, link previews, structured data and bot walls.
77%
publish an llms.txt
12%
block at least one AI crawler
3 of 60
showed my server a bot wall
Which AI crawlers get blocked
Share of the 60 homepages whose robots.txt disallows each crawler. Training crawlers are blocked more often than the search and "user request" crawlers that put a site into answers. Bars use a 0–10% scale.
What I noticed
- 77% publish an llms.txt (8 also have llms-full.txt). Among AI and developer-tool companies it has become standard.
- 12% block at least one AI crawler in robots.txt. Only 1 site blocks a search or user-request crawler, the kind that leads to citations.
- 3 homepages (lovable.dev, canva.com, openai.com) have almost no text before JavaScript runs, so crawlers that don't run scripts see a nearly empty page.
- 3 sites (salesforce.com, midjourney.com, perplexity.ai) answered my datacenter IP with a bot wall, including an AI search company.
- 87% have a large enough og:image (1200×630 or more) and 72% include JSON-LD structured data.
Every site
Click a column to sort. Score is the share of checks passed by the AI readiness checker.
| Site | Score | Blocked crawlers | llms.txt | Needs JS | og:image | JSON-LD | Bot wall |
|---|---|---|---|---|---|---|---|
| cloudflare.com | 100 | none | yes | no | 1400×800 | Organization, WebPage, WebSite | no |
| cohere.com | 100 | none | yes | no | 1200×630 | Organization | no |
| cursor.com | 100 | none | yes | no | 2401×1260 | Organization, WebSite | no |
| datadoghq.com | 100 | none | yes | no | 1200×630 | Offer, Organization, SoftwareApplication, WebSite | no |
| elevenlabs.io | 100 | none | yes | no | 1200×630 | BreadcrumbList, Organization, Organization,Brand, SoftwareApplication, WebPage, WebSite | no |
| fly.io | 100 | none | yes | no | 1200×630 | Organization, Service, SoftwareApplication, WebSite | no |
| framer.com | 100 | none | yes | no | 2400×1260 | Organization | no |
| gitlab.com | 100 | none | yes | no | 1200×630 | Corporation, Organization, WebPage, WebSite | no |
| hubspot.com | 100 | none | yes | no | 2160×1440 | Organization, Product, WebPage, WebSite | no |
| intercom.com | 100 | none | yes | no | 2400×1260 | BreadcrumbList, ItemList, Organization, Product,SoftwareApplication, WebPage, WebSite | no |
| jasper.ai | 100 | none | yes | no | 1200×630 | FAQPage, Organization, SoftwareApplication, WebSite | no |
| mistral.ai | 100 | none | yes | no | 1200×630 | Organization, WebSite | no |
| monday.com | 100 | none | yes | no | 1200×628 | Organization, SoftwareApplication, WebPage, WebSite | no |
| neon.com | 100 | none | yes | no | 1200×630 | Organization | no |
| netlify.com | 100 | none | yes | no | 1200×630 | BreadcrumbList, Organization, SoftwareApplication, WebPage, WebSite | no |
| railway.com | 100 | none | yes | no | 1200×630 | Organization, Service, SoftwareApplication, WebSite | no |
| render.com | 100 | none | yes | no | 1600×900 | Organization, SoftwareApplication, WebSite | no |
| runwayml.com | 100 | none | yes | no | 1200×630 | Organization, SoftwareApplication, WebSite | no |
| sentry.io | 100 | none | yes | no | 1200×630 | Organization, WebPage | no |
| shopify.com | 100 | none | yes | no | 2400×1260 | Corporation | no |
| stripe.com | 100 | none | yes | no | 2048×1024 | Organization, WebSite | no |
| suno.com | 100 | none | yes | no | 1920×1080 | FAQPage, Organization, SoftwareApplication, WebSite | no |
| supabase.com | 100 | none | yes | no | 1200×600 | Organization, Service, SoftwareApplication, WebSite | no |
| twilio.com | 100 | none | yes | no | 2400×1256 | Organization, VideoObject, WebPage, WebSite | no |
| vercel.com | 100 | none | yes | no | 2400×1350 | Organization, Service, SoftwareApplication | no |
| webflow.com | 100 | none | yes | no | 2400×1256 | Organization, SoftwareApplication, WebPage, WebSite | no |
| windsurf.com | 100 | none | yes | no | 1200×630 | SoftwareApplication | no |
| wix.com | 100 | none | yes | no | 1200×630 | FAQPage, HowTo, Organization, VideoObject | no |
| zapier.com | 100 | none | yes | no | 1200×630 | Organization, SoftwareApplication, WebPage, WebSite | no |
| atlassian.com | 88 | none | yes | no | 811×812 | Organization, SoftwareApplication, WebPage, WebSite | no |
| bolt.new | 88 | none | no | no | 2400×1200 | Organization, WebApplication | no |
| cal.com | 88 | none | yes | no | 1200×630 | none | no |
| calendly.com | 88 | CCBot | yes | no | 1200×630 | Organization, WebSite | no |
| linear.app | 88 | none | yes | no | 1200×630 | none | no |
| loom.com | 88 | GPTBot | yes | no | 2400×1254 | Corporation | no |
| lovable.dev | 88 | none | yes | yes | 1200×629 | Organization, SoftwareApplication, WebSite | no |
| notion.com | 88 | meta-externalagent | yes | no | 2400×1250 | Organization, SoftwareApplication, WebSite | no |
| posthog.com | 88 | none | yes | no | 1024×537 | Organization, SoftwareApplication | no |
| replit.com | 88 | none | yes | no | 1200×650 | none | no |
| slack.com | 88 | none | yes | no | 1200×628 | none | no |
| squarespace.com | 88 | none | yes | no | 1200×630 | FAQPage, HowTo, Organization, WebSite | no |
| together.ai | 88 | Google-Extended | yes | no | 1200×630 | Organization, WebSite | no |
| v0.app | 88 | none | no | no | 1200×630 | Organization, WebApplication | no |
| zoom.com | 88 | none | no | no | 1200×630 | Corporation, WebPage, WebSite | no |
| airtable.com | 75 | none | no | no | 1200×630 | none | no |
| anthropic.com | 75 | none | no | no | 2400×1260 | none | no |
| github.com | 75 | Bytespider | yes | no | 1200×630 | none | no |
| huggingface.co | 75 | none | no | no | 1200×648 | none | no |
| miro.com | 75 | none | no | no | 1200×630 | none | no |
| clickup.com | 67 | none | yes | no | 2400×1260 | Organization | no |
| canva.com | 62 | GPTBot, ClaudeBot, Applebot-Extended, meta-externalagent, CCBot, Bytespider | no | yes | 1200×630 | Organization | no |
| figma.com | 62 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, CCBot, Bytespider | no | no | none | Organization | no |
| openai.com | 62 | none | no | yes | 1600×900 | none | no |
| groq.com | 56 | none | no | no | 1200×630 | none | no |
| replicate.com | 50 | none | yes | no | none | none | no |
| character.ai | 44 | none | no | no | 1200×630 | none | no |
| asana.com | 25 | none | yes | no | none | none | no |
| salesforce.com | 25 | none | yes | no | none | none | Akamai |
| midjourney.com | 17 | none | no | no | none | none | Cloudflare |
| perplexity.ai | 17 | none | no | no | none | none | Cloudflare |
Method and limits
- One run per homepage on October 19, 2026, through Shotlane's
/v1/report. Only homepages, not docs or blogs. - robots.txt rules were evaluated for the homepage path. A site can allow a crawler on its homepage and block it elsewhere.
- "Content needs JavaScript" means fewer than 30% of the words were in the HTML before scripts ran.
- Bot walls are what my server saw from a datacenter IP in Germany. Visitors on home connections, and verified crawlers, may get the real page.
- llms.txt counts a Markdown file at
/llms.txt; every detected file was checked by hand to be the real format.
Raw data: ai-readiness-2026-10.json (CC BY 4.0: reuse it with a link back).
Check your own site
The same report, free in your browser, or as JSON from the API for 1 credit.
Run the AI readiness checker