COMMERCIAL OPTION
CrawlerCheck
Checks robots.txt, meta robots and X-Robots-Tag access for search and AI crawlers.
Optional shortcut, not the starting point. Last checked: 2026-07-15.
- Best for
- Technical access diagnosis
- Price / access
- Free
- Data location
- Vendor-hosted service.
- Engine coverage
- Googlebot · GPTBot · ClaudeBot · PerplexityBot · 169 crawlers
Limitations to keep visible
Allowed access does not prove indexing or citation; safety scores are methodology outputs.
Official sources
Read this CrawlerCheck review alongside the price, capabilities, and limitations shown here.