Allowlist legitimate crawlers and AI agents so they are not flagged or blocked. Each bot needs a User-Agent substring plus one verification method: a reverse-DNS suffix (search and SEO crawlers) or published IP ranges (AI agents that have no crawler reverse DNS).
Built-in bots (Googlebot, Bingbot, Applebot, DuckDuckBot, GPTBot, ClaudeBot, PerplexityBot and others) are always verified and need no entry here. Reverse-DNS suffixes must be registrable domains; shared-hosting suffixes and over-broad or non-public IP ranges are rejected so the list cannot become a bypass.
Built-in AI-crawler IP ranges
Published ranges for AI agents (GPTBot, PerplexityBot and others) refresh automatically from the vendor feeds. Run csm update-bot-ranges to refresh now. This view is read-only.
Auto-update
—
Interval
—
Last refresh
—
Prefixes loaded
—
Loading...
No operator-defined verified bots
Add a custom crawler or AI agent with the button above.