A Guth Labs publication ↗
Back to News

Cloudflare expands Radar bot profiles with identity and content-use fields

Cloudflare’s public Radar Bots Directory now draws on BotBase and presents more detail about how tracked bots and agents identify themselves and use web content. The November 9 update adds an identity section to bot profiles, with access level, content-use classifications, Web Bot Auth information and user-agent strings. Announcement

The accompanying API changes matter for developers who consume the directory programmatically. A bot can carry multiple categories, including commerce. Access level distinguishes direct access from an intermediary acting for another agent or person. Content-use values distinguish retained or reused content, references back to a source and immediate use for a request. The older singular category and operator URL fields are deprecated in favor of new fields. API changes

Cloudflare distinguishes the public directory from its account-level BotBase interface. The latter is available to Enterprise Bot Management customers and supports investigating traffic and copying detection identifiers into security rules. The documentation explicitly describes BotBase as a visibility tool; mitigation happens through separate security rules or AI traffic controls. BotBase documentation

Its traffic labels also require care. A successful request means an edge response in the 2xx or 3xx range, while an unsuccessful request can include an origin error. Those outcomes do not, by themselves, say whether a site owner intentionally blocked a bot. Metric definitions

The update gives publishers a more useful vocabulary for deciding how different automated visitors should be treated. An assistant fetching a page for a person and a crawler retaining material for future use can raise different operational questions. But classification, identity verification and permission are separate issues: a profile does not by itself establish that a request is authorized for every purpose. For teams maintaining automated access policies, the immediate work is to review the changed schema and connect directory information to their own documented rules, rather than treating a familiar bot name as an unrestricted allowlist entry.

Sources

Sources and citations

Revision history

Published .