Cloudflare introduces continuous trust scoring for Verified bots based on robots.txt compliance and HTTP 429 response
Cloudflare announced a system that continuously evaluates whether Verified bots keep honoring website preferences after their identity is verified. The system produces an explainable "trust band" from observed behavior. It currently looks at two signals: whether a bot's traffic is consistent with published robots.txt rules (RFC 9309), and whether it slows or pauses after receiving HTTP 429 (Too Many Requests) responses. Cloudflare says it already uses the system internally and plans to publish scores on Radar. It also plans to let customers set a trust threshold that controls how Verified bot traffic is handled. As precedent, Cloudflare cites de-listing Perplexity's bot in 2025 for accessing opted-out sites and trying to circumvent controls once blocked. This is product and policy news with no vulnerability, exploitation or active threat.
Detalhes técnicos
- Bots sign requests with Web Bot Auth to establish identity, and BotBase catalogues them in an open format.
- Repeated patterns of disregarding robots.txt lower trust, while isolated violations are not treated as significant.
- Consistently slowing down or pausing after a 429 raises trust, and repeatedly ignoring it lowers trust.
- The system works alongside Bot Preference Sync and Bot Management to form an accountability loop for automated traffic.
Linha do tempo
-
Cloudflare introduces continuous trust scoring for Verified bots based on robots.txt compliance and HTTP 429 response
ler a análise desta fonte ocultar
Cloudflare announced a system that continuously evaluates whether Verified bots keep honoring website preferences after their identity is verified. The system produces an explainable "trust band" from observed behavior. It currently looks at two signals: whether a bot's traffic is consistent with published robots.txt rules (RFC 9309), and whether it slows or pauses after receiving HTTP 429 (Too Many Requests) responses. Cloudflare says it already uses the system internally and plans to publish scores on Radar. It also plans to let customers set a trust threshold that controls how Verified bot traffic is handled. As precedent, Cloudflare cites de-listing Perplexity's bot in 2025 for accessing opted-out sites and trying to circumvent controls once blocked. This is product and policy news with no vulnerability, exploitation or active threat.
Technical Details
- Bots sign requests with Web Bot Auth to establish identity, and BotBase catalogues them in an open format.
- Repeated patterns of disregarding robots.txt lower trust, while isolated violations are not treated as significant.
- Consistently slowing down or pausing after a 429 raises trust, and repeatedly ignoring it lowers trust.
- The system works alongside Bot Preference Sync and Bot Management to form an accountability loop for automated traffic.