Continuous accountability for Verified bots

Continuous Accountability for Verified Bots

A driver’s license proves identity and competence, but it doesn’t guarantee safe driving forever. The same principle applies to automated agents on the web. Verified bots at Cloudflare are designed to answer two core questions: Who are you? and Can we trust you? While identity is established through Web Bot Auth and catalogued in BotBase, trust hinges on consistent, non‑abusive behavior. This article explains how Cloudflare adds a layer of continuous accountability, turning a one‑time verification into an ongoing relationship.

Why Ongoing Monitoring Matters

Even a correctly identified bot can cause harm if it ignores a site’s preferences, overwhelms an origin, or unintentionally bypasses controls. Such issues often stem from misconfiguration or bugs rather than malicious intent, but they still erode trust. Cloudflare therefore distinguishes risk—the moment‑by‑moment chance of harmful activity—from trust, which is built through a history of reliable actions. A bot that repeatedly respects site directives earns higher trust; a single deviation can trigger a risk flag.

The Three Pillars of Verified Bot Status

1. Accurate, Secure Identification – Bots sign each request with Web Bot Auth, allowing BotBase to maintain an open, auditable registry.
2. Behavioral Consistency – Bots must act in line with the promises they made during verification (e.g., rate limits, respecting robots.txt).
3. Zero‑Abuse Commitment – Any form of abusive behavior—ignoring opt‑out signals, scraping disallowed content, or attempting to circumvent blocks—results in immediate de‑listing.

Measuring Respect for Site Preferences

Cloudflare’s new monitoring system tracks how often a bot follows explicit site signals:

• Robots.txt compliance – Requests that match disallowed paths are logged and counted.
• Rate‑limit adherence – When a site publishes a crawl‑delay or custom throttling rule, the system measures deviation.
• Opt‑out handling – Sites can signal “no‑scrape” via HTTP headers; violations are flagged.

These metrics are aggregated into an explainable trust score for each Verified bot. The score is transparent to both the bot operator and the site owner, enabling quick remediation when a bot’s behavior drifts.

Real‑World Example: A Trust Revocation

In 2025, Perplexity’s crawler was removed from the Verified list after it repeatedly accessed sites that had opted out of scraping and then attempted to bypass the resulting blocks. The incident illustrates how a single bot’s deviation can trigger a trust revocation, reinforcing the principle that verification is not a permanent badge but a conditional status.

What Bot Operators Need to Do

• Implement robust error handling to avoid unintended request spikes.
• Honor all site‑provided signals (robots.txt, crawl‑delay, custom headers).
• Monitor the trust score via the BotBase dashboard and address any warnings promptly.

By treating verification as a continuous contract rather than a one‑off check, Cloudflare helps maintain a healthier web ecosystem where automation can thrive without compromising site owners’ expectations.

---

Bottom line: Continuous accountability transforms Verified bots from a static label into a dynamic trust relationship, ensuring that the answer to “Can we trust you?” stays a confident “Yes.”