Brave Search Implements Encrypted Redirects to Mitigate AI and Automated Scraper Traffic
Following Google's deployment of anti-scraping measures that encrypt result links and funnel clicks through intermediate goto URLs, Brave Search plans to adopt a similar approach to protect its search infrastructure. This involves encrypting individual search result links using unique cryptographic keys and employing an intermediate redirect mechanism requiring real-time decryption. These measures aim to disrupt AI agents and high-frequency web crawlers that impose heavy traffic loads on search engines. Furthermore, Brave offers a commercial Search API with free and paid tiers to support legitimate high-volume queries, encouraging authorized access over scraping.
Detalhes técnicos
Google Search's anti-scraping defenses route select result links through an intermediate google.com/goto/ URL which decrypts the real destination at visit time, imposing a verification delay on suspicious traffic. Brave's upcoming solution will use a similar structure (search.brave.com/go/v1#) where each result has a unique cryptographic key. When a user clicks a link, the browser requests decryption parameters from Brave servers with a randomized payload, then locally decrypts the destination URL. Intermediate decryption tokens expire quickly, limiting the utility of bulk harvested links.
Remediação
By encrypting redirect links and integrating real-time decryption combined with risk-based delays, Brave Search seeks to mitigate abusive automated scraping. The availability of a commercial API provides a legitimate channel for heavy automated access, balancing performance for human users against operational strain caused by bots and AI agents.
Linha do tempo
-
Brave Search Implements Encrypted Redirects to Mitigate AI and Automated Scraper Traffic
ler a análise desta fonte ocultar
Following Google's deployment of anti-scraping measures that encrypt result links and funnel clicks through intermediate goto URLs, Brave Search plans to adopt a similar approach to protect its search infrastructure. This involves encrypting individual search result links using unique cryptographic keys and employing an intermediate redirect mechanism requiring real-time decryption. These measures aim to disrupt AI agents and high-frequency web crawlers that impose heavy traffic loads on search engines. Furthermore, Brave offers a commercial Search API with free and paid tiers to support legitimate high-volume queries, encouraging authorized access over scraping.
Technical Details
Google Search's anti-scraping defenses route select result links through an intermediate google.com/goto/ URL which decrypts the real destination at visit time, imposing a verification delay on suspicious traffic. Brave's upcoming solution will use a similar structure (search.brave.com/go/v1#) where each result has a unique cryptographic key. When a user clicks a link, the browser requests decryption parameters from Brave servers with a randomized payload, then locally decrypts the destination URL. Intermediate decryption tokens expire quickly, limiting the utility of bulk harvested links.
Remediation
By encrypting redirect links and integrating real-time decryption combined with risk-based delays, Brave Search seeks to mitigate abusive automated scraping. The availability of a commercial API provides a legitimate channel for heavy automated access, balancing performance for human users against operational strain caused by bots and AI agents.