Cloudflare, responsible for managing a significant portion of internet traffic, has set a September deadline for AI crawlers. These crawlers must differentiate between bots gathering content for search and those collecting data for AI training. Failure to comply will result in blockage from pages hosting advertisements.
On March 2025, the Cloudflare office in San Francisco reacted to AI web crawlers’ behavior, emphasizing a need for clear separation of their functions.
By enforcing this deadline, Cloudflare aims to control the impact of AI crawlers on its advertising-hosted pages, ensuring content is gathered appropriately.

Concerns Rise Over Security of U.S. Water Systems Amid Cyberattacks
Chinese Vessel Activity Raises Concerns Over Undersea Cables
Fusion Energy Milestone: Breaking New Ground
Trump’s Weekend AI Image Spree on Truth Social
The Modern Challenge of Satellite Navigation and the Path Forward
AI Experiment at Bayreuth Festival Sparks Mixed Reactions