The bot battle: How to defend against scraping attacks

In our digital first economy, organisations hold a treasure trove of valuable data, including product catalogs, pricing models, customer reviews, dynamic content and more—all of which helps differentiate them from competitors. Today, this valuable currency is a prime target for malicious bot attacks. What makes the situation particularly concerning is that the easy accessibility to more advanced attack tools and generative AI models has enhanced bot evasion techniques to the extent that traditional security solutions are being rendered ineffective.

Not only are bots getting smarter and more sophisticated, but they are increasing in number. According to Radware's recent Global Threat Analysis Report, bad bots made up 71% of all bot traffic in 2024; and malicious bot transactions increased 35% between 2023 and 2024. The growing number of bad bots come in various forms, including distributed denial of service attacks, credential stuffing, account takeover, ad fraud, scalping, and web scraping.

Scraping is one of the most common bot attacks. Understanding what scraping is and how to counter it is key to preventing it from eroding business value, compromising user trust, and destabilising digital assets.

What is scraping?

Scraping is the automated extraction of data from websites or applications, typically executed by bots (software scripts) that mimic human behaviour to bypass security measures. These malicious bots crawl through web pages, APIs, or mobile apps, harvesting information at scale—often without the target's consent.

Scraping generally can be classified into two main types. Price Scraping is the automated process of collecting pricing data from other websites to inform marketing and pricing strategies. Content Scraping involves lifting and often republishing original content like articles, catalogs, images, or videos from other websites without permission.

Under attack

Scraping by malicious bots is a powerful tool that comes with disadvantages and risks that businesses need to be aware of. Scraping can lead to:

• Loss of competitive advantage: Competitors can scrape pricing data to undercut prices and reset pricing strategies to gain an unfair advantage, harming legitimate businesses.

• Revenue loss: Scraping can enable other fraudulent activities like scalping where malicious bots buy and hoard inventory and later resell the goods at a higher price.

• Poor website performance: If bots are continuously deployed by competitors and bad actors working at scale to scrape content and price information, it can slow down website performance, degrade the customer experience, and adversely affect a brand.

• Compliance and regulatory risks: With data privacy laws like GDPR in place, scraping personal data without consent can result in significant fines and damage to brand reputation.

Bypassing basic defenses

To mitigate scraping attacks, implementing techniques such as rate limiting to control request frequency or using CAPTCHA services can offer a basic layer of defence. However, these techniques are no longer effective against modern bots that have evolved their evasion techniques and often cause friction for legitimate users. Savvy attackers are using a variety of tactics to bypass basic defences such as:

• IP-based blocking: Attackers now have access to resources that can help rotate across a multitude of IPs, making IP-based blocking ineffective.

• User-agent and header-based filtering: Advanced scrapers can dynamically adjust their headers and user agents to appear as legitimate browsers and avoid signature-based detection.

• Rate limiting: Advanced Bots can mimic natural browsing patterns, distributing requests across multiple sessions and devices, making rate-limiting ineffective.

Ramping up protection

To keep up with scraping techniques, businesses need more advanced prevention and mitigation strategies. Managing bots effectively involves striking a balance between allowing beneficial bots to operate while blocking harmful bots without disrupting customer services.

To tackle this management challenge, modern day solutions need to take a more holistic approach to security—one that uses a combination of techniques. This should include:

• Real-time behavioural-based analysis and AI-powered machine learning: Detects even the most sophisticated human-like bots by analysing their intent and behaviour.
• Signature-based detection: Maintains a constantly updated database of known bot signatures, enabling quick identification and blocking of known malicious bots.
• Dynamic CAPTCHAs: Adjusts in difficulty based on the bot's behaviour, making it challenging for bots to solve while remaining user-friendly for real users.
• Automated mitigation: Blocks malicious bots in real-time without requiring manual intervention.
• Device fingerprinting: Identifies bots across multiple visits, even if they attempt to change their characteristics.
• Threat intelligence feeds: Helps preemptively block known scraping bot sources, preventing an overload of the application infrastructure and ensuring optimum website performance.

Web scraping is a constant problem, especially for organisations that develop proprietary, time-sensitive, paid, or otherwise hard-to-obtain content, e-commerce firms, travel and airline agencies, and classified ad listings just to name a few. With cybercriminals utilising more advanced attack tools and techniques, traditional defences are being outpaced and out maneuvered. By investing in advanced security solutions and staying vigilant against evolving attack vectors, organisations can stay one step ahead in the battle against bad bots.