How to Read Server Logs and Identify Malicious Bot IPs

An access log line already contains most of what you need: the client IP, timestamp, method and path, status code, response size, referrer and user agent. Automated traffic rarely hides in a single field, but it shows a pattern across several of them.

Request rate is the first signal. A browser loads a page and then a burst of assets; a scraper or scanner requests page after page at a steady interval, often around the clock. Look at what is requested too. Probes for wp-login.php, xmlrpc.php, .env, phpMyAdmin, backup archives and admin paths on a site that has none of them are scanners working from a list.

Scrapers add their own tells. They skip the navigation a person would follow and go straight to the pages that hold structured data, such as search results, category listings, product or profile pages and API-like endpoints, request them in numbered or alphabetical sequence, and rotate user agents while the shape and timing of the requests stay identical.

Status codes tell the same story from the other side. Long runs of 404, 401 or 403 from one address mean guessing. A stream of 200 responses for HTML pages with no requests for CSS, JavaScript or images means something is fetching pages without rendering them.

User agents are the weakest evidence because they are trivial to fake, but the defaults are still common: python-requests, curl, Go-http-client, empty strings, or a Chrome version several years old on every request.

Start by counting requests per address over a fixed window. On a Linux host this is a one-liner: extract the first column of the access log, sort it, count unique values and sort by count. The top of that list is where to look first.

Distributed activity does not show up that way. A credential-stuffing run or a scraping job spread over a hosting provider produces hundreds of addresses with a few requests each. Aggregate again by /24 range and by ASN, the network that announces the address. When one hosting ASN accounts for a large share of traffic to a login or checkout endpoint, you are looking at one operation, not many users.

Take the top addresses through an IP lookup and read the results together. A hosting or datacenter classification, a VPN or Tor flag, and a presence on abuse blocklists make a strong case when they line up with aggressive paths and no asset requests. A residential ISP with one short burst is a different situation: carrier-grade NAT can place hundreds of subscribers behind one address, so blocking it punishes people who did nothing.

Expect the capable operators to arrive from residential addresses. Scraping campaigns increasingly run through residential proxy networks, so a consumer ISP classification does not clear an address that acts like a scraper; the pattern still decides. Look at the spread as well: many addresses in many countries sharing one exact request pattern and timing are one operation, however ordinary each address looks on its own.

Rule out the traffic you want before acting. Search engine crawlers publish their address ranges or verify through reverse DNS. Uptime monitors and security scanners you contracted announce themselves. If your site sits behind a CDN or reverse proxy, the client column may hold the proxy's address, and the real client is in the X-Forwarded-For header; blocking the proxy address blocks everyone.

Match the response to the evidence. Rate limits and challenges handle most scrapers without collateral damage. Expiring blocks on a hosting range stop a scanner for the duration of its run. Reserve permanent blocks for addresses that attack repeatedly over weeks, and record why each rule exists so you can remove it later. Cooperative crawlers can be steered with robots.txt; it is the automation that ignores it that needs the limits and challenges, and the abusive patterns that keep returning that justify stronger bot detection.

Log the decision alongside the evidence: the address, ASN, the signals that drove it, and the time. When a customer reports being blocked, that record is how you find and reverse the mistake quickly.

Blocking on the user agent alone stops honest tools and no attackers. Blocking a CDN or proxy address takes down access for every user behind it. Permanent bans on dynamic addresses outlive the attacker and land on the next subscriber. And one burst of 404s is not an attack; a pattern of probes, failures and missing assets across time is.