User-agent feed
Block bots and crawlers by their user-agent string. Use it with the IP block lists.
The user-agent feed lists user-agent (UA) strings used by non-human traffic. Bots often use UA strings that reveal what they are, so blocking those strings is another way to keep non-human traffic out.
What’s in the feed
The list starts from the IAB’s list of valid UA strings and its list of known crawlers and spiders (the IAB/ABC Spiders & Bots List). Pixalate adds UA strings that its own data shows performing fraud, so it can catch new bot UA strings before they reach the IAB list.
Every UA on the list is a deterministic, high-risk entry (equivalent to a probability of 1.0). Entries generally fall into three groups:
- Known bots, spiders and crawlers, including those on the IAB/ABC Spiders & Bots List.
- UAs linked to pervasive sophisticated invalid traffic (SIVT), often from complex bot networks (botnets).
- UAs linked to hyperactive users, with prolonged activity and outlier traffic beyond Pixalate’s thresholds.
In practice that covers:
- Search engine crawlers that declare who they are, such as Googlebot.
- Custom crawlers, for example UA strings containing keywords such as
grabberorpython. - UA strings that are missing the “must-have” keywords that, per IAB guidelines, characterize a valid UA string.
UA strings are easy to spoof or self-declare, so use the most recent file you have, and combine the feed with post-bid measurement to tune it for your traffic.
At a glance
The feed comes in a daily and a weekly version. To confirm which one you receive, ask your Pixalate account manager.
Schema
The file has a single column. There’s no probability or fraud type column, because every entry is deterministic.
Example rows (illustrative):
Apply it pre-bid with the IP lists
The UA feed and the IP block lists are separate lists. Use them together as a two-step filter: each makes the other more effective.
Block a request if its user agent is on the UA list OR its IP is on the IP list (at or above your IP threshold). Don’t block only when both match, and don’t build a combined list of UA and IP pairs.
Here’s why. Say Googlebot crawls from one of Google’s IP addresses, 198.51.100.66 in this example:
If you match on the pair, the crawler gets through as soon as it changes either its UA or its IP. With two separate lists in OR logic, the UA list catches Googlebot from any IP in the world, and the IP list catches any crawler from that IP, even with a UA string it hasn’t used before.
The order of the two checks doesn’t matter, as long as you combine them with OR.
Check for new files
- Daily list: check once an hour. Download only when a new file is available. Keep checking hourly after you download, because more than one file is sometimes uploaded in a day.
- Weekly list: on Fridays, check once an hour. Download only when a new file is available, then stop checking until the next week.
For access and file locations, see Delivery and integration.