Find out which bot protection a site uses before you retry
To detect antibot protection reliably, you need more than a 403: you need to know that a shield answered instead of the site, and which one. Crawlers, link preview services, SEO tools and data pipelines all treat those failures as random noise and retry them blindly. is-antibot, the open-source library Microlink uses for this check, names the provider from the response you already have, so each blocked URL gets the right next step.
A blocked response looks like a success until you know who answered
Blocks come from named providers, and each one blocks differently. Cloudflare serves a “Just a moment” interstitial, DataDome scores request signatures in real time, Akamai Bot Manager blocks datacenter IPs at the edge, PerimeterX leans on client-side fingerprinting, and a reCAPTCHA or hCaptcha widget can sit in front of all of them. Some of these arrive as a 403, some as a 429, and some as a 200 with a challenge page for a body.
Hand-written detection does not keep up. A check for “Just a moment” in the HTML catches one Cloudflare mode and misses the rest; a status code check misses every challenge served on a 200. And treating all failures alike is costly: a retry strategy that works against one system can make the next request look more suspicious to another.
is-antibot reads the status code, headers, cookies, body markers and URL of a response and returns whether a block was detected, the provider behind it and which signal matched. It covers 30+ antibot and CAPTCHA providers, it is dependency-free, and it does not try to solve challenges. The Microlink API runs the same detection as one of its first checks: free requests fail with EPROXYNEEDED, and on Pro the blocked request is routed through the proxy.
How to detect which antibot protection a site uses
Classify your own responses with is-antibot, and let the API handle the ones that are blocked. The API tells you that a URL is protected; only the library tells you by whom.
import isAntibot from 'is-antibot'
const url = 'https://www.example.com/pricing'
const response = await fetch(url)
const { detected, provider, detection } = isAntibot({
url: response.url,
statusCode: response.status,
headers: response.headers,
html: await response.text()
})
console.log(detected, provider, detection)
// => true 'cloudflare' 'html'Install it with npm install is-antibot. detected says whether a shield answered, provider names it and detection reports which signal matched: statusCode, headers, cookies, html or url.
curl 'https://api.microlink.io/?url=https%3A%2F%2Fwww.example.com%2Fpricing&meta=false'Without an API key, a protected target fails with EPROXYNEEDED: “The URL provided uses antibot protection. Upgrade to a Pro plan.” It confirms protection is there, but the response does not name the provider.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
import isAntibot from 'is-antibot'
const read = async url => {
const response = await fetch(url)
const html = await response.text()
const { detected } = isAntibot({
url: response.url,
statusCode: response.status,
headers: response.headers,
html
})
return detected ? microlink.html(url) : html
}Your own fetch serves the open sites. When a shield answers, the same URL goes to the API with your Pro key, where the proxy route is automatic, and resolves to the rendered HTML.
The full list of error codes, EPROXYNEEDED and EPROXY included, is in the error codes reference. If a provider keeps blocking after you know its name, the fix is usually on the routing side: see how to fix 403 and 429 errors.
Why antibot detection comes before any retry
Detection is the decision point. Knowing who answered tells you whether to route, wait, skip or stop spending requests on a URL.
The antibot feature page lists the providers and explains how the API uses the result.
For the most common case, scraping Cloudflare-protected sites walks through what happens after detection.
When not to: if every URL you fetch already goes through the API on a Pro plan, you do not need client-side detection at all, because the proxy route is automatic. Compare plans on the pricing page.
FAQ
How do I detect if a site uses Cloudflare bot protection?
Does the Microlink API tell me which antibot provider blocked a request?
Which bot protection providers can is-antibot detect?
Can is-antibot get my scraper past a bot challenge?
What does EPROXYNEEDED tell me about a site’s bot protection?
Solve the next problem with the same API
Scrape Cloudflare-protected websites
Fix 403 and 429 scraping errors
Link previews for bot-protected sites
Ready to see who is blocking you?
Name the provider with is-antibot, then let a Pro key route the blocked URLs through the proxy.