Fix 403 Forbidden and 429 Too Many Requests in your scraper
Scraping 403 Forbidden and 429 Too Many Requests errors look like the same problem, but they rarely have the same cause or the same fix. One is an antibot wall judging your IP and fingerprint, one is a site throttling repeat traffic, and one can be your own scraping quota. This page sorts them by where they come from, so you stop retrying the wrong way and your scraper only waits when waiting actually helps.
Your scraper is blocked, and the status code alone does not say why
A 403 from a protected site usually means an antibot service scored the request as automated: a datacenter IP, a header set that does not match a real browser, a TLS handshake that gives the client away. A 429 means the site is rate limiting the address the requests come from. Both arrive as a failed fetch, and a scraper that logs only the status code treats them as the same flaky error.
The reflex fixes make it worse. Retrying a 403 immediately, or with a new user agent on the same IP, sends the same signals again and looks more suspicious each time; a real browser profile is the whole header set, not one string. Retrying a 429 in a tight loop extends the throttle. And if you call a scraping API, its own 429 for an exhausted quota is a third case that no proxy will fix.
On Pro plans the antibot 403 is handled before it reaches you: automatic proxy resolution escalates through proxy tiers up to residential IPs and remembers per domain which one worked. The free endpoint names the same situation with EPROXYNEEDED instead of a raw 403. A Microlink 429 is ERATE, your quota, with a reset time. Caching cuts the repeat hits that trigger throttling.
How to fix 403 and 429 errors when scraping
Classify the failure by error code, then reduce how often you hit the target. The troubleshooting guide lists every plan and auth error alongside these.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const read = async url => {
try {
return { url, page: await microlink.metadata(url, { retry: 3 }) }
} catch (error) {
if (error.code === 'ERATE') return { url, blocked: 'quota' }
if (error.code === 'EPROXYNEEDED') return { url, blocked: 'antibot' }
throw error
}
}ERATE is the API’s own 429: your plan quota ran out. EPROXYNEEDED is an antibot wall hit on the free endpoint. With a Pro key the antibot case is resolved through the proxy, so it stops showing up here.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const page = await microlink.metadata('https://example.com/pricing', {
ttl: '1d',
staleTtl: 0
})ttl keeps the response for a day and staleTtl: 0 serves the cached copy while a fresh one is fetched in the background. Cache hits never reach the target site and never count toward your quota.
curl 'https://pro.microlink.io/?url=https%3A%2F%2Fexample.com%2Fpricing&ttl=1d&staleTtl=0&retry=3' \
-H 'x-api-key: $MICROLINK_API_KEY'Sent to the Pro endpoint with your x-api-key header, the request needs no proxy parameter. The payload includes statusCode, the HTTP status the target answered with.
- retry Server-side retries with exponential backoff instead of your own tight loop. Default 2.
- ttl Cache lifetime from 1 minute to 31 days, so repeat reads skip the target. Pro plans.
- staleTtl Serves the cached copy while revalidating in the background. Pro plans.
- proxy Automatic on Pro. Pass it only to pin a country or to use your own proxy server.
When the quota is the limit, the rate limit docs explain the reset window. When a site keeps answering 403 and you want to know who is behind it, detect the antibot system first.
Why sorting 403 and 429 errors beats retrying them
Every retry of a blocked request costs time and makes the next one look worse. Knowing which of the three blocks you hit tells you whether to route, wait or cache.
For the Cloudflare case specifically, see scraping Cloudflare-protected sites.
The pricing page lists every plan, and automatic proxy resolution comes with all of the paid ones.
When not to: a 403 from a login page or a paywall is an access decision, not a bot score. Forward your own session with private pages patterns where you are allowed to, and leave it alone otherwise.
FAQ
Why does my scraper get 403 Forbidden on some sites?
How do I fix 429 Too Many Requests when scraping?
Does changing the user agent fix a 403 when scraping?
What does EPROXYNEEDED mean in a scraping response?
Is a 429 from the scraping API the same as a 429 from the site?
Solve the next problem with the same API
Scrape Cloudflare-protected websites
Detect antibot protection
Rotating proxy alternative
Link previews for bot-protected sites
Markdown from bot-protected pages
PDFs of bot-protected pages
Ready to stop retrying blocked requests?
Pro plans route antibot 403s through the proxy automatically and cache the result, so blocked targets stop being a retry loop.