Replace rotating proxies with an API that picks the route
A rotating proxy API sells you exits; you still have to run the browser, notice the block, choose the next IP and decide when a site deserves a residential one. A scraping API with built-in proxies moves those decisions to the other side of the request. On Pro plans, Microlink routes blocked requests through its proxy tiers automatically and learns per domain which route works, so your code asks for a page and gets the page.
A rotating proxy gives you IP addresses, not pages
Proxy rotation for scraping starts simple: point the HTTP client at a gateway and let it hand out a new exit per request. Then a target serves a challenge on a 200, another only lets residential traffic through, a third works on datacenter IPs and does not need the slower route at all. The rotation layer cannot see any of that, because it moves bytes and never looks at the page.
So the logic lands in your scraper: detect the block, pick a pool, retry, remember which domains needed what, and keep a headless browser running for the sites that render in JavaScript. Every new target is a small research project, and the knowledge of what works where lives in code you maintain or in someone’s head.
Microlink puts the proxy behind the request instead of in front of it. On Pro plans, a request that hits a 403 antibot wall escalates through the proxy tiers, with residential IPs as the last one: the slowest, and the one that usually gets through. The tier that worked is cached per domain, so later requests go straight to it. Browser rendering, retries and caching come in the same call.
How to scrape without managing proxy rotation
Drop the proxy gateway from your client and call the API with a Pro key. What remains is reading which domains took the proxy route, and pinning a country where it matters. The proxy guide lists the headers involved.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const urls = [
'https://news.example.com/story',
'https://shop.example.com/item/42',
'https://docs.example.com/guide'
]
for (const url of urls) {
const markdown = await microlink.markdown(url, { ttl: '1d' })
const mode = microlink.last.response.headers.get('x-fetch-mode')
console.log(new URL(url).hostname, mode, markdown.length)
}Each call resolves to the page as Markdown. x-fetch-mode ends in -proxy for the domains that needed the proxy route and not for the rest, so the log shows which targets are protected without any detection code on your side.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const markdown = await microlink.markdown('https://shop.example.co.uk/item/42', {
proxy: { location: 'gb' }
})proxy.location is the one routing decision left to you: a two-letter country code for sites that serve regional content. Everything else about the route stays automatic.
curl 'https://pro.microlink.io/?url=https%3A%2F%2Fshop.example.com%2Fitem%2F42&data.markdown.attr=markdown&meta=false&ttl=1d' \
-H 'x-api-key: $MICROLINK_API_KEY'Call the Pro endpoint with your x-api-key header. There is no proxy parameter and no gateway URL: automatic resolution is the default on Pro.
- proxy Automatic on Pro. Set it only to pin a country or to use your own proxy server.
- proxy.location ISO 3166-1 alpha-2 country code for the exit, case-insensitive. Default us.
- ttl Caches each page from 1 minute to 31 days so repeat reads skip the fetch. Pro plans.
- retry Server-side retries with exponential backoff. Default 2.
x-fetch-mode is informational: a value like prerender-proxy needs no change to your request. If you already pay for a provider you must keep, bring your own proxy with proxy.url instead.
Why built-in proxy tiers beat a rotating proxy pool
Rotation is a means. What a scraper needs is the right route for each site, found once and reused. That takes a system that sees the response, not just the connection.
Detection is one of the first checks in the request flow, as the antibot feature page describes.
Combined with the response cache, where hits never count toward your quota, a crawler that revisits a site pays for neither the search nor the fetch.
When not to: if a contract requires a specific proxy provider, or you only need raw HTTP through many IPs with no browser at all, a proxy service fits better. For the first case, route through your own proxy. Plans are on the pricing page.
FAQ
What is a good alternative to a rotating proxy API for scraping?
Does the scraping API use residential proxies?
How do I see which domains needed the proxy route?
Can I still choose the exit country without a proxy pool?
Do I need to enable the proxy on every scraping request?
Solve the next problem with the same API
Scrape Cloudflare-protected websites
Bring your own proxy
Fix 403 and 429 scraping errors
Fetch geo-blocked websites
Screenshot blocked websites
Markdown from bot-protected pages
Ready to retire your proxy rotation code?
Ask for the page, get the page. Proxy tiers, residential escalation and per-domain memory are included in Pro.