Skip to content
Proxy · Use case

Scrape Cloudflare-protected websites and get the real page back

To scrape a Cloudflare-protected website, your request first has to reach the page instead of the “Just a moment” interstitial. Price trackers, lead enrichment, research crawlers and RAG pipelines all run into it sooner or later. With a Pro key, the Scraping API handles the wall for you: no proxy option, no challenge solver, just the extraction rules you already wrote.

See how the proxy works
The problem

Cloudflare answers your scraper with a challenge, not the page

Cloudflare, like other antibot systems, scores a request on its IP reputation, its headers, its TLS handshake and the JavaScript it runs. Datacenter traffic from a headless browser fails that score, so it gets a 403 or a JavaScript interstitial titled “Just a moment”. Your selectors then run against the challenge, return nothing, and the job logs a success with empty fields.

The usual fixes are fragile. A stealth plugin patches the browser fingerprint until the next detection update. A proxy subscription changes the IP but leaves you deciding when to switch exits, which tier to pay for and how to tell a challenge from a real page. Retrying the same blocked request the same way mostly teaches the shield that you are a bot.

On Pro plans, automatic proxy resolution is on by default. When a request hits a 403 antibot wall, the API escalates through its proxy tiers, ending with residential IPs, the slowest route and the one that usually gets through. The tier that worked is remembered per domain, so the next request to that site goes straight to it. It is well tested against the 500 most popular websites, not guaranteed for every site on the web.

How it works

How to scrape a Cloudflare-protected site with extraction rules

Write the rules for the page you want and send them with a Pro key. The unblocking happens before the rules run, so the code is the same as for an unprotected site. The proxy guide covers the signals and headers in more depth.

1 · Extract fields from the protected page
import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const product = await microlink.extract(
  'https://shop.example.com/product/42',
  {
    title: { selector: 'h1', attr: 'text' },
    price: { selector: '[itemprop="price"]', attr: 'content', type: 'number' }
  },
  { retry: 3, ttl: '1h' }
)

The call resolves to an object with one key per rule, read from the real product page. retry: 3 absorbs intermittent challenges, and ttl keeps the result cached for an hour so repeat reads skip the escalation.

2 · See which route the request took
import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const mode = microlink.last.response.headers.get('x-fetch-mode')

console.log(mode, mode.endsWith('-proxy'))
// => 'prerender-proxy' true

A value that ends in -proxy, such as prerender-proxy or fetch-proxy, means the request went through the proxy route. It is informational: nothing in your request needs to change.

3 · The same request as a URL
curl 'https://pro.microlink.io/?url=https%3A%2F%2Fshop.example.com%2Fproduct%2F42&data.title.selector=h1&data.title.attr=text&data.price.selector=%5Bitemprop%3D%22price%22%5D&data.price.attr=content&data.price.type=number&meta=false&retry=3' \
  -H 'x-api-key: $MICROLINK_API_KEY'

The Pro endpoint with your x-api-key header is all it takes. There is no proxy parameter in the URL because automatic resolution is the default on Pro.

Parameters used
  • data The extraction rules: a CSS selector, the attribute to read and the type to cast it to.
  • retry Server-side retries with exponential backoff. Default 2; 3 helps with intermittent challenges.
  • ttl Caches the unblocked result from 1 minute to 31 days. Pro plans.
  • proxy.location Optional. Pins the exit country when the site also serves regional content. Default us.

If a site keeps failing, the troubleshooting guide explains the debug headers, and detecting which antibot system blocks you tells you whether it is Cloudflare or another provider.

Why it works

Why scrape Cloudflare sites through the API instead of a stealth browser

A stealth setup is a race against detection updates. Moving the unblocking into the API turns it into someone else’s maintenance and leaves your code about the data.

01 · Escalation, not guesswork
The cheapest route that works, then remembered.
Requests go direct first and only climb the proxy tiers when a 403 wall answers. Residential exits are the last step, and once a tier works for a domain, later requests to that domain start there instead of rediscovering it.

The same mechanism is what makes it a rotating proxy alternative: you never pick exits yourself.

02 · An explicit answer
Real data or a clear error, never challenge HTML.
On the free endpoint a Cloudflare-protected target fails with EPROXYNEEDED, “The URL provided uses antibot protection”, instead of handing your rules a challenge page. On Pro the same URL is resolved through the proxy.

Automatic proxy resolution is part of every paid plan on the pricing page, starting with Pro at 46,000 requests a month.

03 · Know the limits
No CAPTCHA solving, no login.
The proxy gets a request judged as legitimate traffic. It does not solve a CAPTCHA or a Turnstile widget that the site shows to every visitor, and it does not sign in to anything.

When not to: pages that need an account are a session problem, not a proxy problem. Forward your own session with private pages patterns, and only where you are allowed to access the content.

FAQ

Can I scrape a Cloudflare-protected website with an API?

Yes. Send your extraction rules to the Pro endpoint and automatic proxy resolution handles the Cloudflare wall before the rules run. No proxy parameter is needed, and the response is the data from the real page.

Why does my scraper return a Cloudflare “Just a moment” page?

Cloudflare decided the request looked automated, usually because of a datacenter IP or a headless fingerprint, and served its interstitial instead of the page. Your scraper parsed the interstitial. Through the API on a Pro plan, that 403 triggers the proxy escalation instead of reaching your parser.

Does the scraping API solve Cloudflare Turnstile challenges?

No. There is no CAPTCHA or Turnstile solving. The proxy route is built so requests are judged as legitimate traffic in the first place; a challenge shown to every visitor is not something it clicks through.

Will every Cloudflare-protected site scrape successfully?

Not every one. Automatic proxy resolution is well tested against the 500 most popular websites worldwide, and protection settings vary from site to site. Check the x-fetch-mode header and the returned fields on a sample before you scale a job to a new domain.

What happens when I scrape a Cloudflare site on the free plan?

The request fails with the EPROXYNEEDED error code: the URL uses antibot protection and needs a Pro plan. Nothing else changes when you upgrade: the same request, sent with an API key, goes through the proxy automatically.
Related use cases

Solve the next problem with the same API

Fix 403 and 429 scraping errors

Tell an antibot 403, a throttling 429 and your own quota apart, then route, cache or wait accordingly.
Fix 403 and 429 errors 

Detect antibot protection

Name the provider behind a block, from Cloudflare to DataDome to reCAPTCHA, with the open-source is-antibot library.
Detect antibot protection 

Rotating proxy alternative

Skip the proxy pool: the API escalates to residential IPs only when a site blocks you and remembers what works per domain.
Replace your rotating proxy 

Screenshot blocked websites

When a site blocks headless browsers, a Pro key escalates the capture through proxy tiers automatically.
Screenshot blocked websites 

Markdown from bot-protected pages

Convert pages behind Cloudflare, DataDome or Akamai: on a Pro key the built-in proxy resolves automatically.
Convert bot-protected pages 

Link previews for bot-protected sites

Unfurl links to sites behind Cloudflare or DataDome: on a Pro key the built-in proxy resolves automatically.
Preview bot-protected sites 

Ready to scrape behind Cloudflare?

Keep your extraction rules, add a Pro key and let the API climb to residential IPs only when a site demands it.

Scrape protected sites