Convert bot-protected and blocked pages to Markdown
Convert a blocked page to Markdown with an API key instead of a proxy subscription and a rotation script. News sites, marketplaces and documentation behind a CDN shield are the pages that agents, RAG crawlers and research tools most want to read, and the ones most likely to answer automated traffic with a challenge. Automatic proxy resolution handles the unblocking inside the same Markdown request, with no extra parameter.
A blocked page converts to a challenge screen, not Markdown
Antibot services such as Cloudflare, DataDome and Akamai answer datacenter traffic with a JavaScript challenge, a CAPTCHA or a bare 403. Converted to Markdown, that is a heading that says “Just a moment” and no content. The pipeline records a success, moves on, and your index now has a hole where the article should be.
The usual workaround is a proxy vendor plus your own rotation logic: a list of exits to maintain, retries to tune per target and a second bill. It also splits one job across two systems, so when a conversion fails you have to work out whether the proxy, the browser or the parser was at fault. Scraping a Cloudflare site to Markdown should not need three moving parts.
With an API key, the free one included, automatic proxy resolution is on by default. When a fetch hits a 403, Microlink identifies the antibot provider and escalates through proxy tiers, ending on residential exits, the slowest and most reliable route, then caches the tier that worked for that domain. On the keyless endpoint the same target fails with the EPROXYNEEDED error code instead of returning junk, so you know exactly which URLs need an API key.
How to convert a blocked page to Markdown through the proxy
Send protected pages with your API key, or start on the keyless endpoint and move only the EPROXYNEEDED URLs to the key, then cache the result so the proxy path runs once per URL. The proxy guide lists the other signals that a target is blocking you, such as empty results on a known-good URL.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const markdown = await microlink.markdown('https://hard-target.com/article', {
selector: 'article'
})The client carries your API key, so a blocked fetch escalates through the proxy tiers with no parameter, and the selector scopes the conversion to the article element. The call resolves to the Markdown string.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const freeTier = createClient()
const convert = async url => {
try {
return await freeTier.markdown(url)
} catch (error) {
if (error.code !== 'EPROXYNEEDED') throw error
return microlink.markdown(url, { retry: 3, ttl: '1d' })
}
}The keyless endpoint runs first. Only URLs that fail with EPROXYNEEDED move to the keyed client, where the proxy resolves automatically, with three server-side retries for intermittent challenges and a one-day ttl so repeat reads come from the cache.
curl 'https://pro.microlink.io/?url=https%3A%2F%2Fhard-target.com%2Farticle&data.markdown.attr=markdown&meta=false&embed=markdown' \
-H 'x-api-key: $MICROLINK_API_KEY'The proxy comes with every API key, the free one included, so the URL targets the pro endpoint with your API key in the x-api-key header and no proxy parameter. embed=markdown returns the Markdown itself with a text/markdown content type.
- proxy Automatic with any API key, no value needed. Pass location to pin a country or url for your own proxy server.
- retry Server-side retries with exponential backoff for intermittent failures. Default 2.
- ttl Cache lifetime of the converted page: 24 hours by default, 1 minute to 31 days with any API key.
- headers Forward an accept-language or user-agent header when the target expects one. Any API key.
Confirm the route with the x-fetch-mode response header: any value ending in -proxy, such as fetch-proxy or prerender-proxy, means the request went through the proxy. x-cache-status tells you whether the response was a MISS, a HIT or a BYPASS.
Why the proxy belongs inside the Markdown API
Reaching the page and reading it are one problem for an agent. Splitting them across vendors doubles the failure modes, and troubleshooting a failed request is much shorter when one request owns the whole path.
The same API key unblocks screenshots of blocked websites and link previews for bot-protected sites, so one fix covers every workflow that touches the same domain.
The signal surfaces on the keyless endpoint. Automatic proxy resolution comes with every API key, the free one included, and the pricing page lists the plans.
When not to: the proxy does not sign you in or get you past a paywall. Pages behind authentication need forwarded headers, and content you are not permitted to access should stay out of the pipeline.
FAQ
How do I convert a Cloudflare-protected page to Markdown?
Why does a blocked page convert to a “Just a moment” screen in Markdown?
How do I know a Markdown request went through the proxy?
Can I use my own proxy for Markdown conversions?
Does the proxy let a Markdown conversion bypass paywalls or logins?
Solve the next problem with the same API
Markdown from JavaScript-rendered pages
Bulk Markdown conversion with caching
YouTube transcripts as Markdown
LLM context from any URL
Screenshot blocked websites
Link previews for bot-protected sites
Ready to read hard targets?
No option, no proxy list, clean Markdown from the pages that block everyone else. Get your free API key and unblock your pipeline.