Convert bot-protected and blocked pages to Markdown
Convert a blocked page to Markdown with one extra option instead of a proxy subscription and a rotation script. News sites, marketplaces and documentation behind a CDN shield are the pages that agents, RAG crawlers and research tools most want to read, and the ones most likely to answer automated traffic with a challenge. The proxy parameter handles the unblocking inside the same Markdown request.
A blocked page converts to a challenge screen, not Markdown
Antibot services such as Cloudflare, DataDome and Akamai answer datacenter traffic with a JavaScript challenge, a CAPTCHA or a bare 403. Converted to Markdown, that is a heading that says “Just a moment” and no content. The pipeline records a success, moves on, and your index now has a hole where the article should be.
The usual workaround is a proxy vendor plus your own rotation logic: a list of exits to maintain, retries to tune per target and a second bill. It also splits one job across two systems, so when a conversion fails you have to work out whether the proxy, the browser or the parser was at fault. Scraping a Cloudflare site to Markdown should not need three moving parts.
On Pro plans Microlink includes automatic proxy resolution. Add proxy: true and the request leaves through a managed, rotating proxy pool, after Microlink identifies the antibot provider and picks the resolution path for it. When a target needs the proxy and you did not ask for it, the API fails with the EPROXYNEEDED error code instead of returning junk, so you know exactly which URLs to retry.
How to convert a blocked page to Markdown through the proxy
Try the direct request first, retry with the proxy only on EPROXYNEEDED, and cache the result so the proxy path runs once per URL. The proxy guide lists the other signals that a target is blocking you, such as empty results on a known-good URL.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const markdown = await microlink.markdown('https://hard-target.com/article', {
proxy: true,
selector: 'article'
})proxy: true routes the request through the managed pool, and the selector scopes the conversion to the article element. The call resolves to the Markdown string.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const convert = async url => {
try {
return await microlink.markdown(url)
} catch (error) {
if (error.code !== 'EPROXYNEEDED') throw error
return microlink.markdown(url, { proxy: true, retry: 3, ttl: '1d' })
}
}The direct call runs first. Only targets that reject it take the proxy path, with three server-side retries for intermittent challenges and a one-day ttl so repeat reads come from the cache.
curl 'https://pro.microlink.io/?url=https%3A%2F%2Fhard-target.com%2Farticle&data.markdown.attr=markdown&meta=false&proxy=true&embed=markdown' \
-H 'x-api-key: $MICROLINK_API_KEY'proxy is a Pro option, so the URL targets the pro endpoint and authenticates with your API key in the x-api-key header. embed=markdown returns the Markdown itself with a text/markdown content type.
- proxy true for automatic proxy resolution, an object with location to pin a country, or your own proxy URL. Pro plans.
- retry Server-side retries with exponential backoff for intermittent failures. Default 2.
- ttl Cache lifetime of the converted page: 24 hours by default, 1 minute to 31 days on Pro plans.
- headers Forward an accept-language or user-agent header when the target expects one. Pro plans.
Confirm the route with the x-fetch-mode response header: any value prefixed with proxy-, such as fetch-proxy or prerender-proxy, means the request went through the proxy. x-cache-status tells you whether the response was a MISS, a HIT or a BYPASS.
Why the proxy belongs inside the Markdown API
Reaching the page and reading it are one problem for an agent. Splitting them across vendors doubles the failure modes, and troubleshooting a failed request is much shorter when one request owns the whole path.
The same option unblocks screenshots of blocked websites and link previews for bot-protected sites, so one fix covers every workflow that touches the same domain.
The signal surfaces on every plan, including the free endpoint. Routing through the proxy is a Pro capability, and the pricing page lists the plans.
When not to: the proxy does not sign you in or get you past a paywall. Pages behind authentication need forwarded headers, and content you are not permitted to access should stay out of the pipeline.
FAQ
How do I convert a Cloudflare-protected page to Markdown?
Why does a blocked page convert to a “Just a moment” screen in Markdown?
How do I know a Markdown request went through the proxy?
Can I use my own proxy for Markdown conversions?
Does the proxy let a Markdown conversion bypass paywalls or logins?
Solve the next problem with the same API
Markdown from JavaScript-rendered pages
Bulk Markdown conversion with caching
YouTube transcripts as Markdown
LLM context from any URL
Screenshot blocked websites
Link previews for bot-protected sites
Ready to read hard targets?
One option, no proxy list, clean Markdown from the pages that block everyone else. Get a Pro key and unblock your pipeline.