Convert URLs to Markdown in bulk, in parallel and cached
Bulk URL to Markdown conversion is a loop over your URL list: the Markdown API takes the batch as parallel requests, caches every page and refreshes it in the background. Building a knowledge base, feeding a Markdown RAG pipeline or migrating a documentation site means converting a whole site once, then converting it again next week. The second pass should only cost you what changed.
Bulk Markdown conversion is a capacity and freshness problem
A thousand pages are a thousand fetches, some of them browser renders, all of them needing retries, timeouts and cleanup. Run them one by one and the job takes hours. Run them all at once on your own machines and you are now operating a browser pool.
A scraper fleet has to be sized for the initial crawl and then sits idle until the next one. Every re-crawl renders pages that did not change, because nothing remembers the last result. Add a rate limiter on the API side and a one-hour job turns into a day of backoff logic.
Microlink applies no throttling, so parallel requests run as fast as your quota allows. Every URL is cached for 24 hours by default and for up to 31 days with ttl, and staleTtl serves the cached copy instantly while refreshing behind it. Cache hits do not count against your quota, so re-indexing an unchanged site is close to free.
How to convert a list of URLs to Markdown in bulk
The crawl is a map over URLs with bounded concurrency on your side. Scoping, caching and retries are request options, and the caching patterns guide explains how ttl and staleTtl interact.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const pool = async (items, limit, worker) => {
const results = []
let cursor = 0
const run = async () => {
while (cursor < items.length) {
const index = cursor++
results[index] = await worker(items[index])
}
}
await Promise.all(Array.from({ length: limit }, run))
return results
}
const documents = await pool(urls, 25, url =>
microlink.markdown(url, {
selector: 'main',
meta: false,
ttl: '7d',
staleTtl: 0,
retry: 3
})
)Twenty-five workers pull from the list until it is empty, and each call resolves to the Markdown of the main element. The pool protects your memory and the target site; Microlink itself does not throttle.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const urls = await microlink.links('https://docs.example.com', {
selectorAll: 'nav a'
})links() returns absolute, deduplicated URLs. Scope it to the navigation and you get the site map of a documentation site in one request.
curl 'https://pro.microlink.io/?url=https%3A%2F%2Fdocs.example.com%2Fguide&data.markdown.selector=main&data.markdown.attr=markdown&meta=false&ttl=7d&staleTtl=0' \
-H 'x-api-key: $MICROLINK_API_KEY'Any language with an HTTP client can run the batch. ttl and staleTtl require a Pro key, so the URL targets the pro endpoint, and the Markdown comes back in the data.markdown field of the JSON response.
- ttl Cache lifetime per URL: 24 hours by default, 1 minute to 31 days on Pro plans.
- staleTtl Serve the cached copy instantly and refresh it in the background. Cannot exceed ttl. Pro plans.
- retry Server-side retries with exponential backoff. Default 2.
- meta false skips metadata detection on every conversion when you only index the body.
- force Bypass the cache and store a fresh copy for a URL you know has changed.
- cacheKey Append an identifier to the cache key to keep separate entries per index or per tenant. Pro plans.
Log the x-cache-status header of every response. A high ratio of HIT on the second crawl is the signal that the cache is doing its job, and BYPASS should only appear on the URLs you forced.
Why per-URL caching changes the cost of a Markdown crawl
The first crawl is a fixed cost. Every crawl after that should cost only what changed, which is the same logic behind configurable cache TTLs on every Microlink request.
The same fan-out pattern drives bulk PDF generation and screenshots under traffic spikes, with the options of each API.
Pick ttl by how often the source changes: an hour or less for feeds, one to seven days for blogs and docs, the 31-day maximum for stable references.
When not to: Microlink converts the URLs you give it and does not follow links on its own. For a recursive crawl across thousands of unknown pages, pair a crawler or a queue for discovery with the API for the conversion step. Clean Markdown without boilerplate covers the scoping side.
FAQ
How many URLs can I convert to Markdown in parallel?
Does re-converting a cached URL to Markdown use my quota?
How do I force a fresh Markdown conversion for a page that changed?
Can Microlink crawl a whole site and return Markdown for every page?
What happens if my bulk Markdown job exceeds the plan quota?
Solve the next problem with the same API
PDF and office documents to Markdown
Markdown from bot-protected pages
Clean Markdown, no boilerplate
PDFs in bulk
Screenshots under traffic spikes
Link previews at scale
Ready to convert at crawl scale?
Parallel, cached and refreshed in the background. Pick a Pro plan sized for your index and convert your first thousand pages today.