Skip to content
Metadata API · Use case

Serve link previews at scale, without throttling or stale cards

Running a link preview API at scale means absorbing bursts: every pasted link in a chat, a note or a feed becomes a metadata request, and the same link gets pasted by thousands of users in the same hour. The Metadata API fits that shape, with parallel requests limited only by your quota and a cache that serves the repeats without counting them.

Start with the Metadata API
The problem

Link preview traffic is bursty and repetitive

A trending article produces a spike of identical requests within minutes. Messaging apps, social feeds, comment systems and editors all see it: traffic arrives in bursts, and most of it asks for links someone else already pasted.

A self-hosted unfurler handles this badly. Either it rate-limits your own users to protect its workers, or it crawls the same page over and over until the site on the other end starts blocking you. Adding a cache helps, until you also need invalidation, background refresh and per-tenant separation.

Microlink applies no throttling: you can run as many parallel requests as your quota allows. Responses are cached for 24 hours by default, staleTtl serves the cached card instantly while refreshing behind it, and scoping meta keeps each uncached request light.

How it works

How to cache and refresh unfurled links in production

Three options define a production unfurler: which fields to detect, how long to cache them, and whether to serve stale while revalidating. The caching patterns guide explains each control in depth.

1 · A lean preview request
import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const { title, description, image } = await microlink.metadata(url, {
  meta: { title: true, description: true, image: true },
  ttl: '1d',
  staleTtl: 0
})

Only the three preview fields are detected. The card is cached for a day, and with staleTtl: 0 every hit is served from the cache while a background refresh keeps it current.

2 · Unfurl a burst in parallel
import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const previews = await Promise.all(
  links.map(link =>
    microlink.metadata(link, { ttl: '1d', staleTtl: 0, retry: 3 })
  )
)

There is no per-second throttling, so the whole burst goes out at once. retry handles transient browser errors server-side, with exponential backoff.

3 · The same request as a URL
curl 'https://pro.microlink.io/?url=https%3A%2F%2Fexample.com%2Farticle&meta.title=true&meta.description=true&meta.image=true&ttl=1d&staleTtl=0' \
  -H 'x-api-key: $MICROLINK_API_KEY'

ttl and staleTtl need a Pro key, so the URL targets pro.microlink.io with the key sent as the x-api-key header.

Parameters used
  • meta Detect only the fields the card renders.
  • ttl Cache lifetime from 1 minute to 31 days. Default 24 hours. Pro plans.
  • staleTtl Serve the cached preview instantly and revalidate in the background. Cannot exceed ttl. Pro plans.
  • cacheKey Appends a custom identifier to the cache key, for separate entries per tenant or workspace. Pro plans.
  • force Bypasses the cache and returns a fresh copy, for a user-initiated refresh.
  • retry Server-side retries with exponential backoff. Default 2.

Log x-cache-status and x-rate-limit-remaining: the hit ratio tells you how much of the burst the cache absorbed, and the remaining quota tells you when to slow down. The production patterns guide shows how to back off on a 429.

Why it works

Why the cache does most of the work in a link preview API

Most preview requests are for links someone else already pasted. Serving those from the cache is the whole optimization.

01 · No throttling
Bursts are limited by quota, not by a rate limiter.
A thousand users pasting the same link produce a thousand requests that the cache turns into one fetch and many hits. Cache hits do not count against your quota, and when the quota does run out you get an explicit HTTP 429 with ERATE rather than a silent slowdown.

The same behavior backs screenshots under traffic spikes and bulk Markdown conversion.

02 · Instant and fresh
staleTtl: 0 serves now and refreshes later.
Users never wait on a fetch: the cached card returns immediately and a background refresh keeps it current. Set ttl by how fast the source changes, anywhere from 1 minute to 31 days.

Keep force for a user-initiated refresh; it always bypasses the cache and fetches the page again.

03 · Light by design
Field selection keeps each request small.
A card needs title, description and image. Restricting meta to those skips logo, author and date detection, which makes every cache miss faster and every payload smaller.

When not to: a low-volume internal tool with a handful of links a day does not need cache tuning. The defaults and the free endpoint, with 25 requests per day, are enough. For sustained volume above the listed plans, talk to us about enterprise.

FAQ

Is there a per-second rate limit on metadata API requests?

No. Microlink applies no throttling: you can run as many parallel requests as your quota allows. The free endpoint has a soft limit of 25 requests per day, and paid plans use a monthly quota that starts at 14,000 requests.

How do I keep link previews fresh without re-fetching every time?

Set ttl to the freshness you need and staleTtl to 0. Every request is served from the cache instantly and refreshed in the background, so the next visitor gets the newer card. force refreshes a single preview on demand.

Do cached link previews count against my quota?

No. Cache hits do not count against your quota. Only a cache miss, which actually fetches the page, uses a request, so a link pasted a thousand times in a day costs one.

Can I separate link preview cache entries per workspace?

Yes, on Pro plans. cacheKey appends a custom identifier to the cache key, so the same URL can have independent entries per tenant, workspace or variant.

What happens when a site blocks my link preview requests?

A detected block returns the EPROXYNEEDED error code. On Pro plans, retry with proxy: true and the request routes through the managed proxy pool; see link previews for bot-protected sites.
Related use cases

Solve the next problem with the same API

Only the fields you need

Include or exclude normalized fields per request and trim the JSON for faster, lighter metadata calls.
Request only the fields you need 

Link previews for bot-protected sites

Unfurl links to sites behind Cloudflare or DataDome by routing the metadata request through the built-in proxy.
Preview bot-protected sites 

Fix missing or wrong og:image

Override any normalized field with a rule and chain fallbacks so link previews never render empty.
Fix a broken og:image 

Screenshots under traffic spikes

Absorb bursts of screenshot traffic without a browser pool: no throttling, parallel requests and a cache whose hits are free.
Handle traffic spikes 

Bulk Markdown conversion with caching

Convert thousands of URLs in parallel, cached per URL and refreshed in the background for cheap re-indexing.
Convert URLs in bulk 

PDFs in bulk

Render thousands of documents from URLs in one job: parallel requests, server-side retries and per-document caching.
Generate PDFs in bulk 

Ready to unfurl at any volume?

No throttling, a cache that serves the repeats, and a 99.9% SLA on every paid plan. Pick a plan sized for your traffic and ship previews that never lag.

Scale your link previews