Skip to content
Scraping API · Use case

Fetch a JSON endpoint and cache the response for every repeat call

To fetch JSON from a URL through an API, you want the body parsed, the shape untouched and repeat calls answered from cache. Public datasets, third-party APIs with tight rate limits, config files and the JSON endpoints behind a single-page app all fit the pattern. The Scraping API reads them with one rule and caches the result.

Start with the Scraping API
The problem

Every client that calls a slow JSON API pays for it again

A dashboard that reads a public API on every page view, a build step that pulls the same dataset a hundred times, a frontend that hits a rate-limited endpoint directly: each request goes all the way to the origin, waits for it and burns its quota, even when the answer has not changed in hours.

The usual fix is a small caching service in front of the API. Now you run a server, choose a store, write expiry logic and handle the stampede when a popular key expires. Treating the endpoint as a web page to scrape is worse: a browser wraps the JSON in markup that you then have to strip.

A rule with attr: 'json' parses the response body with JSON.parse and returns it with its original shape, no URL rewriting and no value normalization. prerender: false fetches it without a browser, and every response is cached for 24 hours by default, tunable with ttl and served stale while refreshing with staleTtl.

Open the live parsed JSON
How it works

How to scrape a JSON endpoint and cache the response

One rule reads the body, two options control freshness. The extract JSON section of the defining rules guide documents the parsing behavior.

1 · Parse the endpoint
import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const { content } = await microlink.extract(
  'https://pokeapi.co/api/v2/pokemon',
  { content: { attr: 'json' } },
  { prerender: false }
)

console.log(content.count, content.results.length)

content is the parsed body as native objects and arrays. Microlink reads the body directly, or the contents of a pre element when a browser wrapped it, so the rule works either way.

2 · Cache it for a day, refresh in the background
import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const { content } = await microlink.extract(
  'https://pokeapi.co/api/v2/pokemon',
  { content: { attr: 'json' } },
  { prerender: false, ttl: '1d', staleTtl: 0 }
)

ttl keeps the response for a day and staleTtl 0 serves the cached copy instantly while a fresh one is generated in the background. Both are Pro options.

3 · Keep only the part you need
curl 'https://api.microlink.io/?url=https%3A%2F%2Fpokeapi.co%2Fapi%2Fv2%2Fpokemon&data.content.attr=json&meta=false&prerender=false&filter=content.results'

filter with dot notation trims the payload to content.results, so a large endpoint returns only the array your client reads.

Parameters used
  • attr json parses the whole body. It is whole-page only and cannot be combined with selector.
  • prerender false fetches the endpoint with a plain HTTP request, no browser.
  • ttl How long the response stays cached, from 1 minute to 31 days. Default 24 hours. Pro plans.
  • staleTtl Serves the stale copy while revalidating in the background. Cannot exceed ttl. Pro plans.
  • filter Keeps only the listed fields of the response, with dot notation for nested ones.
  • force Bypasses the cache for one request when you need the origin’s current answer.

The x-cache-status response header reads HIT when a response came from cache and MISS when it was fetched fresh, and x-cache-ttl shows the effective lifetime. The caching patterns guide lists every cache header and recommended TTLs by content type.

Why it works

Why a cached JSON fetch beats running your own caching proxy

The cache, the expiry and the background refresh are already there. You pick the lifetime per request instead of deploying a service to hold it.

01 · Shape preserved
The JSON comes back exactly as the origin sent it.
No URL rewriting, no array compaction, no value normalization. Strings that look like HTML pass through unchanged, and the parsed value can be any JSON type, not just an object.

When the JSON sits inside an HTML page instead of a dedicated endpoint, go back to selectors in scrape any website to JSON.

02 · Cache included
Repeat calls are served from the edge.
The first request creates a shared copy, and later requests are served from it and from the nearest edge node. Cache hits never count against your quota, so a popular endpoint costs roughly one request per cache window.

Rate-limited or slow upstreams benefit most. For crawl-scale refresh schedules, bulk Markdown conversion with caching shows ttl and staleTtl at volume.

03 · Private endpoints too
Forward a token without putting it in the URL.
Authenticated APIs take the same rule. Send the token as an x-api-header-authorization request header on Pro and the target receives it as a regular authorization header.

When not to: an endpoint you own is better cached with its own HTTP cache headers. For per-user APIs, cached copies are keyed by URL and query parameters, not by token, so read scraping behind a login and add a cacheKey per user.

FAQ

How do I fetch JSON from a URL through the API?

Send the endpoint URL with a data rule that has attr: 'json' and no selector, plus prerender: false. The response field holds the parsed JSON with its original shape.

Can I combine attr json with a CSS selector?

No. json is whole-page only and always parses the entire response body. To read JSON embedded in an HTML page, use an evaluate rule or a selector with attr text and parse it on your side.

How long are cached API responses kept?

Every response is cached for 24 hours by default. On a Pro plan, ttl sets any lifetime from 1 minute to 31 days, staleTtl serves the cached copy while refreshing it, and force: true skips the cache for a single request.

Do cached JSON responses count against my quota?

No. Cache hits never count against your quota. Only the requests that go to the origin do, so a long ttl on a popular endpoint keeps usage close to one request per cache window.

Can I trim a large JSON API response to the fields I need?

Yes. Add filter with a dot-notation path such as content.results or content.count, and the payload keeps only those fields. Delivery and response shaping compares filter with the other response models.
Related use cases

Solve the next problem with the same API

Any website to JSON

Declare the fields you want as CSS selector rules and get typed JSON back, with null for anything the page does not have.
Turn a website into JSON 

Scraping behind a login

Forward a session cookie or bearer token as a request header and extract data from pages only your users can see.
Scrape authenticated pages 

JavaScript with npm packages

Require any npm package inside a remote function, pin its version, and skip the browser entirely when the code does not need one.
Run JavaScript remotely 

Tables and lists to JSON arrays

Turn table rows, product grids and search results into an array of objects, one nested rule per column.
Scrape tables into JSON 

Bulk Markdown conversion with caching

Convert thousands of URLs in parallel, cached per URL and refreshed in the background for cheap re-indexing.
Convert URLs in bulk 

Link previews at scale

Unfurl links at any volume: no throttling, background refresh and cache hits that never count against your quota.
Scale your link previews 

Ready to cache any JSON endpoint?

Parsed JSON with its shape intact and a cache you set per request. Start on the free tier and add ttl when you move to Pro.

Fetch a JSON endpoint