Scraping API · Use case

Turn any website into JSON with CSS selector rules

A website to JSON API should return the fields you asked for, typed, and nothing else. Catalog imports, lead lists, content migrations and internal dashboards all start with the same job: read a few values off a page and store them as data. With the Scraping API you describe each field as a rule and the response is the JSON object you described.

Start with the Scraping API

The problem

HTML is a document, not the JSON your code expects

Every page mixes the three values you need with navigation, scripts, ads and markup. Getting from that to a clean object means fetching the page, parsing it, walking the tree, trimming whitespace and converting strings into numbers, URLs and dates, once per site.

The usual stack is an HTTP client plus an HTML parser, and it quietly fails on pages that build their content in the browser, where the fetched HTML is an empty shell. Adding a headless browser fixes that and hands you a fleet to run. Either way the output is loose strings, so a missing element surfaces later as undefined deep inside your code.

The data parameter turns that into a schema. Each key is a rule: a CSS selector for the element, an attr for what to read and a type to validate it. Microlink fetches the page, renders it in a browser only when it needs to, applies the rules and returns one key per rule. The extract method is the same grammar from the SDK.

Open the live JSON for the Hacker News front page

How it works

How to convert a website to JSON with CSS selector rules

Start with one field, check it, then add the rest. The defining rules guide walks through single values, collections, nested objects and fallbacks in that order.

1 · Declare the fields

import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const { title, author, published } = await microlink.extract(
  'https://example.com/blog/post',
  {
    title: { selector: 'h1', attr: 'text' },
    author: { selector: '[rel=author]', attr: 'text', type: 'author' },
    published: { selector: 'time', attr: 'datetime', type: 'date' }
  }
)

extract resolves to an object with exactly one key per rule and none of the normalized metadata. A rule that matches nothing, or whose value fails its type, comes back as null.

2 · Build nested objects

import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const { stories } = await microlink.extract('https://news.ycombinator.com', {
  stories: {
    selectorAll: '.athing',
    attr: {
      title: { selector: '.titleline > a', attr: 'text' },
      url: { selector: '.titleline > a', attr: 'href', type: 'url' }
    }
  }
})

An object under attr is evaluated relative to each element matched by selectorAll, so the result is an array of objects. Nested rules can nest again to describe a whole section of the page.

3 · The same schema as a URL

curl 'https://api.microlink.io/?url=https%3A%2F%2Fexample.com&data.title.selector=h1&data.title.attr=text&meta=false'

Rules flatten to data.title.selector and data.title.attr query parameters, so any HTTP client can call it. meta: false skips the metadata pass, usually the biggest speedup for data-only requests.

Parameters used

data The output schema: one key per field, one rule per key.
selector The first element matching a CSS selector. An array of selectors acts as a fallback list.
attr Any HTML attribute, or text, html, outerHTML, markdown, json or val. Default html.
type Validates and normalizes the value: number, url, date, email, image and more. Default auto.
meta Set to false to return only your fields and skip the normalized metadata.
filter Keeps only the listed fields in the payload, with dot notation for nested ones.

Rules run on the rendered page when one is needed: prerender defaults to auto, so client-rendered pages get a browser and static ones do not. For apps that need a wait before the data exists, see scraping JavaScript-rendered pages.

Why it works

Why a declarative JSON schema beats a hand-written parser

The rule set is the scraper. It lives in one object you can version, review and reuse across pages that share a template.

01 · Typed output

Numbers are numbers, URLs are absolute.

type validates each value before it reaches you. A url rule resolves relative hrefs to absolute ones, a number rule returns a number, and an image rule expands into an object with url, width, height and size.

A value that fails its type becomes null, which is what lets fallback rules move on to the next candidate.

02 · Predictable shape

Every key is always present.

Missing elements come back as null instead of throwing or disappearing, so destructuring is always safe and a schema change on the target site shows up as a null you can alert on.

Lists and rows use the same grammar: scrape tables and repeated lists shows nested rules turning each row into an object.

03 · One endpoint

Fetch, render and extract in one call.

There is no parser to install and no browser to host. The same request decides whether the page needs rendering, applies adblock by default and returns the JSON.

When not to: if you want the whole article as text for a model or a search index, a schema is overkill. Convert the page to clean Markdown instead.

FAQ

How do I convert a website to JSON with an API?

Send the URL with a data object where each key is a field and each value is a rule with a CSS selector, the attr to read and an optional type. The response contains one key per rule. From JavaScript, microlink.extract(url, rules) does the same and returns only your fields.

What happens when a CSS selector rule matches nothing?

The field resolves to null, and so does a value that fails its type. The request still succeeds and every other field is returned, so one broken selector never takes down the whole object.

Can I use the HTML to JSON API without the JavaScript SDK?

Yes. The rules flatten into query parameters such as data.title.selector=h1 and data.title.attr=text, so any language or HTTP client can call the API. The data extraction guide shows the raw URL form next to every example.

How do I get only my own fields in the JSON response?

Pass meta: false. Without it the API also returns normalized metadata such as title, description and image next to your fields. The SDK extract method skips it for you, while metadata() keeps both in one response.

Does scraping a website to JSON work on the free tier?

Yes. The API works without a key for 25 requests per day, enough to build and test a schema. A Pro plan adds configurable cache TTL, custom headers and automatic proxy resolution for sites that block automated traffic, see pricing.

Related use cases

Solve the next problem with the same API

Tables and lists to JSON arrays

Turn table rows, product grids and search results into an array of objects, one nested rule per column.

Scrape tables into JSON

Product prices and stock

Read prices as numbers and stock as text from any product page, with fallback rules that survive template changes.

Scrape product prices

Custom fields from JavaScript apps

Render React, Vue or Angular apps in a real browser, wait for the element you need, then run your rules.

Scrape JavaScript-rendered pages

Cached JSON endpoints

Fetch any JSON endpoint without a browser, keep its shape intact and serve repeat calls from the cache.

Fetch and cache JSON APIs

Custom fields alongside metadata

Get prices, ratings, headings or any CSS selector, typed and returned next to the normalized metadata.

Extract custom fields

Clean Markdown, no boilerplate

Convert only the article body: one selector keeps navigation, ads and widgets out of the Markdown.

Get clean Markdown

Ready to turn any page into JSON?

Declare the fields, get typed JSON back. Start on the free tier and write your first rule in a minute.

Scrape your first page