Skip to content
API
GUIDES
SDK
CARDS
Skills

extract

Copy for LLM
|
View as Markdown
Typed values pulled with your own rules. Declare the data you want from a page — a CSS selector, the attribute to read, the type to validate it as — and get it back normalized. It takes the rules as its second argument and resolves to an object with one key per rule:
import createClient from 'microlink.io'

const microlink = createClient()

const { image } = await microlink.extract('https://microlink.io', {
  image: {
    selector: 'meta[property="og:image"]',
    attr: 'content',
    type: 'image'
  }
})

console.log(image.url, image.width, image.height)
For any url, Microlink API already returns normalized data fields extracted from Open Graph, JSON-LD, and a series of DOM fallbacks. Rules let you go further: extract any value present on any website, overwrite a normalized field whose source is wrong, and build your own API on top of any site. Every example on this page assumes the microlink client above.

A rule

A rule is defined by a handful of primitives. Each one answers a single question about the value you want:
PrimitiveQuestion it answersExample
selectorWhich element? The first match of a CSS selector'meta[property="og:image"]'
selectorAllWhich elements? Every match, returning a collection'.athing'
attrWhat to read from it: an HTML attribute, or text, html, markdown, json'content'
typeHow to validate and normalize the value'image'
evaluateRun JavaScript in the page instead of querying the DOM'window.next.version'
A rule needs at least a query — selector, selectorAll, or evaluate — or an attr alone to serialize the whole page. Omitted primitives fall back to attr: 'html' and type: 'auto'.
Rules compose in two ways: an object under attr builds nested structures, and an array of rules defines fallbacks evaluated in order until one yields a value.

selector

Type:
<string> | <string[]>

Values: CSS selector
It defines the HTML element you want to pick from the HTML markup over the url:
import createClient from 'microlink.io'

const microlink = createClient()

const github = username =>
  microlink.extract(`https://github.com/${username}`, {
    avatar: {
      selector: 'meta[property="og:image"]:not([content=""])',
      attr: 'content',
      type: 'image'
    }
  })

const username = 'kikobeats'
const { avatar } = await github(username)

console.log(`GitHub avatar for @${username}: ${avatar.url} (${avatar.size_pretty})`)
It's equivalent to Document.querySelector() and any CSS selector can be specified, such as:
  • An HTML tag (e.g., 'img').
  • A CSS class or pseudo-class, id or data-attribute (e.g., '#avatar').
  • A combination of both (e.g., 'img:first').
When selector is omitted, attr operates on the entire page — see whole-page serialization. The same selector is what the markdown, html, text, and links methods accept as an option to scope their extraction.

Fallback selectors

A collection of selectors is an array of fallback rules: the first selector that yields a typed value wins.

selectorAll

Type:
<string> | <string[]>

Values: CSS selector
It's the same as selector but it returns a collection of results, being equivalent to Document.querySelectorAll():
import createClient from 'microlink.io'

const microlink = createClient()

const hackerNews = () =>
  microlink.extract('https://news.ycombinator.com/', {
    posts: {
      selectorAll: '.athing',
      attr: {
        title: {
          type: 'title',
          selector: '.titleline > a',
          attr: 'text'
        },
        url: {
          type: 'url',
          selector: '.titleline > a',
          attr: 'href'
        }
      }
    }
  })

const { posts } = await hackerNews()

console.log('latest hacker news posts:', posts)
Without a nested attr, each match contributes one plain value, which is how links and the other sweep methods work:
const { links } = await microlink.extract('https://news.ycombinator.com/', {
  links: {
    selectorAll: '.titleline > a',
    attr: 'href',
    type: 'url'
  }
})

console.log(links) // => ['https://…', 'https://…', …]

attr

Type:
<string> | <string[]>

Default: 'html'
Values:
tagName | nodeName | 'html' | 'outerHTML' | 'text' | 'markdown' | 'json' | 'val'
It specifies how the value should be extracted from the matched selector:
import createClient from 'microlink.io'

const microlink = createClient()

const github = username =>
  microlink.extract(`https://github.com/${username}`, {
    avatar: {
      selector: 'meta[property="og:image"]:not([content=""])',
      attr: 'content',
      type: 'image'
    }
  })

const username = 'kikobeats'
const { avatar } = await github(username)

console.log(`GitHub avatar for @${username}: ${avatar.url} (${avatar.size_pretty})`)
Any HTML attribute is supported, plus the following special cases:
  • 'html': Get the inner HTML content of the matched selector.
  • 'outerHTML': Get the outer HTML of the matched selector, including the element itself.
  • 'text': Returns the combined text content, including its descendants, by removing leading, trailing, and repeated whitespace.
  • 'markdown': Converts the HTML content into Markdown, preserving headings, links, and formatting.
  • 'json': Parses the page body as JSON and returns structured data. Whole-page only — do not combine with selector. See Extract JSON in the Data extraction guide.
  • 'val': Get the current value of the matched selector, oriented for select or input fields.

Whole-page serialization

When selector is omitted, attr operates on the entire page. This is useful for serializing a full page into a new output format:
const { content } = await microlink.extract('https://example.com', {
  content: {
    attr: 'markdown'
  }
})

console.log(content)
// => '# Example Domain\n\nThis domain is for use in illustrative examples…'
The markdown, html, and text methods are shortcuts over exactly this rule.
You can also scope the conversion to a specific element by combining selector with attr:
const { article } = await microlink.extract('https://example.com', {
  article: {
    selector: 'article',
    attr: 'markdown'
  }
})

console.log(article)
// => '# Article Title\n\nArticle content as markdown…'
For JSON endpoints, use attr: 'json' to parse the response body as structured data:
const { content } = await microlink.extract('https://pokeapi.co/api/v2/pokemon', {
  content: {
    attr: 'json'
  }
})

console.log(content)
// => { count: 1302, next: '…', results: [ … ] }
json is whole-page only — it cannot be combined with selector. See Extract JSON for the full walkthrough.

Nested rules

An object under attr maps a data structure over the same property key. Each nested rule is evaluated relative to the element matched by the parent selector, and the result is an object with one key per nested rule:
const github = username =>
  microlink.extract(`https://github.com/${username}`, {
    stats: {
      selector: '.application-main',
      attr: {
        followers: {
          selector: '.js-profile-editable-area a[href*="tab=followers"] span',
          type: 'number'
        },
        following: {
          selector: '.js-profile-editable-area a[href*="tab=following"] span',
          type: 'number'
        },
        stars: {
          selector: '.js-responsive-underlinenav a[data-tab-item="stars"] span',
          type: 'number'
        }
      }
    }
  })

const username = 'kikobeats'
const { stats } = await github(username)

console.log(`GitHub stats for @${username}:`, stats)
// => { followers: 1234, following: 56, stars: 789 }
The same structure applies to every item of a list when the parent uses selectorAll, which is how you turn repeated markup into an array of objects:
const { stories } = await microlink.extract('https://news.ycombinator.com', {
  stories: {
    selectorAll: '.athing',
    attr: {
      title: { selector: '.titleline > a', attr: 'text' },
      url: { selector: '.titleline > a', attr: 'href', type: 'url' }
    }
  }
})

console.log(stories[0]) // => { title: '…', url: 'https://…' }
Nested rules can nest again, so a parent rule can describe a whole section of a page as one JSON document.

Fallback values

Multiple attr values use the same fallback form: the first attribute that resolves a value is used.

type

Type:
<string> | <string[]>

Default: 'auto'
Values:
'audio' | 'author' | 'auto' | 'boolean' | 'date' | 'description' | 'email' | 'image' | 'ip' | 'lang' | 'logo' | 'number' | 'object' | 'publisher' | 'regexp' | 'string' | 'title' | 'url' | 'video'
It defines how the value extracted should be considered.
import createClient from 'microlink.io'

const microlink = createClient()

const productHunt = id =>
  microlink.extract(`https://www.producthunt.com/posts/${id}`, {
    name: {
      selector: 'h1 a',
      attr: 'text',
      type: 'string'
    },
    upvotes: {
      selector: '.bigButtonCount_10448',
      attr: 'text',
      type: 'number'
    }
  })

const productSlug = 'microlink-2-0'
const { name, upvotes } = await productHunt(productSlug)

console.log(`'${name}' has ${upvotes} upvotes`)
The data shape ensures that the extracted value will only be considered as valid when it's of the declared shape: a rule whose value doesn't match its type resolves to null, which is what lets fallback rules move on to the next candidate.
Media types do more than validate. 'image', 'video', 'audio', and 'logo' resolve the value to an absolute URL and expand it into an asset object with url, type, width, height, size, and size_pretty, the same shape the normalized data fields use:
const { cover } = await microlink.extract('https://www.youtube.com/watch?v=9P6rdqiybaw', {
  cover: {
    selector: 'meta[property="og:image"]',
    attr: 'content',
    type: 'image'
  }
})

console.log(cover.width, cover.height, cover.size_pretty)

evaluate

Type:
<string> | <function>

It evaluates the JavaScript provided inside the browser context over the target URL, returning the result.
It's quite similar to selector, but designed to specify the value to be obtained in a JavaScript-like way.
import createClient from 'microlink.io'

const microlink = createClient()

const getNextVersion = url =>
  microlink.extract(url, {
    version: {
      evaluate: 'window.next.version',
      type: 'string'
    }
  })

const { version } = await getNextVersion('https://vercel.com')

console.log(`Next.js version is: ${version}`)
You can combine evaluate with types for data correctness.
It can evaluate anything browser compatible in the JavaScript context. A function is serialized to its source before being sent, so it can be as long as you need — but it runs in the page, not in your process, so it can only reach what the page can:
const getExcerpt = url =>
  microlink.extract(url, {
    excerpt: {
      evaluate: async () => {
        const response = await window.fetch(
          'https://cdn.jsdelivr.net/npm/@mozilla/readability/Readability.js'
        )
        const script = await response.text()
        window.eval(script)
        const reader = new window.Readability(window.document)
        return reader.parse().excerpt
      },
      type: 'string'
    }
  })

const { excerpt } = await getExcerpt('https://levelup.gitconnected.com/how-to-load-external-javascript-files-from-the-browser-console-8eb97f7db778')

console.log(excerpt)
When the logic outgrows a single expression — clicks, waits, npm packages — reach for function, which gives the function full Puppeteer access instead of a page-side evaluation.

Options

A third argument takes the shared options, useful for pairing rules with prerender, waitForSelector, ttl, or headers forwarded to the target page:
const { price } = await microlink.extract(
  'https://example.com/product',
  { price: { selector: '.price', attr: 'text', type: 'number' } },
  { prerender: true, waitForSelector: '.price', ttl: '1h' }
)

Result

It resolves to an object with one key per rule. Values are normalized by their type: a 'string' is a string, a 'number' is a number, and 'image', 'video', 'audio', and 'logo' become asset objects carrying url, type, width, height, size, and size_pretty. A rule that matches nothing, or whose value fails its type, resolves to null, so destructuring is always safe.
The complete API response — its status, the payload, and the HTTP response with its headers — stays available on microlink.last after every call; see inspect the last request. Failures throw a MicrolinkError like every other method.

Rules alongside metadata

extract returns only your fields. To evaluate rules next to the normalized data fields in one request, pass them as the data option of metadata:
const { title, description, price } = await microlink.metadata('https://example.com/product', {
  data: {
    price: { selector: '.price', attr: 'text', type: 'number' }
  }
})
A rule named after a normalized field — title, image, author — overrides that field, which is how you fix a page whose metadata is wrong or missing.

Fallback rules

A field can be defined by more than one rule. Pass an array and the rules are evaluated respecting the order: if the first one fails, the second is tried, then the third, and so on. The value is the one obtained by the first rule that succeeds:
const github = username =>
  microlink.extract(`https://github.com/${username}`, {
    avatar: [
      {
        selector: 'meta[name="twitter:image:src"]:not([content=""])',
        attr: 'content',
        type: 'image'
      },
      {
        selector: 'meta[property="og:image"]:not([content=""])',
        attr: 'content',
        type: 'image'
      }
    ]
  })

const username = 'kikobeats'
const { avatar } = await github(username)

console.log(`GitHub avatar for @${username}: ${avatar.url} (${avatar.size_pretty})`)
A rule fails when its query matches nothing or when the value doesn't pass its type, so a fallback chain is also how you make a rule resilient to pages whose markup varies. The same array form works for selector and attr on their own.

Everywhere else

The markdown, html, text, and links methods are shortcuts over rules like these, and accept the same primitives as options to scope their extraction. The same grammar — historically known as MQL, the Microlink Query Language — is what the data query parameter takes when you call the API directly, and what the CLI accepts as JSON through extract --data.
Underneath, the SDK talks to the API through @microlink/mql, the low-level client it ships as a dependency; you never need to install or import it yourself. Reading a response body as a stream or a buffer is the one thing that still lives there rather than in the SDK.
See the data extraction guide for defining rules, page preparation, response shaping, and troubleshooting.