Skip to content
Scraping API · Use case

Scrape a JavaScript-rendered page once the app has loaded

To scrape a JavaScript-rendered page you need the DOM the browser builds, not the HTML the server sends. Single-page apps, dashboards, store fronts and job boards built with React, Vue or Angular ship an empty root element and fill it in later. The Scraping API renders the page in a real browser, waits for your element and then runs your rules.

Start with the Scraping API
The problem

The HTML of a React app has no data in it

Fetch a single-page app with an HTTP client and you get a script tag and an empty div. Every selector you wrote against the page in your browser returns nothing, because the content only exists after the JavaScript runs and the API calls behind it return.

Running headless Chrome yourself fixes the rendering and brings new problems: when is the page done? Waiting for the load event fires too early on apps that fetch data afterwards, a fixed sleep is either too short or wastes seconds on every request, and the browser fleet needs memory, updates and crash handling.

The API decides per page whether it needs a browser: prerender defaults to auto, and true forces rendering. Then waitForSelector holds extraction until the element you are about to read exists, so the rules run against the same DOM a visitor sees. The page preparation guide explains each wait option.

Open the live JSON from a client-rendered demo page
How it works

How to scrape a JavaScript-rendered page with a real browser

Force the browser, wait for the element, extract. Then tighten the wait so each request spends as little time as possible.

1 · Render and wait for the content
import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const { quotes } = await microlink.extract(
  'https://quotes.toscrape.com/js/',
  {
    quotes: {
      selectorAll: '.quote',
      attr: {
        text: { selector: '.text', attr: 'text' },
        author: { selector: '.author', attr: 'text' }
      }
    }
  },
  { prerender: true, waitForSelector: '.quote' }
)

This demo page builds its quotes with JavaScript, so the plain HTML has none. With prerender and waitForSelector, quotes resolves to an array of { text, author } objects.

2 · Navigate fast, then wait for the element
import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const { jobs } = await microlink.extract(
  'https://app.example.com/jobs',
  { jobs: { selectorAll: '[data-testid=job-title]', attr: 'text' } },
  {
    prerender: true,
    waitUntil: 'domcontentloaded',
    waitForSelector: '[data-testid=job-title]'
  }
)

waitUntil domcontentloaded ends navigation as soon as the DOM is parsed, and the selector wait covers the data that loads afterwards. This is usually faster than waiting for the network to go idle.

3 · Read the app state instead of the DOM
import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const { props } = await microlink.extract(
  'https://nextjs-app.example.com/product/42',
  {
    props: {
      evaluate: "JSON.parse(document.getElementById('__NEXT_DATA__').textContent).props.pageProps",
      type: 'object'
    }
  },
  { prerender: true }
)

evaluate runs JavaScript in the page and returns the result. Many frameworks embed their initial data as JSON, which is often cleaner than the rendered markup.

Parameters used
  • prerender auto decides per page. true forces a headless browser, false a plain HTTP fetch.
  • waitForSelector Waits until an element matching the CSS selector appears. The most reliable wait.
  • waitUntil Navigation event to wait for: auto, load, domcontentloaded, networkidle0 or networkidle2.
  • waitForTimeout A fixed delay for pages with no stable selector. Capped by the request timeout.
  • evaluate A rule that runs JavaScript in the page context instead of querying the DOM.
  • javascript Set false to render without running scripts when the page does not need them.

The x-fetch-mode response header tells you which path ran: prerender when a browser rendered the page, fetch for a plain request. If fields are still null after the wait, the troubleshooting guide walks through wrong selectors, page variants and timeouts.

Why it works

Why rendering on the API side beats running your own browser

The browser is the expensive part of scraping a React website. Moving it behind the request leaves you with the part that is specific to your job: the selectors.

01 · Right DOM
Rules run after the app has rendered.
The page loads in a real browser with ads and trackers blocked by default, the selector wait resolves when your element exists, and only then do the rules read it. Nothing runs against the empty shell.

Pages that only reveal content after a click or a scroll need one more step: scrape paginated lists and Load more buttons covers both.

02 · Only when needed
Static pages skip the browser.
With prerender on auto, server-rendered pages are fetched without a browser and client-rendered ones get one. Set false explicitly for targets you know are static and they return faster.

The caching and performance guide lists the extraction speedups in order of impact, starting with meta: false.

03 · No fleet
Every request gets a fresh browser.
There is no pool to size, no Chrome version to update and no zombie process to reap. Each call runs in its own isolated browser that is destroyed afterwards, and the request timeout is 30 seconds on free and 60 on Pro.

When not to: if you only need the page title, description and image of an app, metadata from single-page apps returns them without writing rules. For the full text, use Markdown from JavaScript-rendered pages.

FAQ

How do I scrape a React website that returns empty HTML?

Add prerender: true to force a headless browser and waitForSelector with a selector for the content you need. The rules then run against the rendered DOM instead of the empty root element the server sends.

Why does my SPA scraping request return null fields?

Usually the rules ran before the data existed. Add waitForSelector for the exact element you extract; if fields stay null, open the page in your browser and check that the selector matches the rendered DOM and that the same page variant, device or locale, is being loaded.

Should I use waitForSelector or waitForTimeout on a JavaScript page?

Prefer waitForSelector. It resolves as soon as the element appears, while waitForTimeout always waits the full delay and can still be too short on a slow load. Keep waitForTimeout for pages with no stable selector to wait on.

How do I know whether the page was scraped with a browser?

Read the x-fetch-mode response header. prerender means a headless browser rendered the page and fetch means a plain HTTP request was enough. x-fetch-time shows how long that step took.

Can I scrape the JavaScript state of a page instead of its DOM?

Yes. An evaluate rule runs JavaScript in the page and returns its result, so it can read a global variable or parse the JSON a framework embeds for hydration. For clicks, loops or npm packages, move to a remote Puppeteer function.
Related use cases

Solve the next problem with the same API

Any website to JSON

Declare the fields you want as CSS selector rules and get typed JSON back, with null for anything the page does not have.
Turn a website into JSON 

Pagination and Load more buttons

Scrape numbered pages in parallel, one call per page, or click Load more inside a function until the list is complete.
Scrape paginated lists 

Puppeteer without hosting Chrome

Send a Puppeteer function with a URL and get its return value back. The browser, the sandbox and the cleanup run on Microlink.
Run Puppeteer remotely 

Metadata from single-page apps

Render React, Vue or Angular apps in a headless browser, wait for the tags, then read the normalized metadata.
Read metadata from single-page apps 

Markdown from JavaScript-rendered pages

Render single-page apps in a real browser, wait for the content, then convert the finished DOM to Markdown.
Convert JavaScript-rendered pages 

Screenshots of JavaScript-rendered pages

Wait for a selector, a lifecycle event or a delay so single-page apps and lazy sections finish rendering before capture.
Capture JavaScript-rendered pages 

Ready to scrape JavaScript apps?

A real browser, the right wait and your rules in one request. Start on the free tier and scrape your first single-page app today.

Scrape a JavaScript page