Scraping API · Use case

Scrape data from pages behind a login without scripting the sign-in

To scrape a website behind a login, the request has to arrive with a session the site already trusts. Account balances, order histories, analytics dashboards, supplier portals and internal tools all sit behind authentication. The Scraping API forwards your cookie or token as a real request header, so the rules run on the page your user sees.

Start with the Scraping API

The problem

An anonymous scraper only ever sees the login form

A fresh browser has no cookies. Request an authenticated URL and the site redirects to its sign-in page, which returns 200 like any other page, so your rules run against the login form and every field comes back null without an error.

Scripting the login looks like the fix and is the fragile path. You store a real password next to the scraper, replay a form whose markup changes, and run into the multi-factor prompts and bot checks that exist precisely to stop automated sign-ins. Each run pays for the whole login flow before extracting anything.

If your application already holds a valid session or API token, forward it. Any request header prefixed with x-api-header- reaches the target with the prefix stripped, so x-api-header-cookie arrives as cookie and x-api-header-authorization as authorization. The private pages guide shows the pattern for data extraction.

How it works

How to scrape authenticated pages with a forwarded session

Headers are a Pro feature, so authenticated requests go to pro.microlink.io with your API key. Keep them on your backend, never in client-side code.

1 · Forward a session cookie

import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const { balance, orders } = await microlink.extract(
  'https://app.example.com/account',
  {
    balance: { selector: '[data-testid=balance]', attr: 'text', type: 'number' },
    orders: { selectorAll: '.order-row .order-id', attr: 'text' }
  },
  {
    headers: {
      'x-api-header-cookie': `session=${process.env.SESSION_COOKIE}`
    },
    cacheKey: `user-${userId}`,
    waitForSelector: '[data-testid=balance]'
  }
)

The SDK sends headers as real HTTP request headers, never in the URL. Forwarded headers are not part of the cache key, so cacheKey keeps each user in a separate cache entry, and waitForSelector on an element that only exists when signed in doubles as a check that the session was accepted.

2 · A bearer token for an authenticated API

import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const { content } = await microlink.extract(
  'https://api.example.com/v1/invoices',
  { content: { attr: 'json' } },
  {
    prerender: false,
    headers: {
      'x-api-header-authorization': `Bearer ${process.env.APP_TOKEN}`
    },
    cacheKey: `tenant-${tenantId}`
  }
)

For endpoints that answer with JSON, attr json parses the whole body and returns it with its original shape. prerender: false skips the browser because there is nothing to render, and cacheKey scopes the cached response to the tenant that owns the token.

3 · The same request with curl

curl -G https://pro.microlink.io \
  -d url=https://app.example.com/account \
  -d 'data.balance.selector=[data-testid=balance]' \
  -d data.balance.attr=text \
  -d meta=false \
  -d cacheKey=user-42 \
  -H "x-api-key: $MICROLINK_API_KEY" \
  -H 'x-api-header-cookie: session=abc123'

x-api-key authenticates you against pro.microlink.io and the x-api-header-cookie value is what the target receives. The rules travel as data query parameters and the secret only as a header.

Parameters used

headers Public request shaping such as a language. Secrets go in x-api-header-* request headers instead. Pro plans.
cacheKey Adds a custom identifier, such as a user id, so each user gets a separate cache entry. Pro plans.
waitForSelector Waits for an element that only renders for a signed-in user.
data The extraction rules, identical to any public page.
force Skips the cache when a private page must be read fresh.

Sending x-api-key to the free endpoint fails with EPRO, and the headers parameter without a Pro plan returns EHEADERS. The SDK options reference explains how the headers option is routed.

Why it works

Why forwarding a session beats automating the login

The session is the smallest credential that makes the page render for a user. Forwarding it keeps the password out of your scraper entirely.

01 · No password

Skip the form, the redirect and the second factor.

Your app already has a session from a real sign-in or an OAuth token from the user. Forward it and the browser lands on the authenticated page directly, with nothing to replay and nothing for MFA to block.

The same header pattern captures the page as an image in screenshots behind a login and as a document in PDF invoices from authenticated pages.

02 · Isolated

The session never outlives the request.

Every call runs in its own browser that is destroyed when the response is sent, so no cookies or storage are shared between calls and one user’s session can never leak into another request.

The request isolation page describes the one-browser-per-call model, and the custom headers feature shows how values ride the HTTP layer.

03 · Separate caches

One cache entry per user with cacheKey.

The cache key is derived from the URL and the query parameters, and forwarded headers are not query parameters. Two users requesting the same account URL can share a copy, so add a cacheKey with the user id to keep their data apart.

When not to: Microlink does not log in for you. If all you have is a username and password, sign in with your own backend first and forward the resulting session. Never forward a session that belongs to someone who has not authorized it.

FAQ

Sign in once with your own application, then send the session cookie as an x-api-header-cookie header on a request to pro.microlink.io. Microlink forwards it to the target as a regular cookie, and your extraction rules run on the authenticated page.

Can I scrape an API that needs a bearer token?

Yes. Send x-api-header-authorization with the value Bearer followed by the token, and use a rule with attr json to parse the response body. Add prerender: false so the endpoint is fetched without a browser.

Why does my authenticated scrape return null for every field?

The rules probably ran on the login page. Check that the cookie name and domain match what the site sets, that the session has not expired, and that the request goes to pro.microlink.io with a valid key. A waitForSelector on a signed-in-only element makes this failure explicit.

No. Query parameters are public and end up in logs, history and shared links. Use the headers parameter only for harmless values such as accept-language, and x-api-header-* request headers for cookies and tokens.

No. Forwarding headers requires a Pro plan and the pro.microlink.io endpoint. The free tier is fine for building and testing your rules on public pages first, then you add the session header on Pro. See pricing.

Related use cases

Solve the next problem with the same API

Screenshots behind a login

Forward a session cookie or a bearer token as a request header and capture pages only your users can see.

Capture pages behind a login

PDF invoices from authenticated pages

Print the invoice page your app already renders: forward the session, hide the chrome, name the file.

Generate PDF invoices

Any website to JSON

Declare the fields you want as CSS selector rules and get typed JSON back, with null for anything the page does not have.

Turn a website into JSON

Cached JSON endpoints

Fetch any JSON endpoint without a browser, keep its shape intact and serve repeat calls from the cache.

Fetch and cache JSON APIs

Custom fields from JavaScript apps

Render React, Vue or Angular apps in a real browser, wait for the element you need, then run your rules.

Scrape JavaScript-rendered pages

Puppeteer without hosting Chrome

Send a Puppeteer function with a URL and get its return value back. The browser, the sandbox and the cleanup run on Microlink.

Run Puppeteer remotely

Ready to scrape authenticated pages?

Forward the session, keep the password out of the scraper. Get a Pro key and extract data from the pages only your users can see.

Scrape behind a login