Skip to content
Scraping API · Use case

Scrape paginated results and the items behind a Load more button

To scrape a Load more button or a paginated website you need every item, not the first twenty. Catalogs, job boards, review pages, directories and search results all split their lists across pages or hide them behind a button. The Scraping API covers both: one request per page for numbered pagination, and a short function for lists that grow on click.

Start with the Scraping API
The problem

The first request only returns the first page of results

Your rules work, the array comes back, and it has exactly as many items as the first page shows. The rest sit behind a page 2 link, a ?page= parameter or a Load more button that fetches the next batch with JavaScript, and none of them are in the HTML you scraped.

Writing a crawler for that means a queue, a loop and a browser you keep alive between pages, plus logic to notice the end of the list. For Load more buttons it means Puppeteer code that clicks, waits for the new items and repeats, running on infrastructure you have to host and scale.

Split the problem by pagination type. Numbered pages have their own URLs, so each one is a normal extraction request and they run in parallel. A Load more button needs clicks inside one browser session, which is what the function parameter is for: your Puppeteer code runs on Microlink’s browser, clicks until the list is complete and returns the items.

Open the live JSON for page 2 of a paginated demo
How it works

How to scrape a paginated website and a Load more button

Use the lightest tool that reaches every item. Page URLs first, a function only when the list grows in place. The browser interaction guide documents the click-and-wait pattern.

1 · Numbered pages, in parallel
import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const rules = {
  quotes: {
    selectorAll: '.quote',
    attr: {
      text: { selector: '.text', attr: 'text' },
      author: { selector: '.author', attr: 'text' }
    }
  }
}

const pages = [1, 2, 3, 4, 5].map(n => `https://quotes.toscrape.com/page/${n}/`)
const results = await Promise.all(pages.map(url => microlink.extract(url, rules)))
const quotes = results.flatMap(result => result.quotes)

Each page is one request with the same rules, and the calls run concurrently with no throttling. An empty quotes array on a page past the end tells you where to stop.

2 · Load more, clicked inside a function
import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const loadAll = async ({ page, clicks }) => {
  for (let i = 0; i < clicks; i++) {
    const button = await page.$('button.load-more')
    if (!button) break
    const count = await page.$$eval('.results li', items => items.length)
    await button.click()
    await page.waitForFunction(
      n => document.querySelectorAll('.results li').length > n,
      {},
      count
    )
  }
  return page.$$eval('.results li', items => items.map(el => el.textContent.trim()))
}

const { isFulfilled, value } = await microlink.function(
  'https://example.com/catalog',
  loadAll,
  { clicks: 10 }
)

The function clicks, waits until more items exist than before, and stops when the button disappears or after the number of clicks you pass. clicks is a custom option, forwarded to the function as a named argument.

3 · Or one click with page preparation
import createClient from 'microlink.io'

const microlink = createClient({
  apiKey: process.env.MICROLINK_API_KEY
})

const { items } = await microlink.extract(
  'https://example.com/catalog',
  { items: { selectorAll: '.results li', attr: 'text' } },
  { click: 'button.show-all', waitForSelector: '.results li' }
)

click runs before the rules without writing a function. It fits a single Show all toggle or an accordion; a button that must be pressed repeatedly needs step 2.

Parameters used
  • function Runs your Puppeteer code on the page. 15 seconds on free, up to 60 seconds on Pro.
  • click Clicks the elements matching a CSS selector before extraction.
  • waitForSelector Waits for the list items before the rules read them.
  • scroll Scrolls one element into view. It triggers lazy content once, it does not scroll forever.
  • selectorAll Returns every item on the page as an array.

Size the function to its time budget: every click waits for a network round trip on the target, so ten clicks on a slow site can exceed the 15 second free limit and return a TimeoutError. The function troubleshooting guide covers timeouts and the other plan-aware errors.

Why it works

Why one request per page beats a long-lived crawler

Independent requests fail independently, cache independently and run in parallel. Keep the browser session only where the site forces you to.

01 · Parallel by default
Pages are requests, not steps in a loop.
When page URLs are predictable, fire them all at once. There is no throttling on the API, each page is cached on its own, and one failed page is one retry rather than a restarted crawl.

Each page uses the same nested rules as scraping tables and repeated lists, so one rule set covers the whole listing.

02 · Real clicks
Load more runs in a real browser.
The function gets the full Puppeteer page after navigation, so clicks, waits and scrolling behave exactly as they do locally. You write the loop and Microlink runs the browser.

The same runtime handles any browser automation you would write locally: run Puppeteer without hosting Chrome covers the runtime and its limits.

03 · Bounded
Every run has a clear ceiling.
The free plan gives a function 15 seconds and 64 MB, Pro up to 60 seconds and 128 MB. Hitting a limit returns isFulfilled false with a named error instead of hanging.

When not to: infinite feeds with thousands of items do not fit in one call. Look for the JSON endpoint the page calls as you scroll and read it page by page as cached JSON instead.

FAQ

How do I scrape a Load more button?

Send a function that clicks the button, waits until more items are in the DOM, repeats until the button is gone, and returns the items. It runs in Microlink’s browser, so there is no Puppeteer to host.

Can I scrape a paginated website in parallel?

Yes. When pages have their own URLs, build the list of URLs and send one extraction request per page at the same time. The API applies no throttling; the free tier allows 25 requests per day in total.

Why does scroll not load every item on an infinite scroll page?

The scroll parameter scrolls a single element into view once. That triggers one batch of lazy content, not an endless feed. Scroll in a loop inside a function, or read the JSON endpoint the page requests as it scrolls.

How many times can a function click Load more before it times out?

As many as fit in the time limit: 15 seconds on the free plan and up to 60 seconds on Pro. Pass a click count as a custom option, and check profiling in the response to see how long each run took.

Does each scraped page count as a separate request?

Yes. Each API call is one request, whether it reads page 1 or runs a function that clicks ten times. Cache hits never count against your quota, so re-reading unchanged pages within the cache lifetime is free. See pricing.
Related use cases

Solve the next problem with the same API

Tables and lists to JSON arrays

Turn table rows, product grids and search results into an array of objects, one nested rule per column.
Scrape tables into JSON 

Puppeteer without hosting Chrome

Send a Puppeteer function with a URL and get its return value back. The browser, the sandbox and the cleanup run on Microlink.
Run Puppeteer remotely 

Custom fields from JavaScript apps

Render React, Vue or Angular apps in a real browser, wait for the element you need, then run your rules.
Scrape JavaScript-rendered pages 

Any website to JSON

Declare the fields you want as CSS selector rules and get typed JSON back, with null for anything the page does not have.
Turn a website into JSON 

Bulk Markdown conversion with caching

Convert thousands of URLs in parallel, cached per URL and refreshed in the background for cheap re-indexing.
Convert URLs in bulk 

PDFs in bulk

Render thousands of documents from URLs in one job: parallel requests, server-side retries and per-document caching.
Generate PDFs in bulk 

Ready to scrape every page of the list?

Parallel requests for numbered pages, a function for Load more. Start on the free tier and get past page one today.

Scrape a paginated list