Turn any URL into context for an LLM or agent
Turning a URL into LLM context takes more than the page text: an agent also needs to know what the page is, where it links, how to contact the company and what it runs on. Research agents, sales enrichment, support bots and browsing tools all rebuild that picture by hand. The Microlink SDK exposes each part as a method, so one URL becomes one structured context object.
Raw HTML is expensive and incomplete website context for an LLM
Feeding HTML to a model wastes most of the tokens on markup and still misses what is not in the visible text: the canonical title, the author, the outbound links, the email in a footer, the framework the site runs on. The model pays for the noise and guesses at the rest.
The do-it-yourself version is a fetch, an HTML parser, a readability pass, a link extractor, a regular expression for emails and a metadata library, each with its own failure mode. It breaks on client-rendered pages, and every new question the agent asks about a page means another parser.
Each Microlink SDK method returns one clean facet of the page, so you extract links and emails with the same API that converts the body. markdown() gives the body, links() every absolute URL, emails() the addresses, metadata() the normalized fields and technologies() the stack. They all take the same URL and the same shared options, so you combine them and the agent gets a page it can reason about.
How to build LLM context from a URL with the SDK
Every method takes the same URL and the same shared options, so the calls run in parallel and each one is cached on its own.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const url = 'https://example.com'
const [markdown, links, emails, meta, technologies] = await Promise.all([
microlink.markdown(url, { selector: 'main' }),
microlink.links(url),
microlink.emails(url),
microlink.metadata(url),
microlink.technologies(url)
])Five methods, one URL. markdown() resolves to a string, links() and emails() to arrays of strings, metadata() to the normalized fields and technologies() to an array of detections with a confidence score.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const context = {
title: meta.title,
description: meta.description,
author: meta.author,
published: meta.date,
content: markdown,
links: links.slice(0, 50),
contacts: emails,
stack: technologies
.filter(({ confidence }) => confidence === 100)
.map(({ name }) => name)
}Trim the links and keep only the confident technology detections so the context stays small and reliable. Fields the page does not expose come back as null, so the object always has the same keys.
import createClient from 'microlink.io'
const microlink = createClient({
apiKey: process.env.MICROLINK_API_KEY
})
const readPage = {
name: 'read_page',
description: 'Fetch a URL as Markdown with its links and metadata',
execute: async ({ url }) => {
const [content, meta] = await Promise.all([
microlink.markdown(url, { selector: 'main' }),
microlink.metadata(url)
])
return { title: meta.title, content }
}
}Wrap the calls as a tool so the model decides when to read a page. If your agent speaks the Model Context Protocol, the Microlink MCP server exposes Markdown, metadata and scraping as ready-made tools.
- markdown The page as clean Markdown, scoped with selector or selectorAll.
- links Every link as an absolute, deduplicated URL. mailto and javascript hrefs are dropped.
- emails Email addresses from mailto links and plain text, as bare strings.
- metadata title, description, lang, author, publisher, date, image, logo and url, normalized across sites.
- technologies The tech stack with name, categories and a confidence score from 0 to 100 per detection.
- proxy Add it to any method when a site blocks automated access. Pro plans.
Obfuscated addresses such as name [at] domain are not detected by emails(). Run your own logic against the rendered page with the function method when you need them.
Why separate facets make better LLM context than one big scrape
A page is several kinds of information. Asking for each one explicitly is cheaper and more accurate than parsing everything and hoping the model sorts it out.
The Markdown API page puts the saving at up to 80% fewer tokens than raw HTML for the body alone.
Every method accepts the shared options, so a proxy, a wait or a cache TTL applies uniformly. When a site rejects automated traffic, convert the blocked page through the proxy.
When not to: if the model only needs a summary of the text, markdown() alone is enough, ideally scoped to the content. Add facets when the task asks for links, contacts or the stack.
FAQ
How do I turn a URL into Markdown context for an LLM?
Do the Markdown, links and metadata facets count as one request or several?
Can I get Markdown and metadata for an agent in a single request?
How do AI agents get Markdown from pages that block bots?
Is there a ready-made Markdown integration for AI agents?
Solve the next problem with the same API
Clean Markdown, no boilerplate
Markdown with metadata frontmatter
Markdown from bot-protected pages
YouTube transcripts as Markdown
PDF and office documents to Markdown
Custom fields alongside metadata
Ready to give your agent the whole page?
Markdown, links, emails, metadata and stack from one URL and one client. Start on the free tier and build your first read_page tool today.