Microlink Proxy:
 How it works

June 7, 2021 ()

All
Microlink Pro
 plans come with a built-in proxy layer that does automatic proxy resolution and rotation based on the input URL.

What's wrong with URLs

The Internet is a wild place where every URL is different. That's an issue especially when you are doing things at scale.
A
metascraper issue
 facing with fetching problems.
When you are surfing the net and visit a website, there are situations you can't handle at all:
  • The are geographic location restrictions (e.g., visiting a site from China).
  • Your IP address is blacklisted (e.g., performing a request from a data center).
  • You have to validate your identity (e.g., filling a CAPTCHA).
You can quickly face these problems when sending enough traffic to popular sites, like Instagram or Bloomberg, which, in turn, makes getting a successful response inconsistent.

Resolving URLs gracefully

Wouldn't it be nice if every time you want to retrieve data from any URL, the Microlink API just returned the data all while handling any of the problems above transparently?
No additional setup needed, just a Microlink Pro plan.
That's exactly what Microlink Proxy layer does. No failures. No maintenance. You don't need to do anything additional. It just works™.
The Microlink Proxy is enabled for all Microlink Pro plans and it will handle all the situations for the
Top 500
 most popular worldwide websites, in a transparent way, by default.
Just for informational purposes, you can check if a specific request was resolved using the proxy layer by checking x-fetch-mode response header.
$ microlink https://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance

 SUCCESS  3.64 kB in 329ms

 cache HIT (19h 20m 54.8s left)
  mode prerender-proxy (8.5s)
   uri https://pro.microlink.io/?url=https://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance&filter=statusCode
    id iad:paw98q36bHEIlAgsFBzy5
If the value there is prefixed by `proxy` then it was handled by the proxy layer.
As you can see, the proxy layer respects other query parameters, such as prerender or ttl.
Also, you can always provide your own proxy.
Last but not least, Microlink Proxy is a cross-feature: it's available for all the Microlink products.

Join the community

All of these improvements or features are community driven: We listen to your feedback and act accordingly.
Whether you are are building a product and you need fancy previews, you’re an indie hacker or simply you like frontend stuff, come chat with us 🙂.