Product brief #5:
user agent and video parameters
metascraper v3.11 brought Custom Rules to the API
May 30, 2018
This product brief added two API parameters, a configurable User Agent and an opt-in video, and deployed metascraper v3.11 to the Microlink API. That update is what made Custom Rules possible: the ability to tell the API new rules to extract new data.
The API examples highlight the term they document
Microlink API examples was improved,
highlighting the referenced term in each section.
Each section of the Microlink API documentation started highlighting, in its example, the term the section references.
The user agent became an API parameter
The User Agent of the request sent to the target website became configurable as an API parameter.
You rarely need to set it. Some websites return a different output based on the user agent of the incoming request. A common case is a site that serves images in WebP format when it detects Google Chrome.
Video detection became opt-in
The previous product brief introduced the first iteration of video support as one of the API data types.
The
video field is widely used when you consume the response through the SDK. If you only need the rest of the extracted data, such as title or image, detecting the video made response times heavier.The new video parameter moved that decision to you, with a default value of `false`. Link previews rendered with the SDK had no breaking changes. If you call the API directly, pass
video to keep the behavior you had until now.metascraper v3.11 made the extraction engine smaller and faster
metascraper is the extraction engine behind the API, and every improvement to it reaches the API responses. The changes since the last brief:
- Improved URL detection: a WHATWG URL parser, contributed by Stefan Andres Charsley.
- Optimized bootstrap: the configuration file is loaded just once.
- Faster core rules: removing non really necessary steps left the core smaller and faster.
- Rules parameter support: load a specific set of rules out of the box.
- Exported core validators: reuse them to build consistent third-party rule sets.
Custom Rules reached the API
These changes were deployed to the Microlink API, and together they add Custom Rules: you send the API a rule, and it returns the new data field that rule describes.
A dedicated post explains how to write one: a
selector to pick the element, an attr to pick its property, and a type to validate the value.Join the community
All of these improvements or features are community driven: We listen to your feedback and act accordingly.
Whether you are building a product, an indie developer, or just interested in web technologies, come chat with us.