Javascript

Intoli Smart Proxies

Want to use the smartest web scraping proxies available?

Get started now and find out why Intoli is the best in the business!

Performing Efficient Broad Crawls with the AOPIC Algorithm

September 16, 2018

This article explains how the Adaptive On-Line Page Importance Computation (AOPIC) algorithm works. AOPIC is useful for performing efficient broad crawls of large slices of the internet. The key idea behind the algorithm is that pages are crawled based on a continuously improving estimate of page importance. This effectively allows the user of the algorithm to allocate the bulk of their limited bandwidth on the most important pages that their crawler encounters.

User-Agents — Generating random user agents using Google Analytics and CircleCI

By Evan Sangaline

August 30, 2018

If you’re in a hurry, you can head straight to the user-agents repository for installation and usage instructions! While web scraping, it’s usually a good idea to create traffic patterns consistent with those that a human user would produce. This of course means being respectful and rate-limiting requests, but it often also means concealing the fact that the requests have been automated. Doing so helps avoid getting blocked by overzealous DDOS protection services, and allows you to successfully scrape the data that you’re interested in while keeping site operators happy.

No API Is the Best API — The elegant power of Power Assert

By Evan Sangaline

July 24, 2018

One of the core ideas behind Facebook’s React library is that there should be no need to learn a new API for things that you already know how to do in vanilla JavaScript. Why bother memorizing Angular’s ng-repeat syntax when you can just use good-old Array.map()? That’s a good idea, and it’s a big part of what made the project so appealing to developers in the first place. So then why should Jest–the same company’s JavaScript testing framework, and a popular choice among React developers–encourage you to learn a new assertion API, and to write code like

Recreating Python's Slice Syntax in JavaScript Using ES6 Proxies

By Evan Sangaline

June 28, 2018

I’ve noticed that JavaScript proxies seem to have been getting an increasing amount of attention recently. They were introduced by ECMAScript 2015 (ES6) several years ago, but they remain one of the less well-known features of the language. That’s a real shame because proxies are pretty awesome. They give you a level of flexibility that simply didn’t exist previously in JavaScript, and have allowed for projects like Remote Browser to become possible.

Building a YouTube MP3 Downloader with Exodus, FFmpeg, and AWS Lambda

By Evan Sangaline

May 21, 2018

Let’s focus on the easy part first: what we’ll be building in this tutorial. The end result will be a browser bookmarklet which can be used to convert YouTube videos to MP3s and download them. The basic interaction flow is that you click on the bookmarklet while on the page for a specific video, a new tab opens and displays a progress bar for the conversion, and then the download starts automatically as soon as it’s ready.

Running FFmpeg on AWS Lambda for 1.9% the cost of AWS Elastic Transcoder

By Evan Sangaline

May 2, 2018

Building a Media Transcoder with Exodus, FFmpeg, and AWS Lambda When delivering media content over the internet, it’s important to keep in mind that factors like network bandwidth, screen resolution, and codec support will vary drastically between different devices and connections. Certain media encodings will be better suited for certain viewers, and transcoding source media to multiple formats is a must in order to ensure that you’re delivering the best possible experience to your users.

Using Ant Design in Sass-Styled Projects

By Andre Perunicic

April 24, 2018

I recently started using Ant Design as my go-to React component library over ported frameworks like React-Bootstrap or React Material UI. There’s a lot to love about Ant Design: it follows a collection of well thought out design principles, has a comprehensive component library, and can easily be customized through a simple theming system. It also uses Less as its styling language, which is unfortunate if you want to transition an existing Sass-based project to Ant Design, or if you simply prefer using Sass to style your components.

Extending CircleCI's API with a Custom Microservice on AWS Lambda

By Evan Sangaline

February 20, 2018

There’s a lot to love about CircleCI. First of all, continuous integration is just awesome in general. You can certainly develop fine software without it, but a good CI configuration can really make your life easier. Beyond that, CircleCI has a generous free tier, provides four free containers per open source project, allows the use of custom Docker images, and is reasonably easy to configure. There’s unfortunately also some stuff not to love about CircleCI.

It is not possible to detect and block Chrome headless

By Evan Sangaline

January 18, 2018

A few months back, I wrote a popular article called Making Chrome Headless Undetectable in response to one called Detecting Chrome Headless by Antione Vastel. The one thing that I was really trying to get across in writing that is that blocking site visitors based on browser fingerprinting is an extremely user-hostile practice. There are simply so many variations in browser configurations that you’re inevitably going to end up blocking non-automated access to your website, and–on top of that–you’re really not accomplishing anything in terms of blocking sophisticated web scrapers.

JavaScript Injection with Selenium, Puppeteer, and Marionette in Chrome and Firefox

By Evan Sangaline

December 29, 2017

Browser automation frameworks–like Puppeteer, Selenium, Marionette, and Nightmare.js–strive to provide rich APIs for configuring and interacting with web browsers. These generally work quite well, but you’re inevitably going to end up running into API limitations if you do a lot of testing or web scraping. You might find yourself wanting to conceal the fact that you’re using a headless browser, extract image resources from a web page, set the seed for Math.

Our residential proxies do what other proxies do, and a whole lot more. We intelligently route your requests through residential proxies to outsmart bot-mitigation strategies, and automatically retry any requests that fail. We also support optionally rendering requests in remote headless browsers, so that you don’t have to deal with the overhead of managing thousands of browser instances yourself.

Intoli Smart Proxies

Performing Efficient Broad Crawls with the AOPIC Algorithm

User-Agents — Generating random user agents using Google Analytics and CircleCI

No API Is the Best API — The elegant power of Power Assert

Recreating Python's Slice Syntax in JavaScript Using ES6 Proxies

Building a YouTube MP3 Downloader with Exodus, FFmpeg, and AWS Lambda

Running FFmpeg on AWS Lambda for 1.9% the cost of AWS Elastic Transcoder

Using Ant Design in Sass-Styled Projects

Extending CircleCI's API with a Custom Microservice on AWS Lambda

It is not possible to detect and block Chrome headless

JavaScript Injection with Selenium, Puppeteer, and Marionette in Chrome and Firefox

Search

Tags

Intoli Smart Proxies