How To

Have a great idea for an article?

We're always looking for guest contributors or article suggestions. Shoot us an email at blog@intoli.com, we would love to hear yours!

Building a YouTube MP3 Downloader with Exodus, FFmpeg, and AWS Lambda

Let’s focus on the easy part first: what we’ll be building in this tutorial. The end result will be a browser bookmarklet which can be used to convert YouTube videos to MP3s and download them. The basic interaction flow is that you click on the bookmarklet while on the page for a specific video, a new tab opens and displays a progress bar for the conversion, and then the download starts automatically as soon as it’s ready.

Continue reading

Running FFmpeg on AWS Lambda for 1.9% the cost of AWS Elastic Transcoder

Building a Media Transcoder with Exodus, FFmpeg, and AWS Lambda When delivering media content over the internet, it’s important to keep in mind that factors like network bandwidth, screen resolution, and codec support will vary drastically between different devices and connections. Certain media encodings will be better suited for certain viewers, and transcoding source media to multiple formats is a must in order to ensure that you’re delivering the best possible experience to your users.

Continue reading

Extending CircleCI's API with a Custom Microservice on AWS Lambda

There’s a lot to love about CircleCI. First of all, continuous integration is just awesome in general. You can certainly develop fine software without it, but a good CI configuration can really make your life easier. Beyond that, CircleCI has a generous free tier, provides four free containers per open source project, allows the use of custom Docker images, and is reasonably easy to configure. There’s unfortunately also some stuff not to love about CircleCI.

Continue reading

It is *not* possible to detect and block Chrome headless

A few months back, I wrote a popular article called Making Chrome Headless Undetectable in response to one called Detecting Chrome Headless by Antione Vastel. The one thing that I was really trying to get across in writing that is that blocking site visitors based on browser fingerprinting is an extremely user-hostile practice. There are simply so many variations in browser configurations that you’re inevitably going to end up blocking non-automated access to your website, and–on top of that–you’re really not accomplishing anything in terms of blocking sophisticated web scrapers.

Continue reading

Installing Google Chrome On CentOS, Amazon Linux, or RHEL

The easiest way to install the latest Chrome version on RHEL, CentOS, and Amazon Linux versions 6.X and 7.X. # This installs Chrome on any RHEL/CentOS/Amazon Linux variant. curl https://intoli.com/install-google-chrome.sh | bash A Universal Installation Script for Google Chrome on Amazon Linux and CentOS 6 CentOS, Amazon Linux AMI, and Red Hat Enterprise Linux are three closely related GNU/Linux distributions which are all popular choices for server installations. They offer excellent performance and stability, but package availability can often be lacking.

Continue reading

Making Chrome Headless Undetectable

Detecting Headles Chrome A short article titled Detecting Chrome Headless popped up on Hacker News over the weekend and it has since been making the rounds. Most of the discussion on Hacker News was focused around the author’s somewhat dubious assertion that web scraping is a “malicious task” that belongs in the same category as advertising fraud and hacking websites. That’s always a fun debate to get into, but the thing that I really took issue with about the article was that it implicitly promoted the idea of blocking users based on browser fingerprinting.

Continue reading

Patching a Linux Kernel Module

A Bug on Linux? Why, I never! I’ve been using GNU/Linux for about fifteen years and, I’ve got to admit, it used to be pretty rough around the edges (to put it lightly). A lot can change over fifteen years though; most of the things that were once major problem areas haven’t required a second thought in years. Laptop suspension, WIFI, advanced function keys, sound, and pretty much everything else all typically “just work” these days, and this has been the case for quite a while.

Continue reading

Running Selenium with Headless Firefox

Update: This article is regularly updated in order to accurately reflect improvements in Firefox’s headless browsing capabilities. Note: Check out Running Selenium with Healdess Chrome if you’d rather use Google’s browser. Using Selenium with Headless Firefox (on Windows) Ever since Chrome implemented headless browsing support back in April, the other major browsers started following suit. In particular, Mozilla has since then expanded support for Firefox’s headless mode from Linux to its Windows and macOS builds, and fixed a number of bugs that might have been in the way of early adopters.

Continue reading

Running Selenium with Headless Chrome

UPDATE: This article is updated regularly to reflect the latest information and versions. If you’re looking for instructions then skip ahead to see Setup Instructions. NOTE: Be sure to check out Running Selenium with Headless Chrome in Ruby if you’re interested in using Selenium in Ruby instead of Python. Background It has long been rumored that Google uses a headless variant of Chrome for their web crawls. Over the last two years or so it had started looking more and more like this functionality would eventually make it into the public releases and, as of this week, that has finally happened.

Continue reading

Implementing a Custom Waiting Action in Nightmare JS

Nightmare is a popular browser automation library specifically designed with ease of use in mind. A typical Nightmare script chains together semantically named user actions like goto and click to perform any given task, resulting in simple and readable code. These actions of course include a few methods for waiting on the page to fully load: you can wait for a selector to become available, for all static resources to load, or simply wait for a fixed amount of time.

Continue reading