Intoli Blog

Have a great idea for an article?

We're always looking for guest contributors or article suggestions. Shoot us an email at blog@intoli.com, we would love to hear yours!

Using Puppeteer to Scrape Websites with Infinite Scrolling

Infinite scrolling has become a ubiquitous design pattern on the web. Social media sites like Facebook, Twitter, and Instagram all feature infinitely scrolling feeds to keep users engaged with an essentially unbounded amount of content. Here’s what that looks like on Instagram, for example. This mechanism is typically implemented by using JavaScript to detect when the user has scrolled far enough down the existing feed, and then querying an underlying API endpoint for the next batch of data that gets processed and dynamically injected into the page.

Continue reading

Implementing a Custom Waiting Action in Nightmare JS

Nightmare is a popular browser automation library specifically designed with ease of use in mind. A typical Nightmare script chains together semantically named user actions like goto and click to perform any given task, resulting in simple and readable code. These actions of course include a few methods for waiting on the page to fully load: you can wait for a selector to become available, for all static resources to load, or simply wait for a fixed amount of time.

Continue reading

Using Firefox WebExtensions with Selenium

The WebExtensions API In 2015, Mozilla announced that they would be deprecating XPCOM and XUL based addons in favor of their new WebExtensions API based on the Google Chrome Extension API. There were some vocal critics of this shift because it meant that some existing add-ons would be discontinued, but this was tremendously positive news for add-on and extension developers. Writing cross-browser extensions had previously been an absolutely miserable experience, and many developers understandably chose to only target Chrome due to its market share and relatively pleasant API.

Continue reading

Using Google Chrome Extensions with Selenium

Running Google Chrome with an extension installed is quite simple because Chrome supports a --load-extension command-line argument for exactly this purpose. This can be specified before launching Chrome with Selenium by creating a ChromeOptions instance and calling add_argument(). from selenium import webdriver from selenium.common.exceptions import NoSuchElementException # Configure the necessary command-line option. options = webdriver.ChromeOptions() options.add_argument('--load-extension=path/to/the/extension') # Initalize the driver with the appropriate options. driver = webdriver.Chrome(chrome_options=options) The above code will setup a Selenium driver for Chrome with the extension located at path/to/extension preinstalled.

Continue reading

Terminal Recorders: A Comprehensive Guide

Recording a terminal session and converting into a nice animated GIF to embed on a website. Sounds pretty simple, right? I’ve occasionally wanted to embed terminal recorders into blog posts, but it wasn’t until recently that I decided to actually look into some of the tools available to do it. It turns out that there are a lot of them. That alone isn’t necessarily a bad thing, but it also unfortunately turns out that most of them have some pretty serious issues.

Continue reading

Fantasy Football for Hackers II — An Interactive Visualization of Average Draft Position vs Season Projections

ADP vs Season Projections In the first part of this series, Fantasy Football for Hackers I, I walked through the process of coming up with my own draft strategy using scraped projections and simulated rosters. A lot of people pointed out that I probably would have done better if I had just looked up the average draft positions and picked the best available players. As one user on /r/fantasyfootball so eloquently put it:

Continue reading

Running Selenium with Headless Chrome in Ruby

NOTE: Be sure to check out Running Selenium with Headless Chrome if you’re interested in using Selenium in Python instead of Ruby. Since Google added support to run Chrome and Chromium in headless mode as of version 59, it has become a popular choice for both testing and web scraping. There are a few Chrome-specific automation solutions out there, such as Puppeteer and Chrome Remote Interface, but Selenium remains a popular choice due to it’s uniform API across web browsers and it’s support for multiple programming languages.

Continue reading

Closing the Tech Industry’s Gender Gap

In this guest post, Jackie Young shares her thoughts about closing the tech gender gap. After a career that started in software development and IT teaching, Jackie Young now focuses on writing about tech, software and innovations for a range of publications. When she’s not writing, she’s a mom to two girls, one of whom is on the autism spectrum. Jackie also has autism and in her free time supports a number of organizations and charities to help other people with the condition.

Continue reading