Python

Intoli Smart Proxies

Want to use the smartest web scraping proxies available?

Get started now and find out why Intoli is the best in the business!

Recreating Python's Slice Syntax in JavaScript Using ES6 Proxies

By Evan Sangaline

June 28, 2018

I’ve noticed that JavaScript proxies seem to have been getting an increasing amount of attention recently. They were introduced by ECMAScript 2015 (ES6) several years ago, but they remain one of the less well-known features of the language. That’s a real shame because proxies are pretty awesome. They give you a level of flexibility that simply didn’t exist previously in JavaScript, and have allowed for projects like Remote Browser to become possible.

Continue reading

What's New in Exodus 2.0

By Evan Sangaline

March 8, 2018

Exodus started with one simple goal in mind: to make it as easy as possible for a user to relocate working binaries from one Linux machine to another. For example, say that your laptop has a more recent version of gzip than what’s available though your server’s package manager, but that you really want to use a command-line flag that the older version doesn’t support. exodus gzip | ssh intoli.

Continue reading

JavaScript Injection with Selenium, Puppeteer, and Marionette in Chrome and Firefox

By Evan Sangaline

December 29, 2017

Browser automation frameworks–like Puppeteer, Selenium, Marionette, and Nightmare.js–strive to provide rich APIs for configuring and interacting with web browsers. These generally work quite well, but you’re inevitably going to end up running into API limitations if you do a lot of testing or web scraping. You might find yourself wanting to conceal the fact that you’re using a headless browser, extract image resources from a web page, set the seed for Math.

Continue reading

Building Data Science Pipelines with Luigi and Jupyter Notebooks

By Mattia Ciollaro

November 28, 2017

In this guest post, Mattia Ciollaro writes about how to get started with the Luigi task runner, and highlights his own contribution to Luigi through a special use case geared towards running Jupyter notebooks in your workflows. Mattia holds a PhD in statistics from Carnegie Mellon University and is working on improving American healthcare at Spreemo Health. You can get in touch with him via LinkedIn. Data Science meets plumbing In many data science projects, we often start by developing code to solve specific small tasks.

Continue reading

Dangerous Pickles — Malicious Python Serialization

By Evan Sangaline

October 17, 2017

What’s so dangerous about pickles? Those pickles are very dangerous pickles. I literally can’t begin to tell you how really dangerous they are. You have to trust me on that. It’s important, Ok? – “Explosive Disorder” by Pan Telare Before we get elbow deep in opcodes here, let’s cover a little background. The Python standard library has a module called pickle that is used for serializing and deserializing objects.

Continue reading

A Brief Tour of Grouping and Aggregating in Pandas

By Andre Perunicic

October 13, 2017

If you work with data in Python, chances are that you’ve heard of the pandas data manipulation library. You can think of pandas as a way to programmatically interact with spreadsheets. It works well with huge datasets, unlike its desktop counterparts like Google Sheets and Microsoft Excel, and implements a number of common database operations like merging, pivoting, and grouping. Moreover, being backed by numpy and efficient algorithm implementations makes it fast and easily integrated with other tools in the vast Python data science landscape.

Continue reading

Analyzing One Million robots.txt Files

By Evan Sangaline

September 19, 2017

One Million robots.txt Files The idea for this article actually started as a joke. We do a lot of web scraping here at Intoli and we deal with robots.txt files, overzealous ip bans, and all that jazz on a daily basis. A while back, I was running into some issues with a site that had a robots.txt file which was completely inconsistent with their banning policies, and I suggested that we should do an article on analyzing robots.

Continue reading

Fantasy Football for Hackers

By Evan Sangaline

September 7, 2017

There’s a First Time for Everything Like some 75 million other Americans, I am playing fantasy football this year. Unlike most of them, I know virtually nothing about football. I would estimate that I’ve watched somewhere around five games total in my life, most of them Super Bowls. I don’t know the rules beyond the very basics and I can’t name a single NFL player off the top of my head.

Continue reading

Scraping User-Submitted Reviews from the Steam Store

By Andre Perunicic

August 16, 2017

This article was originally published as a guest post on ScrapingHub’s blog. ScrapingHub is the company that wrote Scrapy, which this article is about, so read on to see why they liked it! Introduction The Steam game store is home to more than ten thousand games and just shy of four million user-submitted reviews. While all kinds of Steam data are available either through official APIs or other bulk-downloadable data dumps, I could not find a way to download the full review dataset.

Continue reading

Making Chrome Headless Undetectable

By Evan Sangaline

August 9, 2017

Detecting Headles Chrome A short article titled Detecting Chrome Headless popped up on Hacker News over the weekend and it has since been making the rounds. Most of the discussion on Hacker News was focused around the author’s somewhat dubious assertion that web scraping is a “malicious task” that belongs in the same category as advertising fraud and hacking websites. That’s always a fun debate to get into, but the thing that I really took issue with about the article was that it implicitly promoted the idea of blocking users based on browser fingerprinting.

Continue reading

Search

Tags

Browse All Tags