Data Analysis

Have a great idea for an article?

We're always looking for guest contributors or article suggestions. Shoot us an email at blog@intoli.com, we would love to hear yours!

A Brief Tour of Grouping and Aggregating in Pandas

If you work with data in Python, chances are that you’ve heard of the pandas data manipulation library. You can think of pandas as a way to programmatically interact with spreadsheets. It works well with huge datasets, unlike its desktop counterparts like Google Sheets and Microsoft Excel, and implements a number of common database operations like merging, pivoting, and grouping. Moreover, being backed by numpy and efficient algorithm implementations makes it fast and easily integrated with other tools in the vast Python data science landscape.

Continue reading

Analyzing One Million robots.txt Files

One Million robots.txt Files The idea for this article actually started as a joke. We do a lot of web scraping here at Intoli and we deal with robots.txt files, overzealous ip bans, and all that jazz on a daily basis. A while back, I was running into some issues with a site that had a robots.txt file which was completely inconsistent with their banning policies, and I suggested that we should do an article on analyzing robots.

Continue reading

Fantasy Football for Hackers

There’s a First Time for Everything Like some 75 million other Americans, I am playing fantasy football this year. Unlike most of them, I know virtually nothing about football. I would estimate that I’ve watched somewhere around five games total in my life, most of them Super Bowls. I don’t know the rules beyond the very basics and I can’t name a single NFL player off the top of my head.

Continue reading

Finding Pareto Optimal Blogs on Hacker News

Introduction I’ve been doing a lot of technical writing recently and, with that experience, I’ve grown to more deeply appreciate the writing of others. It’s easy to take the effort behind an article for granted when you’ve grown accustomed to there being new high-quality content posted every day on Hacker News and Twitter. The truth is that a really good article can take days or more to put together and it isn’t easy to write even one article that really takes off, let alone a steady stream of them.

Continue reading

Fantasy Football for Hackers II — An Interactive Visualization of Average Draft Position vs Season Projections

ADP vs Season Projections In the first part of this series, Fantasy Football for Hackers I, I walked through the process of coming up with my own draft strategy using scraped projections and simulated rosters. A lot of people pointed out that I probably would have done better if I had just looked up the average draft positions and picked the best available players. As one user on /r/fantasyfootball so eloquently put it:

Continue reading