Visualization

Intoli Smart Proxies

Want to use the smartest web scraping proxies available?

Get started now and find out why Intoli is the best in the business!

Performing Efficient Broad Crawls with the AOPIC Algorithm

This article explains how the Adaptive On-Line Page Importance Computation (AOPIC) algorithm works. AOPIC is useful for performing efficient broad crawls of large slices of the internet. The key idea behind the algorithm is that pages are crawled based on a continuously improving estimate of page importance. This effectively allows the user of the algorithm to allocate the bulk of their limited bandwidth on the most important pages that their crawler encounters.

Continue reading

Fantasy Football for Hackers II — An Interactive Visualization of Average Draft Position vs Season Projections

ADP vs Season Projections In the first part of this series, Fantasy Football for Hackers I, I walked through the process of coming up with my own draft strategy using scraped projections and simulated rosters. A lot of people pointed out that I probably would have done better if I had just looked up the average draft positions and picked the best available players. As one user on /r/fantasyfootball so eloquently put it:

Continue reading