Performing Efficient Broad Crawls with the AOPIC Algorithm
Learn how to estimate page importance and allocate bandwidth during a broad crawl.
User-Agents — Generating random user agents using Google Analytics and CircleCI
A free dataset and JavaScript library for generating random user agents that are always current.
How F5Bot Slurps All of Reddit
The creator of F5Bot explains in detail how it works, and how it’s able to scrape million of Reddit comments per day.
A Slack Community for Developers to Discuss Web Scraping
Intoli is launching a new Slack community called Web Scrapers where developers can chat about web scraping.
Analyzing One Million robots.txt Files
Insights gathered from analyzing the robots.txt files of Alexa’s top one million domains.
Scraping User-Submitted Reviews from the Steam Store
An introduction to Scrapy though a realistic project.
Finding Pareto Optimal Blogs on Hacker News
An analytical approach to finding the best blogs out there.
Scraping and Parsing Sitemaps in Bash
A guide to using bash and common command-line utilities for quickly parsing sitemaps without specialized tools.