In this guest post, Lewis Van Winkle talks about F5Bot, a free service that emails you when selected keywords are mentioned on Reddit, Hacker News, or Lobsters. He explains in detail how F5Bot is able to process millions of comments and posts from Reddit every day on a single VPS. You can check out more of Lewis Van Winkle’s writing at codeplea.com, and his open source contributions at github.com/codeplea.

I run a free service, F5Bot. The basic premise is that you enter a few keywords to monitor, and it’ll email you whenever those keywords show up on Reddit or Hacker News. I’m hoping to add a few more sites soon.

This post is going to be about how I built the service. Specifically, we’re going to focus on scraping every post and comment from Reddit in real time.

F5Bot is written in PHP. It’s programmed in a very boring, straight-forward manner. I’m not doing anything special, really, but it does manage to scan every Reddit post and comment in real-time. And it does it all on a tiny low-end VPS. Sometimes people are surprised when I tell them this. “Reddit has huge volume,” they say, “Reddit runs a room-full of super computers, and they still have downtime like every day! How do you it?” Well, I’m going to tell you.

Reddit gets about 500,000 new posts and 4,000,000 new comments every day. Combined, that’s about 50/second. The posts don’t come in at a constant rate, though, so we’d better be able to handle a few hundred a second. You may think PHP is slow, but it can definitely handle that. I mean how long does it take you to search a few hundred files on your computer? So really, the only key thing here is that we need to write the program in a careful way. We can’t waste resources, but if we’re careful, we’ll be able to handle the traffic.

So let’s get started!

Reddit’s Listing API

Reddit offers a nice JSON API. There’s some good documentation here. I’m pretty sure their site uses it internally to render the HTML pages for visitors. So we’ll just use that to grab the most recent posts. It’s at https://www.reddit.com/r/all/new/.json, and I recommend you open that up in your browser to follow along.

In PHP we’ll just grab it with a simple curl and then decode it with json_decode(). (As a side note, I’m going to completely skip over error handling because it’s so boring. I mean it’s the most important part of running an actual real-live server, but it’s boring so I won’t talk about it anymore here.)

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.reddit.com/r/all/new/.json");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'testbot2000'); /* Don't forget to name yourself! */
$posts = curl_exec($ch);
curl_close($ch);
$posts = json_decode($posts, true);
print_r($posts);

Here’s an example showing the basic structure of the JSON you’ll get back. I have trimmed it down by leaving only 2 of the 25 posts and by removing most of the key/value pairs (there’s a lot of them).

{
  "kind": "Listing",
  "data": {
    "modhash": "5b1dwrfbn4a9f8c200824114ecc7110ee2baa4d95dc2c106b9",
    "dist": 25,
    "children": [
      {
        "kind": "t3",
        "data": {
          "approved_at_utc": null,
          "subreddit": "Guitar",
          "selftext": "I practice my guitar every day.   Most every day, I ...",
          "user_reports": [],
          "saved": false,
          "mod_reason_title": null,
          "gilded": 0,
          "clicked": false,
          "title": "[DISCUSSION] Thanks to everyone here",
          "subreddit_name_prefixed": "r/Guitar",
          "hidden": false,
          "id": "91sed5",
          /* Many more key/value pairs. */
        }
      },
      {
        "kind": "t3",
        "data": {
          "approved_at_utc": null,
          "subreddit": "gpdwin",
          "selftext": "What would be the cheapest way to purchase a unit in ...",
          "user_reports": [],
          "saved": false,
          "mod_reason_title": null,
          "gilded": 0,
          "clicked": false,
          "title": "Looking for advice on buying a WIN 2 in the UK",
          "subreddit_name_prefixed": "r/gpdwin",
          "hidden": false,
          "id": "91secm",
          /* Many more key/value pairs. */
        },
        /* Many more objects. */
      }
    ],
    "after": "t3_91secm",
    "before": null
  }
}

Depending on how you have PHP setup, curl may have trouble with the secure HTTPS connection. If that’s the case you can bypass it with curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false).

Now we have all the recent posts in $posts as a nice clean PHP array. This is going to be a short article, we’re pretty much done.

But, wait, how many posts did we get? I only want to my scraper to run every few minutes, so it’d better be a lot.

print("Downloaded " . count($posts['data']['children']) . " posts.\n");

Downloaded 25 posts.

Well, that’s a bit tedious. I only want to run my scraper every few minutes, not constantly. So we’d better get at least a minute worth of posts. That’s 300 posts, let’s grab 1,000 just to be safe. We can just add a ?limit=1000 to the end of the URL.

...
curl_setopt($ch, CURLOPT_URL, "https://www.reddit.com/r/all/new/.json?limit=1000");
...
print("Downloaded " . count($posts['data']['children']) . " posts.\n");

Downloaded 100 posts.

Turns out that Reddit has a limit. It’ll only show you 100 posts at a time.

There’s an easy solution though. If you look closely at the JSON you’ll see that it has an after field with an ID. The after field tells us which post id came before the oldest post here. So we’re saved, really. We can just call the API again, but ask it for the 100 posts starting at that post id!

$after = $posts['data']['after'];
...
curl_setopt($ch, CURLOPT_URL, "https://api.reddit.com/api/info.json?limit=100&id=$after");
...

This works well up to a point. That point is about 1,000 posts. After that it will either loop back to the beginning and start showing you posts you’ve already seen, or it will just stop returning anything. I’m not a big fan of that, because if my scraper has a little down time I’d like it to go back and grab the old posts too, but maybe you’re not worried about that.

Now everything I’ve said here about posts is also true of comments. The only difference is that the new ones are at https://www.reddit.com/r/all/comments/.json instead of https://www.reddit.com/r/all/new/.json, and they come in much quicker.

But we’ve got an even bigger problem. Each request takes a few seconds (or much longer if Reddit servers are loaded), and that means we can’t pull comments as quickly as they’re coming in. Remember, comments get posted at a rate of 50 per second, and it can be much more during peak traffic.

Making Multiple Simultaneous Connections

So we’ll have to make a bunch of connections all at once. PHP makes this easy enough with its curl_multi functions, but we’re getting ahead of ourselves. If we’re just pulling the listings, how do we know the id to start the next request on if the current request hasn’t finished?

So each Reddit thing, like a post or comment, has a unique ID. Post IDs start with t3_ and comment IDs start with t1_. The API is very inconsistent about whether it uses the prefix or not. Some places of the API require the prefix, but some just use the ID with no prefix.

The IDs themselves are encoded in base 36 - therefore they use the ten digits, 0-9, and the twenty six letters, a-z. In PHP we can use base_convert() to convert them to decimal and do math with them.

So here we get the most recent post ID, and convert it to decimal.

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.reddit.com/r/all/new/.json?limit=1");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$posts = curl_exec($ch);
curl_close($ch);

$posts = json_decode($posts, true);
$id = $posts['data']['children'][0]['data']['id'];
$id = base_convert($id,36,10);

From that you can easily do math on $id and then convert it back to Reddit’s format with: base_convert($id,10,36). It’s pretty convenient.

The best part is all the bandwidth you’re going to save by using base 36 instead of 10! You’re going to need it too, since almost the entire Reddit API consists of crap you’ll almost certainly never need. I mean do you really want subreddit name and subreddit_name_prefixed? They’re the same, one just has an “r/” in front of it. In fact, take a look at that JSON again: https://www.reddit.com/r/all/new/.json. Basically I use the selftext, subreddit, permalink, url and title. The other 95% of it is just wasted bandwidth. Actually, over half of the reply is JSON key text. So even if you’re going to use every value, you’re spending more bandwidth just on JSON key names than on the actual data. Oh well, I guess it’s still better than XML.

Anyway, now we can manipulate Reddit IDs, and it turns out that both comments and posts are assigned IDs more or less serially. They’re not strictly monotonic in the short term, but they are in the long term. (i.e. a post from an hour ago will have a lower ID than a post right now, but a post from 2 seconds ago may not)

So we can find where to start each API call by simply subtracting 100 from a starting ID. Using this method we can download a bunch of posts all at once. Here’s the latest 1,000 posts, downloaded simultaneously. It assumes you’ve already loaded the ID of the latest post into $id (in decimal).

$mh = curl_multi_init();
$curls = array();

for ($i = 0; $i < 10; ++$i) {
  $ch = curl_init();
  $url = "https://www.reddit.com/r/all/new/.json?limit=100&after=t3_" . base_convert($id,10,36);
  print("$url\n");
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

  curl_multi_add_handle($mh, $ch);
  $curls[] = $ch;
  $id -= 100;
}

$running = null;
do {
  curl_multi_exec($mh, $running);
} while ($running);

echo "Finished downloading\n";

foreach ($curls as $ch) {
  $response_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
  $response_time = curl_getinfo($ch, CURLINFO_TOTAL_TIME);
  $posts = curl_multi_getcontent($ch);
  $posts = json_decode($posts, true);

  print("Response: $response_code, Time: $response_time, Post count: " .
    count($posts['data']['children']) . ", Starting id: " .
    $posts['data']['children'][0]['data']['id'] . "\n");

  curl_multi_remove_handle($mh, $ch);
}

curl_multi_close($mh);

If you try to push that back past about 1,000 posts you’ll just get empty replies. Assuming that you’re ok with the 1,000 post limit, we’re done. We’ve got the API totally figured out, scraping is a solved problem, and you’re ready to finish building your bot.

Except… if you look at the post IDs you’re getting, you’ll find some funny business. Some posts are repeated. That’s not a big deal, we can ignore them, but also some posts are missing. That’s a big deal! We aren’t slurping all of Reddit if we’re missing posts.

The missing posts are caused by the IDs not being totally in order. Or maybe it’s just some other bug. I don’t know. In any case, there’s nothing we can do about it here that I know of. We’ll need another solution. The 1,000 post limit was really annoying anyway. And the 1,000 comment limit meant we were going to need to scrape every 10 seconds just to avoid missing anything. That wasn’t really viable long-term.

A Completely New Approach

Reddit’s listing APIs are terrible for trying to see every post and comment. To be fair, they probably weren’t designed for that. I guess they are just designed to work with their website to render pages. No real human would want to keeping clicking past 1,000 posts.

So here’s the approach I ended up using, which worked much better: request each post by its ID. That’s right, instead of asking for posts in batches of 100, we’re going to need to ask for each post individually by its post ID. We’ll do the same for comments.

This has a few huge advantages. First, and most important, we won’t miss any posts. Second, we won’t get any post twice. Third, we can go back in time as far as we want. This means we can run our scraper in huge batches every few minutes. If there is downtime, we can go back and get the old posts that we missed. It’s perfect.

Here’s how we do it. We find the starting post ID, and then we get posts individually from https://api.reddit.com/api/info.json?id=. We’ll need to add a big list of post IDs to the end of that API URL.

So here are 2,000 posts, spread out over 20 batches of 100 that we download simultaneously. It assumes you’ve already got the last post ID loaded into $id base-36.

print("Starting id: $id\n");

$urls = array();
for ($i = 0; $i < 20; ++$i) {
  $ids = array();
  for ($j = 0; $j < 100; ++$j) {
    $ids[] = "t3_" . $id;
    $id = base_convert((base_convert($id,36,10) - 1), 10, 36);
  }
  $urls[] = "https://api.reddit.com/api/info.json?id=" . implode(',', $ids);
}



$mh = curl_multi_init();
$curls = array();

foreach ($urls as $url) {
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch, CURLOPT_USERAGENT, "This is actually needed!");
  print("$url\n");
  curl_multi_add_handle($mh, $ch);
  $curls[] = $ch;
}

$running = null;
do {
  curl_multi_exec($mh, $running);
} while ($running);

print("Finished downloading\n");

foreach ($curls as $ch) {
  $response_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
  $response_time = curl_getinfo($ch, CURLINFO_TOTAL_TIME);
  $posts = json_decode(curl_multi_getcontent($ch), true);

  print("Response: $response_code, Time: $response_time, Post count: " .
    count($posts['data']['children']) . ", Starting id: " .
    $posts['data']['children'][0]['data']['id'] . "\n");

  curl_multi_remove_handle($mh, $ch);
}
curl_multi_close($mh);

We’ll end up with some pretty long URLs, such as this, but Reddit doesn’t seem to mind.

Also please note that we actually need to set the user-agent for the “info.json” API point. Apparently Reddit has blocked PHP/Curl from it. It’s probably a good idea to set the user-agent for everything; I’ve only omitted it above to shorten the code samples. See Reddit’s API access rules for more info.

From the above script, you’ll get an output like this:

Finished downloading
Response: 200, Time: 5.735, Post count: 96, Starting id: 90yv0w
Response: 200, Time: 7.36, Post count: 99, Starting id: 90yuy4
Response: 200, Time: 6.532, Post count: 100, Starting id: 90yuvc
Response: 200, Time: 6.547, Post count: 99, Starting id: 90yusk
Response: 200, Time: 7.344, Post count: 99, Starting id: 90yups
Response: 200, Time: 7.25, Post count: 94, Starting id: 90yun0
Response: 200, Time: 6.672, Post count: 83, Starting id: 90yuk8
Response: 200, Time: 7.469, Post count: 97, Starting id: 90yuhg
Response: 200, Time: 6.344, Post count: 91, Starting id: 90yueo
Response: 200, Time: 7.187, Post count: 97, Starting id: 90yubw
Response: 200, Time: 22.734, Post count: 96, Starting id: 90yu94
Response: 200, Time: 7.453, Post count: 93, Starting id: 90yu6c
Response: 200, Time: 7.359, Post count: 99, Starting id: 90yu3k
Response: 200, Time: 7.812, Post count: 100, Starting id: 90yu0s
Response: 200, Time: 7.703, Post count: 100, Starting id: 90yty0
Response: 200, Time: 6.375, Post count: 99, Starting id: 90ytv8
Response: 200, Time: 6.734, Post count: 97, Starting id: 90ytsg
Response: 200, Time: 7.328, Post count: 97, Starting id: 90ytpo
Response: 200, Time: 7.812, Post count: 98, Starting id: 90ytmw
Response: 200, Time: 7.812, Post count: 98, Starting id: 90ytk4

So even though we asked for posts in batches of 100, many batches are short. There are two reasons for this. First, some posts are going to be in private communities you don’t have access too. You won’t be able to see them, there’s nothing you can do about that. Second, because post IDs aren’t assigned in perfect order, it could be that some of the missing posts haven’t been written yet. In that case, we just wait a few seconds and do another request with the missing IDs. Not a big deal.

By the way, the above code also works perfectly for comments too. The only difference is that comment IDs start with t1_ instead of t3_.

The other issue you may run into when loading comments, is that the API doesn’t return the title of the post that the comment is attached to. If you need this (F5Bot does) you’ll have to use the parent_id field to get the parent post’s ID. Then you’ll need a separate call to load up that post just so you can grab its title. Luckily you can batch a bunch of these calls together into one request.

Working the Data

After I got it figured out, scraping all of Reddit in real-time wasn’t that tough. I launched F5Bot and it worked fine for a long time. Eventually, however, I ran into a second problem. Processing the data became the bottle-neck. Remember, I’m doing this on a tiny VPS. It has a fast connection, but an anemic CPU.

F5Bot has to search every post and comment for all of the keywords that all of my users have. So it started out as something like this:

foreach ($new_posts as $post) {
    foreach ($all_keywords as $keyword) {
        if (strpos($post, $keyword) !== FALSE) {
            ... found a relevant post ...
        }
    }
}

As you can imagine, I get more users, more keywords, and eventually I’m searching each post for thousands of keywords. It gets a bit slow.

Eventually I converted over to using the Aho-Corasick string searching algorithm. It’s really slick. You put your keywords into a tree structure as a pre-processing step. Then you only need to look at each post one time to see which keywords it contains.

I couldn’t find an Aho-Corasick implementation in PHP, so I wrote my own. It’s up on Github here.

Conclusion

Scraping all of Reddit in real-time doesn’t require a lot of processing power. You just have to be careful and know how to work around their APIs. If do you implement any of these ideas, please be sure to follow Reddit’s API access rules. It’s cool of them to offer a public API.

I hope you enjoyed the write-up! Special thanks to Intoli and Evan Sangaline for the idea to write this article and for hosting it here.

If you’d like to know when people on Reddit or Hacker News are talking about you, your company, or your product, please give F5Bot a try. It’s free.

If you want to read more of my posts you can check out my blog at https://codeplea.com or follow me on Github.