Ocelot: The story of a torrent tracker

What.CD is a private tracker. Thus, the entire site, staff, and community all revolve around a common piece of software - the tracker backend. Complementing the site frontend, which you're looking at now, the tracker itself handles connections between peers.

With over five million peers, our tracker receives an average of 3,500 hits per second, although after a period of tracker downtime, load can spike up to past 12,000 hits per second. This means that, when your client announces, the tracker has 80 microseconds to search through its database of over 900,000 torrents and 5,000,000 peers, compute a response, and send it back to you. That's a lot of stress on a piece of software!

We anticipated this problem, of course, back before the site even started. That's why we elected to use what was then the fastest private tracker backend in the world - XBTT.

Lauded for its speed, XBTT handled the peers very well for the first few months of the site's existence. We brought on a developer - asm - whose job was to tune it and modify it as needed, and he was able to do that just fine - for a few months. However, asm was reluctant to make any major changes. When we asked why, his response was that XBTT's code was too weird, and that he was afraid he'd break something.

A bit surprised, we lead site developers peered into the bowels of XBTT for the first time, and we found that he was correct. XBTT's internal code worked fine in practice, but strange/outdated design decisions and the inclusion of thousands of lines of unnecessary code gave us worries about how well it would scale to a swarm of the size we had planned, as well as whether we'd be able to continue modifying it to our needs.

So a plan was formed. We would create a tracker of our own.

Late winter 2007

It made perfect sense. We were already replacing the outdated TBDev source with our own new Gazelle source, so why not replace XBTT with another piece of software as well? Make it fast, make the code pretty, give it a cool-sounding exotic animal name, and we'd be set. It couldn't possibly take very long - trackers are very simple pieces of software, after all. The only problem was that XBTT had scared asm into hiding, the other developers were all php developers (php is a language that is fast to write and slow to run) and we wanted the tracker coded in C++ (slow to write, fast to run). The solution was thus to outsource.

January 2008

Our first developer choice was a young developer called rootkit. Immensely intelligent, but perhaps not the greatest people person in the world, rootkit decided that he wanted to write the tracker in haskell instead. We weren't too excited to have the tracker written in a weird language that no one understood, but he promised that it'd be fast so we let him go at it. We don't think he ever wrote more than a hundred lines of it before he gave up and stepped down.

While we searched for a new developer, WhatMan decided to try an experiment - to see if a php tracker could outperform XBTT. He hacked away for a weekend and created Lioness - a beautiful little tracker, no doubt one of the fastest php trackers ever made. Unfortunately, it wasn't quite fast enough for our needs - upon testing, the swarm crushed our poor webserver, and we were forced to go back to XBTT.

By this time, XBTT was barely able to keep up with the load. The timeouts had already started, and we did whatever we could, but in the end, the only thing that really helped was when we moved to our new (then) ridiculously oversized server in Canada.

March - May 2008

Another developer had been found! The guy was smart, mature, well educated, fluent in C++, and seemingly very able. We told him what we needed, and he started coding. A month later, the new dev - lenrek - had created the first tracker to call itself Ocelot.

lenrek's ocelot looked promising. It was new, shiny, and multithre...