It's cool that he got so much press in such a short time, but applauding technology choices that allowed him to sustain 8 requests per second (assuming this was over a 12 hour period) is ridiculous.
The PHP script is super simple, so lighttpd (which is a great web server, in the same performance realm of nginx) and php-fpm on a shared VPS, is plenty of power and resources.
Never seen $found = `grep $udid FILE` using back-ticks in PHP land. Is that the same as the exec() function?
I've seen this used in a PHP git class to call git and capture the results. (Obviously) Just make sure you're not including any user input data in any variables you put in there.
It's also grossly insecure unless you sanitize $udid, which thankfully the author of this script does.
It's still relatively unsafe, as it invokes external programs through a shell, so you're dependent on the shell's environment, which can diverge on different systems. Last I checked, PHP did not have an easy, obvious way to safely invoke another program. You can do it manually with a mix of pcntl_fork and pcntl_exec, but capturing the output using that is more difficult.
Escaping is difficult to get right, especially in PHP... witness things like magic quotes that existed for so long.
The right thing to do is provide an output capturing API that does not require escaping because it takes a list of strings passed to the exec(2) system call.
All those blogs die because people don't bother to use WP Super Cache or the equivalent.
Just for comparison, in 1998/99 I ran an mp3 search engine that handled 200k daily queries at the peak. It was a Pentium with 128MB of memory. The search server was written in C, and it used an inverted index. The system load rarely went over 1.
You missed the real catch that he mentioned towards the end - Quoting from the article -
"The lesson is really simple. Make it work, put it out. You can make it pretty later, maybe."
My http://pastehtml.com/udid got 130k hits yesterday. PasteHTML.com itself and this UDID tool are running on a moderate dedicated server, nginx/rails/postgresql.
Security-wise, the traffic to his server is http, meaning any man-in-the-middle can collect tuples of UDIDs, IP and browser fingerprints, even if he doesn't collect anything.
If you worry about UDIDs being tracked I guess you should worry about the http to his page too.
However, according to somebody who apparently tested it a year ago (and posted it on HN), half of the iPhone apps pass UDID over the same http, allowing collection of the very same or similar data.
Which makes the whole subject of a government agency having UDID database without other convenient data, ummm, quite less probable. IMHO most of the "business entities" are able to collect much more than presented here.
What someone needs to do - ASAP - is do another one where people whose UDID's were on the list can list all the apps they've bought (it should be possible to get these out of itunes somehow) and we can use that to narrow down which ones most people have in common, and track down the organisation responsible for collecting this data in the first place. I would do it myself but I really don't have the free time at the moment.
Please describe how this script is crap. It's short and to the point, it does its job, handles the load, and it doesn't pass bogus characters to the command line.
Perhaps it needs commenting? Could be formatted more nicely? I'm truly interested in your characterization because I'm just not seeing it.
I don't know if the 350k hits run that script, let's say they did.
The OP is praising the performance of lighttpd + php5-fpm, but the script actually spawns an external process to search for the UUID. That's like running an old GGI that needs to be executed with every request.
I may be wrong but using lighttpd and php5-fpm (with fastcgi) may not be as relevant as it seems. I'd say that the operating system is caching everything.
EDIT: yes, php-fom is "a simple and robust FastCGI Process Manager for PHP". I'm no doing much PHP lately!
I agree. I had never heard of the site this post is referring to. I used a different site to check UDID's that let you only enter in part of the UDID. This site looks like it REQUIRED you to send the ENTIRE UDID. Not very good.
I also had a blog post hit the front page of Reddit as well as Gizmodo. ~400K hits over 24 hours. Page size was roughly 700KB in size. All hosted on a shared hosting plan at HostGator.
How? A free CloudFlare account to cache every image/css/js resource on the page, plus a custom page rule to force cashing of the actual html response from the server.
Did I think my situation was impressive? Not really. Just an average number of pageviews for a front page article.
The page size is just shy of 3kb. At 350k hits, that's around 1gb of data transfer.
If he had 100 requests in one second (average is 8/ps over 12 hours), that's 300kb per second transfer rate.
1gb of data transfer for a single day is relatively small. Server upload rates of <1mbps is also small.
If we make the page 100kb in size now, for example with a few nice images 350k hits turns to 34gb of data transfer. 100 requests in one second would require a 10mbps transfer rate. A $15p/m host likely wouldn't be able to cope with this.
> I think the combination of Lighttpd+php5-fpm is underappreciated...
So for me the real reason the site stayed up was not because he picked PHP or any other technology, but because the page was barebones and didn't put any strain on the server at all. Page file size is an underappreciated advantage for websites!
I had basically the same experience a few years ago when the Wikileaks stuff was first released and I hacked together a basic search engine for it. Interestingly, I got most traffic through Twitter which I hardly ever use personally.
Anyway, that was an EC2 Micro running PHP-FPM on Nginx, and it stayed up without problems through millions of hits over a few days as well. It's easy to write a well-scaling single-function site like that.
Yet a lot of blogs and more complex sites die instantly when they cross a couple of dozen hits per second. I suspect there are multiple reasons for it, but the most egregious one coming to mind is the typical PHP setup within shared hosting environments where every single request means about 30 file compilations and a lot of DB requests. Caching is probably not very popular either.
33 comments
[ 5.7 ms ] story [ 37.9 ms ] threadConsidering that lots of peoples blogs die because they get 10k visitors from HN. His small $15 vps did fine.
Never seen $found = `grep $udid FILE` using back-ticks in PHP land. Is that the same as the exec() function?
It's still relatively unsafe, as it invokes external programs through a shell, so you're dependent on the shell's environment, which can diverge on different systems. Last I checked, PHP did not have an easy, obvious way to safely invoke another program. You can do it manually with a mix of pcntl_fork and pcntl_exec, but capturing the output using that is more difficult.
Is there any shell on which those functions are unsafe ? I think they assume cmd.exe on windows and sh on others.
The right thing to do is provide an output capturing API that does not require escaping because it takes a list of strings passed to the exec(2) system call.
Just for comparison, in 1998/99 I ran an mp3 search engine that handled 200k daily queries at the peak. It was a Pentium with 128MB of memory. The search server was written in C, and it used an inverted index. The system load rarely went over 1.
If you worry about UDIDs being tracked I guess you should worry about the http to his page too.
However, according to somebody who apparently tested it a year ago (and posted it on HN), half of the iPhone apps pass UDID over the same http, allowing collection of the very same or similar data.
Which makes the whole subject of a government agency having UDID database without other convenient data, ummm, quite less probable. IMHO most of the "business entities" are able to collect much more than presented here.
EDIT: Re-reading, that sounds a little bitchy.. I didn't mean it that way. Sorry
Perhaps it needs commenting? Could be formatted more nicely? I'm truly interested in your characterization because I'm just not seeing it.
The OP is praising the performance of lighttpd + php5-fpm, but the script actually spawns an external process to search for the UUID. That's like running an old GGI that needs to be executed with every request.
I may be wrong but using lighttpd and php5-fpm (with fastcgi) may not be as relevant as it seems. I'd say that the operating system is caching everything.
EDIT: yes, php-fom is "a simple and robust FastCGI Process Manager for PHP". I'm no doing much PHP lately!
When the system begins to show signs of stress, you look at where it's spending its time, then alleviate the bottleneck.
When I first visited the site, before he started putting more text on the page, I don't think I would have run into the issue.
Browser window isn't maximized; if I scroll all the way up it cuts off the last two paragraphs/sentences (link to his wife's blog and the results).
Otherwise it's simple; it doesn't need to be flashy when it's a one-stop site (unless they release the remaining ones).
I also had a blog post hit the front page of Reddit as well as Gizmodo. ~400K hits over 24 hours. Page size was roughly 700KB in size. All hosted on a shared hosting plan at HostGator.
How? A free CloudFlare account to cache every image/css/js resource on the page, plus a custom page rule to force cashing of the actual html response from the server.
Did I think my situation was impressive? Not really. Just an average number of pageviews for a front page article.
If he had 100 requests in one second (average is 8/ps over 12 hours), that's 300kb per second transfer rate.
1gb of data transfer for a single day is relatively small. Server upload rates of <1mbps is also small.
If we make the page 100kb in size now, for example with a few nice images 350k hits turns to 34gb of data transfer. 100 requests in one second would require a 10mbps transfer rate. A $15p/m host likely wouldn't be able to cope with this.
> I think the combination of Lighttpd+php5-fpm is underappreciated...
So for me the real reason the site stayed up was not because he picked PHP or any other technology, but because the page was barebones and didn't put any strain on the server at all. Page file size is an underappreciated advantage for websites!
Anyway, that was an EC2 Micro running PHP-FPM on Nginx, and it stayed up without problems through millions of hits over a few days as well. It's easy to write a well-scaling single-function site like that.
Yet a lot of blogs and more complex sites die instantly when they cross a couple of dozen hits per second. I suspect there are multiple reasons for it, but the most egregious one coming to mind is the typical PHP setup within shared hosting environments where every single request means about 30 file compilations and a lot of DB requests. Caching is probably not very popular either.