Those aren't "system calls"! They're shell-outs. Replacing a shell-out with File.read doesn't get rid of system calls; File.read("/etc/termcap") incurs, at best, 4 system calls (stat, open, read, close).
I'm surprised that Github was sloppy enough to be dropping to the shell to get things done.
Considering Git is written primarily in C, it's not that surprising, and the fact that they're no longer using the canonical Git implementation but a Ruby-based clone makes me worried about future incompatibilities and bugs.
I don't know Ruby, but wouldn't the better solution be some variant of C's exec call that executes the instruction directly without interpreting the shell string? This would reduce some parsing overhead and still allow the site to use the official version of Git.
Not really. I mean, with the backtick, you're doing 2x the execve's (you're exec'ing the shell), but it's still just very slow to fork and exec another process to get things done for you.
The real solution to calling C code from within Ruby is to call into the C code, either using DL/Win32API (tricky but doable) or by writing Ruby bindings.
Unfortunately, the implementation of Git makes writing bindings to it - in Ruby or Python or something similar -very problematic. None of it is really meant to be a long running process, so it will exit program execution on lots of different errors, etc.
There was a Google Summer of Code project that tried to address this 'libification' problem, but it did not get very far - at least, not far enough to do what something like GitHub needs to do.
Actually, the Ruby implementation for most of these operations is pretty fast (though not as fast as C) and the object specs are incredibly slow changing. The commands change around a lot, but the objects themselves are pretty simple and have not changed much in years. Doing this is far less prone to git changing than depending on command line (or possibly even binding) calls.
Converting over to a pure-Ruby solution has taken us weeks. If we had put forth this effort pre-launch, it would have either a) cost us integral features or b) pushed back the launch.
I thought (hazy memory here) that git was implemented as a bunch of perl wrappers on some C code. Wouldn't it be better to call into that C code, rather than implementing git in Ruby?
Well, it's sh and perl wrappers around C code, but the important point is that those wrappers don't FFI into the C code, they shell out to it.
So, yes, a hypothetical reimplentation of git in ruby as a C FFI interface would probably be faster than ruby reading the .git files itself; however exactly as demostrated in the blog entry shelling out to perl/sh code shelling out to C is much slower than the ruby program reading the .git files itself.
The improvement is noticeable. As a paying user, I appreciate it. I also appreciate that R1.0 was out the door so fast. So no demerits for the shell calls. :) Glad they're gone though.
When I am hacking something together and can't find a library or don't have time to write one to do something that UNIX does really well already, I use the shell too.
When performance becomes an issue, you rip it out and do it better.
I appreciate the visibility into the service issues and the related fix. That was a good move on GitHub's part.
Much appreciated! I just finished moving all of my projects onto GitHub and I couldn't be happier. Of course, I was formerly ball-and-chained to CVS which was just awful, so anything is an improvement. But GitHub really does take things up quite a few notches.
Don't know much about it, but perhaps speeding up shell calls would have been another possible approach? Something akin to fast_cgi for shells, I suppose?
13 comments
[ 3.7 ms ] story [ 34.7 ms ] threadI'm surprised that Github was sloppy enough to be dropping to the shell to get things done.
I don't know Ruby, but wouldn't the better solution be some variant of C's exec call that executes the instruction directly without interpreting the shell string? This would reduce some parsing overhead and still allow the site to use the official version of Git.
The real solution to calling C code from within Ruby is to call into the C code, either using DL/Win32API (tricky but doable) or by writing Ruby bindings.
There was a Google Summer of Code project that tried to address this 'libification' problem, but it did not get very far - at least, not far enough to do what something like GitHub needs to do.
http://code.google.com/soc/2007/git/appinfo.html?csaid=5DB94...
Actually, the Ruby implementation for most of these operations is pretty fast (though not as fast as C) and the object specs are incredibly slow changing. The commands change around a lot, but the objects themselves are pretty simple and have not changed much in years. Doing this is far less prone to git changing than depending on command line (or possibly even binding) calls.
So, yes, a hypothetical reimplentation of git in ruby as a C FFI interface would probably be faster than ruby reading the .git files itself; however exactly as demostrated in the blog entry shelling out to perl/sh code shelling out to C is much slower than the ruby program reading the .git files itself.
When I am hacking something together and can't find a library or don't have time to write one to do something that UNIX does really well already, I use the shell too.
When performance becomes an issue, you rip it out and do it better.
I appreciate the visibility into the service issues and the related fix. That was a good move on GitHub's part.
GitHub is a stellar example of how to start a software company.