Interesting at their scale, they haven't needed to shard PostgreSQL. +1 for PostgreSQL.
Also, would be super curious to see and hear more about their patch to SSHD to allow it to read from a database instead of disk. This would be super useful for us as well (https://commando.io).
Lastly, it was really interesting that switching to bcrypt took their servers down, and they were forced to write a custom sha1 to bcrypt cache. One solution would be to not hash (bcrypt) API tokens, but instead just encrypt them and store in PostgreSQL. The side benefit of this, is users could view their API tokens again if needed.
Obviously it is less secure, but Stripe for example allows you to view your API tokens, and thus they are not hashing them.
Note that since openssh 6.2 you can plug in to sshd for the keys without patching it with the AuthorizedKeysCommand , and there's several utilities around for pulling the authorized keys from LDAP or similar services.
Unfortunately, this doesn't scale, as with the AuthorizedKeysCommand, you are required to output all the keys for that user on stdout. Outputing all of the "git" user's authorized keys lines would be an extremely expensive operation.
From the sshd_config man page: "Specifies a program to be used for lookup of the user's public keys. The program will be invoked with its first argument the name of the user being authorized, and should produce on standard output AuthorizedKeys lines"
Random tidbit I learned at Atlassian Road Trip NYC last night: apparently Stash is written in Java instead of Python, and BitBucket and Stash are developed semi-independently.
8 comments
[ 3.2 ms ] story [ 28.3 ms ] threadAlso, would be super curious to see and hear more about their patch to SSHD to allow it to read from a database instead of disk. This would be super useful for us as well (https://commando.io).
Lastly, it was really interesting that switching to bcrypt took their servers down, and they were forced to write a custom sha1 to bcrypt cache. One solution would be to not hash (bcrypt) API tokens, but instead just encrypt them and store in PostgreSQL. The side benefit of this, is users could view their API tokens again if needed.
Obviously it is less secure, but Stripe for example allows you to view your API tokens, and thus they are not hashing them.
It's based on this patch: https://github.com/wuputahllc/openssh-for-git
Either way, if you want to tinker with opensshd I can recommend it as a starting point. It's very small, readable and easily tweaked.
From the sshd_config man page: "Specifies a program to be used for lookup of the user's public keys. The program will be invoked with its first argument the name of the user being authorized, and should produce on standard output AuthorizedKeys lines"
https://news.ycombinator.com/item?id=8165745