Sounds like utter chaos for any seasoned professional. No QA? No real product managers (or at least none with any teeth)? Design as optional resource? Ops managing user engagement metrics? Where's the security audit?
I'm a little blown away at this myself. I knew they were extremely agile, able to make changes on a whim, but this sounds a bit crazy. Might check in with my buddy who works at FB to see if this matches up with reality. The sheer stress of working in an environment where perfection is not only desired, but required, would drive me crazy. Some of the processes, though, sound like they would be a dream. Working on things I want to work on would make me 100x more productive. Of course, it also means some things may never get done.
It would be interesting to get some verification, because no one could work in this environment for very long. Regardless of compensation. The stress and associated error rates that would accompany it would've been a problem long ago for Facebook. Particularly with no QA.
I cannot imagine Facebook existing and prospering for this many years without any QA team.
Perhaps this is a good example of how much of that professional "seasoning" is actually unnecessary or gets in the way of building successful products and high-caliber responsible teams, i.e. "throwing over the wall to QA" etc.
edit: Can I ask why this is being downvoted? Because I am disagreeing with the parent poster? I don't think I am raising poor points or somehow not contributing to the discussion. Facebook seems to not do a lot of "standard" things because they view them as counter to their goals of building their product. I am merely stating that perhaps there is a lesson in there about the value of what some of the convential wisdom in our industry dictates.
Upmodded... with that said, I do think that this is very industry dependent. They are a free product that people don't necessarily "rely" on. If status messages were severely delayed or dropped, it wouldn't be a huge deal (assuming they fixed it when noticed).
There's very little that they can't fix after deployed, and they have no SLA with consumers (advertisers may be different). In essence they have 500 million testers.
This is a bit harder to do with a database or compiler.
I'm especially wondering: who will take this platform to the next level? Is there anybody besides mark zuckerberg in charge for designing and testing out new ideas. I don't mean just new features like questions or places, but really significant changes like a facebook phone, or a new friends model to better graph you real network.
Copying the main points here, from Reddit user "epriest":
The article is has a few inaccuracies:
There is mandatory code review for all changes (i.e., by one or more engineers). I think the article is just saying that Zuck doesn't look at every change personally.
People do not get called out for introducing bugs. They only get called out if they ask for changes to go out with the release but aren't around to support them in case something goes wrong (and haven't found someone to cover for you).
Getting blamed will NOT get you fired. We are extremely forgiving in this respect, and most of the senior engineers have pushed at least one horrible thing, myself included. As far as I know, no one has ever been fired for making mistakes of this nature.
We have automated testing, including "push-blocking" tests which must pass before the release goes out.
We absolutely do not believe "most engineers are capable of writing bug-free code", much less that this is a reasonable notion to base a business upon. If this is a real quote, it comes from a very junior engineer. It is empirically untrue.
The nine push phases are not concentric. There are three concentric phases (p1 = internal release, p2 = small external release, p3 = full external release). The other six phases are auxiliary tiers like our internal tools, video upload hosts, etc.
OK, so according to him there's code review and some automated tests.
But that still doesn't address wide open access to the live DB by brand-new devs, and the complete disregard for having a dedicated QA team.
Why QA? I would think FB engineers are more busy doing real work at work than fooling around on their own site (heh, the irony) and finding out that Junior Dev 39's change to the thingamabob broke chat for the umpteenth time in IE7. And as many people have pointed out here, they are a platform for many other businesses now--there's no dogfooding that--and the API stinks in terms of stability and documentation.
It depends. If your API was built underneath your own site, then it's easy. FB's API doesn't seem to actually support anything on the main site, e.g., I doubt their own "apps" actually run on the FB Apps API. And there's no incentive for any of the engineers to build them like that, because they have such lateral access to internal code and data.
This seems to fit how I view Facebook as a whole. I find some of their technology extremely impressive such as their blazing fast search or HipHop. However, a lot of their user experience is extremely flawed and there is a big lack of consistency and intuitiveness. Examples: clicking your friend requests on the right side of the home page pops something up that tells you to click somewhere else, events are no longer notified so you have to manually check to see if you received new ones, the distinction between adding a page to your interests versus Like'ing it is not well defined, or if it is, many users are unaware, making an event for a Facebook Page is very convoluted... etc etc. I feel that issues like these are the reason why product managers and designers exist. I think Facebook is starting to really fall behind in this regard as more and more features start to clutter and complicate the site.
Yeah, I wouldn't want to run my company like this:
re: surprise at lack of QA or automated unit tests — “most engineers are capable of writing bug-free code. it’s just that they don’t have an incentive to do so at most companies. when there’s a QA department, it’s easy to just throw it over to them to find the errors.”
It does explain how facebook ships with bad documentation etc. -- well, now we have enough knowledge to know how to compete with it :)
Kasparov often remarked how a good process is more important than the actual participants. An average human and pretty good computer with a great system won the championship against great computers and against great grandmasters.
Google believes in this, and their products are very well engineered, with full documentation, videos etc. (although admittedly, many haven't taken off). Yahoo definitely understands this. But these are the same companies that are losing to facebook because of social.
If google and yahoo understood the dynamics of social, we would all be better off.
I'm really surprised that all engineers have free access to the Facebook database. That's really scary. I know at other organizations, with much less sensitive data, this stuff was under lock and key.
I have a hard time believing many aren't abusing this, just out of human nature ("did she mention me to any of her friends?").
That seems way to risky to me. You basically have to watch server logs for some interesting patterns. And of course, if they're smart they simply oversubscribe in a query (or temporally space out a query), and then pinpoint the data they are looking for locally.
Given they have 2000+ employees this is just a process that is looking to be constantly abused. I surely don't expect people aren't talking about it, but I'm nearly as confident that it is happening in a way that most people aren't aware of. No evidence, but again, I've seen people do worse, with less access.
Whenever I meet anyone who works at facebook, my first instinct is to extricate myself from that conversation.
Otherwise we might get to know each other, and become friends, and I'd never be sure that they weren't looking me up. My strategy for avoiding the prying eyes of facebook engineers is to be personally uninteresting to any facebook engineers.
I wonder how/if the access is restricted? Seems like a good compromise might be:
* Read-only all the time.
* Access through a special shell which logs your actions remotely.
* Seemingly in keeping with the aggressive name & shame culture, if you join across all tables or forget a limit or something else which affects live operation, you get called out.
I imagine that the database is enormous and constantly changing, so it's probably impractical to have development copies that are sufficient to test your code against realistically. I'll bet Google engineers have access to their full database too for the same reason.
On Reddit, someone mentioned that 'becoming someone', which I presume means making the software think you've logged in as that person, is unusual and must be justified and closely tracked when it happens. So I'm guessing that you can't just pull up records for some person from the database; you need to 'be' that person to get their data. The free access is probably just for stuff that is shared via the privacy controls.
> the database is enormous and constantly changing, so it's probably impractical to have development copies that are sufficient to test your code against realistically
Considering they use mysql how feasible is it for the database to be constantly changing? I think ive read that they use it more as a key value store then a relational store but still.
I don't know. We use artificially populated data for performance testing. for user behaviour and other metrics, they would have programs running against the actual production database, not a stale developer copy of it.
Agreed. Every large company I've worked with has had scripts that create sanitized versions of the production DB, with PCI and PII stripped out or masked to allow for testing against full size datasets. It's not hard to do.
> I'll bet Google engineers have access to their full database too for the same reason.
No, we don't. We have test instances of everything. There are a privileged few who have access to sensitive production data, and their access is carefully monitored.
Extracting terabytes of data is difficult to go un-noticed.
How do you transfer the data? rsync through external network is too slow.
The most convenient way would be to mount physical drive on one of database server and copy things directly. But that would require cage access, which I assume not many employees have.
That's an excellent interview question! "How do you get the maximum of data out before getting noticed?". I guess the most effective way is to tamper with source code and smuggle data out along with legitimate traffic.
Wikileaks exercises caution in what it leaks. I highly doubt wikileaks would leak personal information of millions of users. Their aim is to open up governments and public agencies, not people's personal lives.
That was their story last week. This week their story is "We're going to leak a bunch of folks' Swiss bank account details because some of them might be avoiding taxes!"
They didn't redact informants and collaborators names from many of the documents they leaked. I would hardly call that exercising caution.
I understand that in some cases redacting names weakens the impact of the leak and its political statement. However, releasing the name of some random informant working on the ground in Iraq has no substantial effect on the impact of the whole leak, but can have an enormous effect on that person's life, especially if he's killed.
I think we can probably agree that if the access is available but carefully guarded, a clever and patient black-hat could make use of their access to get the data they want; but the white-hat who just wants to do their job gets blocked. While it's not the only valid choice, it strikes me as reasonable to just grant access and impress upon everyone the incredible responsibility they now have not to do unethical things.
It seems, from the article at least, that the're very much a culture of blame, which I find quite surprising. I've never been a fan of the practices of practices such as making people wear a hat if they break the build and the like. It seems counter-productive to getting things built and shipped, and just seems to make an unhappy workplace (in my mind, at least)
The more i read about facebook engineering the less impressed i am. It used to be interesting to hear about their engineering challenges. Now its just scary to hear anything about them. And i use it as a rulebook of how not to do youre infrastructure. Im amazed facebook works at all frankly. I also think its one of the ugliest sites ive ever used. Its what 5 years old now and they still dont even have some basic functionality on it. Plus as another person said its often broken.
So about 1000 people have unfettered access to all of the live data, but there are absolutely no safeguards against them modifying it, copying it elsewhere, or peeking at it? That's terrifying. I'm under the impression that Google has more a stringent philosophy about this, and even then, one of the few SRE's that they do give broad access to was caught stalking teens via GMail data. Imagine the range of bad behavior an FB engineer can get away with after just 4 weeks of "Boot Camp".
Part of the reason I never made a real FB account was my general feeling in 2005 that securing a MySQL/PHP site against internal abuse takes incredible effort, and they wouldn't ever bother to do it. According to this article, that's likely true. And the "everybody can modify anything anytime" philosophy with little QA explains why, after 6 years, FB is still a gaping maw of security holes.
after boot camp, all engineers get access to live DB (comes with standard lecture about “with great power comes great responsibility” and a clear list of “fire-able offenses”, e.g., sharing private user data)
This is through a third-party, so I'm not shaking the finger at Facebook outright yet, but the wording of that — sharing private user data — is kind of frightening. What about merely accessing private user data? Like you mentioned, I know Google is super-paranoid about even simple data access.
And now someone from Facebook chimes in and replies to this and talks more about their developer privacy safeguards and makes us all feel better. Go!
> And now someone from Facebook chimes in and replies to this and talks more about their developer privacy safeguards and makes us all feel better. Go!
Former Facebook dev here. That's all I have to say in response.
I left late last year because I got a better offer and was fully vested. I suppose I was in a small minority that wasn't drinking the koolaid. The perks were nice, but ultimately I felt I wanted more stimulation than working on a social network, and didn't like some of the features that were being pushed (face recognition on photos, etc.) I'm grateful for the opportunity, and can't say much else.
From the people I've talked to at facebook, this is not true. Just because engineers have access doesn't mean the company doesn't have controls in place. As I understand it, access is carefully logged and you better be able to explain exactly why you needed to get specific information at any given time.
“most engineers are capable of writing bug-free code. it’s just that they don’t have an incentive to do so at most companies. when there’s a QA department, it’s easy to just throw it over to them to find the errors.”
Bull. Shit. This one line makes me think that this entire post is completely hear-say. A 500 person engineering team that writes bug-free code on a system as a large as facebook? Give me a break.
I have my Facebook set on Spanish to keep my Spanish language neurons firing even when I'm surrounded by Gringos, and their localization definitely has frequent bugs. There's been some issues for months with the birthday wall post aggregation they've been testing in production. I've frequently seen stuff like "So-and-so has written on so-and-so's muro for their birthday."
Almost all of Facebook's internationalization is crowd-sourced. What you're seeing probably isn't a bug, the string just hasn't been translated by a user yet.
Facebook's language translations are actually crowdsourced. It's funny that so many people would approve errors like this. There is a brief blog post about it:
http://blog.facebook.com/blog.php?post=10056937130
The whole system is pretty interesting. They maintain a mapping of every token on the site to every language and only user translators verify portions.
Multiple mentions of "public shaming". Doesn't seem so good. A good engineer is anyway going to have a sense of shame inside when he/she screws up. I really don't think there's any need for "public shaming".
So, so true. The tendency for perfectionism necessary to write code is already sufficient to instill a professional ethic as far as quality of work. Public shaming would be intolerable, I'd think.
You don't get shamed for a bug in your code - you get shamed if you aren't willing to go on-call to support your own code. That IMHO is a very healthy attitude.
As someone who works for an online gambling company where we are not even allowed to use the product we are building (legal/trust issues i guess), it would _rock_ to be able to have this kind of impact on features
Is the author of this piece an engineer working at Facebook? This part makes me think not:
>I’m fascinated by the way Facebook operates. It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and release software.
So I'd take whatever is written here with a grain of salt. My communication with friends working at Facebook yielded similar thoughts but nothing that comes to what's written that implies a callous recklessness. I know for a fact that they have some code-review tools and blocking tests.
Anyway, my point is that the author doesn't seem to be embedded too deeply in the engineering at Facebook and his notes are, while not outright false, definitely misleading.
This is very close to how we develop at TripAdvisor, though we have a suite of tests that get run during a branch merge and code review (of at least one more engineer) of anything going into the livesite code base.
We also have very small QA team. They work to make sure that the buttons that make us money aren't screwed up.
The key to this sort of uber-agile development is that you have very, very talented engineers, a release process flexible enough to deal with errors, and a management that buys into moving so fast that mistakes will happen.
I have to say that it is simultaneously exhilarating, humbling and a little terrifying to work like this. Luckily, mostly the first two.
I know all of this certainly has worked and does work for you, but the first thing I thought about when reading "small QA team" and "the key...talented engineers" is that QA is not very important to the company. A QA team that only checks the money buttons tells me that traditional QA has been incorporated into the developer positions themselves, which creates a conflict of interest.
Couching my conclusion in the topic of this post, it would seem that Facebook gives extraordinary freedom to developers, but also hangs everything around the developers neck. Screwing up in this area sounds pretty fatal, and not an aspect of a company I would want to work for. I'm kinda old, though.
"If lots of [employees] are flocking to a new business unit, that's a good sign that the opportunity is a good one. . . . If a business unit can't attract people very easily, that's a good sign that it's a business Enron shouldn't be in."
"Resourcing for projects is purely voluntary.
-a PM lobbies group of engineers, tries to get them excited about their ideas.
-Engineers decide which ones sound interesting to work on.
-Engineer talks to their manager, says “I’d like to work on these 5 things this week.”
-Engineering Manager mostly leaves engineers’ preferences alone, may sometimes ask that certain tasks get done first.
-Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything they need themselves."
Good god. Does this strike anyone else as disturbing? Surely every single piece of work being taken on should have the USERS needs and concerns as top priority and not sexy stuff that can attract sufficient engineering interest?
What if there's important problems that are really bothering lots of users and a PM can't get anybody interested (or no-one decides to take on the problem?)
Here's a complaint from a guy about maintaining a personal page and fan page:
"As a programmer myself, I can’t fathom that it would take much technical and design effort to address these issues, and Facebook is flooded with complaints from users begging them to fix these headaches. From my perspective as a Facebook user with a very active personal page and fan page, I can’t help but get the impression that Facebook deliberately wants to make some basic admin tasks (like blocking spammers) difficult or impossible in order to compel you to spend more time on the site. There doesn’t seem to be any other logical reason for these glaring design flaws that I can comprehend, other than pure incompetence, and based on their success in other areas, it seems more likely that these choices are deliberate."
"Surely someone on their team is aware of all the complaints and requests to fix the broken elements. So why do they seem to ignore what appear to be such glaring (and fixable) problems?"
"I thought that Facebook would be an interesting place to share inspirational messages and build more community around growth-oriented people. But the current implementation of Facebook can’t handle the way I’ve been trying to use it without creating more headaches than it’s worth, and their momentum appears to be headed in the wrong direction for me to expect that these problems would be fixed anytime soon."
"So I’ve crossed the threshold where Facebook’s value isn’t worth the hassle to use it. I concluded that the best choice was to simply drop the service altogether and invest my time elsewhere."
Who will take on these issues?
Again from the Malcom Gladwell article linked above:
"You might expect a C.E.O. to say that if a business unit can't attract customers very easily that's a good sign it's a business the company shouldn't be in. A company's business is supposed to be shaped in the direction that its managers find most profitable. But at Enron the needs of the customers and the shareholders were secondary to the needs of its stars."
Facebook should wake up in my opinion. They did really well to reach 600m users (or whatever the figure is n...
I think the comparison here to Enron is a bit of a stretch.
From what I know of Google, their internal staffing of projects is done much the same way: you're assigned to certain teams when you first join the company, but after that the various Product Owners try to recruit and advertise within the company for engineers to join their team.
When I was interviewing there I recall seeing flyers posted in common areas advertising different projects that were in need of engineers to come join them ("Interested in _____? Google ___ could use your help!").
"I think the comparison here to Enron is a bit of a stretch."
Could you clarify why this is? I'm not trying to imply anything, certainly not that Facebook is Enron, just that (to me) they have similar attitudes to assigning staff to work duties. When I read the Facebook quote, I immediately thought of Skilling's quote.
It seems to me and many others here on HN that I've read over the last few months have problems with the way Facebook operate e.g. those who need to use their API to integrate with their own service, privacy concerns, spam, etc; and Facebook doesn't seem to address these issues at all. That was my ultimate point - how they address with these issues, and could the fact they are not dealt with in a timely manner actually be a symptom of letting engineers pick their own work?
> Could you clarify why this is? I'm not trying to imply anything, certainly not that Facebook is Enron, just that (to me) they have similar attitudes to assigning staff to work duties. When I read the Facebook quote, I immediately thought of Skilling's quote.
Let's see how I can put this. What you've just done is the business equivalent of Godwin's law. It would be like comparing US customs to the Nazis because, you know, both want to look at your papers.
What makes Enron Enron isn't how they did staffing, it's how they cooked their books and defrauded people. Their name is synonymous with fraud and corruption, and so invoking them in conversation about similar staffing practices seems... misleading.
Well, sort of. Most projects at Google have headcount budgets that they can't go over without management approval. But even if you have the headcount, you still need to find people.
Doesn't seem like he's calling Facebook a fraud. But Enron had a famously superstar-driven culture. In particular, they had a fairly strict employee ranking system. Every year, each employee would be assigned a quintile based on performance. The top 20% of performers would always get raises, and the bottom 20% would always get fired. Sounds like a great system in theory, but from what I hear from people who were there, it led to a poisonous and cutthroat culture. Not sure if the same will happen to Facebook, but there certainly are some dangers.
The problem with that is the relative performance. If you have no shot at being top 20% all you need to do to survive is pick one of four peers and "outperform" him or her. No wonder cutting throats occurred...
As you said, Facebook has 500+M users, among those you'll find people annoyed by bugs and willing to: "to simply drop the service altogether and invest my time elsewhere."
But obviously those users are a tiny minority.
I challenge your conclusion that they should get their priority straight: if their goal is to keep growing at current rate as you seem to suggest, then they probably shouldn't change a damn thing, because they are darn good at it.
This strategy may work in the short term but I can't imagine how it will be sustainable. As engineers continue to flock to the hot new thing, they will be leaving a pile of bugs in their wake. The article mentions that developers are responsible for fixing their own bugs but that just won't scale. Once a developer has been there for two or three years, their body of work will have grown so large that they will no longer have the capacity to perform maintenance whilst working on new features. Your example of bugs in fan pages is a problem that will only get worse.
Perhaps they just need to sustain this model long enough to get to an IPO.
As someone who develops on top of the Facebook Platform, I'm not surprised. Huge, obvious bugs that affect many applications are released far too often.
Serious question. Why doesnt Facebook do any engineering talks anymore? Some of the coolest talks on infrstructure from the past few years have been from FB. And yet the only thing i can remember really hearing about their stack recently is that they were using HBase. Which really came off as more about marketing and hype about some products they were working on then truly geeky stuff thats come out from FB in the past. Anybody know? I wonder if its because Fb is no longer a startup? Which might the reason why we never hear about Youtube or Twitter anymore either.
Oh my other comment was just meant in jest. But thanks for downvoting it and killing all the worthless and meaningless HN karma points i had accrued!
121 comments
[ 3.2 ms ] story [ 181 ms ] threadScary, even for a developer like myself.
I cannot imagine Facebook existing and prospering for this many years without any QA team.
edit: Can I ask why this is being downvoted? Because I am disagreeing with the parent poster? I don't think I am raising poor points or somehow not contributing to the discussion. Facebook seems to not do a lot of "standard" things because they view them as counter to their goals of building their product. I am merely stating that perhaps there is a lesson in there about the value of what some of the convential wisdom in our industry dictates.
There's very little that they can't fix after deployed, and they have no SLA with consumers (advertisers may be different). In essence they have 500 million testers.
This is a bit harder to do with a database or compiler.
The article is has a few inaccuracies:
There is mandatory code review for all changes (i.e., by one or more engineers). I think the article is just saying that Zuck doesn't look at every change personally.
People do not get called out for introducing bugs. They only get called out if they ask for changes to go out with the release but aren't around to support them in case something goes wrong (and haven't found someone to cover for you).
Getting blamed will NOT get you fired. We are extremely forgiving in this respect, and most of the senior engineers have pushed at least one horrible thing, myself included. As far as I know, no one has ever been fired for making mistakes of this nature.
We have automated testing, including "push-blocking" tests which must pass before the release goes out.
We absolutely do not believe "most engineers are capable of writing bug-free code", much less that this is a reasonable notion to base a business upon. If this is a real quote, it comes from a very junior engineer. It is empirically untrue.
The nine push phases are not concentric. There are three concentric phases (p1 = internal release, p2 = small external release, p3 = full external release). The other six phases are auxiliary tiers like our internal tools, video upload hosts, etc.
But that still doesn't address wide open access to the live DB by brand-new devs, and the complete disregard for having a dedicated QA team.
Why QA? I would think FB engineers are more busy doing real work at work than fooling around on their own site (heh, the irony) and finding out that Junior Dev 39's change to the thingamabob broke chat for the umpteenth time in IE7. And as many people have pointed out here, they are a platform for many other businesses now--there's no dogfooding that--and the API stinks in terms of stability and documentation.
re: surprise at lack of QA or automated unit tests — “most engineers are capable of writing bug-free code. it’s just that they don’t have an incentive to do so at most companies. when there’s a QA department, it’s easy to just throw it over to them to find the errors.”
It does explain how facebook ships with bad documentation etc. -- well, now we have enough knowledge to know how to compete with it :)
Kasparov often remarked how a good process is more important than the actual participants. An average human and pretty good computer with a great system won the championship against great computers and against great grandmasters.
Google believes in this, and their products are very well engineered, with full documentation, videos etc. (although admittedly, many haven't taken off). Yahoo definitely understands this. But these are the same companies that are losing to facebook because of social.
If google and yahoo understood the dynamics of social, we would all be better off.
I have a hard time believing many aren't abusing this, just out of human nature ("did she mention me to any of her friends?").
Given they have 2000+ employees this is just a process that is looking to be constantly abused. I surely don't expect people aren't talking about it, but I'm nearly as confident that it is happening in a way that most people aren't aware of. No evidence, but again, I've seen people do worse, with less access.
I'm gathering that this will never change.
Otherwise we might get to know each other, and become friends, and I'd never be sure that they weren't looking me up. My strategy for avoiding the prying eyes of facebook engineers is to be personally uninteresting to any facebook engineers.
This might not work so well if I were a chick.
* Read-only all the time. * Access through a special shell which logs your actions remotely. * Seemingly in keeping with the aggressive name & shame culture, if you join across all tables or forget a limit or something else which affects live operation, you get called out.
I'm fairly certain if you give me the schema, I can be very inconspicuous over a given amount of time.
On Reddit, someone mentioned that 'becoming someone', which I presume means making the software think you've logged in as that person, is unusual and must be justified and closely tracked when it happens. So I'm guessing that you can't just pull up records for some person from the database; you need to 'be' that person to get their data. The free access is probably just for stuff that is shared via the privacy controls.
What the hell? why not?
No, we don't. We have test instances of everything. There are a privileged few who have access to sensitive production data, and their access is carefully monitored.
How do you transfer the data? rsync through external network is too slow.
The most convenient way would be to mount physical drive on one of database server and copy things directly. But that would require cage access, which I assume not many employees have.
It would have to be code rarely seen by others, e.g. Apache's mod_rewrite.
"Thousands of random users' data has been downloaded from Facebook and published on Wikileaks" would be enough for a good media scare story.
I think he'll be losing some of his libertarian fans if he publishes information of people who simply want to be more free.
I understand that in some cases redacting names weakens the impact of the leak and its political statement. However, releasing the name of some random informant working on the ground in Iraq has no substantial effect on the impact of the whole leak, but can have an enormous effect on that person's life, especially if he's killed.
(I don't speak for my employer, Facebook.)
Part of the reason I never made a real FB account was my general feeling in 2005 that securing a MySQL/PHP site against internal abuse takes incredible effort, and they wouldn't ever bother to do it. According to this article, that's likely true. And the "everybody can modify anything anytime" philosophy with little QA explains why, after 6 years, FB is still a gaping maw of security holes.
This is through a third-party, so I'm not shaking the finger at Facebook outright yet, but the wording of that — sharing private user data — is kind of frightening. What about merely accessing private user data? Like you mentioned, I know Google is super-paranoid about even simple data access.
And now someone from Facebook chimes in and replies to this and talks more about their developer privacy safeguards and makes us all feel better. Go!
Former Facebook dev here. That's all I have to say in response.
Bull. Shit. This one line makes me think that this entire post is completely hear-say. A 500 person engineering team that writes bug-free code on a system as a large as facebook? Give me a break.
It explains a lot, really.
So yeah, lack of QA => lots of bugs.
The whole system is pretty interesting. They maintain a mapping of every token on the site to every language and only user translators verify portions.
"resourcing for projects is purely voluntary."
As someone who works for an online gambling company where we are not even allowed to use the product we are building (legal/trust issues i guess), it would _rock_ to be able to have this kind of impact on features
>I’m fascinated by the way Facebook operates. It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and release software.
So I'd take whatever is written here with a grain of salt. My communication with friends working at Facebook yielded similar thoughts but nothing that comes to what's written that implies a callous recklessness. I know for a fact that they have some code-review tools and blocking tests.
Anyway, my point is that the author doesn't seem to be embedded too deeply in the engineering at Facebook and his notes are, while not outright false, definitely misleading.
We also have very small QA team. They work to make sure that the buttons that make us money aren't screwed up.
The key to this sort of uber-agile development is that you have very, very talented engineers, a release process flexible enough to deal with errors, and a management that buys into moving so fast that mistakes will happen.
I have to say that it is simultaneously exhilarating, humbling and a little terrifying to work like this. Luckily, mostly the first two.
Couching my conclusion in the topic of this post, it would seem that Facebook gives extraordinary freedom to developers, but also hangs everything around the developers neck. Screwing up in this area sounds pretty fatal, and not an aspect of a company I would want to work for. I'm kinda old, though.
Couldn't the other corollary of this be that the company/development team has invested heavily in automated testing?
I'm still not sure having the product team responsible for hours and hours of black-box QA testing each week was an effective use of resources.
- Jeff Skilling, former president of Enron
http://www.gladwell.com/2002/2002_07_22_a_talent.htm
Contrast that with the Facebook approach:
"Resourcing for projects is purely voluntary. -a PM lobbies group of engineers, tries to get them excited about their ideas.
-Engineers decide which ones sound interesting to work on.
-Engineer talks to their manager, says “I’d like to work on these 5 things this week.”
-Engineering Manager mostly leaves engineers’ preferences alone, may sometimes ask that certain tasks get done first.
-Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything they need themselves."
Good god. Does this strike anyone else as disturbing? Surely every single piece of work being taken on should have the USERS needs and concerns as top priority and not sexy stuff that can attract sufficient engineering interest?
What if there's important problems that are really bothering lots of users and a PM can't get anybody interested (or no-one decides to take on the problem?)
Here's a complaint from a guy about maintaining a personal page and fan page:
http://www.stevepavlina.com/blog/2011/01/leaving-facebook/
Some quotes from that piece:
"As a programmer myself, I can’t fathom that it would take much technical and design effort to address these issues, and Facebook is flooded with complaints from users begging them to fix these headaches. From my perspective as a Facebook user with a very active personal page and fan page, I can’t help but get the impression that Facebook deliberately wants to make some basic admin tasks (like blocking spammers) difficult or impossible in order to compel you to spend more time on the site. There doesn’t seem to be any other logical reason for these glaring design flaws that I can comprehend, other than pure incompetence, and based on their success in other areas, it seems more likely that these choices are deliberate."
"Surely someone on their team is aware of all the complaints and requests to fix the broken elements. So why do they seem to ignore what appear to be such glaring (and fixable) problems?"
"I thought that Facebook would be an interesting place to share inspirational messages and build more community around growth-oriented people. But the current implementation of Facebook can’t handle the way I’ve been trying to use it without creating more headaches than it’s worth, and their momentum appears to be headed in the wrong direction for me to expect that these problems would be fixed anytime soon."
"So I’ve crossed the threshold where Facebook’s value isn’t worth the hassle to use it. I concluded that the best choice was to simply drop the service altogether and invest my time elsewhere."
Who will take on these issues?
Again from the Malcom Gladwell article linked above:
"You might expect a C.E.O. to say that if a business unit can't attract customers very easily that's a good sign it's a business the company shouldn't be in. A company's business is supposed to be shaped in the direction that its managers find most profitable. But at Enron the needs of the customers and the shareholders were secondary to the needs of its stars."
Facebook should wake up in my opinion. They did really well to reach 600m users (or whatever the figure is n...
From what I know of Google, their internal staffing of projects is done much the same way: you're assigned to certain teams when you first join the company, but after that the various Product Owners try to recruit and advertise within the company for engineers to join their team.
When I was interviewing there I recall seeing flyers posted in common areas advertising different projects that were in need of engineers to come join them ("Interested in _____? Google ___ could use your help!").
Could you clarify why this is? I'm not trying to imply anything, certainly not that Facebook is Enron, just that (to me) they have similar attitudes to assigning staff to work duties. When I read the Facebook quote, I immediately thought of Skilling's quote.
It seems to me and many others here on HN that I've read over the last few months have problems with the way Facebook operate e.g. those who need to use their API to integrate with their own service, privacy concerns, spam, etc; and Facebook doesn't seem to address these issues at all. That was my ultimate point - how they address with these issues, and could the fact they are not dealt with in a timely manner actually be a symptom of letting engineers pick their own work?
Let's see how I can put this. What you've just done is the business equivalent of Godwin's law. It would be like comparing US customs to the Nazis because, you know, both want to look at your papers.
What makes Enron Enron isn't how they did staffing, it's how they cooked their books and defrauded people. Their name is synonymous with fraud and corruption, and so invoking them in conversation about similar staffing practices seems... misleading.
If facebook is starting to go down this same road they might arrive at the same end.
But obviously those users are a tiny minority.
I challenge your conclusion that they should get their priority straight: if their goal is to keep growing at current rate as you seem to suggest, then they probably shouldn't change a damn thing, because they are darn good at it.
This goes back to the well-known advice of "make a product you want to use"
Perhaps they just need to sustain this model long enough to get to an IPO.
As someone who develops on top of the Facebook Platform, I'm not surprised. Huge, obvious bugs that affect many applications are released far too often.
Oh my other comment was just meant in jest. But thanks for downvoting it and killing all the worthless and meaningless HN karma points i had accrued!
Or it is labeled "unsexy" to work on it.