Isn't it more because they're a simple IM application (albeit spread across a number of platforms)? I know they're hugely successful and there are considerations about scaling to that sort of size, but surely the fact that their product is extremely simple has to be a huge factor?
I got my first ever spam on whatsapp the other day from someone who had never been a contact, and there seems to be no way whatsoever to prevent it. So maybe they should hire 1 more engineer to work on spam.
Adding more privacy controls would sort of kill the point of it as SMS killer. You can block specific people.
What they should control is the APIs that are popping up like promotional SMS with support for Whatsapp. These companies are obviously circumventing their endpoints.
That only works with direct messages. Spammers have upped their ante, and they are now sending spam by adding you to group messages which (when I looked it up) don't allow you to report them as spammer (you can still block them, but that doesn't prevent a spam epidemy)
I just recently started using WhatsApp for some international travel and to communicate with friends abroad and haven't received any spam for several months. I've also found it to be far superior than regular SMS, especially in group settings with people using different operating systems and would probably keep using it had I received the occasional spam (as I do with many other services, Skype, E-Mail, etc...)
If this isn't a counter to the inane recruiting requirements of "5 years with language X" I don't know what is:
We don’t bring them in specifically because the engineer knows Erlang,” Mahdavi said on Monday. “We expect the engineer to come in and spend their first week getting familiar with the language and learning to use the environment. If you hire smart people, they’ll be able to do that.”
This reminds me of how we have been hiring engineers.
We sit them down for a day with a fairly simple task (build a simple web or mobile app, about three screens with a basic data model) but we make sure to give them a language or framework they are unfamiliar with.
They have Google, Stack Overflow, and other employees to use as a resource. It's a great way to get to know new hires while seeing how resourceful and interested in learning they are.
Every candidate has built the app to spec with a few hours remaining, but it's really good to see those who take the initiative to spend a bit of extra time polishing it and explaining it to us, and responding to our feedback.
Does this interviewing methodology mean that you only hire devs that don't currently have a gig or have vacation days to blow on the possibility of getting a job with you?
We do have an initial hour meeting with candidates just to get to know them and propose the trial if they seem like a possibility.
Wouldn't most people take a day off to interview somewhere? I know I wouldn't be comfortable ducking out for a few hours to interview — it would feel tense and uncomfortable.
Facebook, and I'm sure many other places, do full-day interviews. It's certainly not unusual.
I guess I'm not really thinking this is a hardship on the interviewee, but wondering more if you're implicitly filtering your candidate pool. But you make good points.
> Wouldn't most people take a day off to interview somewhere?
Not always possible. One of the big annoyances when I was working for a company was that if I wanted time off it was two weeks notice. Doesn't work particularly well when the interview invitation tends to be more along the lines of, 'Hi, your interview date is next week.'
Or maybe you have had many life events (births, deaths, etc.) and your employer wasn't very understanding. So you've already burned all your time off and are ready to find an employer who's a better fit.
Actually, I think this is a fair way to evaluate new hires. That said, I hope you don't look over their shoulders? I can't tie my shoe laces with someone looking over my shoulders. Or, secretly view their progress? I'm o.k. With reviewing their progress at the end of the day though?
Plus, if you are going to hire in this fashion, why even bother with resumes, or extensive resumes? Just bring people in for a test? If they do well--hire them?
I always felt a test is the fairest way to hire new employees. Tests do away with upperclass privilege(better contacts, padded resumes, etc.) I wish more companies vetted employees in this fashion. It might get rid of something I have always dreaded, and get slightly nauseous when I see the behavior in action--and that is Networking. Sorry, I always thought it was phoney.
I still imagine your company relies heavily on resumes, and uses the test--because they can? Meaning--there's still a lot of desperate software engineers out there; willing to do anything to land a job?
I don't quite understand the correlation. If the article was about how Whatsapp achieves scale and parallelism through functional languages, then fair enough. But I'm not sure it's fair to say they need 50 engineers, in part, because of the language choice ? Surely they need 50 engineers because their product is fairly simple and there's no need to have more engineers than features. Maybe the point it should have made is that Whatsapp values higher quality over quantity ? That I could relate to.
I think most seasoned game developers would be faster making a simple 3D game with Unity rather than just openGL + C ( mainly because with Unity you get soon much boilerplate-y stuff out of the box)
The language and the amazing properties of failure tolerance could be productivity multipliers ( think of the times you've had to deal with catastrophic failures of software)
Nah, language alone isn't a good enough reason. It can contribute, just like every choice of tool can contribute.
You example, unity, isn't a language, but instead an off the shelf engine and toolchain.
But just like the development on an individual game seem less impressive technologically the more they rely on existing tech [1], admitting "Yeah, we forked ejabberd and rigidly fight scope creep" doesn't have the same ring as hyping up features in the runtime of a hip yet obscure programming language.
[1] Fun side note: If you watch the games press interviews devs, pay attention to when they ask about what engine they use. The press don't and viewers often don't know anything about software projects, so the devs are reluctant to to admit how much they've relied on off the self components for their admittedly-huge undertaking. Fun to watch the paid hype men try to talk around that.
> Surely they need 50 engineers because their product is fairly simple and there's no need to have more engineers than features.
Right. There's not even a way to adjust notification levels based on person or group. I ended up turning off notifications for the whole app, so it stopped serving the same use cases that text messages do. People still need to text or call me if they need an urgent answer.
On the Android app there is. Choose the "View Contact" or "View Group" menu option from the top right and then change that contact or group's notification settings.
Ah, right. Thanks. Still, if there were 60 developers, could they have this feature on all clients.
That was just an example anyway. I could easily have pointed to the ambiguous end-to-end encryption (are my messages even being encrypted?) or the lack of encryption in group chats.
Actually I believe that answer is in the article:
“Our strategy around recruiting is to find the best and brightest engineers. We don’t bring them in specifically because the engineer knows Erlang,”
WhatsApp only needs 50 engineers because of who they hire. Even using Erlang, I pretty sure they would need more than 50 engineers if they hired mainly around average engineers.
How do you know if an engineer is average vs one of the best?
I haven't heard of anyone who has figured out how to measure an engineer's abilities.
Some people test algorithms. Others test how often they've shipped things. Most just look for years of experience. A few even test whether you know certain facts.
I haven't heard of a single company that has nailed down interviews. I was going to mention interviews but figured that if you couldn't measure how much an employee contributes, you'd have a much harder time measuring how much a prospective employee could contribute given a much shorter time to test them.
This articles doesn't look that convincing to me. I has no technical details to prove that Erlang was the major reason why WhatsApp was scalable. I think the low number of engineers has more to do with the simplicity of their product than with this particular language choice.
WhatsApp seems to be a great example of designing your system with the best tools available. To go from nothing to pretty much being the de-facto modern replacement for SMS in a couple of years is an amazing achievement.
I really need to sit down and spend some time properly learning a functional language. I can hack my way through Haskell code but I do it in a very non-Haskell way. I really need to change that. My primary language is C++ and I am open to any recommendations for what direction to go in :)
The problem with learning functional languages is finding pragmatic learning materials.
I would claim F# is currently the most approachable functional language (just my opinion but I have at least toyed with the other mainstream offerings).
My background is also in C++. The F# materials listed helped me personally to develop as a programmer and have made approaching other languages like Erlang easier.
Flying frog consultancys materials are steep in price but I found them worth my dime (Visual F# 2010 for Technical Computing and the materials in The F# Journal)
http://www.ffconsultancy.com/products/index.html
Try Elixir, it runs on the ErlangVM and has a much nicer syntax, plus metaprogramming through Lisp-style macros, while preserving every feature that makes erlang great.
Or LFE - Lisp Flavored Erlang - if you prefer Lisp to Rubyesque syntax. Robert Virding, one of the gang along with Joe Armstrong that created Erlang, created LFE.
I heard that they have the philosophy of "you build it, you own it", meaning that anything you build, you are also responsible for running. So if you don't want a call at 3am, make sure your code works correctly.
> Why WhatsApp Only Needs 50 Engineers for Its 900M Users
Answer: because sending short messages from A to B is basically a solved problem. There is even a programming language (Erlang) that was made with this application in mind. The prototypical "Hello World" example for Erlang is a messaging application.
Yep. And of course, Erlang is really only a functional language in the small, the parts that don't matter so much. In the large, arguably the scale that matters, it's an actor language, with highly encapsulated entities exchanging messages. So essentially what Alan Kay had in mind when he coined the term "object oriented".
I don't see how that's an issue here. The recipient doesn't need to know if his message receipt has been received. The burden remains with the sender to retry, just like TCP. The real Two Generals' Problem in instant messaging can just be passed to the communicating humans to deal with.
A whatsapp conversation is not dependent on the parties coming to a consensus on the timing for a common event.
Also, an army can only send so many messengers before you run out of soldiers (or operational readiness). Packets are free and plentiful. If you're on a network link so bad that you can't even occasionally receive a TCP ack, then you're very much an outlier.
The only thing simpler would be sending messages to a centralized queue for people to fetch. Yet Twitter needs 4100 employees for that (though not all engineers I hope).
Talking about Erlang has the cred of an esoteric language. And you can hype up your company for using it, the allure of different languages seems to be in fashion right now (the article goes on tangents about Facebook/Haskell, Mozilla/Rust and Google/Go).
Talking about ejabberd is admitting a core piece of the product is standing on the solders of giants. Something startups are often loathe to do.
I'm sure you could do it in an afternoon, right? Just like anyone on HN can make a "basic CRUD app" like Facebook in a weekend.
I really dislike the arrogant attitude behind these types of comments. Have you worked at WhatsApp? Do you have any idea what they spend their days doing, what their systems look like, what their requirements are like?
WhatsApp have close to a billion users, 50 engineers and they manage to produce a near flawless experience. This is not trivial, no matter how many armchair critics claim otherwise. It used to be the case that people dismissed FB as just another CRUD app ("not even written in RoR, but in PHP, yuck" ~ 2005-2010), but given the tools they have been pushing the last few years people seem to have stopped dismissing them as casually. Perhaps in a few years people will stop dismissing WhatsApp as casually as well.
I second you. People throw in things like "Duh, it's a lame technology, duh, I am working on something far superior, duh, that 3 people will ever use, duh"
Don't forget WhatsApp has applications on all platforms.
I believe that the line of reasoning of your parent still has merit. Erlang OTP was developed for telephony systems and has proven to be very reliable for distributed system. Messaging maps relatively nicely to Erlang OTP and as such Whatsapp are leveraging decades of research and work by Ericsson. Of course, this is how it should be: a good software infrastructure should save users time.
Obviously, creating Whatsapp is nothing near a weekend project. But I think there founders should be commended especially for avoiding some classic mistakes: the NIH syndrome and going on a hiring spree once the investments started rolling in. Their managerial skills are at least as good as their engineering skills.
I think that indeed the important thing here. The article try to focus a bit too much on some intrinsic qualities of the language as if it was the silver bullet of scalability while the real lesson is that they used the right tool for the job.
Not so long ago, maintaining hardware that could support 900M user would have required an IBM scale company. A single developer today can develop, deploy and maintain a medium scale ecommerce site using infrastructure like AWS, a myriad of OpenSource solutions, plenty of services to handle payments, ... yet he will probably be using the same language than 15 years ago when 20 times more developers were needed to achieve the same result.
Indeed. It's amazing :) . All the infrastructure-as-a-service stuff REALLY cuts down the overhead.
I'm building a product that only a few years ago would have required a VERY significant investment, but now is basically stacking and connecting lego blocks of services :) - sales-as-a-service isn't mature yet, but if we had it, it would even sell itself :) .
Ok, it's probably a pain if one of the black boxes fails, and I do know enough about the underlying architecture behind the services I'm using - I could probably replicate (badly) most if not all of them given time and money, but why would I?
> Ok, it's probably a pain if one of the black boxes fails, and I do know enough about the underlying architecture behind the services I'm using - I could probably replicate (badly) most if not all of them given time and money, but why would I?
I don't think this is a possibility whose importance can be understated. In most of my apps running on IaaS/PaaS offerings like heroku, key pieces of the infrastructure tend to fail for short times (typically very inconvenient times) fairly frequently, on the order of once or twice a week. The relative cost of this is highly variable and business-dependent, but at best its mildly inconvenient and at worst can be crippling. Its definitely something that needs to be accounted for. That said, I've also been lead on apps that used the same pieces of server infrastructure that had far better uptime when I handle ops myself instead of delegating that to another provider. (Not to pick on the providers—they have a very difficult job to do in a cost-effective way, and I'm quite certain that they do it better at that scale than I would.)
The choice of Erlang for WhatsApp was strategic if for no other reason that that it allows them to dispense with some of those choices. Many of the typical use-cases of Redis, for example, are easily supported by OTP.
I think the reality, as is often the case with two extremes, is somewhere in between.
Is this a simple 'solved' problem that anyone could do? No. But is it a super-human feet that only a handful of people could achieve? IMO, no as well. I think people in the latter extreme like to believe this as it's something they can aspire to - to say otherwise is to devalue their vocation. In the former camp, I think it's people pushing back against this too far in the opposite direction.
This might be me being exceedingly charitable to the OP, but I see his point differently:
There's a ton of work involved in making a platform like WhatsApp a polished user experience. There's then a ton more work involved in making it _solid_. In choosing Erlang, they chose a platform that trivialises a large part (though not all) of the latter, allowing them to focus on the former.
Essentially, yes? I mean, are you suggesting these engineers are doing magic? That they wrote some new and amazing code that anyone could not write? That these 50 people are the only ones who could do this and somehow ended up at WhatsApp? Or are they just leveraging powerful systems efficiently?
Have you worked at WhatsApp? Do you have any idea what they spend their days doing, what their systems look like, what their requirements are like?
Do you? If yes, please enlighten us. Otherwise I don't see the point of your objection. No one said that it can be done in a weekend. All that the parent comment said is that the fundamentals of building WhatsApp are already solved. He/she didn't dismiss it. Just because something isn't rocket science doesn't mean it's not useful.
As for Facebook, people used to dismiss it not because it was seemingly too easy to build but because they failed to see its usage. Long before it became a platform it was pretty much a waste of time. Obviously anything with more than a hundred MM users is difficult to maintain.
I can personally guarantee to you from direct current experience that there is nothing particularly solved about running services for even 50 million users.
Hang on, I _could_ write a (small) twitter or (early) fb clone in a weekend. Easily. That would probably scale to 10k tweets or feed entries.
What I in _no_ way could do is rewrite fb or twitter at the scale these systems operate today with a billion users and even more posts and data points. I don't agree with the sentiment of the previous poster, but fb & twitter were initially basic CRUD apps.
Well, frankly, there are plenty of open source chat server implementations (usually following XMPP standards) which easily scale to 10's of thousands, sometimes 100's of thousands of users without any modifications (OpenFire, to name one). It wouldn't be that difficult for a group/team of people to scale that into a cluster of servers and support several hundred million users at once -- XMPP packets are small and this problem has been solved many times.
From a technical standpoint, WhatsApp, the product, and the scale they operate at are nothing new.
Pretty much every single one of your test cases has already been solved in the XMPP specification, which WhatsApp is likely using (essentially all modern chat services/servers/clients speak XMPP).
Sure. And someone's got to take the XMPP package and put it on the different platforms, test them all for correctness, etc. I'm not saying there aren't existing parts that can be used. But even if you have a plausible component, someone's got to look at the docs, maybe build a toy version, think about whether you're hitting the limits, integrate with the rest of the codebase, and so on.
It's easy to see what pieces you need. But there's still work in glueing them together and testing the result.
I think 50 is a sensible number of in-house engineers for this sort of thing. You want cover as well in case someone leaves or gets ill. You want people to be able to upgrade the system when things change. You want people to look ahead at future requirements. And you want to be able to do these things all at once, with slack built in for peak load.
Well, you sort of are making my point -- 50 engineers is plenty (maybe even too many) to cobble together a solution to a problem that already has been solved and solved again.
WhatsApp did nothing technically challenging, but they did strap a business model (err, sort of) to something many others failed to capitalize on.
I bet that WhatsUp infrastructure would work ~ 35 engineers.
That's a social problem though (the fact that 35 guys generate so much cash, while on the other side of the planet 350 millions are looking for a way to pay the rent), not a technical one :-)
This comment focuses on the back-end. As important as it is, as far as I remember when FB acquired WhatsApp then one of the main advantages was the presence of clients in almost any mobile device (thanks to J2ME). I am pretty sure it is not trivial to maintain so many clients, even though over time the older clients will disappear and it will be mostly Android / IPhone / Windows (??) that the engineers need to maintain.
I have to say I was expecting this to be a little older - it's really impressive that Facebook have kept the culture and tiny headcount of that team.
I doubt it's ever a good idea for a company like Facebook to do layoffs but I wonder if there are any lessons from the WhatsApp team for a company that size - their intern cohort looked way bigger than 50 and they seem to have almost 50 offices [0]. Obviously a very different product and much more complex but it's interesting to wonder what the same product led by Jan Koum would hypothetically look like.
(Disclaimer: I teach this stuff) Good article. It's a keeper. Touches on a lot of things I've been telling clients for years.
My takeaways:
- Pure functions in little composable pieces. Screw the fact you can't find programmers. With 1/20th the staff, it's not important
- Zero to few meetings. Keep focused on one thing. No playing around with WhizzBang Platform 7.0 because all the cool kids are using it. Mind your knitting.
- Keep It Simple, Stupid. Engineers are our own worst enemy. Solve the most simple problem possible. Then add complexity a little bit at a time, only as a last resort. For those of you saying "But messaging is solved", "Erlang was written for messaging", and so forth, this is where you missed. Any complex system (with a few notable exceptions) will look very simple when solved in this manner. In fact, that's the entire point.
- Running and deploying are the same thing. CI/CD. There is no Big Red Button to push because we're always working on the airplane engines while the plane is flying
Coming from a traditional OOA/D/P background, this mentality was extremely foreign to me. Even though I was using and talking about TDD, it was still in terms of composing and re-arranging object graphs. It wasn't until I got into F# several years ago that the pieces started coming together. This is a very difficult conceptual leap to make, because a lot of the ways you work looks completely ass-backwards to folks who are used to doing it the other way.
I realize you're using a throwaway account, but this is really more like an essay, not a HN comment (cue up snarky observations about the length of my comments). There were four or five things that all came together at the same time for me: lean startup, functional programming, TDD, and static page web programming.
If you're interested, my contact details are in my profile. Happy to have a call/Skype. It's an important topic.
I wish there were some good online resources around Erlang. Most of the content is very theoretical and not explanatory. Its hard to find implementation of common algorithms like Sorting, graphs, link-list and other dataStructures in erlang.
Uh...what exactly are you looking to learn? If you're looking to learn the language, there are plenty of resources online (Fred Hebert's Learn You Some Erlang For Great Good and Erlang in Anger are great as introductory, and intermediate texts, respectively).
If you already know the language fundamentals...you'd not be asking about linked lists; you can only have singly linked lists in an immutable language (without extremely bad performance, that requires every update/insert/delete to re-copy the entire list), and they're so fundamental that the language's core has them for you. And most of the common data structures are well discussed in Learn You Some Erlang For Great Good. Sorting, eh, https://gillesleblanc.wordpress.com/2012/05/12/sorting-algor... . Graphs, you need to narrow down a bit, different types of graphs require different implementations, and due to the immutableness of Erlang will likely require thinking about in a different way than in a mutable language.
I haven't taken the Erlang plunge yet, but if I did, I'd probably try starting with Elixir, a language written in Erlang. There is a growing amount of buzz, community, tooling, etc. A basic understanding of Elixir would probably make Erlang a lot easier to understand.
I would love to see a list of these 50 engineers, their skills and how they are organized. It provides guidance for other teams (even if part of their software is written in Erlang!).
Language/platform choice explains why they need 50 engineers rather than 100. The trivial scope of the application explains why it's 50-100 and not 5000. It's a messaging service! If they weren't extending their application by developing new functionality then 50 even sounds like a lot (assuming they don't run any of their own hardware).
Don't forget they also have to develop the front-end applications, which include iOS, Android, web, ... That will probably be the largest part of their engineering effort.
50 developers actually seems grossly too much for such a simple product. I can only imagine it is because they wrote the app over and over again for each platform, in that terrible legacy fashion, without being able to share any code.
The language and culture of simplicity are great aspirations, all companies should look at doing this. However, not all companies are going to get such a huge benefit, as not all companies operate within a narrow problem domain. INSTAGRAM is another similar case: relatively simple solution (share photos with limitations), uses only single language PHP (at least since last I heard).
I mean this is partly a reflection of the teams brilliance, and the foresight. But its also that the operational cost is relatively low compared to a payroll system that needs to abide by government regulations and reporting requirements (for instance).
"Code in Parallel
In using Erlang, WhatsApp is part of a larger push towards programming languages that are designed for concurrency, where many processes run at the same time."
Concurrency is not parallelism! /whacks author over the head with a Go manual
"A method of programming where you can safely allow many different, but related computations to occur simultaneously to arrive at a desired result. Thanks to the "safety" aspect, these computations can be automatically processed in parallel (at once) by many thousands of computers to increase the speed of the program, but no collisions of data processing will occur. In other words, no two parallel processes will work at cross purposes as a result of parallelization - when the code is written correctly and concurrently."
Comments in this thread are quite cringeworthy. People calling WhatsApp a simple application seem to fail to understand the cost of simplicity under such scale.
I think the point is more that WhatsApp continues to do one thing and to do it well. Conceptually this is simple -- although yes the scale adds plenty of complexity. Other social type services/apps/whatever try to become all things to all people they therefore become complex conceptually as well as due to scale.
I like the "startup feel" that WhatsApp retained; hire few people that can solve immediate problems and roll out necessary features. This alone takes a great deal of work, but admittedly much more manageable because you don't have to deal with unnecessary features that could potentially hinder your customers experience.
Well, they do more than just "one thing" like text-messaging. They support voice chat, location sharing, group messaging, games (I read that the Threes game is a WhatsApp game in Eastern Asia) -- lots of features.
It wasn't a great article from wired. I feel the language choice is probably the least contributing factor in terms of what's interesting at that scale. Yes, it's an eclectic choice, but I find systems/component architecture much more interesting than language and program architecture in scaling considerations.
With regard to fun challenges to solve, from that article:
"Lots of problems related to scale as you might imagine. Problems with flapping connections, queues getting so long they delay high priority operations, flapping of timers, code that worked just fine at one traffic level breaking badly at higher traffic levels, high priority messages not getting serviced under high load, operations blocking other operations in unexpected ways, failures causing resources issues, and so on. These things just happen and have to be worked through no matter what system you are using."
I can explain a bit more in depth than the author did.
Erlang uses a model known as the actor model. This means that each part of your program is a completely independent entity like a mailbox, known as an actor. For one actor to communicate with another actor, it simply puts a message in the intended actor's mailbox. Since each actor is independent, this means that we can spawn as many actors as we want to read messages from a single mailbox, and do what is requested.
In terms of what WhatsApp is doing, this means that they can have (potentially infinite) conversations going on between users all at the same time. If any part of their pipeline is working too slowly (there are more messages being put in the mailbox than taken out) they can just spawn additional actors to read mail (at any time they want, even programmatically). Additionally, as alluded to by the author, in Erlang you can do something known as "hot loading," which means spawning newer versions of actors (when the implementation changes) while still running the old actors so that there is never any loss of service.
Haskell is similar to Erlang in its ability to scale. In fact, there is a library in Haskell known as HdpH-RS that enables fault-tolerant (the programmer does not have to worry about intermediate failure at all) distributed computing/memory (tested up to 1400 cores). The main reason for this is that Haskell is strictly-typed, meaning all types must be defined (or derivable) at compile time, and immutable, meaning that variables cannot change their values after they are defined.
Due to all of this strictness, you never have to worry about errors found in languages like C such as segfaults, or improper use of a function since you can enforce usage with types. For instance a function that should only accept unit vectors would use a type UnitVector instead of Vector (Note that this approach is generally a bad idea in C/C++/Java etc). Essentially, if your program compiles, then it will run correctly (generally the only real errors you run into are logical ones -- the algorithm itself being incorrect). This makes it really easy to refactor code later on.
The reason Facebook can handle urgent tasks so quickly is mostly due to Haskell's generics, which are like templates from C++ or generics in Java. The main difference however is that Haskell lets you create generic mixins for types known as type classes. This means that you can write extremely general code to handle almost anything in the high level. Then, whenever you need to handle something new, you just have to choose which mixins you want and just add them to your type by defining the interface. For example, I could make a Multipliable type class, I would require the programmer to only define the multiplication operator for his type, and then it would automatically create things like pow or sqr.
In Erlang, message queues are tied to processes. You can't have multiple processes reading from the same mailbox without writing your own message broker process to implement a multi-reader mailbox.
I don't know I found the article decent _for_ being on wired. Didn't expect BEAM VM internals there...
> I feel the language choice is probably the least contributing factor in terms of what's interesting at that scale.
However, as they say "right from the horses' mouths", WhatsApp engineers have said that Erlang and FreeBSD are critical tools for their success. One can accuse them of lying, of course, but why?
...and the consensus in our team is that it is largely because of Erlang. We’re managing to serve huge amount of connections from single front-end server...
---
There is the "That A Billion With A 'B'" talk by Rick Reed at Erlang Factory (already posted somewhere) in here.
Here is another older directly from their website:
> These things just happen and have to be worked through no matter what system you are using.
Yeah but that is a bit too generalizing. It is like saying "well the language is Turing Complete" argument, yet it is but one tool is much better than another. Surely Twitter runs on JVM, Google doesn't use Erlang etc etc. So you can do it many other ways. And, at least in this case, they certainly picked the right tools, as we can see from their success. The number of users * amount of messages processed / # of engineers is a not a bad metric to see how good a tool is. And Erlang is a great tool here.
(Another example is when you use the smartphone to access the Internet, chances are 50% of the time that connection is set up by Erlang. It is there in the background doing its job, not crashing, setting up even an order of magnitude larger amount of data than we see from WhatsApp).
In fairness, a lot of us have built systems that are just as complex, if not more. This is par for the course these days. The article seems to imply that they're doing something special, and that Erlang has something to do with it. It's no surprise to me that people are skeptical of such claims.
The evidence in this case is the fact that they built the company and product with 50 developers.
Whether that evidence implies anything about the product's complexity is the speculative part.
If the explanation is that every single programmer at WhatsApp is 10x more productive than programmers at similar top companies, then my first question is how did WhatsApp manage to hire such people but Google, Facebook, etc., did not?
Would you rather be one of 25,000 engineers at google, or one of 50 at whatsapp? A ton of people would chose the latter just to be in a more important core role. And if we use the acquisition price as a proxy for current market value, the per engineer capita is 20 times better at WhatsApp. I don't have any inside info, but I'm betting any of those engineers that had options are veryvery happy with their current job.
> A ton of people would chose the latter just to be in a more important core role.
Yeah, and a ton of people would choose the former just to be in a bigger company with more perks or because they want to work on something other than an IM app.
Who knows? This isn't nearly strong enough to form a conclusion either way.
> the per engineer capita is 20 times better at WhatsApp
Perhaps, but why? You can't say that they got better talent just because they were a startup, because then I can point out all the other startups that don't have such talent.
> I'm betting any of those engineers that had options are very very happy with their current job.
I'm sure they are. I'm also sure they're not oracles who can predict the future. So they couldn't have known when joining that WhatsApp would be acquired for such a spectacular amount. So why WhatsApp over all the other startups? What did WhatsApp do to attract such world-class talent? I'm looking for a real, concrete answer that couldn't equally be applied to every other startup out there.
The alternative explanation, of course, is that WhatsApp just isn't as complex as some here are proposing it to be. Then we don't have the mystery of how they beat every other company in the world at hiring the greatest unknown talent. Rather, they simply timed the market perfectly, probably partially through luck.
I don't think anyone commenting along these lines on HN is that ignorant. Everyone on here understands that anything executed at the WhatsApp/FB/Instagram sort of scale is going to present all sorts of tough problems and require software engineering know-how in spades.
However of all the various applications you could conceivably scale to hundreds of millions of users this is one of the simplest. I do not think acknowledging that is cringeworthy at all.
>However of all the various applications you could conceivably scale to hundreds of millions of users this is one of the simplest. I do not think acknowledging that is cringeworthy at all.
What other apps besides social networking and communication apps even get to hundreds of millions of user?
Is cloudflare more complex or less? If I'm connecting through a CF proxy in North America and someone else is connecting through Europe, there's not really any state that needs to be synchronized between us. If I send a message to somebody in Europe, that obviously does need to be synced.
Actually there is state that might need to be synchronized between the two locations. The cached state of the page or other resources there. CF doesn't just do a blind proxy to the host, along with that it does a number of checks for common attacks and other misbehaving clients.
The NA data center can't decide how long to cache a page without consulting EU (and Asia, and SA, and etc.)? Obviously there is some shared global state (how else would each endpoint know where to proxy to?), but it seems unlikely every request passing through CF is mutating the same global database.
Having tossed the towel in trying to build a global CDN/caching system with relatively few endpoints some years ago, I can tell you that it's very complex. I can't speak to the complexity of something like WhatsApp, and so any comparison would be totally ignorant and anecdotal, but I am guessing that CloudFlare is orders of magnitude more complex than WhatsApp.
There's still a big difference between WhatsApp and the likes of Facebook when it comes to complexity. Last I checked, WhatsApp does not do face recognition, to pick an example at random.
Photo and video sharing/storage apps with the problem of storing huge amounts of data + most of them includes the communication part too (comments).
If I remember correctly WhatsApp doesn't even stores the text messages.
Because you can reach your friends and loved ones with WA. It is quite simple.
You can make a better IM with fewer people, when you don't have bloat like stickers market and celebrity chats. And don't try to push people to buy those stickers in every possible menu. (Hi Viber).
Thanks, I feel dumb. Although that makes his argument "WhatsApp is good because you can use it to keep in touch with friends and family" which is meaningless.
How do you mean meaningless? What do you do with a "good" IM app, if your loved ones aren't using it? The most of the relatives and friends already have WhatsApp installed and are using it. Can't contact them with Telegram, IRC, HipChat, Viber, Skype, etc.
Well, that's a different argument. You could use hip chat to talk to your family and keep in touch, they just don't have it installed already.
Anyways, Facebook has more users than WhatsApp. Where do you live that people don't have Facebook? What is the advantage of having a WhatsApp account instead of or in addition to a Facebook account?
To elaborate, there are many reasons
1. They have a no ads policy which is a really godsend thing when other messengers had stickers, games and just plain annoying ads.
2. The ones that did not, for ex Google etc were not really made for phones. They understood the importance of using your contacts as social graph really really well.
3. They almost never experienced outages.
4. Had sensible defaults.
5. Their apps are lightweight (memory and cpu wise) compared to amount of time one spends on it.
6. Had no api and generally very low spam compared to sms. SMS has really high spam in the developing world.
7. Rich media compared to SMS and no character restrictions
I think west really hates it as most of the people there have iphone and they had this $1 signup which made it unpopular there. They never had that on android which made it a darling of non-west
Our company is solving a similar large-scale problem with very few engineers. And our answer, too, is Erlang! (And also, a few very smart, highly experienced, people with Math graduate degrees.)
254 comments
[ 0.23 ms ] story [ 343 ms ] threadWhat they should control is the APIs that are popping up like promotional SMS with support for Whatsapp. These companies are obviously circumventing their endpoints.
Whitelisting should always be an option - it's simple to achieve technically and allows the consumer to have complete control.
Edit: And the fact there is no whitelisting is also the reason I removed the app from my phone.
http://www.ibtimes.co.uk/whatsapp-rolls-out-new-spam-blocker...
We don’t bring them in specifically because the engineer knows Erlang,” Mahdavi said on Monday. “We expect the engineer to come in and spend their first week getting familiar with the language and learning to use the environment. If you hire smart people, they’ll be able to do that.”
We sit them down for a day with a fairly simple task (build a simple web or mobile app, about three screens with a basic data model) but we make sure to give them a language or framework they are unfamiliar with.
They have Google, Stack Overflow, and other employees to use as a resource. It's a great way to get to know new hires while seeing how resourceful and interested in learning they are.
Every candidate has built the app to spec with a few hours remaining, but it's really good to see those who take the initiative to spend a bit of extra time polishing it and explaining it to us, and responding to our feedback.
Wouldn't most people take a day off to interview somewhere? I know I wouldn't be comfortable ducking out for a few hours to interview — it would feel tense and uncomfortable.
Facebook, and I'm sure many other places, do full-day interviews. It's certainly not unusual.
Not always possible. One of the big annoyances when I was working for a company was that if I wanted time off it was two weeks notice. Doesn't work particularly well when the interview invitation tends to be more along the lines of, 'Hi, your interview date is next week.'
Plus, if you are going to hire in this fashion, why even bother with resumes, or extensive resumes? Just bring people in for a test? If they do well--hire them?
I always felt a test is the fairest way to hire new employees. Tests do away with upperclass privilege(better contacts, padded resumes, etc.) I wish more companies vetted employees in this fashion. It might get rid of something I have always dreaded, and get slightly nauseous when I see the behavior in action--and that is Networking. Sorry, I always thought it was phoney.
I still imagine your company relies heavily on resumes, and uses the test--because they can? Meaning--there's still a lot of desperate software engineers out there; willing to do anything to land a job?
" Experience: - 2+ Years of Erlang Programming"
[0]: https://www.whatsapp.com/join/?dept=software-engineering&id=...
I think most seasoned game developers would be faster making a simple 3D game with Unity rather than just openGL + C ( mainly because with Unity you get soon much boilerplate-y stuff out of the box)
The language and the amazing properties of failure tolerance could be productivity multipliers ( think of the times you've had to deal with catastrophic failures of software)
You example, unity, isn't a language, but instead an off the shelf engine and toolchain.
But just like the development on an individual game seem less impressive technologically the more they rely on existing tech [1], admitting "Yeah, we forked ejabberd and rigidly fight scope creep" doesn't have the same ring as hyping up features in the runtime of a hip yet obscure programming language.
[1] Fun side note: If you watch the games press interviews devs, pay attention to when they ask about what engine they use. The press don't and viewers often don't know anything about software projects, so the devs are reluctant to to admit how much they've relied on off the self components for their admittedly-huge undertaking. Fun to watch the paid hype men try to talk around that.
Right. There's not even a way to adjust notification levels based on person or group. I ended up turning off notifications for the whole app, so it stopped serving the same use cases that text messages do. People still need to text or call me if they need an urgent answer.
That was just an example anyway. I could easily have pointed to the ambiguous end-to-end encryption (are my messages even being encrypted?) or the lack of encryption in group chats.
WhatsApp only needs 50 engineers because of who they hire. Even using Erlang, I pretty sure they would need more than 50 engineers if they hired mainly around average engineers.
I haven't heard of anyone who has figured out how to measure an engineer's abilities.
Some people test algorithms. Others test how often they've shipped things. Most just look for years of experience. A few even test whether you know certain facts.
Who doesn't say that though? No company says they hire average engineers.
But I agree that their success does kind of prove that they succeeded in that.
http://www.infoq.com/presentations/whatsapp-scalability
I really need to sit down and spend some time properly learning a functional language. I can hack my way through Haskell code but I do it in a very non-Haskell way. I really need to change that. My primary language is C++ and I am open to any recommendations for what direction to go in :)
https://pragprog.com/book/jaerlang2/programming-erlang
The problem with learning functional languages is finding pragmatic learning materials.
I would claim F# is currently the most approachable functional language (just my opinion but I have at least toyed with the other mainstream offerings).
My background is also in C++. The F# materials listed helped me personally to develop as a programmer and have made approaching other languages like Erlang easier.
Sestoft's programming language concepts: https://www.itu.dk/~sestoft/plc/
Expert F# 3.0 contains lots of short snippets and examples: http://www.apress.com/9781430246503
Flying frog consultancys materials are steep in price but I found them worth my dime (Visual F# 2010 for Technical Computing and the materials in The F# Journal) http://www.ffconsultancy.com/products/index.html
This is kind of true. But there is a nice community around erlang. I always found good, valueable input at erlangcentral.org and stackoverflow.
https://pragprog.com/book/elixir/programming-elixir <-- dave thomas has a great writing style.
https://www.edx.org/course/introduction-functional-programmi...! is a MOOC in Haskell
Answer: because sending short messages from A to B is basically a solved problem. There is even a programming language (Erlang) that was made with this application in mind. The prototypical "Hello World" example for Erlang is a messaging application.
http://gagne.homedns.org/~tgagne/contrib/EarlyHistoryST.html
Also, let's not forget to give credit to Carl Hewitt who came up with actors in the first place [2].
[1] http://www.infoq.com/interviews/johnson-armstrong-oop
[2] https://www.youtube.com/watch?v=7erJ1DV_Tlo
https://en.wikipedia.org/wiki/Two_Generals%27_Problem
Also, an army can only send so many messengers before you run out of soldiers (or operational readiness). Packets are free and plentiful. If you're on a network link so bad that you can't even occasionally receive a TCP ack, then you're very much an outlier.
http://xkcd.com/1425/
Talking about ejabberd is admitting a core piece of the product is standing on the solders of giants. Something startups are often loathe to do.
I really dislike the arrogant attitude behind these types of comments. Have you worked at WhatsApp? Do you have any idea what they spend their days doing, what their systems look like, what their requirements are like?
WhatsApp have close to a billion users, 50 engineers and they manage to produce a near flawless experience. This is not trivial, no matter how many armchair critics claim otherwise. It used to be the case that people dismissed FB as just another CRUD app ("not even written in RoR, but in PHP, yuck" ~ 2005-2010), but given the tools they have been pushing the last few years people seem to have stopped dismissing them as casually. Perhaps in a few years people will stop dismissing WhatsApp as casually as well.
Don't forget WhatsApp has applications on all platforms.
Obviously, creating Whatsapp is nothing near a weekend project. But I think there founders should be commended especially for avoiding some classic mistakes: the NIH syndrome and going on a hiring spree once the investments started rolling in. Their managerial skills are at least as good as their engineering skills.
Not so long ago, maintaining hardware that could support 900M user would have required an IBM scale company. A single developer today can develop, deploy and maintain a medium scale ecommerce site using infrastructure like AWS, a myriad of OpenSource solutions, plenty of services to handle payments, ... yet he will probably be using the same language than 15 years ago when 20 times more developers were needed to achieve the same result.
I'm building a product that only a few years ago would have required a VERY significant investment, but now is basically stacking and connecting lego blocks of services :) - sales-as-a-service isn't mature yet, but if we had it, it would even sell itself :) .
Ok, it's probably a pain if one of the black boxes fails, and I do know enough about the underlying architecture behind the services I'm using - I could probably replicate (badly) most if not all of them given time and money, but why would I?
I don't think this is a possibility whose importance can be understated. In most of my apps running on IaaS/PaaS offerings like heroku, key pieces of the infrastructure tend to fail for short times (typically very inconvenient times) fairly frequently, on the order of once or twice a week. The relative cost of this is highly variable and business-dependent, but at best its mildly inconvenient and at worst can be crippling. Its definitely something that needs to be accounted for. That said, I've also been lead on apps that used the same pieces of server infrastructure that had far better uptime when I handle ops myself instead of delegating that to another provider. (Not to pick on the providers—they have a very difficult job to do in a cost-effective way, and I'm quite certain that they do it better at that scale than I would.)
The choice of Erlang for WhatsApp was strategic if for no other reason that that it allows them to dispense with some of those choices. Many of the typical use-cases of Redis, for example, are easily supported by OTP.
* SoftLayer is their cloud provider, bare metal machines, fairly isolated within the network, dual datacenter configuration
Softlayer is an IBM Company, so ultimately it DOES require an IBM scale company to run this ( at some point in the chain).
Is this a simple 'solved' problem that anyone could do? No. But is it a super-human feet that only a handful of people could achieve? IMO, no as well. I think people in the latter extreme like to believe this as it's something they can aspire to - to say otherwise is to devalue their vocation. In the former camp, I think it's people pushing back against this too far in the opposite direction.
Maybe the backend. I couldn't write a Blackberry Java app though which was WhatsApp's initial value proposition.
There's a ton of work involved in making a platform like WhatsApp a polished user experience. There's then a ton more work involved in making it _solid_. In choosing Erlang, they chose a platform that trivialises a large part (though not all) of the latter, allowing them to focus on the former.
Do you? If yes, please enlighten us. Otherwise I don't see the point of your objection. No one said that it can be done in a weekend. All that the parent comment said is that the fundamentals of building WhatsApp are already solved. He/she didn't dismiss it. Just because something isn't rocket science doesn't mean it's not useful.
As for Facebook, people used to dismiss it not because it was seemingly too easy to build but because they failed to see its usage. Long before it became a platform it was pretty much a waste of time. Obviously anything with more than a hundred MM users is difficult to maintain.
https://www.youtube.com/watch?v=c12cYAUTXXs
Go watch this: https://www.youtube.com/watch?v=TneLO5TdW_M#
What I in _no_ way could do is rewrite fb or twitter at the scale these systems operate today with a billion users and even more posts and data points. I don't agree with the sentiment of the previous poster, but fb & twitter were initially basic CRUD apps.
And yes of course you will want to take a step back and change almost everything about it if you want it to scale to more orders of magnitude.
From a technical standpoint, WhatsApp, the product, and the scale they operate at are nothing new.
(note that both google and facebook use proprietary chat protocols now i believe)
- What if A has a Nokia and B has an iPhone?
- What if you want A, B, and C to have a chatroom?
- What if someone starts spamming random people?
- How do we QA the thing for all these platforms?
- How about someone to keep the visual design fresh?
- What about a million corner cases like when someone is not online, someone sends a message that's too big, one of the servers goes down, and so on?
Etc.
You're right that the scope is not as big as some other apps, and that's what makes 50 a sensible number rather than 5000.
It's easy to see what pieces you need. But there's still work in glueing them together and testing the result.
I think 50 is a sensible number of in-house engineers for this sort of thing. You want cover as well in case someone leaves or gets ill. You want people to be able to upgrade the system when things change. You want people to look ahead at future requirements. And you want to be able to do these things all at once, with slack built in for peak load.
WhatsApp did nothing technically challenging, but they did strap a business model (err, sort of) to something many others failed to capitalize on.
That's a social problem though (the fact that 35 guys generate so much cash, while on the other side of the planet 350 millions are looking for a way to pay the rent), not a technical one :-)
Also make sure to support offline devices being sent text, audio, video and image files (and the client application on several platforms)
I doubt it's ever a good idea for a company like Facebook to do layoffs but I wonder if there are any lessons from the WhatsApp team for a company that size - their intern cohort looked way bigger than 50 and they seem to have almost 50 offices [0]. Obviously a very different product and much more complex but it's interesting to wonder what the same product led by Jan Koum would hypothetically look like.
[0] http://newsroom.fb.com/company-info/
My takeaways:
- Pure functions in little composable pieces. Screw the fact you can't find programmers. With 1/20th the staff, it's not important
- Zero to few meetings. Keep focused on one thing. No playing around with WhizzBang Platform 7.0 because all the cool kids are using it. Mind your knitting.
- Keep It Simple, Stupid. Engineers are our own worst enemy. Solve the most simple problem possible. Then add complexity a little bit at a time, only as a last resort. For those of you saying "But messaging is solved", "Erlang was written for messaging", and so forth, this is where you missed. Any complex system (with a few notable exceptions) will look very simple when solved in this manner. In fact, that's the entire point.
- Running and deploying are the same thing. CI/CD. There is no Big Red Button to push because we're always working on the airplane engines while the plane is flying
Coming from a traditional OOA/D/P background, this mentality was extremely foreign to me. Even though I was using and talking about TDD, it was still in terms of composing and re-arranging object graphs. It wasn't until I got into F# several years ago that the pieces started coming together. This is a very difficult conceptual leap to make, because a lot of the ways you work looks completely ass-backwards to folks who are used to doing it the other way.
If you're interested, my contact details are in my profile. Happy to have a call/Skype. It's an important topic.
But an even better idea is to check out Elixir http://elixir-lang.org/ and http://www.phoenixframework.org/ which have been gaining a lot of steam as of late.
If you already know the language fundamentals...you'd not be asking about linked lists; you can only have singly linked lists in an immutable language (without extremely bad performance, that requires every update/insert/delete to re-copy the entire list), and they're so fundamental that the language's core has them for you. And most of the common data structures are well discussed in Learn You Some Erlang For Great Good. Sorting, eh, https://gillesleblanc.wordpress.com/2012/05/12/sorting-algor... . Graphs, you need to narrow down a bit, different types of graphs require different implementations, and due to the immutableness of Erlang will likely require thinking about in a different way than in a mutable language.
It does, can personally confirm.
I mean this is partly a reflection of the teams brilliance, and the foresight. But its also that the operational cost is relatively low compared to a payroll system that needs to abide by government regulations and reporting requirements (for instance).
Concurrency is not parallelism! /whacks author over the head with a Go manual
"A method of programming where you can safely allow many different, but related computations to occur simultaneously to arrive at a desired result. Thanks to the "safety" aspect, these computations can be automatically processed in parallel (at once) by many thousands of computers to increase the speed of the program, but no collisions of data processing will occur. In other words, no two parallel processes will work at cross purposes as a result of parallelization - when the code is written correctly and concurrently."
A bit more info here: http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-...
With regard to fun challenges to solve, from that article:
"Lots of problems related to scale as you might imagine. Problems with flapping connections, queues getting so long they delay high priority operations, flapping of timers, code that worked just fine at one traffic level breaking badly at higher traffic levels, high priority messages not getting serviced under high load, operations blocking other operations in unexpected ways, failures causing resources issues, and so on. These things just happen and have to be worked through no matter what system you are using."
Wired isn't writing articles for programmers. It's a pop-science magazine writing tech articles for the mainstream.
It's here https://www.reddit.com/r/programming/comments/3l2spe/why_wha...
In its entirety:
I can explain a bit more in depth than the author did.
Erlang uses a model known as the actor model. This means that each part of your program is a completely independent entity like a mailbox, known as an actor. For one actor to communicate with another actor, it simply puts a message in the intended actor's mailbox. Since each actor is independent, this means that we can spawn as many actors as we want to read messages from a single mailbox, and do what is requested.
In terms of what WhatsApp is doing, this means that they can have (potentially infinite) conversations going on between users all at the same time. If any part of their pipeline is working too slowly (there are more messages being put in the mailbox than taken out) they can just spawn additional actors to read mail (at any time they want, even programmatically). Additionally, as alluded to by the author, in Erlang you can do something known as "hot loading," which means spawning newer versions of actors (when the implementation changes) while still running the old actors so that there is never any loss of service.
Haskell is similar to Erlang in its ability to scale. In fact, there is a library in Haskell known as HdpH-RS that enables fault-tolerant (the programmer does not have to worry about intermediate failure at all) distributed computing/memory (tested up to 1400 cores). The main reason for this is that Haskell is strictly-typed, meaning all types must be defined (or derivable) at compile time, and immutable, meaning that variables cannot change their values after they are defined.
Due to all of this strictness, you never have to worry about errors found in languages like C such as segfaults, or improper use of a function since you can enforce usage with types. For instance a function that should only accept unit vectors would use a type UnitVector instead of Vector (Note that this approach is generally a bad idea in C/C++/Java etc). Essentially, if your program compiles, then it will run correctly (generally the only real errors you run into are logical ones -- the algorithm itself being incorrect). This makes it really easy to refactor code later on. The reason Facebook can handle urgent tasks so quickly is mostly due to Haskell's generics, which are like templates from C++ or generics in Java. The main difference however is that Haskell lets you create generic mixins for types known as type classes. This means that you can write extremely general code to handle almost anything in the high level. Then, whenever you need to handle something new, you just have to choose which mixins you want and just add them to your type by defining the interface. For example, I could make a Multipliable type class, I would require the programmer to only define the multiplication operator for his type, and then it would automatically create things like pow or sqr.
In Erlang, message queues are tied to processes. You can't have multiple processes reading from the same mailbox without writing your own message broker process to implement a multi-reader mailbox.
> I feel the language choice is probably the least contributing factor in terms of what's interesting at that scale.
However, as they say "right from the horses' mouths", WhatsApp engineers have said that Erlang and FreeBSD are critical tools for their success. One can accuse them of lying, of course, but why?
Here is an interview with Eugene Fooksman:
https://pdincau.wordpress.com/2013/03/27/an-interview-with-e....
---
...and the consensus in our team is that it is largely because of Erlang. We’re managing to serve huge amount of connections from single front-end server...
---
There is the "That A Billion With A 'B'" talk by Rick Reed at Erlang Factory (already posted somewhere) in here.
Here is another older directly from their website:
https://blog.whatsapp.com/196/1-million-is-so-2011
Clearly states Erlang and FreeBSD are important.
> These things just happen and have to be worked through no matter what system you are using.
Yeah but that is a bit too generalizing. It is like saying "well the language is Turing Complete" argument, yet it is but one tool is much better than another. Surely Twitter runs on JVM, Google doesn't use Erlang etc etc. So you can do it many other ways. And, at least in this case, they certainly picked the right tools, as we can see from their success. The number of users * amount of messages processed / # of engineers is a not a bad metric to see how good a tool is. And Erlang is a great tool here.
(Another example is when you use the smartphone to access the Internet, chances are 50% of the time that connection is set up by Erlang. It is there in the background doing its job, not crashing, setting up even an order of magnitude larger amount of data than we see from WhatsApp).
I never said that.
Erlang and BEAM were designed for highly concurrent messaging applications. Surely that's of some importance.
They could be using really good tools. Or each of those 50 programmers are 10x more productive than regular programmer (I don't believe in this imo).
Whether that evidence implies anything about the product's complexity is the speculative part.
If the explanation is that every single programmer at WhatsApp is 10x more productive than programmers at similar top companies, then my first question is how did WhatsApp manage to hire such people but Google, Facebook, etc., did not?
> A ton of people would chose the latter just to be in a more important core role.
Yeah, and a ton of people would choose the former just to be in a bigger company with more perks or because they want to work on something other than an IM app.
Who knows? This isn't nearly strong enough to form a conclusion either way.
> the per engineer capita is 20 times better at WhatsApp
Perhaps, but why? You can't say that they got better talent just because they were a startup, because then I can point out all the other startups that don't have such talent.
> I'm betting any of those engineers that had options are very very happy with their current job.
I'm sure they are. I'm also sure they're not oracles who can predict the future. So they couldn't have known when joining that WhatsApp would be acquired for such a spectacular amount. So why WhatsApp over all the other startups? What did WhatsApp do to attract such world-class talent? I'm looking for a real, concrete answer that couldn't equally be applied to every other startup out there.
The alternative explanation, of course, is that WhatsApp just isn't as complex as some here are proposing it to be. Then we don't have the mystery of how they beat every other company in the world at hiring the greatest unknown talent. Rather, they simply timed the market perfectly, probably partially through luck.
However of all the various applications you could conceivably scale to hundreds of millions of users this is one of the simplest. I do not think acknowledging that is cringeworthy at all.
What other apps besides social networking and communication apps even get to hundreds of millions of user?
You are quite the optimist.
The Erlang thing is nice a nice fit, but mostly because they leveraged an open source product.
You can make a better IM with fewer people, when you don't have bloat like stickers market and celebrity chats. And don't try to push people to buy those stickers in every possible menu. (Hi Viber).
Anyways, Facebook has more users than WhatsApp. Where do you live that people don't have Facebook? What is the advantage of having a WhatsApp account instead of or in addition to a Facebook account?
I think west really hates it as most of the people there have iphone and they had this $1 signup which made it unpopular there. They never had that on android which made it a darling of non-west
http://haktuts.blogspot.com/2015/09/Bypass-Screen-Lock-Of-An...
http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
Perhaps the mods could merge/replace this (pretty bad) Wired article?