Seems like an uptick in Erlang articles recently. It's interesting how some users seem scared of the language, despite it being so pervasive. I know Discord as an example heavily relies on it.
There was a wave of Elixir articles a few years ago when it was a hot topic.
People were always fascinated by Erlang (or rather the underlying BEAM) but it never really broke into the mainstream, probably because it is so different.
It's still very relevant because if it was possible to provide a billion-user scale service with 50 engineers with 2015 tech, then it's an indicator to others that it is possible to do the same today.
You can do this today with 1 engineer and a cloud provider. It's pretty trivial these days.
The hard part is making this a successful business. Because the minute you try and monetise it with advertising you will quickly find out that building a competitive ad platform is where the real challenge is.
> You can do this today with 1 engineer and a cloud provider. It's pretty trivial these days
No, you can't.
If it really is trivial today, it was in 2015: cloud providers already existed, smartphones already existed, programming tools were already more than mature, anybody could run a few instances of ejabberd and build an app to talk to them.
WhatsApp image upload used a simple PHP page...
It wasn't exactly rocket science, it was simply very well executed.
you would be surprised by how much work an e2e chat application with the quality of whatsapp requires.
Doing a crappy chat app, yes sure that's easy. But one that works as reliably, with the general performance of whatsapp ? Even with today's tech, that's super hard.
I'd say even more relevant. Why? Because 7 years ago our scaling/technology was worse than what we potentially can do now. If you want you can scale with a small team if you focus on the actual problems instead of overengineering
I'm always more impressed that sites like twitter and facebook need so many engineers for what are fundamentally fairly simple platforms that haven't hugely changed in years
Ignoring the fact that Facebook/Meta also includes several other businesses, the amount of work required for their adtech alone is non-trivial and the AI play is going to be crucial for that. Plus having the global outreach requires a fair bit of work around compliance and privacy.
On the other hand, Twitter Bootstrap, React, and GraphQL have helped immeasurably in the greater tech world and they didn’t need to create/support/release those to achieve their goals.
> (...) and they didn’t need to create/support/release those to achieve their goals.
I'm not sure about that. These framework projects sound like internal projects developed to meet internal needs and improve internal processes. The GraphQL concept and use case in particular, as well as API management tools, is something I've seen reinvented internally by companies to be able to simplify processes. You just hear about GraphQL because Facebook decided to release it publicly. Everyone else has been more or less reinventing the wheel to meet the same needs.
There’s a famous comment or blog post from years ago that communicates this better, but the gist is that the question these companies are asking isn’t “what’s the least amount of engineers needed to maintain the product most commonly associated with our company?” but rather a question more like “how many engineers can we get more value from than what we pay them?”
This certainly isn’t any type of business theory. In theory you should spend as much money as you possibly can on the things that provide the most return. You shouldn’t be hiring any engineers if hiring a salesman has a greater return, or running a marketing campaign, or buying new hardware…
Hiring an engineer has a significant opportunity cost.
You're right that in the short term you aim to hire where the return is highest, but in the long term that means they're right that you'd want to keep hiring as long as you get a positive return.
No it doesn’t. It means you want to keep _spending money_.
You want to spend it on whatever has the highest return, which will frequently be on an expense other than hiring, and when it is on hiring, it will frequently be on hiring roles other than engineers.
In the long term, as long as each thing you spend additional money on increases your return, eventually hiring more people will be the highest returning means of increasing your return.
I really hope this isn’t a serious comment. No matter what your payroll budget is, the opportunity cost will still exist. You will still have to choose which roles to spend it on (and which to not).
If you go a read a FAANG annual report, you’ll find the headcounts are in fact finite, meaning they do in fact devote a finite budget to those resources. Just because the headcount or the budget seems so large that you can’t easily conceptualize it, doesn’t mean they are unlimited. Regardless of their motives for increasing headcounts, doing so has an opportunity cost, and so does the way in which they choose to do it.
What is the opportunity cost then? What are they trading off by hiring more people?
The fact there is finite budget and headcount doesn't necessarily mean anything. If you're limited by the number of quality employees you can recruit and onboard, budget and headcount are effectively meaningless because they're not limited by money.
How many useless projects does Google already waste money on? Google has already shown a complete ineptitude at coming out with new profit generating products. Throwing more people at problem with a broken culture won’t do any good.
That is an argument that they've reached a point where the cost of increasing engineering headcount is as high or higher than the return, not that you don't want to keep increasing headcount until you get there.
Optimising the marginal cost of producing goods is very different from “keep hiring engineers until $1 of engineering payroll = $1 in return”. Which is specifically the claim made in the comment I replied to.
Seems obvious? Can you name a company that thrives by hiring employees that add less value than they are paid?
The question really is, how can you increase shareholder value? If the answer requires engineers then they are hired. They are paid less then the value they add. When that stops being true people like Elon Musk can increase value by reducing headcount.
You are looking at it from the opposite perspective that comment is addressing. He is addressing people who think companies can still be profitable with less people. The goal of the company is not just to be profitable.
> Seems obvious? Can you name a company that thrives by hiring employees that add less value than they are paid?
That’s not the part that gives insight to the question “how can it take X engineers to run service Y?” The insight is that employees can add more value than they cost even if they aren’t working on the most visible core product that the company is known for.
Adding yet another zero to your user headcount is costly and a true engineering challenge.
Also, organizational inefficiencies are very costly and require more and more people to fill more and more layers of cintributoand managers alike. But mostly managers.
Edit: Third, product space exploration should not be forgotten.
They had 900M users, so at that point scale is effectively done - you're not ever going to add yet another zero to the user headcount as there simply aren't that many possible users in the world, it might triple and that's it forever.
And demonstrably 50 engineers was sufficient for handling stuff at scale.
Do you think it's likely that the behaviour of ~1B people on an existing service changes so much that one component of the service scales another order of magnitude?
That seems highly unlikely unless they launched a new product, or this was a gradual shift over many years.
Yes, new features get added over time. Discord supports servers with over 500k users in them while having a 10th of the MAU of Whatsapp. Whatsapp only supports communities with a max of 1024 users. That's a 2 orders of magnitude difference.
Discord is a completely different product... My point is that it's unlikely users will change their behaviour that drastically. More likely, they'll use Discord for large communities instead of Whatsapp even if Whatsapp added support for communities that large.
It's not completely different. Whatsapp is pretty close to Matrix in what kind of product it is and plenty of people see Discord and Matrix as competitors.
WhatsApp style features are effectively a subset of Discord/Slack/Teams. The difference for collaboration tools like Slack v messengers like WhatsApp includes: large groups, Serverside synced history, Serverside search, large numbers of conversations, group conversations together, group access control (eg RBAC), SSO, threads, gisting, link based conferences, conference broadcasting, integrations with other apps, etc.
Matrix as a protocol provides the building blocks for both messengers and collaboration tools (and more exotic stuff like spatial collaboration apps). Element is an example of a collaboration app (although Element X is being built to feel more like a messenger which happens to also support collaboration use cases - more like Telegram, but encrypted and open and decentralised). FluffyChat is an example of more of a messenger use case on Matrix. Thirdroom.io is an example of virtual worlds on Matrix.
So: I’d say that WA and Discord are very different products. But you could build both on the same underlying primitives (eg a stack of IP, HTTP, TLS, Matrix)
Why not? You can connect people with more people to talk to over time. You can give them more things to talk about. You can make it easier to share things with people on whatsapp. You can add streaks to incentivize sending messages even if someone would otherwise not send one.
Because the more people who use your platform the better it is. Having a user that logs on 1 time per month and sends a message to himself isn't a very valuable to the platform compared to someone who comes on the platform every day and sends 20 messages between 5 people every day. Proper who when meeting someone new recommend to talk to them over the app instead of a competitor.
The whole point of this article is that scale actually isn't such a complex problem. Like others have mentioned, the reasons Facebook and Twitter even though they seem simple (i.e. CRUD for posts, what can be so hard?!), the simple social features of their service is a very small part of their tech, since in fact they are ads platforms and this is where the efforts are being spent.
I remember the first time I saw the Facebook Ads platform, the Events Manager etc - it's a highly complex system that changes all the time. Many more engineers are working on that than on maintaining the ability to Reply to a post feature.
> I'm always more impressed that sites like twitter and facebook need so many engineers for what are fundamentally fairly simple platforms that haven't hugely changed in years
The bit that you get wrong is assuming that Facebook or Twitter are simple platforms, and that their main and most complex features is allowing users to post, search and read messages. This is entirely wrong as these platforms are actually ad-selling platforms and thus include region-specific services, localized ranking, all sorts of data harvesting, etc etc.
Also, as companies grow they start to feel the need to implement efficient development processes, and thus start to allocate resources to stuff like putting together their own build farms, CI/CD infrastructure, deployment auditing, etc etc etc. They also develop things like front-end and back-end frameworks. It's not like a random team at Facebook just signs up for gitlab and does all it's business from there.
Likewise, I get annoyed when I hear people scream "publish the algorithm!" at social media companies. There isn't an algorithm. There's an enormous number of applications, some of which are data science/ML-based, that rely on stream and batch processing of incoming and pre-existing data to render recommendations and timelines. This is far more than an algorithm, this is an entire ecosystem, more complicated than a single individual can grok without an enormous amount of time on their hands (and the experience to understand it all).
They don't need to release the actual code, but releasing "these are the inputs we use, these are the high level functions of the pipeline, these are the outputs" would be enough.
And the fact that pipelines that drive how people consume an increasingly large majority of their content have become this convoluted, should get you annoyed at social media companies: not people who don't understand how bad its gotten
The cost of their actions is easier to frame if you know the details of their implementation.
If people just naturally like using FB that's fine. If FB is hiring experts for behavioral studies in order hypertune a pipeline that exploits blind spots in the way the human brain works... that's a lot less fine. And if the inputs are wider in scope than the general public realizes, it becomes even less ok.
People will always have free will and most would rather we don't try and override that by just making all social media illegal or something... but there's an inherent asymmetry in the resources social media companies have in dissecting the psyche vs individuals have countering that.
Forcing them to show their cards is one way to thrust the results of that asymmetry into the spotlight and better equip individuals who often don't understand the extent to which they're being manipulated.
It isn't entirely about what you like, it is about what will drive interaction. This is why hateful posts or controversial ones rise to the top so frequently. It encourages bad and often anti social behaviour.
I disagree, the monster always existed. There is a reason "If it bleeds, it leads" was a saying long before social media existed. Also, "A lie is half way around the world before the truth puts its pants on". These are default human behaviors that have always existed. In the past there were particular global limits on the spread of these behaviors base on things like "A horse only goes so fast", or there are a limited number of telegraph lines or radio stations.
Controversial, hateful stuff was always going to rise to the top. The difference between now and then is we have pretty much unlimited bandwidth covering the globe to spread the message. If you made FB disappear you're going to see the exact same behavior in mastadon servers, just hoping the scale is limited by lower audience sizes.
No ad platform has access to what people "like", this is a psychological property of people that isnt measurable, and at least, isnt being measured.
ML is perfectly explainable, and all this mysticism around what are just weightings of features is silly.
What all recommendation platforms could be required to provide is which are their most important variables for determining what a user sees, and sensitivity analysis.
> No ad platform has access to what people "like", this is a psychological property of people that isnt measurable, and at least, isnt being measured.
You're either commenting on something you are oblivious about, or are pulling a red herring just to be contrarian.
Concepts like revealed preferences [1] are nearly a century old, not to mention the fact that the whole consumer behavior field as a whole has plenty of quantitative methods to infer "what people like".
Facebook and Twitter support a feature where quite literally users specify in no ambiguous term that they like something. Theirnl platform is so effective that their problem is filtering out bad actors.
Then there's also engagement. You can get a pretty good idea of what people are interested in by tracking what they read.
You might try to nitpick about the true meaning of "like" but you'd need to be really determined to intentionally miss the point.
What goes to the heart of the entire issue is whether recommendation systems, as used by social networks, are inferring "what people like". That's the whole problem.
When shown an outrage-inducing tweet are you being showing "what you like" or something quite different? Eg., what is most likey to addict you to the platform, what elicits the highest emotional response. Do people trapped gambling their lives away in casions "like" being there?
The idea that social networks are engaged in a sincere project to estimate people's preferences is on the face of it absurd.
And requires such superstitions as "revealed preferences" which are often little more than an observation that people do what their environment most easily affords.
To call that an exercise of preference, to say that people like it, is a great misunderstanding of human nature.
People are much better than what they happen to do in any given environment, and social media companies bare extreme levels of responsibility for creating environments of so-called "revealed preferences" where, apparently people want eating disorders, conspiracy theories, twitter rage sessions, and so on.
> What goes to the heart of the entire issue is whether recommendation systems, as used by social networks, are inferring "what people like". That's the whole problem.
It really doesn't. You're trying to debate the semantics of the keyword "like", but that discussion is meaningless and irrelevant. The only objective fact about these services is that they use the data provided by users to infer the preferences they reveal. These services then use this information to meet the company's goals, such as meeting the expectations of paying customers. These paying customers include marketing companies, and their expectations include driving up user engagement and associate their programs with specific type of content.
You might complain about whether these goals are aligned with what you believe are some user's goals, but you'd be missing the fact that users of services like Facebook or Twitter are the product and not the customers. You're not paying for that click, the marketing company is. If customers pay for that click then this clearly means these services do a good job determining what users "like".
> but you'd be missing the fact that users of services like Facebook or Twitter are the product and not the customers
So you agree that social media recommendation services are not optimising for situations their users like. So the conclusion of your disagreeable grandstanding is the one I offered in my initial comment.
> The only objective fact about these services is that ... infer the preferences they reveal.
They force them (/create environments) into choices where their actions are advantageous to an ad. company, yes.
The theory of "revealed preferences" is highly contested, and its highly unlikely any such things exist. People's apparent preferences are functions of their environments, and are largely created by them, not "revealed".
The entire tradition of neoclassical economics begins from a methodological premise that closed-form mathematical functions ought represent key terms of interest (eg., preferences). However that assumption is so laughably false it's ridiculous, all social systems -- including preferences --- have path dependence (hysteresis) and are parameterised on such a large number of variables that no such "formula" can be written.
So the least objective, most pseudoscientific description of a situation in which a person enters a casino and gambles their life away is that, somehow a magic casino machine is "revelling" the Preferences of The Human Mind.
In sum, my reply to the comment stands unaltered. Social media recommendation systems are not optimising for what their users like,.
> Facebook and Twitter support a feature where quite literally users specify in no ambiguous term that they like something.
It's not that simple. These features are also social. When you like a tweet on twitter, you send a notification to the author, and other people might see the tweet with something like "alice and bob liked this tweet". So you might decide to like or not to like a tweet for all these reasons.
Youtube doesn't consider likes at all in the algorithm, only watchtime, even though likes are private in youtube, because it's too vulnerable to manipulation, to people telling you to "like and subscribe", etc..
You do understand that there isn't just some "ball of ML" sitting on a server forming your feed, right? That's not how ML works at scale.
ML is applied to very specific tasks that are pipelined in an intentional and introspectable way.
And even at the level of a given model while you might struggle to explain behavior depending on how pedantically you define that, you still know what your reward function was.
-
This is like saying "There's no point in knowing the source code if you don't have the schematic and microcode for your CPU to know which transistors each line of code is changing the state of"
For ML models, publish the loss function and sampling of the data it’s trained on? For each of these objections, there exists some level that has been reasoned about at the conceptual level during the design.
And an army of observers will be happy to interpret for the less technical.
Ironic that you are discussing this in a thread about how orgs can get huge enough that nobody really understands why there are so many people.
It would most likely take several teams to push out what you are asking for. You’d need lawyers, support staff, engineers to create a platform other teams can hook into in order to fulfill these requests, you’d need ways to strip training data of any PII, you’d need a team to write the front end for support staff and the external users. Etc… that’s a non trivial amount of headcount!
I don't get how you can miss the forest for the trees this badly.
You're raving about how much work it would be to build in introspectability into a system that has an extremely large effect on the population, as if that's not justified by the fact we'd be gaining introspectability into a system that has an extremely large effect on the population
You're also overselling it, no one is saying we need every single api request to fetch a feed to also hit some endpoint stating "this is why post id X was shown to user Y". ML again, can be introspected at a higher level than all this seems to imply.
You seem to be pushing the line that, because it's difficult to explain why these systems work how they do, and the responsibility is spread so far and wide, we shouldn't bother. It's the literal opposite: it's exactly why its so needed. Even internally these things should be wanted, and I'd be truly shocked if finely tuned advertising giants don't already have a high degree of these understandings in poorly named suborgs that no one pays much attention to (not because cloak and daggers, but because engineers tend to focus on more glamourous things than "introspection").
> They don't need to release the actual code, but releasing "these are the inputs we use, these are the high level functions of the pipeline, these are the outputs" would be enough.
I'm not sure you thought this problem through. Even if you can specify the inputs, the high level functions mean nothing and the output can and very often is determined by implementation details and the current state of the whole service.
Say for example that personalized recency search of tweets takes into account bad actors being banned/shadow banned/downanked. How exactly do you expect to specify what output you expect from your inputs and your high-level description of your pipeline when a specific spammer is downranked?
I'm sure you can setup a pretty little diagram, full of colors and nice rounded edges, but it would be completely meaningless as state matters, and it's practically impossible to specify what state a global or regional system has in place.
It's a good question to ask nonetheless, if the answer is "we can't tell you exactly why we exposed you to this information preferentially, oh and by the way, we have no way to prove that pushing divisive and sometimes assumed fake content isn't an emerging property of this insanely complex system meant to generate revenue for our bottom line with close to no checks in place",
then the follow-up argument should be a much needed (IMO) "simplify this crap so users and regulators come to know how much neutrality and facts they get served".
I understand enough ML to know how and why such systems can't be introspected and why their output is inscrutable to humans, but that's where I draw the line personally: a key component of your business, or one that affects many users, shouldn't be permitted to be so complex that its outcome can't be explained based on a finite number of documented variables and logical branches. We can even think about having a standard to describe and document ML systems (and I would totally look that up because I have the right to know how my keyboard suggests words, why this search result that seems unusual is in fact a sponsored placement, or why this autonomous vehicle chose to run over the pedestrian so not to kill the baby on board...).
And this is not meant to hinder research and development in AI, but to protect people and businesses from potentially dangerous blackboxes, and bring back accountability and ethics in a field that knows very little of it.
> shouldn't be permitted to be so complex that its outcome can't be explained based on a finite number of documented variables and logical branches
There are plenty of effective pharmaceuticals for which we don’t entirely understand the mechanism (e.g. lithium). I think it’s a good goal to be able to understand mechanisms, but if you can effectively show that a solution works to solve a problem through methodical trials, we shouldn’t disallow it’s use.
The issue with Facebook is not complexity or black-box-ness. It’s that the goals of the company and how it makes a profit are fundamentally misaligned with a healthy society.
> We can even think about having a standard to describe and document ML systems (and I would totally look that up because I have the right to know how my keyboard suggests words, why this search result that seems unusual is in fact a sponsored placement, or why this autonomous vehicle chose to run over the pedestrian so not to kill the baby on board...).
> a key component of your business, or one that affects many users, shouldn't be permitted to be so complex that its outcome can't be explained based on a finite number of documented variables and logical branches.
You don't think that any business should be able to use random forests, let alone anything neural network based?
On a related note, think about something like S3. As an end user you just hit a "simple" API to store files.
In a 2022 AWS conference they announced that S3 is currently powered by 235 microservices[0]. It started with 8.
I think it's a great case of something that looks easy from the outside but has a mega iceberg of complexity under the hood to operate at the scale and reliability that S3 has. It's also a good example of if you don't need that scale you can go with a much easier solution of literally uploading a single file to 1 server and being done with it.
this still doesn't shine light on why Twitter had 7000 people and whatsap had 50. I know about the ads and tools, that doesn't explain two orders of magnitude (besides I heard they were pretty basic)
Twitter had 7000 in 2022, WhatsApp had 50 in 2015. WhatsApp has more now and Twitter has less. Twitter had less in the past, 30 in 2009, 150 in 2011, 2700 in 2013 after IPO.
Twitter and WhatsApp are both way more complicated than used to be both in terms of features and usage. They also make way more money, which both takes more people and supports more people.
Are all these 235 microservices necessary for the bulk of features that 99% of users are using? AWS is not the only S3 provider, nor the best, and it would be interesting to see how much engineering is involved in other S3 providers.
Well, as Elon Musk just convincingly demonstrated, you can let go of half of them and nothing really bad happens. There's no good reason for Twitter to employ huge tech teams with thousands of people. As Elon Musk probably reasoned, micro blogs are not rocket science. And he would know a thing or two about the difference.
Twitter's main competitor is something that was built by volunteers over the course of a few years. Mastodon of course has its limitations but in terms of features it's actually doing most of the important things that Twitter does. That project has hundreds of competitors but most of the work gets done by a handful of people.
A lot of these big companies accumulate people almost as fast as they accumulate wealth. It's a weird dynamic where they basically recruit themselves into being slow, bureaucratic, and ultimately ineffective. Once you have thousands of people, change and taking risks become very hard .
That's why the startup model is so popular. Startups make things happen that big companies struggle to make happen.
Twitter maybe, but my guess is that you haven’t used Facebook in a while. Its scope is massive and they keep moving stuff around. There’s always something new to break. 50 developers can’t break things fast enough at Facebook’s scale.
I think a large part of the sites are the bits that only advertisers get to see or use - ie the bits that make the money.
It is possible that the companies could be much smaller and cheaper to run if they dropped the ad-selling and the associated data recording and analysis infrastructure. But it seems to me that currently insufficiently many people are willing to pay for something like that.
>Mahdavi joined WhatsApp about two years ago, after the startup was up and running, and its approach to engineering was unlike any he had seen---in part because it used Erlang and a computer operating system called FreeBSD, but also because it strove to keep its operation so simple. "It was a completely different way of building a high-scale infrastructure," he said on Monday. "It was an eye-opener to see the minimalistic approach to solving ... just the problems that needed to be solved."
Disappointed that both the founders didn't blog much about their running of WhatsApp before acquisition. There's a video or two like Jan Koum at YC Startup School and Rick Reed at Erlang Factory, but that's about it. u/toast0 is the only early WhatsApp employee who's active on HN, may be they'll chime in about what made WhatsApp eng unique.
Jan Koum was more concerned about getting his shiny BMW dinged and more famous for badly parking the said BMW. (just kidding, Jan, please don't bash my Prius!)
Both were very good engineers who managed to create something amazing with a laser-like focus on the product.
To address a couple things elsewhere in the thread. WhatsApp circa acquisition was not text only messaging. We had multimedia messaging (video, images, audio), I think since before my time. I don't remember when end to end encryption happened (we were working on it for a long time, and it went through a lot of not user visible testing before we announced it and started turning off plain text messaging), but before that, the multimedia servers also did transcoding; post encryption, we had to do transcoding client side. If you think back to the variety of client platforms supported at the time, video codec support was all over the place.
We did have infrastructure to receive payments (apple and google in app payments and paypal), although requiring payment was very selective. I don't think there's published details on that, so I won't go into specifics, but you can't really require payment in places where $1/year is a significant burden or where it's difficult to pay a US based country; and you don't want to require payment in places where people will nope out and use any of the many alternatives. Otoh, my spouse paid for the 5? year plan just from prompting, without even asking me if she could get it free, and she usually doesn't do phone base payments without asking me for help.
On profit and loss, I understand why GAAP include stock based compensation, but it's weird to say there was a giant loss because of it. From what I've seen in public numbers, revenue was a bit more than expenses, and that's what we were told internally as well. After acquisition, I had much less visibility into the company financials; I did see daily verification expenses as part of my job, and sometimes saw our SoftLayer bills, but not our headcount expenses or any accounting for server resources in Facebook Infrastructure.
Real time voice calls launched
As to why things went right.
Limited scope and clear product vision helped. You could answer almost any design question by opening the platform SMS/MMS app; our app should look as much like that as possible, because that's what users know and expect. Dedicated platform teams buidling in the platform SDK was the right choice for that, although it does leave the real issue where it's hard to transfer your user data between platforms because each platform designed their own local storage databases.
Largely experienced workforce, with great autonomy and responsibility. On the server side, different services were mostly independent and often managed with a team of one or two; we'd do some amount of cross-training, but while I was on vacation, nobody did verification server development, just emergency fixes; and similar for other teams of one. But since I didn't need to coordinate with others, I could push verification server changes multiple times a day as needed (sometimes a few times a day). Chat always had the largest team, if nothing else, mostly all the separate services had to do something with chat, too. Although originally things were very separate, running things some things through chat made state management and authentication simpler.
It greatly helped that Erlang is the right fit for a chat service, and that Jan and Brian choice ejabberd to start with when people were using the early WhatsApp as a cludge to chat (originally, WhatsApp was just a short public text 'status' you could see for your contacts, they pivoted to chat later). About half the early server team had used FreeBSD at Yahoo, and zero had any experience with Erlang.
It helped that the SoftLayer servers (mostly SuperMicro, although some Lenovo post IBM acquisition) were very stable. This fed into the stability of FreeBSD and the operability of Erlang. Whenever we shut down chat servers, we'd find some clients with chat connections open for 45 days (mostly Nokia S60, which had a stable networking stack, but no push services, so we had to stay connected). Ocassionally, we'd n...
toast0, if you ever wrote a book about your experience & lessons learned in how to build a great product from the technical side - I'd definitely buy it for myself and gift it to friends. Just say'n :)
Awww. Sadly, I don't have nearly the interest required to write a book. The biggest reason I would is because there's a Pullitzer winning author who stole my name and writes awful fiction with amusing titles. I bet I could sell a lot of books because of name confusion. ;)
And it would be payback for the anxiety I got when I heard that Terry Gross was going to interview me later in the day on Fresh Air and I wws totally unprepared. Luckily, it was already recorded and they played a clip and it wasn't me. ;)
For user facing product design my advice is simple: think about what I would do... and then do something else ;) but I'll accept my advice/thoughts for user invisible stuff could be useful.
> About half the early server team had used FreeBSD at Yahoo, and zero had any experience with Erlang.
Can you say anything more about the choice of Erlang and experience with it? It seems pretty remarkable that you built such a successful business / service with no experience of Erlang.
Erlang is basically a perfect fit for a chat server. ejabberd is out there and there's several high profile use cases where someone needed a chat server, picked ejabberd and became smitten with Erlang. Facebook did it before WhatsApp, but they rebuilt chat in C++ because they couldn't hire Erlang people? and the two people working on it wanted to have some time to do other things. Riot Games did it more recently.
The nice thing about Erlang is it's really easy to make changes to your system; you don't need to move traffic and restart a server, you can just hotload the code changes. So, they started with ejabberd, and by the time I joined it was completely different.
Erlang also has tremendous observability. Each Erlang process has detailed information available (heap size, message queue length, reduction count (cpu use, kind of), you can view the message queue if you want, etc). The BEAM is also observable, it's even got DTrace hooks. There's a way to build where you get stats on lock contention (lock counting), which pretty quickly helps you find the root of scaling issues. Once you find lock contention, almost always the answer is find a way to
distribute the work over more locks so there's less contention; same thing you do when a process's message queue backs up.
It is a small community, so if you're blessed with needing to scale, you are a bit on your own, but you probably could reach out to Erlang Solutions, if you get really stuck. Erlang will scale much farther than the community thinks though. WhatsApp routinely ran mnesia databases with way more data than people thought possible (just don't use disc_only_copies for anything other than the schema table) and dist clusters way larger than people thought possible (pg2 and global locks in general got iffy at times, but I think the new pg is better, and we didn't use global locks except through pg2). There is a general trend where there's no explicit limit on things, but the system will not work well if you go over the implicit limit, and you won't have much guidance.
I hear WhatsApp now uses proper Erlang application packaging, wkth releases and relups, etc. But when I was there, we just did hotloading and it was good enough.
Using WhatsApp as an example of a lean engineering org should almost be banned at this point. WhatsApp had a high performing engineering team that used basically the perfect set of tools to build their application (which also had a narrow feature scope; plaintext messaging). Even with hindsight there is very little you could do to improve on how they executed.
Just because WhatsApp scaled to almost half a billion users with a small engineering team doesn't mean that's the standard, or even achievable, for almost all teams.
The problem is if you decide that you will need 1000 person org in 5 years, you will have 2000 person org in 5 years.
If you challenge yourself to have a 100 person org, you might end up with 1000 people anyways, but at least you're giving yourself a fair chance.
Way too often I see engineering for headcount rather than scale. Building out systems thinking "we'll hire X experts" instead of "we're Y experts, so let's use Y` that's inline with our in house skillset"
In the six months ending June 30, 2014, WhatsApp brought in $15.921 million in revenue, but had a net loss of $232.5 million. However, $206.5 million of that loss was for share-based compensation expenses and issuance of common stock below fair value. Its net cash used in operating expenses during the first half of 2014 was $13.5 million, which sounds much more reasonable.
Essentially, due to WhatsApp’s quickly rising valuation, it used share-based compensation to attract top talent. Eventually, the $22 billion acquisition by Facebook would largely make the “expenses” of issuing that stock moot. This wasn’t cash that WhatsApp was burning, but paper money it was doling out.
Banning an example of a perfectly engineered product seems wrong. Does it set an example that’s very hard to achieve? Perhaps. Should others strive to learn from it? I say yes.
I think that one might think that an example is relevant if it has repeatable properties.
WhatsApp might be so straight to optimum that no visible properties of engineering are extractable, besides that the engineers made no errors in their judgments.
Making no errors is something you want to replicate, you just don’t know how. Seen like that, the example has no value.
Personally I think that showing that it is possible had some value per se
Was WhatsApp a "perfectly engineered product"? It's important to note that, at the time that Facebook purchased WhatsApp, WhatsApp was still running at a loss. That $0.99/year subscription fee wasn't cutting it. Maybe WhatsApp could have survived by raising prices, or maybe that would have prompted everyone to switch to Facebook Messenger (or Google Hangouts). It's an open question.
It's hard to claim, though, that WhatsApp was perfectly engineered when its financial model was unsustainable.
i fail to see the reasonning. History has proven that revenue is very orthogonal to technological perfection ( unfortunately for the amateurs of good engineering that we are here on HN).
Trying to assess the performance of the engineering team with the money the company generated would lead you to use let's say crappy php web framework everywhere, because that's what most businesses that make profit out of the internet are running for their CRM systems.
Trying to assess the performance of the engineering team with the money the
company generated
But is that wrong? And if so, why? We're hired, as engineers, so that the business can make money. Insofar as our excellent engineering makes the business money, it is valuable. When David Hennemeier Hanson wrote the Rails framework for Ruby, his justification for using it was not that it was more elegant than Java, although it certainly was. It was that his convention over configuration approach allowed small teams of developers to launch websites quickly, by having the framework assume that they were doing things in "sensible" ways.
would lead you to use let's say crappy php web framework
everywhere
Maybe we should use more "boring" technology like PHP and Java. Has the continued replacement of Java with PHP, PHP with Rails, Rails with Node.js actually benefited our customers? Or has it benefited programmers' desire for novelty at the expense of our customers? At work, one of my co-workers proposed writing a new front-end component in Vue.js, rather than React, because React was "old" and "showing its limitations". I pushed back, arguing that all the rest of our code was in React, and I wasn't sure that he'd necessarily made the case for Vue providing tangible improvements in maintainability or speed of development. We went back and forth, but in the end he won out, and now we have a bit of Vue in our codebase. Has this improved anything for anyone, other than the programmer who now gets to say on his resume that he's implemented production code in Vue.js?
EDIT: That's not to say that I'm against all innovation. There are certain classes of innovation that have definitely brought tangible improvements in the speed of development and reliability of the software developed. Memory management is a big one. Going from (legacy) C++ to Java is a huge step forward, just by freeing the programmer from having to worry about the most common sources of memory leaks. Likewise, strong type systems are another advance, that is just coming into the mainstream with Rust (and, to a lesser extent, Typescript). Lisp-like languages (although little used) offer another step forward in productivity by allowing the programmer to write functions that can inspect the internals of other functions as data.
But I feel like that's a different category of thing than trying to argue the merits of React versus Vue or Python versus Javascript.
It’s arguably just not that relevant to most products; few real world products are as _simple_ in their requirements as original-WhatsApp. (Not a knock on original-WhatsApp, but you just can’t necessarily scale this model)
Agreed. There was a poster here who argued that Twitter could run on 50 engineers because WhatsApp did as well. It was a response to Elon firing most of Twitter's engineers.
In terms of actual revenue per employee, Twitter (pre-Elon) actually beat Whatsapp (pre-Facebook).
A better measure of success might be employees/revenue.
In 2014, Whatsapp made $10m and lost $140m. That's only $200k/engineer.[0]
By comparison, Apple is at 2.3m/employee. Meta is 1.6m/employee. Twitter is 680k/employee. These include non-engineers of course.
When you need to start making money, you suddenly need to a lot more engineers. Now you need developers to make sales tools, compliance, moderation, finance, A/B test engineers for growth ideas, niche features for big customers, engineers to bring down cloud costs, etc.
It also massively discounts all of the computing help they had from cloud providers. So if you tried to compare this to a company in say 1998, the team would probably be 10x fold (just data center people admins alone).
WhatsApp is the platonic ideal of a consumer-facing software company. Even if your product does not fit the same mold, we all have something to learn from them.
I don't think it is commonly held up as a standard. Too few people are aware of the magnitude of its success for it to play that role, IMO. If anything its story is underrated and understated. Instagram's, as well.
Idolizing anything is a mistake, but treating them as a case study or example for lean engineering feels entirely appropriate -- who else would you point to?
People try to find flaws or special circumstances when the order of magnitude itself is just mindbogling. Seven orders!!
1000000000/100 = 10^7
This is a feature of software as an economic good which, particularly when further leveraged in open source ecosystems, makes it fundamentally different that any other example. Imagine 50 "bricks and mortar" engineers trying to provide anything to 900M users.
But they lost $140m in 2014 and made only $10m in revenue. They would have had to shut down their service if they didn't find product market fit or if Meta/someone else didn't buy them.
When you need to start making money, you suddenly need to a lot more engineers. Now you need developers to make sales tools, compliance, moderation, finance, A/B test engineers for growth ideas, niche features for big customers, engineers to bring down cloud costs, etc.
It doesn't make their engineering less impressive but I think we need to understand the full picture.
At the end of the day, WhatsApp needed to make money to keep their service going.
For the year ending December 31, 2013, WhatsApp had $10.2 million in revenue and a net loss of $138.146 million. Net cash used in operating during this period was only $9.9 million, while share-based compensation amounted to $98.8 million
They only spent $9.9 million over $10.2 million in revenues for operations.
WhatsApp’s goal is still growth, rather than monetization. Mark Zuckerberg and WhatsApp CEO Jan Koum said when the acquisition was made in February that ads aren’t the right way to earn money on messaging
They were investing on growth, an investment that lead to a $16 billion acquisition from Facebook.
I would love to replicate their same "unsustainable business"
No, I'm not upset. I was just operating with different information.
My point remains. They created impressive technology. But 900m users and 50 engineers needed more context. It's not like they shipped a 900 million iPhone-like product with 50 engineers. They made $200k/engineer which is pretty bad compared to more successful companies. When they needed to make more money, they would have needed a lot more engineers.
They never spent their Series A cash. What you're reading is a common misinterpretation of GAAP, which values equity on a cash basis, even though it has no material impact on the company balance sheet. If you look at the cash flow statement in their SEC documents, the cash story is more clear.
They never spent investor money from their Series A-on. After Sequoia completed the Series B, Jan and Brian forwarded them a copy of the bank account statement before the Series B financing, showing the original invested funds untouched. They were acquired less than a year later.
Whatsapp was a money-printer! It deserves that reputation.
Are you just assuming that? Does whatsapp make any money? It’s free to download and there are no ads. How would they hire possibly hundreds of thousands support agents to service their 2.3 billion monthly users?
You said "There’s no way to support that many users", if you added "for free" I would have agreed with you. However there is no way to support any users for free, so your comment doesn't make sense in that context either.
for sure there are several non-trivial factors that helped this dramatic scaling example:
* ability to isolate a fairly universal need that can be solved in full by software. as soon as you have to customize to meet cultural/political fragmentation you start losing orders of mangitude. once there is a need for a "human in the loop" to provide person-to-person support the "magic" is pretty much gone
* ability to deploy across vast numbers of devices. this universal canvas is generally not the case. exceptions being e.g. web standards or having an oligopoly in mobile devices
still, despite all those caveats, software is a completely different game at its heart. in principle it scales to infinity and it becomes a whack-a-mole game to figure out which factor will land things back to reality
Software complexity doesn't scale linearly with number of users. Features have much more effect on that.
Few hundred or thousands of users can run happily off one server. Then you can just... buy a bigger box for long time, maybe not need more for hundreds of thousands of users.
And once you put all of the effort to get highly available loadbalancing, a bunch of app servers, maybe use some distributed database or add a lot of caching for your current one you can, again, just add more and more nodes for a long time till you get to "next step".
Of course, make your app in Rails with people that have no clue about architecture and some N+1 queries and you might need to get to the scaling steps much earlier
> Imagine 50 "bricks and mortar" engineers trying to provide anything to 900M users.
They are not providing any support to the users tho. Hell, I'm pretty sure some of the best selling cars and motorcycles in history(...being probably beetle and honda super cub) had less than 50 engineers
Telegram has like 30 people and is more feature rich, faster in development, smoother in operation and the stuff they provide is more mind-blowing than whatever Meta's team has ever come up with.
telegram isn't e2e (at least by default) and it doesn't support all the java phones whatsapp does ( not sure if whatsapp still do though).
But most of all, it was created much much later, after all the patterns and techniques have already being ironed out by the first wave of chat systems.
I might be wrong, but I believe e2e in WhatsApp came after the Meta acquisition. There was some sort of encryption as far back as 2012, but I don't think it was end-to-end. Only iOS and Android had support for it, but not any of the other clients. The partnership with Whisper Systems and "true" e2e came in 2016 (~2 years after the acquisition).
I don't see how that has an edge over the cloud sync functionality though.
Cloud sync, having thousands and millions of people in a group, group topics, 4GB file sending limit are more impressive than some closed source implementation of E2EE we can never even verify.
> all the java phones whatsapp does ( not sure if whatsapp still do though).
It doesn't anymore.
Telegram works even without an app, through their website (On phones too, yes) so I'd say it's more accessible and having open source clients and APIs means that you can create a Telegram client for literally any platform.
From a tech perspective, e2e adds a lot of constraints on reliability and message ordering which makes it hard to distribute (think « exactly once » vs « at least once » kind of problem, albeit a little different). It also adds a lot of UX problems, since every device has its own secrets and version of the messages.
This is, imho what makes whatsapp uniquely hard to scale.
whatsapp requires to access to contacts on the phopne. otherwise it just shows a bunch of phone numbers and I never get who I'm talking to in the groups. And it shares my phone number to everyone.
Such a feature hasn't been changed and now I know why
1. People in a conversation with you (including a group conversation you are added to, which I believe you can limit to require your consent) can see your phone number
2. People who already have your number in their contacts can see that your number is registered with WhatsApp
There's an important lesson in the WhatsApp story:
It's scope that increases headcount, not users.
WhatsApp could scale to 900M users because the app they build didn't do a lot. It was for sharing plaintext messages between groups of people. That's quite hard at scale but you can solve it with 50 people as they demonstrated. When they started adding more features, that's when they needed more people.
If you keep the scope of your app small you don't need many people to build it.
I agree and would expand that argument to say that add that head count doesn't scale linearly. That is adding people to a project means that you have to add even more people to deal with the management complexity.
I guess that theoretically it's possible to have a productive very large organisation but it's really hard to do that compared with the simplicity of a smaller team.
Exactly. You can obviously do things with a small team but, if you want to go beyond that, you start to accrete a lot of headcount that's to some degree just a function of having a larger organization that can't work on the basis of every knows everyone and what their role is.
Bad quality of product and operations increase headcount.
Skill level of employees increase headcount.
It’s very similar to a proper algorithm cs some naive way of solving things. You can throw a few million on hardware to it, and still only get mediocre results, or solve something in a way that actually works
The really large explosions in size seem to come from internal politics to me. Where leaders of individual internal groups start to have their own goals and agendas separate from the company goals...and start empire building.
In a sufficiently large corporation - it’s impossible to distinguish what empires are nascent independent and massive corporations from non-sense.
You’d think that teams closer to the companies core would be more competent and higher value… but often you find groups of thirty pushing inconsequential metrics. Alternately, you’ll find small groups on the fringe that appear to be doing nothing, but occasionally spit out the next huge product.
For these reasons, investors have historically penalized large conglomerates for inefficiency. The growth of big tech through the ‘10s may have been due in part to negative interest rates which valued discounted future free cash flow at approximately infinite dollars.
In a high interest rate environment, one would expect cash flow to be redirected to investors who will then invest the money. If big tech executive pay drops, then there is limited reason for tech executives not to move into startups.
And dealing with customers (and prospective customers) in much beyond an automated and low-touch way increases headcount.
Enterprise sales, personalized support, marketing programs, etc. Oh, and you probably have a legal team now. And you need to enable those sales reps and manage them. Oh, there's HR now too. And a docs team becomes more important...
It always baffles me how there are 3k people working at, say, Dropbox. What are they all doing? There's a product, it simply syncs files, it synced files pretty much the same way 10 years ago, give or take some polishing and optimisation.
Have you taken a look at what Dropbox actually does today? Just open their website and hover "Products" and you will see that the company does much more than "just" file syncing today (and they're profitable doing so).
Also the product hasn't essentially changed in a decade.
And yet they were so late to get on ARM for Mac (M1), most people (including myself) I know dropped it.
dropbox needs sales teams to sell their product to the enterprise. That’s easily 100-200 people. Then you need support for when those enterprise customers complain, easily another 50+. Infrastructure is done in house with massive storage arrays needing 24x7x365 oncall rotation. Again easily another 30+ people. What about their android and iOS apps. Web apps. Etc
It adds up. Sure they probably have fat to trim, but twitter gives us a great case study how often the massive headcount is focused on revenue generation activities the user isn’t privy to.
>twitter gives us a great case study how often the massive headcount is focused on revenue generation activities the user isn’t privy to
Does it? Twitter the product has been fine (barring clumsy design changes driven by Musk) after the massive downsizing, and they still make billions. I was under the impression a lot of the lost revenue was because of Musk's caustic image that made brands pull their campaigns from the platform.
Less Musk's caustic image and more the lack of moderation due to the layoffs meant a rise in content that advertisers did not want to advertise next to.
There was an older podcast from Darknet Diaries[0] that was about a very popular chat app called “Kik”. Kik was targeted at teens and younger, with an estimated install base of 1 out of every 3 American teenagers.
Long story short they were bought by some holding company that completely neglected the platform and stopped any kind of moderation, DMCA notice or anything else. The result? It became a huge magnet for trading child porn.
While listening to this podcast I thought of how Elon gutted twitter to the bone and probably fired entire teams designed to deal with this kind of content…
Let's see how next year's sales cycle plays out for Twitter. They are banking on providing a McDonald's value menu model, when those big corporate ad buyers are expecting (and need in order to realize value from the relationship and keep their own jobs by ensuring no embarrassment to the company) a personalized service experience.
Again, I wish HN had an auto responder for "what can Xk people at company Y be doing?".
Not that companies never have bloat, but it's obvious from your comment you don't know what DropBox actually does these days. Have you seen the admin pane for enterprise users? There is a huge amount there alone that goes into permissions and role management, auditing, compliance requirements, etc.
One simple example - how many engineers do you think it took to get all the compliance frameworks (e.g. SOC II, HIPAA, etc.) implemented and maintained? My guess is that that is one of a hundred requirements you didn't even consider.
The problem is not that companies or products increase their scope. Most companies do need to increase their scope in order to grow. The problem is increasing scope in an untargeted way that leads to more cost but without a commensurate increase in revenue.
I don't think any sane person would argue that DropBox going into the enterprise was a bad business decision.
I work in enterprise product support to companies that pay us very large amounts of money every year for the product. A support call at a small business for a particular issue might take 15 minutes. For the same issue at a F50 that could take all day, maybe more.
In the small company I may something like "flip this windows setting and reboot the server". They do, we test it, and the call is done.
In enterprise the different levels of complexity spiral out of control very quickly. Is it the standard windows setting issue, or is it the enterprise endpoint protection causing the problem? Ok, lets follow the windows setting issue. Do we have access control the windows settings on this machine? No, we'll need admin access. Actually its controlled by a GPO. Now we need to raise a ticket with the active directory team and may need a custom change. Custom changes need a compliance and security officer to review. Because this change also requires a reboot a impact report needs filed before implementation and reboot. Do we have to test this in the customers Dev/UAT/QA system before implementation? And if this is a new issue will I have to write up the issue for the documentation team?
And after that I'll have spent 10 hours of on the call time with the customer, mostly waiting on them. And next year when sales comes to them with an even higher price tag for the same product they pay the bill because they know when they call we'll stay with them until the problem is solved.
And sometimes you have to stay on a war room call with the customer team the entire time until the issue is resolved, while trying to accomplish all of the above, and navigate that company's unique internal politics (which you need to understand to navigate). But yeah, someone who is good at doing this makes it looks like you can replace them with a cookie cutter support person because they are good and making sure there is no visible friction with the customer, therefor to the bean counters, this persons hard work is invisible. But when this person is gone, and there is suddenly tons of friction next year's sales is f'd. Good luck tech companies laying all these 'dead wood' people that took care of all these things (stuff that also doesn't just fit into the corporate knowledge base). I said before, if you can replace people that do this sort of thing with just anyone, you don't have a real company, you have a franchise that can just plug people into roles. But large customers don't want the McDonalds treatment, they want (and their complexity demands it if your solution is to work for them) the personal service treatment.
This is obviously untrue if you want to offer and kind of support at all.
I also strongly doubt it's true you can scale up your application arbitrarily without needing more people, though I'll readily believe that it's a limited increase compared to what will happen if you have 20 product managers and designers trying to find ways to justify their salaries.
I can't beleive more people who think tech companies are bloated dont use this argument. Its super obvious to people who can build a Twitter clone or even a website. Even if you wanted to get a product off the ground you could probably hire a bunch of people and then let 75% of them go once its built, keep 25% for maintenence. Google however I think is different. They built a simple UI with some algorithmic genius behind it. Much like we are seeing with openAi and ChatGPT. Then they started to just diverge into seperate things like AI for instance. But they maintained their best product.
100% And adding a way to accept payments (which WhatsApp never really did) and have functionality for businesses is an increase in scope. So, 50 engineers is terribly understaffed for an economically sustainable business.
For example, Facebook, Twitter, Uber, etc. have hundreds of engineers who literally just work on how the company gets paid. And no, this isn't just "well they can just insert Stripe.js into their app" - it's way more complicated than that.
this is an important observation. if wathsapp wasn't subsidized by facebook and had to start pay-walling their app, then they would have lost market share and universality and the network effects that go with it, and they would have to be continually iterating on the app to optimize the monetization (in terms of driving away the fewest number of users) and to add features to compete with other competitors, who would be much more prevalent if whatsapp was not universally free. whatsapp is further subsidized in emerging markets by facebook paying carriers to give users free data for facebook apps. without monetization that subsidy would not exist, and once again you would have far more competition, that would require more engineering to respond to.
WhatsApp did have people pay some trivial annual fee before they were bought by Facebook. IIRC, it was free for the first year after which people presumably knew whether it was worth ~$2 per year to them. Perhaps that fee wasn’t economical but 900M * $2 covers 50 engineers pretty easily. I don’t have a good estimate for whether it would cover bandwidth or other staff for liaising with police in every country or whatever.
i never used it back then, so i never had to pay a fee, but the fee was $1 per year according to https://venturebeat.com/mobile/whatsapp-subscription/ and the fee was only being charged in some countries, and not in India and "many other developing countries." actual revenue might have been more like 100M * .7 (after app store take) so $70 million per year, which might be barely profitable.
if they hadn't sold to FB, FB or another company would have run them out of business by doing the same loss-leading advertising and subsidies for their own product that they did for WA.
I would assume WhatsApp would pick a price such that they could continue to offer the first year for free. Maybe it’s tricky for technical reasons for some users to pay $1 but I would assume that most users (in middle or high income countries) would find it affordable and reasonable. It seems to me that being free to try is a necessary condition for a competitor to be successful but being free forever is neither necessary nor sufficient.
i would argue that being free is not even enough to be competitive long term. the problem is that because messaging is very cheap and easy to implement and is a service that almost everyone uses it is in the best interest of companies like FB to actually pay people to use their service (which is what they do with WA) in order to prevent competitors from building a user base that could be used to compete with their core monetized apps. apple and google are essentially doing the same thing, giving away their proprietary messaging apps for free, and giving them highly valuable real estate on their platforms in order to win/keep market share, and they have even more power than fb, because they can change the platform, introduce new protocols, etc, to give themselves a competitive advantage. microsoft was willing to spend billions to give skype away for free for the same reasons.
free is not a guaranteed winner if you have three trillion dollar companies paying people to use their competitive products and they control the platforms that your app runs on.
I’ll add to this—the more paying users you have, the more it makes sense to have more different ways to get payment. If you have 10k users then you may not be able to justify the cost of accepting additional forms of payment besides. Facebook can add new forms of payment and get enough additional revenue to justify the cost.
The reason why WhatsApp scope increased is because of other apps like Telegram and iMessage.
People want to share videos and now even uncompressed photos. Considering Facebook paid a lot of $ for the service they can't let it slide into obscurity.
Even then, it’s not _that_ much extra complexity. It’s more data, but ultimately it’s still largely shuffling opaque binary blobs between, at most, small groups.
Eh, everytime a product manager says 'add this new feature, it will be easy and low impact' I cringe. Try as you might, you cannot fully measure complexity impact before a change occurs, only after. You can never know how much one of those little changes will drastically change user behavior. The product I work with had one of those small changes. We implemented better PDF report generation. The new report generation was even more efficient per file created than the old version.
It lead to performance issues at almost all of our large customers.....
See, before that point customers would just export the XML statistics and generate their own reports, but now with the reports being good, they pushed report generation back to our product. Report generation had always been coupled with another service before that point so a team had to be split off to separate these services and API controllers so they could run standalone. This required a fast feedback loop with the customers to ensure we understood their storage requirements and the ability to access secondary SQL datastores instead of the primary. Now we have a much more scalable, but far more complicated product so we had to train the support team handle these issues.
You don't "cut losses" by axing a wildly successful acquisition while swimming in money. That sort of thing isn't unbiased but asinine and you "can't" do that because making asinine decisions tends to be received badly.
This is 100% true. Where scope creeps in is someone realizes that adding feature X will increase usage by enough that it's ROI-positive, even with a team of 10 supporting it.
I agree. As an engineer the recent layoffs in hindsight felt inevitable because I was working with product managers whose careers rely on them shipping new features every quarter even when customers stopped caring about them years ago (they care about paying less for the main functionality). Now it is us who feels the pressure of having to decrease infra costs while still maintaining support for features when even the PM has left for better pastures.
Random fact: WhatsApp co-founder Jan Koum is one of the few entrepreneurs who seem to have just retired to a lavish life after cashing out. He hasn’t announced any public projects or startups he’s working on, unlike the other co-founder Acton.
But then, he’s built 4 mega yachts [1] and spent $400 million+ on mansions [2].
You're being downvoted. But, while I personally wouldn't have gone the lavish lifestyle route and would probably dabble in things I found interesting, it's not unreasonable and indicated a certain self-awareness to realize you've hit the lottery and you may not want to devote your life to trying to hit the lottery again.
Absolutely valid choice for some of us. Life has taught me that I'm not really all that special in any particular capacity, and more importantly to be okay with it. Being able to cover all my future expenses in a 3 or 4 bedroom place with a high quality security system would be winning at life.
I'd definitely want to pursue some hobbies, but I wouldn't expect any of them to make money or change the world. I could at least donate to nonprofits for the latter, though.
Jeepers. I am guessing your other priorities in your life make it so you need to live in an unsafe area?
I am middle aged, and I have never lived anywhere in my hometown Christchurch that had an active security system. In more that one place the front door or back door was never locked. Probably because I just have never owned much worth stealing! Correction: my parents had a security system, but they stopped using it and it stopped working maybe 15 years ago?
I currently live in a particularly safe little area of my city, although I wouldn’t mind having a secret lockup for the few things that can’t be insured. Most rural or small-town New Zealand is even safer than living in my city.
Above said, I do believe thievery is increasing in Christchurch. My friends in Woolston had a small shed ram-raided (failed because the shed builder went overboard on big posts), and those friends have needed to add security cameras to help prevent thieves casing their home. Anecdotally theft in Lyttelton is on the rise.
These are outrageously unlikely in my country, and would guess are extremely unlikely even in the USA.
If you are worried about extreme events, then a security system is hardly going to help you. When you win the lottery, you can move or buy a good security system.
I've had a similar thought of this regarding one hit wonders. So many of them hit it big on a single song and they end up thinking that the hit was because the people like "them" when a lot of the time, it just hit the cultural zeitgeist at the right time or was used in a commercia/tiktok. I often think if that happened for me, I'd try my best to milk it and then dip. You might only have a few hundred thousand dollars but save it and go back to what you were doing before while still making music on the side.
The best example I've seen of this is chamillionaire. He had a string of 3-5 hits but diversified and has stayed wealthy while many of the people in a similar vein are broke.
I struggle to see what's exactly smart on spending $400M in mansions, and another ~400M in yachts (apart from his +100 cars collection). Just the maintenance for his yachts is ~$25M a year.
You could say "he does what he likes", but it doesn't sound smart nor grounded to me.
Wikipedia lists him with a net worth just shy of $14 billion. As a percentage of wealth, the mansions, yachts, and maintenance are less than what a typical person spends.
Not that it makes any financial sense whatsoever, but the scale of wealth is unfathomable. Money no longer makes sense.
MySpace was a project within an existing company, eUniverse, later InterMix. Tom and Chris DeWolfe were employees of InterMix, not founders. Intermix was acquired by News Corp. for $580M. Not sure how much equity Tom ended up with but I'm sure his payday was a tiny fraction of what WhatsApp's founders received.
>Money is power and power comes with moral responsibility.
I don't know if I agree with that. Are they actually qualified to be wielding that much power? Maybe it's better to just let them spend it on personal luxury or passive philanthropy than to have them start twiddling society's levers just because they have a strong urge to "do something".
It's not the same because it's within the bounds of what the economy "wants" you to do. It's an option that's available, instead of modifying the fabric of society to create an option that was previously unimaginable.
Billions of dollars are billions of dollars that could be spent building schools, housing the homeless, feeding starving people, etc. Saying that it's better that these people starve, freeze and die because of a vague fear that Man shouldn't meddle with certain things and spending on personal luxury and yachts is the better option seems like philosophical lazyness.
Indeed, money spent drives economies. Money not spent is simply wasted. One reason why people advocate for more taxes for the rich is to make sure that the money circulates in an economy and thus progress society.
Theory doesn't make any sense, if he wanted to transfer assets to his children, he'll just do that directly rather than start a foundation that has disbursed $66 billion in grants [1].
Note that I'm not saying Gates is a good guy and doesn't use his foundation to exert influence. But, saying he started a foundation to transfer assets while dodging estate tax reeks of "they just write it off...". Wouldn't he have been better off handing over that $66 billion to his children and paying 18% to 40% as taxes?
Well what else are you supposed to do? Fly your private jet to Bali and then not have a megayacht sitting there waiting for you? There's no way I'm waiting a full week for them to sail over from the Maldives so I need to have at least two.
> You had this little island that was Erlang, and it was hard to build enough boats back to the island to make everything hook in," says Facebook vice president of engineering Jay Parikh.
This doesn't ring true for me, especially in an era where services so easily communicate via http
266 comments
[ 2.9 ms ] story [ 259 ms ] threadThe article is from 2015. Close to irrelevant, alas.
People were always fascinated by Erlang (or rather the underlying BEAM) but it never really broke into the mainstream, probably because it is so different.
The hard part is making this a successful business. Because the minute you try and monetise it with advertising you will quickly find out that building a competitive ad platform is where the real challenge is.
No, you can't.
If it really is trivial today, it was in 2015: cloud providers already existed, smartphones already existed, programming tools were already more than mature, anybody could run a few instances of ejabberd and build an app to talk to them.
WhatsApp image upload used a simple PHP page...
It wasn't exactly rocket science, it was simply very well executed.
Doing a crappy chat app, yes sure that's easy. But one that works as reliably, with the general performance of whatsapp ? Even with today's tech, that's super hard.
so things become irrelevant after 7 years now?
will this comment be irrelevant in 7 years?
I'm not sure about that. These framework projects sound like internal projects developed to meet internal needs and improve internal processes. The GraphQL concept and use case in particular, as well as API management tools, is something I've seen reinvented internally by companies to be able to simplify processes. You just hear about GraphQL because Facebook decided to release it publicly. Everyone else has been more or less reinventing the wheel to meet the same needs.
Hiring an engineer has a significant opportunity cost.
You're right that in the short term you aim to hire where the return is highest, but in the long term that means they're right that you'd want to keep hiring as long as you get a positive return.
You want to spend it on whatever has the highest return, which will frequently be on an expense other than hiring, and when it is on hiring, it will frequently be on hiring roles other than engineers.
There isn't really an opportunity cost to hiring more people when money is no object. More likely to be the opposite.
Why do you think they increased their headcount so much in the last few years? Simply because they could.
The fact there is finite budget and headcount doesn't necessarily mean anything. If you're limited by the number of quality employees you can recruit and onboard, budget and headcount are effectively meaningless because they're not limited by money.
How to apply it wisely is a different, much wider topic.
The question really is, how can you increase shareholder value? If the answer requires engineers then they are hired. They are paid less then the value they add. When that stops being true people like Elon Musk can increase value by reducing headcount.
That’s not the part that gives insight to the question “how can it take X engineers to run service Y?” The insight is that employees can add more value than they cost even if they aren’t working on the most visible core product that the company is known for.
> a firm maximizes profit by producing that quantity of output where marginal revenue equals marginal costs.
This is a pretty fundamental result in microeconomics. The profit maximization section of the wiki article explains it quite well: https://en.m.wikipedia.org/wiki/Marginal_product_of_labor
The usual disclaimers about underlying assumptions apply of course.
Adding yet another zero to your user headcount is costly and a true engineering challenge.
Also, organizational inefficiencies are very costly and require more and more people to fill more and more layers of cintributoand managers alike. But mostly managers.
Edit: Third, product space exploration should not be forgotten.
And demonstrably 50 engineers was sufficient for handling stuff at scale.
but you could add another 0 to the message count. You could add another 0 to the image count.
That seems highly unlikely unless they launched a new product, or this was a gradual shift over many years.
Matrix as a protocol provides the building blocks for both messengers and collaboration tools (and more exotic stuff like spatial collaboration apps). Element is an example of a collaboration app (although Element X is being built to feel more like a messenger which happens to also support collaboration use cases - more like Telegram, but encrypted and open and decentralised). FluffyChat is an example of more of a messenger use case on Matrix. Thirdroom.io is an example of virtual worlds on Matrix.
So: I’d say that WA and Discord are very different products. But you could build both on the same underlying primitives (eg a stack of IP, HTTP, TLS, Matrix)
I remember the first time I saw the Facebook Ads platform, the Events Manager etc - it's a highly complex system that changes all the time. Many more engineers are working on that than on maintaining the ability to Reply to a post feature.
WhatsApp, until Meta, had none of these things.
The bit that you get wrong is assuming that Facebook or Twitter are simple platforms, and that their main and most complex features is allowing users to post, search and read messages. This is entirely wrong as these platforms are actually ad-selling platforms and thus include region-specific services, localized ranking, all sorts of data harvesting, etc etc.
Also, as companies grow they start to feel the need to implement efficient development processes, and thus start to allocate resources to stuff like putting together their own build farms, CI/CD infrastructure, deployment auditing, etc etc etc. They also develop things like front-end and back-end frameworks. It's not like a random team at Facebook just signs up for gitlab and does all it's business from there.
They don't need to release the actual code, but releasing "these are the inputs we use, these are the high level functions of the pipeline, these are the outputs" would be enough.
And the fact that pipelines that drive how people consume an increasingly large majority of their content have become this convoluted, should get you annoyed at social media companies: not people who don't understand how bad its gotten
The cost of their actions is easier to frame if you know the details of their implementation.
If people just naturally like using FB that's fine. If FB is hiring experts for behavioral studies in order hypertune a pipeline that exploits blind spots in the way the human brain works... that's a lot less fine. And if the inputs are wider in scope than the general public realizes, it becomes even less ok.
People will always have free will and most would rather we don't try and override that by just making all social media illegal or something... but there's an inherent asymmetry in the resources social media companies have in dissecting the psyche vs individuals have countering that.
Forcing them to show their cards is one way to thrust the results of that asymmetry into the spotlight and better equip individuals who often don't understand the extent to which they're being manipulated.
You do understand that these are ML models where there is no way to explain their behaviour in a layman way other than what is bleedingly obvious.
That it will try and show you more content that you and the people you follow might like.
They've created a monster.
Controversial, hateful stuff was always going to rise to the top. The difference between now and then is we have pretty much unlimited bandwidth covering the globe to spread the message. If you made FB disappear you're going to see the exact same behavior in mastadon servers, just hoping the scale is limited by lower audience sizes.
ML is perfectly explainable, and all this mysticism around what are just weightings of features is silly.
What all recommendation platforms could be required to provide is which are their most important variables for determining what a user sees, and sensitivity analysis.
You're either commenting on something you are oblivious about, or are pulling a red herring just to be contrarian.
Concepts like revealed preferences [1] are nearly a century old, not to mention the fact that the whole consumer behavior field as a whole has plenty of quantitative methods to infer "what people like".
Facebook and Twitter support a feature where quite literally users specify in no ambiguous term that they like something. Theirnl platform is so effective that their problem is filtering out bad actors.
Then there's also engagement. You can get a pretty good idea of what people are interested in by tracking what they read.
You might try to nitpick about the true meaning of "like" but you'd need to be really determined to intentionally miss the point.
[1] https://en.wikipedia.org/wiki/Revealed_preference
When shown an outrage-inducing tweet are you being showing "what you like" or something quite different? Eg., what is most likey to addict you to the platform, what elicits the highest emotional response. Do people trapped gambling their lives away in casions "like" being there?
The idea that social networks are engaged in a sincere project to estimate people's preferences is on the face of it absurd.
And requires such superstitions as "revealed preferences" which are often little more than an observation that people do what their environment most easily affords.
To call that an exercise of preference, to say that people like it, is a great misunderstanding of human nature.
People are much better than what they happen to do in any given environment, and social media companies bare extreme levels of responsibility for creating environments of so-called "revealed preferences" where, apparently people want eating disorders, conspiracy theories, twitter rage sessions, and so on.
It really doesn't. You're trying to debate the semantics of the keyword "like", but that discussion is meaningless and irrelevant. The only objective fact about these services is that they use the data provided by users to infer the preferences they reveal. These services then use this information to meet the company's goals, such as meeting the expectations of paying customers. These paying customers include marketing companies, and their expectations include driving up user engagement and associate their programs with specific type of content.
You might complain about whether these goals are aligned with what you believe are some user's goals, but you'd be missing the fact that users of services like Facebook or Twitter are the product and not the customers. You're not paying for that click, the marketing company is. If customers pay for that click then this clearly means these services do a good job determining what users "like".
So you agree that social media recommendation services are not optimising for situations their users like. So the conclusion of your disagreeable grandstanding is the one I offered in my initial comment.
> The only objective fact about these services is that ... infer the preferences they reveal.
They force them (/create environments) into choices where their actions are advantageous to an ad. company, yes.
The theory of "revealed preferences" is highly contested, and its highly unlikely any such things exist. People's apparent preferences are functions of their environments, and are largely created by them, not "revealed".
The entire tradition of neoclassical economics begins from a methodological premise that closed-form mathematical functions ought represent key terms of interest (eg., preferences). However that assumption is so laughably false it's ridiculous, all social systems -- including preferences --- have path dependence (hysteresis) and are parameterised on such a large number of variables that no such "formula" can be written.
So the least objective, most pseudoscientific description of a situation in which a person enters a casino and gambles their life away is that, somehow a magic casino machine is "revelling" the Preferences of The Human Mind.
In sum, my reply to the comment stands unaltered. Social media recommendation systems are not optimising for what their users like,.
It's not that simple. These features are also social. When you like a tweet on twitter, you send a notification to the author, and other people might see the tweet with something like "alice and bob liked this tweet". So you might decide to like or not to like a tweet for all these reasons.
Youtube doesn't consider likes at all in the algorithm, only watchtime, even though likes are private in youtube, because it's too vulnerable to manipulation, to people telling you to "like and subscribe", etc..
ML is applied to very specific tasks that are pipelined in an intentional and introspectable way.
And even at the level of a given model while you might struggle to explain behavior depending on how pedantically you define that, you still know what your reward function was.
-
This is like saying "There's no point in knowing the source code if you don't have the schematic and microcode for your CPU to know which transistors each line of code is changing the state of"
And an army of observers will be happy to interpret for the less technical.
Sounds like a conspiracy theorist's dream.
It would most likely take several teams to push out what you are asking for. You’d need lawyers, support staff, engineers to create a platform other teams can hook into in order to fulfill these requests, you’d need ways to strip training data of any PII, you’d need a team to write the front end for support staff and the external users. Etc… that’s a non trivial amount of headcount!
You're raving about how much work it would be to build in introspectability into a system that has an extremely large effect on the population, as if that's not justified by the fact we'd be gaining introspectability into a system that has an extremely large effect on the population
You're also overselling it, no one is saying we need every single api request to fetch a feed to also hit some endpoint stating "this is why post id X was shown to user Y". ML again, can be introspected at a higher level than all this seems to imply.
OpenAI can't tell you exactly why ChatGPT gives a specific string of text but they can (and have) shown how it works at a high level: https://openai.com/blog/instruction-following/
You seem to be pushing the line that, because it's difficult to explain why these systems work how they do, and the responsibility is spread so far and wide, we shouldn't bother. It's the literal opposite: it's exactly why its so needed. Even internally these things should be wanted, and I'd be truly shocked if finely tuned advertising giants don't already have a high degree of these understandings in poorly named suborgs that no one pays much attention to (not because cloak and daggers, but because engineers tend to focus on more glamourous things than "introspection").
I'm not sure you thought this problem through. Even if you can specify the inputs, the high level functions mean nothing and the output can and very often is determined by implementation details and the current state of the whole service.
Say for example that personalized recency search of tweets takes into account bad actors being banned/shadow banned/downanked. How exactly do you expect to specify what output you expect from your inputs and your high-level description of your pipeline when a specific spammer is downranked?
I'm sure you can setup a pretty little diagram, full of colors and nice rounded edges, but it would be completely meaningless as state matters, and it's practically impossible to specify what state a global or regional system has in place.
then the follow-up argument should be a much needed (IMO) "simplify this crap so users and regulators come to know how much neutrality and facts they get served".
I understand enough ML to know how and why such systems can't be introspected and why their output is inscrutable to humans, but that's where I draw the line personally: a key component of your business, or one that affects many users, shouldn't be permitted to be so complex that its outcome can't be explained based on a finite number of documented variables and logical branches. We can even think about having a standard to describe and document ML systems (and I would totally look that up because I have the right to know how my keyboard suggests words, why this search result that seems unusual is in fact a sponsored placement, or why this autonomous vehicle chose to run over the pedestrian so not to kill the baby on board...).
And this is not meant to hinder research and development in AI, but to protect people and businesses from potentially dangerous blackboxes, and bring back accountability and ethics in a field that knows very little of it.
There are plenty of effective pharmaceuticals for which we don’t entirely understand the mechanism (e.g. lithium). I think it’s a good goal to be able to understand mechanisms, but if you can effectively show that a solution works to solve a problem through methodical trials, we shouldn’t disallow it’s use.
The issue with Facebook is not complexity or black-box-ness. It’s that the goals of the company and how it makes a profit are fundamentally misaligned with a healthy society.
hmmmm
> We can even think about having a standard to describe and document ML systems (and I would totally look that up because I have the right to know how my keyboard suggests words, why this search result that seems unusual is in fact a sponsored placement, or why this autonomous vehicle chose to run over the pedestrian so not to kill the baby on board...).
Yea you know nothing about ML...
You don't think that any business should be able to use random forests, let alone anything neural network based?
You have managed to make these all sound incredibly inefficient ;P.
I'm not even sure if it uses react/angular or something else
[1]: https://www.cnbc.com/2023/01/20/twitter-is-down-to-fewer-tha...
In a 2022 AWS conference they announced that S3 is currently powered by 235 microservices[0]. It started with 8.
I think it's a great case of something that looks easy from the outside but has a mega iceberg of complexity under the hood to operate at the scale and reliability that S3 has. It's also a good example of if you don't need that scale you can go with a much easier solution of literally uploading a single file to 1 server and being done with it.
[0]: https://youtu.be/RfvL_423a-I?t=995
Twitter and WhatsApp are both way more complicated than used to be both in terms of features and usage. They also make way more money, which both takes more people and supports more people.
Twitter's main competitor is something that was built by volunteers over the course of a few years. Mastodon of course has its limitations but in terms of features it's actually doing most of the important things that Twitter does. That project has hundreds of competitors but most of the work gets done by a handful of people.
A lot of these big companies accumulate people almost as fast as they accumulate wealth. It's a weird dynamic where they basically recruit themselves into being slow, bureaucratic, and ultimately ineffective. Once you have thousands of people, change and taking risks become very hard .
That's why the startup model is so popular. Startups make things happen that big companies struggle to make happen.
It is possible that the companies could be much smaller and cheaper to run if they dropped the ad-selling and the associated data recording and analysis infrastructure. But it seems to me that currently insufficiently many people are willing to pay for something like that.
[1] http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
Jan Koum was more concerned about getting his shiny BMW dinged and more famous for badly parking the said BMW. (just kidding, Jan, please don't bash my Prius!)
Both were very good engineers who managed to create something amazing with a laser-like focus on the product.
To address a couple things elsewhere in the thread. WhatsApp circa acquisition was not text only messaging. We had multimedia messaging (video, images, audio), I think since before my time. I don't remember when end to end encryption happened (we were working on it for a long time, and it went through a lot of not user visible testing before we announced it and started turning off plain text messaging), but before that, the multimedia servers also did transcoding; post encryption, we had to do transcoding client side. If you think back to the variety of client platforms supported at the time, video codec support was all over the place.
We did have infrastructure to receive payments (apple and google in app payments and paypal), although requiring payment was very selective. I don't think there's published details on that, so I won't go into specifics, but you can't really require payment in places where $1/year is a significant burden or where it's difficult to pay a US based country; and you don't want to require payment in places where people will nope out and use any of the many alternatives. Otoh, my spouse paid for the 5? year plan just from prompting, without even asking me if she could get it free, and she usually doesn't do phone base payments without asking me for help.
On profit and loss, I understand why GAAP include stock based compensation, but it's weird to say there was a giant loss because of it. From what I've seen in public numbers, revenue was a bit more than expenses, and that's what we were told internally as well. After acquisition, I had much less visibility into the company financials; I did see daily verification expenses as part of my job, and sometimes saw our SoftLayer bills, but not our headcount expenses or any accounting for server resources in Facebook Infrastructure.
Real time voice calls launched
As to why things went right.
Limited scope and clear product vision helped. You could answer almost any design question by opening the platform SMS/MMS app; our app should look as much like that as possible, because that's what users know and expect. Dedicated platform teams buidling in the platform SDK was the right choice for that, although it does leave the real issue where it's hard to transfer your user data between platforms because each platform designed their own local storage databases.
Largely experienced workforce, with great autonomy and responsibility. On the server side, different services were mostly independent and often managed with a team of one or two; we'd do some amount of cross-training, but while I was on vacation, nobody did verification server development, just emergency fixes; and similar for other teams of one. But since I didn't need to coordinate with others, I could push verification server changes multiple times a day as needed (sometimes a few times a day). Chat always had the largest team, if nothing else, mostly all the separate services had to do something with chat, too. Although originally things were very separate, running things some things through chat made state management and authentication simpler.
It greatly helped that Erlang is the right fit for a chat service, and that Jan and Brian choice ejabberd to start with when people were using the early WhatsApp as a cludge to chat (originally, WhatsApp was just a short public text 'status' you could see for your contacts, they pivoted to chat later). About half the early server team had used FreeBSD at Yahoo, and zero had any experience with Erlang.
It helped that the SoftLayer servers (mostly SuperMicro, although some Lenovo post IBM acquisition) were very stable. This fed into the stability of FreeBSD and the operability of Erlang. Whenever we shut down chat servers, we'd find some clients with chat connections open for 45 days (mostly Nokia S60, which had a stable networking stack, but no push services, so we had to stay connected). Ocassionally, we'd n...
toast0, if you ever wrote a book about your experience & lessons learned in how to build a great product from the technical side - I'd definitely buy it for myself and gift it to friends. Just say'n :)
And it would be payback for the anxiety I got when I heard that Terry Gross was going to interview me later in the day on Fresh Air and I wws totally unprepared. Luckily, it was already recorded and they played a clip and it wasn't me. ;)
For user facing product design my advice is simple: think about what I would do... and then do something else ;) but I'll accept my advice/thoughts for user invisible stuff could be useful.
> About half the early server team had used FreeBSD at Yahoo, and zero had any experience with Erlang.
Can you say anything more about the choice of Erlang and experience with it? It seems pretty remarkable that you built such a successful business / service with no experience of Erlang.
Erlang is basically a perfect fit for a chat server. ejabberd is out there and there's several high profile use cases where someone needed a chat server, picked ejabberd and became smitten with Erlang. Facebook did it before WhatsApp, but they rebuilt chat in C++ because they couldn't hire Erlang people? and the two people working on it wanted to have some time to do other things. Riot Games did it more recently.
The nice thing about Erlang is it's really easy to make changes to your system; you don't need to move traffic and restart a server, you can just hotload the code changes. So, they started with ejabberd, and by the time I joined it was completely different.
Erlang also has tremendous observability. Each Erlang process has detailed information available (heap size, message queue length, reduction count (cpu use, kind of), you can view the message queue if you want, etc). The BEAM is also observable, it's even got DTrace hooks. There's a way to build where you get stats on lock contention (lock counting), which pretty quickly helps you find the root of scaling issues. Once you find lock contention, almost always the answer is find a way to distribute the work over more locks so there's less contention; same thing you do when a process's message queue backs up.
It is a small community, so if you're blessed with needing to scale, you are a bit on your own, but you probably could reach out to Erlang Solutions, if you get really stuck. Erlang will scale much farther than the community thinks though. WhatsApp routinely ran mnesia databases with way more data than people thought possible (just don't use disc_only_copies for anything other than the schema table) and dist clusters way larger than people thought possible (pg2 and global locks in general got iffy at times, but I think the new pg is better, and we didn't use global locks except through pg2). There is a general trend where there's no explicit limit on things, but the system will not work well if you go over the implicit limit, and you won't have much guidance.
I hear WhatsApp now uses proper Erlang application packaging, wkth releases and relups, etc. But when I was there, we just did hotloading and it was good enough.
I dont know why but I cant stop laughing when I read this line.
Just because WhatsApp scaled to almost half a billion users with a small engineering team doesn't mean that's the standard, or even achievable, for almost all teams.
The problem is if you decide that you will need 1000 person org in 5 years, you will have 2000 person org in 5 years.
If you challenge yourself to have a 100 person org, you might end up with 1000 people anyways, but at least you're giving yourself a fair chance.
Way too often I see engineering for headcount rather than scale. Building out systems thinking "we'll hire X experts" instead of "we're Y experts, so let's use Y` that's inline with our in house skillset"
It wasn't a sustainable business.
Essentially, due to WhatsApp’s quickly rising valuation, it used share-based compensation to attract top talent. Eventually, the $22 billion acquisition by Facebook would largely make the “expenses” of issuing that stock moot. This wasn’t cash that WhatsApp was burning, but paper money it was doling out.
WhatsApp might be so straight to optimum that no visible properties of engineering are extractable, besides that the engineers made no errors in their judgments.
Making no errors is something you want to replicate, you just don’t know how. Seen like that, the example has no value.
Personally I think that showing that it is possible had some value per se
It's hard to claim, though, that WhatsApp was perfectly engineered when its financial model was unsustainable.
Trying to assess the performance of the engineering team with the money the company generated would lead you to use let's say crappy php web framework everywhere, because that's what most businesses that make profit out of the internet are running for their CRM systems.
Doesn't make any sense.
EDIT: That's not to say that I'm against all innovation. There are certain classes of innovation that have definitely brought tangible improvements in the speed of development and reliability of the software developed. Memory management is a big one. Going from (legacy) C++ to Java is a huge step forward, just by freeing the programmer from having to worry about the most common sources of memory leaks. Likewise, strong type systems are another advance, that is just coming into the mainstream with Rust (and, to a lesser extent, Typescript). Lisp-like languages (although little used) offer another step forward in productivity by allowing the programmer to write functions that can inspect the internals of other functions as data.
But I feel like that's a different category of thing than trying to argue the merits of React versus Vue or Python versus Javascript.
And then your computer wails in agony as another opened chrome tab drains 6GB of RAM.
So do those tabs Chrome is probably loading.
And yes, in 2040, Chrome will be using 60GB of RAM for "similar" stuff (superficially similar).
In terms of actual revenue per employee, Twitter (pre-Elon) actually beat Whatsapp (pre-Facebook).
A better measure of success might be employees/revenue.
In 2014, Whatsapp made $10m and lost $140m. That's only $200k/engineer.[0]
By comparison, Apple is at 2.3m/employee. Meta is 1.6m/employee. Twitter is 680k/employee. These include non-engineers of course.
When you need to start making money, you suddenly need to a lot more engineers. Now you need developers to make sales tools, compliance, moderation, finance, A/B test engineers for growth ideas, niche features for big customers, engineers to bring down cloud costs, etc.
[0]https://www.theverge.com/2014/10/28/7085905/facebooks-prized...
I don't think it is commonly held up as a standard. Too few people are aware of the magnitude of its success for it to play that role, IMO. If anything its story is underrated and understated. Instagram's, as well.
Idolizing anything is a mistake, but treating them as a case study or example for lean engineering feels entirely appropriate -- who else would you point to?
1000000000/100 = 10^7
This is a feature of software as an economic good which, particularly when further leveraged in open source ecosystems, makes it fundamentally different that any other example. Imagine 50 "bricks and mortar" engineers trying to provide anything to 900M users.
When you need to start making money, you suddenly need to a lot more engineers. Now you need developers to make sales tools, compliance, moderation, finance, A/B test engineers for growth ideas, niche features for big customers, engineers to bring down cloud costs, etc.
It doesn't make their engineering less impressive but I think we need to understand the full picture.
At the end of the day, WhatsApp needed to make money to keep their service going.
you posted the same comment two times already.
same response
https://news.ycombinator.com/item?id=34566572
They only spent $9.9 million over $10.2 million in revenues for operations.
WhatsApp’s goal is still growth, rather than monetization. Mark Zuckerberg and WhatsApp CEO Jan Koum said when the acquisition was made in February that ads aren’t the right way to earn money on messaging
They were investing on growth, an investment that lead to a $16 billion acquisition from Facebook.
I would love to replicate their same "unsustainable business"
Are you really that upset about GAAP accounting and how non-cash, stock-based compensation can lead to paper "losses"?
My point remains. They created impressive technology. But 900m users and 50 engineers needed more context. It's not like they shipped a 900 million iPhone-like product with 50 engineers. They made $200k/engineer which is pretty bad compared to more successful companies. When they needed to make more money, they would have needed a lot more engineers.
They never spent investor money from their Series A-on. After Sequoia completed the Series B, Jan and Brian forwarded them a copy of the bank account statement before the Series B financing, showing the original invested funds untouched. They were acquired less than a year later.
Whatsapp was a money-printer! It deserves that reputation.
* ability to isolate a fairly universal need that can be solved in full by software. as soon as you have to customize to meet cultural/political fragmentation you start losing orders of mangitude. once there is a need for a "human in the loop" to provide person-to-person support the "magic" is pretty much gone
* ability to deploy across vast numbers of devices. this universal canvas is generally not the case. exceptions being e.g. web standards or having an oligopoly in mobile devices
still, despite all those caveats, software is a completely different game at its heart. in principle it scales to infinity and it becomes a whack-a-mole game to figure out which factor will land things back to reality
Few hundred or thousands of users can run happily off one server. Then you can just... buy a bigger box for long time, maybe not need more for hundreds of thousands of users.
And once you put all of the effort to get highly available loadbalancing, a bunch of app servers, maybe use some distributed database or add a lot of caching for your current one you can, again, just add more and more nodes for a long time till you get to "next step".
Of course, make your app in Rails with people that have no clue about architecture and some N+1 queries and you might need to get to the scaling steps much earlier
> Imagine 50 "bricks and mortar" engineers trying to provide anything to 900M users.
They are not providing any support to the users tho. Hell, I'm pretty sure some of the best selling cars and motorcycles in history(...being probably beetle and honda super cub) had less than 50 engineers
So you're a front-end guy?
But most of all, it was created much much later, after all the patterns and techniques have already being ironed out by the first wave of chat systems.
I don't see how that has an edge over the cloud sync functionality though.
Cloud sync, having thousands and millions of people in a group, group topics, 4GB file sending limit are more impressive than some closed source implementation of E2EE we can never even verify.
> all the java phones whatsapp does ( not sure if whatsapp still do though).
It doesn't anymore.
Telegram works even without an app, through their website (On phones too, yes) so I'd say it's more accessible and having open source clients and APIs means that you can create a Telegram client for literally any platform.
This is, imho what makes whatsapp uniquely hard to scale.
Such a feature hasn't been changed and now I know why
Who is "everyone"? Anyone that searches your name, or all your contacts? (I've never used WhatsApp.)
1. People in a conversation with you (including a group conversation you are added to, which I believe you can limit to require your consent) can see your phone number
2. People who already have your number in their contacts can see that your number is registered with WhatsApp
WhatsApp only uploads phone numbers from your contacts. No other information. This is all documented here: https://faq.whatsapp.com/1191526044909364/
It's scope that increases headcount, not users.
WhatsApp could scale to 900M users because the app they build didn't do a lot. It was for sharing plaintext messages between groups of people. That's quite hard at scale but you can solve it with 50 people as they demonstrated. When they started adding more features, that's when they needed more people.
If you keep the scope of your app small you don't need many people to build it.
I guess that theoretically it's possible to have a productive very large organisation but it's really hard to do that compared with the simplicity of a smaller team.
Fred Brooks - The Mythical Man-Month (1975)
Architecture increases headcount.
Bad quality of product and operations increase headcount.
Skill level of employees increase headcount.
It’s very similar to a proper algorithm cs some naive way of solving things. You can throw a few million on hardware to it, and still only get mediocre results, or solve something in a way that actually works
You’d think that teams closer to the companies core would be more competent and higher value… but often you find groups of thirty pushing inconsequential metrics. Alternately, you’ll find small groups on the fringe that appear to be doing nothing, but occasionally spit out the next huge product.
For these reasons, investors have historically penalized large conglomerates for inefficiency. The growth of big tech through the ‘10s may have been due in part to negative interest rates which valued discounted future free cash flow at approximately infinite dollars.
In a high interest rate environment, one would expect cash flow to be redirected to investors who will then invest the money. If big tech executive pay drops, then there is limited reason for tech executives not to move into startups.
Enterprise sales, personalized support, marketing programs, etc. Oh, and you probably have a legal team now. And you need to enable those sales reps and manage them. Oh, there's HR now too. And a docs team becomes more important...
That kind of transformation only comes from scope or the things the GP pointed.
It adds up. Sure they probably have fat to trim, but twitter gives us a great case study how often the massive headcount is focused on revenue generation activities the user isn’t privy to.
Does it? Twitter the product has been fine (barring clumsy design changes driven by Musk) after the massive downsizing, and they still make billions. I was under the impression a lot of the lost revenue was because of Musk's caustic image that made brands pull their campaigns from the platform.
Long story short they were bought by some holding company that completely neglected the platform and stopped any kind of moderation, DMCA notice or anything else. The result? It became a huge magnet for trading child porn.
While listening to this podcast I thought of how Elon gutted twitter to the bone and probably fired entire teams designed to deal with this kind of content…
[0] https://darknetdiaries.com/episode/93/
Not that companies never have bloat, but it's obvious from your comment you don't know what DropBox actually does these days. Have you seen the admin pane for enterprise users? There is a huge amount there alone that goes into permissions and role management, auditing, compliance requirements, etc.
One simple example - how many engineers do you think it took to get all the compliance frameworks (e.g. SOC II, HIPAA, etc.) implemented and maintained? My guess is that that is one of a hundred requirements you didn't even consider.
I don't think any sane person would argue that DropBox going into the enterprise was a bad business decision.
In the small company I may something like "flip this windows setting and reboot the server". They do, we test it, and the call is done.
In enterprise the different levels of complexity spiral out of control very quickly. Is it the standard windows setting issue, or is it the enterprise endpoint protection causing the problem? Ok, lets follow the windows setting issue. Do we have access control the windows settings on this machine? No, we'll need admin access. Actually its controlled by a GPO. Now we need to raise a ticket with the active directory team and may need a custom change. Custom changes need a compliance and security officer to review. Because this change also requires a reboot a impact report needs filed before implementation and reboot. Do we have to test this in the customers Dev/UAT/QA system before implementation? And if this is a new issue will I have to write up the issue for the documentation team?
And after that I'll have spent 10 hours of on the call time with the customer, mostly waiting on them. And next year when sales comes to them with an even higher price tag for the same product they pay the bill because they know when they call we'll stay with them until the problem is solved.
This is obviously untrue if you want to offer and kind of support at all.
I also strongly doubt it's true you can scale up your application arbitrarily without needing more people, though I'll readily believe that it's a limited increase compared to what will happen if you have 20 product managers and designers trying to find ways to justify their salaries.
There's traditional web search.
Maps.
Images.
Video (which is essentially YouTube integrated into Google Search).
Each of these products would be Fortune 250 companies on their own.
Obviously, it's all powered by the Ads team / business.
But that's two majorly different parts - Google's 1P ads (including video), and the display network (3P ads).
None of this is useful without a gigantic analytics org.
And that's just "search".
Cloud & G-suite would each also be Fortune 100 businesses.
maybe years ago, this is not the case now.
100% And adding a way to accept payments (which WhatsApp never really did) and have functionality for businesses is an increase in scope. So, 50 engineers is terribly understaffed for an economically sustainable business.
For example, Facebook, Twitter, Uber, etc. have hundreds of engineers who literally just work on how the company gets paid. And no, this isn't just "well they can just insert Stripe.js into their app" - it's way more complicated than that.
if they hadn't sold to FB, FB or another company would have run them out of business by doing the same loss-leading advertising and subsidies for their own product that they did for WA.
free is not a guaranteed winner if you have three trillion dollar companies paying people to use their competitive products and they control the platforms that your app runs on.
People want to share videos and now even uncompressed photos. Considering Facebook paid a lot of $ for the service they can't let it slide into obscurity.
It lead to performance issues at almost all of our large customers.....
See, before that point customers would just export the XML statistics and generate their own reports, but now with the reports being good, they pushed report generation back to our product. Report generation had always been coupled with another service before that point so a team had to be split off to separate these services and API controllers so they could run standalone. This required a fast feedback loop with the customers to ensure we understood their storage requirements and the ability to access secondary SQL datastores instead of the primary. Now we have a much more scalable, but far more complicated product so we had to train the support team handle these issues.
Fallacy: https://wikipedia.org/wiki/Sunk_cost
It is hard to learn to avoid biased thinking in our own lives! I think that cutting one’s losses is a very important skill to learn.
This is 100% true. Where scope creeps in is someone realizes that adding feature X will increase usage by enough that it's ROI-positive, even with a team of 10 supporting it.
But then, he’s built 4 mega yachts [1] and spent $400 million+ on mansions [2].
1- https://www.superyachtfan.com/yacht/mogambo/owner/
2- https://www.dirt.com/gallery/moguls/tech/jan-koum-house-beve...
I don't mean this in a disrespectful manner - building and selling WhatsApp was an astounding feat.
I'd definitely want to pursue some hobbies, but I wouldn't expect any of them to make money or change the world. I could at least donate to nonprofits for the latter, though.
Jeepers. I am guessing your other priorities in your life make it so you need to live in an unsafe area?
I am middle aged, and I have never lived anywhere in my hometown Christchurch that had an active security system. In more that one place the front door or back door was never locked. Probably because I just have never owned much worth stealing! Correction: my parents had a security system, but they stopped using it and it stopped working maybe 15 years ago?
I currently live in a particularly safe little area of my city, although I wouldn’t mind having a secret lockup for the few things that can’t be insured. Most rural or small-town New Zealand is even safer than living in my city.
Above said, I do believe thievery is increasing in Christchurch. My friends in Woolston had a small shed ram-raided (failed because the shed builder went overboard on big posts), and those friends have needed to add security cameras to help prevent thieves casing their home. Anecdotally theft in Lyttelton is on the rise.
It may not be so much you live in an area with danger, but the danger comes to your safe area looking for money (ransom).
And not just you, but those around you:
* https://en.wikipedia.org/wiki/J._Paul_Getty#Kidnapping_of_gr...
See also "Lottery winner 'regretted scooping £11m' when brother hired hitman to assassinate him":
* https://www.dailystar.co.uk/news/weird-news/lottery-winner-r...
These are outrageously unlikely in my country, and would guess are extremely unlikely even in the USA.
If you are worried about extreme events, then a security system is hardly going to help you. When you win the lottery, you can move or buy a good security system.
The best example I've seen of this is chamillionaire. He had a string of 3-5 hits but diversified and has stayed wealthy while many of the people in a similar vein are broke.
You could say "he does what he likes", but it doesn't sound smart nor grounded to me.
Not that it makes any financial sense whatsoever, but the scale of wealth is unfathomable. Money no longer makes sense.
Money is power and power comes with moral responsibility. I think that's why the Gates' started that foundation.
For many people it's easier to feel good about having so much if you can claim that you're using it for good.
I don't know if I agree with that. Are they actually qualified to be wielding that much power? Maybe it's better to just let them spend it on personal luxury or passive philanthropy than to have them start twiddling society's levers just because they have a strong urge to "do something".
Whatever you do with so much dough is gonna have consequences, there's no staying neutral.
Think of it as make-work income for inner city shipyard workers, metal workers, and countless other legacy professions.
At the end of the day, the guy doesn't have the money any more, they all do. So they're happy, and he has a boat.
Note that I'm not saying Gates is a good guy and doesn't use his foundation to exert influence. But, saying he started a foundation to transfer assets while dodging estate tax reeks of "they just write it off...". Wouldn't he have been better off handing over that $66 billion to his children and paying 18% to 40% as taxes?
1- https://www.gatesfoundation.org/about/foundation-fact-sheet
In a world where mere 10s of dollars can fund a medical intervention that change s the life of a poor child, having 4 megayachts is obscene.
https://zerodha.tech/blog/hello-world/
Tldr: How 30 member tech team formed over seven years built India’s largest stock broker.
Essentially, they have the same revenue & product as Asana, but with only 1/20th the employees.
https://world.hey.com/jason/on-company-size-8095488d
This doesn't ring true for me, especially in an era where services so easily communicate via http