There is another bloated company with an unclear growth story right across the street from Twitter, and it happens share same CEO. How long can Square keep up its act?
Yeah, honestly I'd think it were sad if they were, say, manufacturing employees who were laid off and had poor prospects, but in today's market any decent engineer or product person who worked at Twitter would be snapped up in a second.
A sad day for Twitter, but a dose of reality for a lot of the fluffiness in the market at the moment.
Companies need to have real revenue and produce real profit and then need a valuation based on sane multiples. Twitter got behind that curve badly and ultimately this is what happens.
Yes, it's sad for employees being let go, but it's good news for Twitter. I don't think Twitter is going anywhere; it has a huge user base and a lot of people think of it as a branch of the internet. By cutting surgically (removing, as you put it, the "fluffiness"), Jack might very well be able to bring some fighting spirit back.
Oh I agree, I didn't say I thought the company was going bust... just saying that this is likely the start of a lot more focus on the hard numbers of tech companies and less focus on hype-based valuation (although the later isn't going to disappear entirely). Just that question a a lot of startups hate "So how are you going to make money?" is likely going to start getting asked a lot more.
Yes, it's sad for employees let go, but it's good news for AOL. I don't think AOL is going anywhere; it has a huge user base and a lot of people think of it as the internet. By cutting surgically (removing, as you put it, the "fluffiness"), Jonathan might very well be able to bring some fighting spirit back.
Not trying to be snarky, but when have you ever seen a company do "surgical" layoffs? Anything across the board is not surgical by definition. This is just a plain, simple RIF to stem the losses. There's nothing surgical about it. Surgical would imply killing off specific teams and products. This is across the board.
> Surgical would imply killing off specific teams and products. This is across the board.
Perhaps I missed it but no where did I see any specific mention that this was either surgical or across the board. Care to elaborate? If they didn't specify it certain could go either way.
>I don't think Twitter is going anywhere; it has a huge user base and a lot of people think of it as a branch of the internet.
It's not going anywhere in the next year or two. But in five years? Ten years?
Twitter's main problem is that it has one product, and it's possible to get bored with that product. It's not really possible to get bored of googling things or buying things off amazon, but it's entirely possible to get bored of twitter.
Remember when every man and his dog had a blog? (Or possibly a livejournal?) Remember when the word "blogosphere" was all over the news? What happened to all the blogs?
A format like Twitter requires major infrastructure to work well so being under one company makes complete and necessary sense. This doesn't detract from the fact that it's become a format that a lot of the Internet now depends on for communication and news. Simply saying "no, it's not" doesn't prove anything.
>Remember when every man and his dog had a blog? (Or possibly a livejournal?) Remember when the word "blogosphere" was all over the news? What happened to all the blogs?
You're spot on about Twitter basically being far overvalued. However their problem isn't lack of real revenue. They're tracking to $2+ billion in revenue in the next four quarters. Assuming a significantly slowing growth rate, they can easily double that sales trajectory to $4 billion in three years.
Their problem is that costs are far out of line, which is why they're cutting. Twitter could turn a significant profit if they design for $3 or $4 billion in sales (rather than pretending they're heading toward $20 billion). Their margins should be comparable to Facebook's, their business is no more difficult to optimize.
They have 12% of the sales of Facebook, with ~38% of the employee base. It's an extremely poorly run, bloated business. They're built to scale to Facebook's size, but they will never achieve that size no matter what happens. Long-term they're a $20 or $40 billion market cap business, instead of $250 billion.
You're seeing this from the wrong angle. Twitter's problem is NOT the lack of "real revenue" or "real profit." This is the sort of thinking that stagnates the platform.
Google's original investors were FUMING that the company didn't pursue ads earlier, following with what Yahoo did. In retrospect we now see the insight of that approach.
Twitter's problem is one of product. It's not innovating, it hasn't changed it's core formula in many years. Compare that to Google and Facebook which both made many bold improvements to their platforms after booming into success. (Though facebook one could argue has slowed down in its innovation.)
>Companies need to have real revenue and produce real profit and then need a valuation based on sane multiples.
I wonder how many unicorns lack these criteria. Only Twitter has the discipline of public markets. If it doesn't deliver, public shareholders get burned.
If unicorns don't deliver, private investors get burned, but where do the losses go? VCs, founders, employees, as well as some family offices, banks, university endowments, pension funds...where else? And I sure am curious about the exposure of these last three groups.
Do you get mostly opinions and reactions on Twitter or 140char reporting on breaking stories? I tried using it to follow the Arab spring and just couldn't.
Wish I could find it, but a couple weeks ago I saw someone posting a 20-ish part message on Twitter. At some point you've gotta think maybe this isn't the appropriate way to use the service.
@pmarca is famous for his multipart "tweet storms". Lots of others have followed suit (not sure if he started the trend, but was the first one I saw doing it).
Being a big sports fan (#phinsup) I find that I get breaking news first on Twitter. Player signings, coaches being hired (#dancampbell) and fired (#firephilbin), etc. I also find a lot of interesting links on Twitter, that are things I wouldn't necessarily have found otherwise, because they're from sites that weren't necessarily always on my radar.
Same here. I just don't see the value in Twitter when I can easily get personalized content from Facebook's news feed and use an RSS reader for the sites that I follow.
I agree with @bpicolo when it comes to support. If you have a problem with most of the big companies in terms of support and they are delaying a response, all you have to do is reach out to the support team on Twitter and they respond ASAP.
Companies such as AT& T, Chase, Citi, Bank of America, Amazon etc..
@Bachmeier, I agree. And oh, they are right there with Comcast in my opinion when it comes to bad service. They are very good at making small mistakes like overcharging you know on a bill. So Twitter will do wonders for you :).
Even just setting up a Twitter account for a business feels like it's not something they've put any effort into. You can set up an advertising account with multiple users, but for a regular public-facing Twitter account it's either use TweetDeck or share passwords.
Yep. It sounds like a real, viable market too. They probably have a lot more insight into this than me, it's just something that surprises me as an external party
At first when it appeared I thought it was silly but now I like it since it's a quick simple way to get information without all the bloat. I never thought I'd like it, now I look forward to new technologies that don't exist yet but I think I will need when they arrive.
I have ditched Facebook that's a cesspool of humanity and HR magnet.
I hated the idea of twitter for a long time because I thought it was the shitty part of facebook (incessant updates from people whom I don't care about) x 10, because it's just a feed.
But since I started really getting into it last year, I've become so much more knowledgeable by following the best entrepreneurs, VCs, anyone with interesting insight. It's a surprisingly pure channel of news/knowledge if you use it correctly and not go on a follow spree.
It's definitely not the place for social networking as far as communicating with friends/family goes.
I've been a Twitter user since 2006 and mostly just use it to post sarcastic and pretentious remarks for my own enjoyment; the social aspect of it is almost meaningless to me. Many of my friends in tech do have Twitter accounts, of which they make varying degrees of use (though I'm not sure how many of them would regard it as essential - probably not many).
Most of my non-tech friends in real life broadly do not have Twitter accounts, or if they do, they follow about eight people (usually including Stephen Fry) and have either never tweeted, or only tweeted about four times, usually starting with, "Well, I'm on Twitter now. What do I do next?"
I'm talking about people in their 20s and 30s here. The same set of people do make a lot of use of Facebook (though a few, like me, find it vastly irritating and have deleted their accounts in recent years - but I'm not sure I know anyone female in that age bracket who doesn't have and use a Facebook account).
A bit more anecdata while I'm at it: Instagram seems pretty popular in cities, where people enjoy posting pictures of their pulled pork burgers and cocktails in jam jars - out here in the countryside, I don't hear much about it. The younger generation definitely do use Snapchat and the like, but I think most people over about 25 have no understanding at all of it. WhatsApp seems to have a very broad and universal appeal, and is a real common denominator for anyone who falls under the category of "I want to keep in contact with a friend who lives abroad" (I don't fully understand how it won out over Skype, apart from Skype being a very poor experience on mobile). Lastly, I'm not sure I know anyone who doesn't work in online media who actually uses Pinterest.
This is bad, if not entirely unexpected, news. What is out of the ordinary is the public markets' reaction: the stock is down 3% in after-hours trading.
Typically when a company announces layoffs its stock goes up. More mature companies have sometimes slashed headcount on the way to a better market situation. But when younger companies announce layoffs, the markets stop assuming they have lots of growth ahead and adjust expectations accordingly -- usually resulting in a significant drop in stock price.
Remember, back in July, Twitter announced earnings beating market estimates. This was at 4pm, its stock surged 4%. About half an hour later, CFO Anthony Noto said Twitter would not see "sustained, meaningful" user growth for a "considerable period of time." TWTR promptly plunged 10%.
I suspect that's just people having their delusions of rampant future growth and expansion of twitter being crushed in the gears of mundane reality. These moves are perfectly fine for a normal company that intends to continue operating at about the same level as it is currently indefinitely. But people view every tech company through the lens of apple, google, amazon, etc. They expect growth, they expect new stuff, and so on.
The truth is that with modern technology you can build a company that has hundreds of millions of high-volume users and yet is not very expensive to operate. Companies, and their revenue, don't necessarily scale the way we're used to from the experience of industrial companies.
They turned off their APIs to application developers. What they should have done is ramp that shit up and figure out how to monetize the query traffic or figure out a way to decentralize their infrastructure to allow everyone to run Twitter together.
>What they should have done is ramp that shit up and figure out how to monetize the query traffic or figure out a way to decentralize their infrastructure to allow everyone to run Twitter together
So, I'm genuienly curious. How do they make money by "ramping that shit up"? I've seen a lot of companies that have 'get users' as their M.O., but I have no insight into the "profit!!!" phase. Opening their user base seems contary to "profit!!!", from a laymans perspective.
It seems to me like API/User info availability is their leverage w.r.t. profitability, much as it pains me as a potential developer who would like that information for free. How does decentralization make them (and their investors) money? I can't see a way to do both given the incentives.
While not stated eloquently, I do think there's some truth in that.
1. Open API's, good documentation and community engagement goes a long way in establishing developer trust. I don't believe developer trust is correlated significantly with revenue (directly), but it's absolutely correlated with acquisition. I'd argue that Angular or React were partially successful because of the stickiness of their communities, and that's a powerful self-perpetuating force.
2. Monetization is in the queries. Facebook and Google know this. Throwing aside all of their auxiliary services, emerging markets, hardware, et al... between 70-90% of their respective revenue is from direct advertising. If there's a way to scale or "ramp" their revenue model, this would be the portion to focus on.
Monetization of the queries doesn't appear to be a significant revenue opportunity for either Google or Facebook - why would this be different for Twitter? Not that this can't be a revenue source, but how big is it?
An alternative: I agree with everyone that they should have turned the APIs back on - they just should have made it mandatory to carry advertising as part of the feeds in order to access the feeds. They could have also layered on federated advertising for anyone consuming the API - meaning, allow those folks to show their own advertising in-stream. They simply pay Twitter a cut. Both of these may have helped them both grow the audience and advertising at the same time.
Monetization of the queries isn't a significant revenue opportunity because of scaling payment issues. Solve that, and you solve getting paid for doing something you are good at, which is throwing millions of statuses at millions of users.
I'm not getting why you think this data is so valuable or who you think the audience is - are you suggesting that millions of people would pay for the feed and it is just a problem of being able to pay?
The data is valuable only if someone has fair access to it. What an individual or a thousand may do with that data is an unknown and that is the real opportunity here, not the promise of a million subscribers paying you a dollar each. Twitter took that unknown, turned it off, and tried to own all the opportunity and all the answers. They did this because they IPO'd and then the need by investors to know what's going to happen took over even more than before. If there were answers to be had here, they'd already be doing it. I'm suggesting a move back to innovation, which is where they belong.
The desire by some individuals to make insane amounts of money on an idea are directly responsible for a massive SUCK experience for the end users of the software because those users are then forced to use the software in a way that makes the revenue predictable. I used to love to use Twitter because of all the wide variety of apps and things I could do with it with code. Now it's just unusable and dying. I would pay to make that different, but there isn't anything they offer that's worth my time and effort, so I'll go elsewhere.
"The data is valuable only if someone has fair access to it. What an individual or a thousand may do with that data is an unknown and that is the real opportunity here, not the promise of a million subscribers paying you a dollar each."
Why is the data only valuable if someone has fair access to it? Right now, basically anyone can go license the Twitter data and have access to the full firehose. And I don't know how you know if the data is real opportunity here (vs. advertising).
"The desire by some individuals to make insane amounts of money on an idea are directly responsible for a massive SUCK experience for the end users of the software because those users are then forced to use the software in a way that makes the revenue predictable. I used to love to use Twitter because of all the wide variety of apps and things I could do with it with code. Now it's just unusable and dying. I would pay to make that different, but there isn't anything they offer that's worth my time and effort, so I'll go elsewhere."
Are you offering a better path for Twitter and their shareholders, or are you offering a path that feels better, but may have a much lesser outcome for those parties? If so, why would they take it?
And while I agree with you that Twitter is dying, it's not unusable, it's just not usable in the way that you want to use it.
I'm sorry but I'm not quite sure what you mean by this?
In 2011, 96% of Google's revenue was from AdWords -- in recent years that has lowered slightly due to their diversification and cloud services, but it's still at least a 70% share of their gross revenues.
The original poster is talking about queries in a difference sense than a Google query - he's referring to it as an API call in a "firehose" sense - so I'm buying access to the Twitter data for my application. I'm suggesting that the number of people who will pay for those API calls is limited.
But no doubt that money is in the ads - that's why I suggested that they should turn the API back on, but require clients to display the ads. Sadly, I think this opportunity is gone as they have killed their developer ecosystem.
I'm the original poster, and I was talking about ads, not metered queries. :)
[edit] and perhaps that was a poor choice of words. as a developer of high-volume low-latency exchange feeds, "query" has a bit of a different meaning to me.
Nope - the misinterpretation is my end - my bad. Re-reading your original I think we're on the same page. I think they need to reopen their API (if it isn't too late) and then just send ads as part of that package to clients and require them to display them.
Trust initially makes you decide to use something (marketing) and trust also makes decide to stay with it (loyalty). Marketing itself is mostly related to raising interest, but it frequently 'hacks' trust by using biases to establish quick and cheap trust with consumers. If a company's product is trustworthy (real trust, not biased trust) then the loyalty becomes strong with the customer. Companies who acquire other companies regularly have identified their bad habits of using too much marketing to try to get more customers and instead focus on buying loyalty through an acquisition.
I think it's supposed to work like this: assuming more users == more revenue, then opening up and promoting the use of APIs creates alternative channels to acquire new users (partner apps have their own incentive to attract users etc). Where it can hurt is when the integrating apps either don't substantially move the needle on new user acquisition, or worse they cannibalize your user base by creating reasons for your users to NOT visit your site.
IMO though the API "shutdown" was shortsighted by Twitter. If people are using a competing interface instead of your own, then maybe you should fix your own rather than just shutting down the API. Of course that's easy to say as an outsider. I'm sure the truth is much more nuanced.
Facebook's API gave birth to the billion dollar juggernaut that is Tinder.
Tinder piggybacked off of Facebook OAuth logins in a useful way, and now Facebook has further cemented their permanence as it will now be very difficult for that dating app to sever themselves from Facebook OAuth.
Note I said "or" there. Decentralizing the infrastructure would make things cheaper and might give you revenue opportunity (and growth) that you wouldn't have otherwise. For example, perhaps you get into the business of aggregating ad sales to downstream providers.
As for directly monetizing the APIs, one thought that springs to mind is charging for the firehose in a federated way. You could, for example, have a query endpoint that looks like this to guarantee standard issuance of data, identity and payment:
And yes, I'm proposing Twitter should use Bitcoin to monetize query traffic. Even better, charge people for posting lots of statuses, or following and unfollowing lots of people. Link ask price to signal/noise ratio.
"Note I said "or" there. Decentralizing the infrastructure would make things cheaper and might give you revenue opportunity (and growth) that you wouldn't have otherwise. For example, perhaps you get into the business of aggregating ad sales to downstream providers."
Well, maybe, but you can't save your way to $2b in revenue. Stock prices don't generally go up on cost savings, they go up because of future revenue AND profit expectations. Silicon Valley and Wall Street both prize growth in revenue way over profit. But you're correct that they might discover another revenue stream - but now you're betting a lot on the value of federation.
"Make a payment to the address, then start querying at 50MB/sec for the duration of your valid token (in this example, 2 hours):"
I think you are vastly overstating the number of people or companies that would be willing to pay for this firehose. While it may be useful to some companies, overall the data isn't likely to be enough to be a $1b opportunity. More importantly, the number of people willing to buy isn't likely to grow - so a B2B play can be a minor part of their revenue but I don't believe they'd be able to depend on it for massive growth.
"And yes, I'm proposing Twitter should use Bitcoin to monetize query traffic. Even better, charge people for posting lots of statuses, or following and unfollowing lots of people. Link ask price to signal/noise ratio."
Given the current cost structure of other services (Facebook/Instagram/SnapChat), I'd argue you're unlikely to get anyone to pay for it, and you're reducing both the size and any potential growth of your audience dramatically.
> Data licensing and other revenue totaled $48 million, an increase of 95% year-over-year.
This is from their Q1 2015 report, so data licensing and other revenue is probably $200M ARR now. Clearly people are willing to pay for access. If they improve the way they provide that, it just might be bigger than a $1B opp.
I would be willing to pay for on demand abilities to control who I follow and unfollow without being throttled.
That's approximately 10% of total revenue. I also think you have the growth incorrect - in their 10Q, Twitter said the product was growing 65% YoY. So you're making two assumptions:
- That the opportunity is there - and that the growth will continue. This isn't a given, but it is possible. I think the fact that Twitter is obviously investing more in the advertising business tells us what we need to know about the data business - that it is good, growing, but will always remain a relatively small portion of their overall revenue mix. Now you may argue that that is because they're blocking it - but my assumption is that they are investing based on Total Addressable Market (TAM) and experience.
- That the problem is the payment system. There's little to support this - there's little sign of demand or willingness from individuals or small companies to spend on this at scale.
The fact that you're willing to pay for on demand controls is great and fine, but it does not mean that anyone else is willing in the same way.
> The truth is that with modern technology you can build a company that has hundreds of millions of high-volume users and yet is not very expensive to operate.
Investors had to dump more than a billion dollars into Twitter to get it to where it is today.
Uh, most of that money was from the IPO and that was probably done because founders/early investors wanted to cash in. I'd say the vast majority of that funding was not necessary to scale to where they are today.
Really? Wikipedia shows they raised over a billion from investors before IPO (including individual rounds of $800M and $300M.) Of the $800M, half went to cash out.
I'd agree though that at least technically, to deliver the core user product, this money simply wasn't needed. (It might have been needed for advertising or building up contracts, etc.)
It didn't help improve the interface much. For example, on this site the interface shows me quickly who is replying to whom. A billion dollars in and Twitter still can't do this. They tried to be the world's largest chat room and basically came up AOL.
And the UI changes they have made are demonstrably worse, such as their seemingly random threading, constantly losing page scroll position, and repeating earlier tweets.
How is it even possible for a company like twitter to spend a billion dollar and change in one quarter? That's enough to run a small country complete with golden bathroom for the dictator.
Time to go read about their OPEX I suppose. Anyone got a link to a good write-up?
Twitter is notorious for being ridiculously frivolous[1]. They also pay their employees very very high wages[2], which isn't a bad thing, but I assume that doesn't scale well.
And? Aside from the fact that that's not really true, what does it matter? Twitter already IPO'd, the original investors got their money back, now it's just owners. And it doesn't require a team of 4000 to operate twitter. It requires far fewer, and as time goes on twitter will generally just get cheaper to operate.
Twitter can easily bring in enough revenue to keep doing what it's doing indefinitely. And that's totally fine. But it's unlikely they're going to grow by 10x in a few years or pivot drastically. Yet those unrealistic expectations were built into the current stock price.
Google's stock price would not be at where it is now if the company "just" stopped at search. It's everything else they kept pursuing that's allowed the company to reach stratospheric valuation heights relative to its real revenue.
> What is out of the ordinary is the public markets' reaction?
But is it? Maybe the public is realizing that tweets were just a cool fad for a while. For twitter to be profitable, I am guessing, there has to be people that want to use it, and (very important) people believing that people want to use it (so they'd pay Twitter for advertising).
Compare with Google. Google provides something momre back (gmail, search, calendar, docs, photo storage, directions + many other things I forgot about).
Twitter provides 140 status messages. There is nothing wrong it, that is very cool. But I think there is a realization that it just isn't as cool as it was in 4 years ago. And it just became 3% less cool.
I don't know, I'd be willing to bet twitter usage is up compared to 4 years ago. Also, stock price != coolness, and further coolness != making more money. Isn't that what this story is about?
It seems to me that stock price is correlated with more money: if I have X stock options worth Y dollars each, I effectively have X*Y dollars worth of stock. If stock price increase by 1 dollars, I have X dollars more than I did before. No?
I think I remember that happening to Google at least once. There are expectations and "expectations". Analysts' published expectations are often different than the ones they believe privately, and the expectation might be to exceed the "expectation".
Beating published analyst expectations isn't the same as beating the actual expectations of investors. One might reasonably expect the actual market expectations to be loosely correlated with analyst's published expectations, and they probably are, but its probably only loosely correlated.
> It seems to me that stock price is correlated with more money: if I have X stock options worth Y dollars each, I effectively have XY dollars worth of stock
Sure, but the "market" price of stock isn't really the value of every* share, its roughly the marginal price of the last/next share purchased or sold, but the more you want to trade, the farther from the market price the value will be.
And changes to the market clearing price don't tell you about changes to the supply and demand curves, just where they intersect. So, for any but a small quantity of stock, you can't tell what the real change in the realizable value of your holdings is just from the change in the market price, since there's no guarantee that a change of $1 in market price equates to a change of $1/share in the price you'll get if you try to sell a particular quantity of a stock.
We tend to ignore that when placing a valuation on a portfolio because if you don't ignore it, you can't get a valuation. But those valuations are approximations, at best.
> bet twitter usage is up compared to 4 years ago.
It probably is. But probably not as much as everything believed it would be.
Coolness is about making money in case, but indirectly. If investors see others not being interested in Twitter and not regarding it as cool, they will pull out. I understand Twitter's primary revenue tool is advertisement. That requires users to advertise to.
> Twitter provides 140 status messages. There is nothing wrong it, that is very cool. But I think there is a realization that it just isn't as cool as it was in 4 years ago. And it just became 3% less cool.
I can't tell if you're being ironic or facile, but Twitter has become quite a bit more than "140 status messages"(sic). It's a medium where politicians, dissidents, celebrities, the hoi palloi, brands, and anarchists all interact. Its a place of record quoted in headline news briefs and long-form journalism. It is used to organize rebellion and restricting access to it is viewed by many as a human rights violation.
Twitter as a business has problems, but being a fad isn't one of them.
I'd heavily disagree with that. The fad's not over, but it is a fad. The reason it sticks around is that people have an easy way to communicate with celebrities/popular people, and that was certainly not twitter's goal. The moment something better for this comes along, the fad will quickly fade.
Twitter is actually becoming quite popular in science. It's a great tool to hear of new papers, find out about new findings reported in conferences you can't attend, and communicate in a public forum. Don't think you would have had much of that in MySpace...
> restricting access to it is viewed by many as a human rights violation.
Restricting access to any website (except in some extreme cases) is viewed by many as a human rights violation. I don't see how Twitter is unique in this respect.
While I agree with you, with Twitter the opposite seems to hold true. Many people like Twitter and see it as an integrated part of how they work and interact and that has driven the evaluation of the company. Not once has I heard anyone in the tech press or end in the financial news question Twitter.
Twitter should never have been a publicly listed company. They don't have a business plan, there was never any prof that they would be profitable. But Silicon Vally types, news media and bankers liked using the platform so: hey presto Twitter is a billion dollar company.
Twitter could be profitable if each user paid just $10 per year, but sadly I don't think people like it enough to pay up.
Well a lot of other people are saying "It's crazy that a company with that few people and that little profit can be worth that many billions." Hence Twitter's problem at the moment.
I wonder how many of the 4100 are tech. Probably less than half at this point. The rest are there to work with advertisers, publishers, celebrities. All to maximize profitability and all of them failing at it.
Twitter is doing something vastly more technically challenging that WhatsApp. WhatsApp just delivers messages. It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world.
This isn't a knock on WhatsApp -- I use WhatsApp and don't use Twitter -- but Twitter's scale challenges are vastly more complex than WhatsApp's, which doesn't have many features that can't just very simply horizontally scale.
> It doesn't store them, and it certainly doesn't, say, try to make them available in a searchable away to everyone in the world.
Back of the envelope calculation: If your messages are 256 char messages, on a 128GB ram server you can fit 500 billion of them in memory; incidentally, that's one day's worth of tweets these days. I recently got a quote for a beefy 256GB ram server from Dell for $6,000. To make things easier, let's assume $10,000 with redundant 1TB SSD and everything else you might need. The data is extremely easy to shard.
With proper caching, you can probably keep 99.9999% of all tweets shown in just one such servers, and serve them within a few ms. I don't have recent experience with web frontends, but a properly configured nginx could probably server on meager hardware can do a few tens-of-thousands requests per second. a Seastar server on beefy hardware can probably handle a significant percentage of Twitter's load.
Search is a little harder, but not by much.
Is this engineering trivial? Far from it. But I've worked with stunt programmers who can build this in a few months with a 3 person team. At least one of them commands a salary of about $500K last I heard.
Is that all Twitter is doing? No, I'm sure they have 10 backend "things" for every user visible "thing". They needed something like Bootstrap for internal use, after all. I don't really believe you can run Twitter with just $1M/year in capex, just $1M/year in opex and just $1M/year in engineering salary - you need a whole lot more than that; In fact, to pull in revenue of $2B/year (which Twitter seems to be on track for), you probably need 50 people just in accounting.
What I am saying, though, is that the technical challenges faced by Twitter (at least those that are visible to users) are highly overrated. If you know what you are doing, Twitter can be run on $50,000 worth of hardware; This has been true since 2007 - Computers were weaker and had less memory then, but Twitter had much less traffic too. I've looked at the numbers every single year since 2007, and at any point in time, the "bare bones essential hardware for interactive response" was in the $50,000 range. Which probably means if they spent more than $1M/year on hardware (or depreciation-equivalent-hosting) at any point in time, they should have hired better engineers.
You will also need space for fulltext indexes, indexes per hashtag (a set which grows constantly and never shrinks, but has a power distribution, meaning that some hot tags will overwhelm any single machine and it constantly changes), the ability to call up any tweet from any point in history (growing by one beefy Dell server per day), the ability to fan out to 1 or 10 or 10 million followers and so on. The fanout problem turns sharding into a lumpy problem because again, there is a power law distribution that can overwhelm any single machine.
Then there's the fun part that Twitter is so reliable these days that when something else on the internet breaks, we check Twitter for updates. Robustness at that scale is more than buying a pair of SSDs and setting up RAID.
One thing I've learnt from being a consulting engineer is to stop assuming other engineers are morons. Sometimes we do things out of ignorance, or out of fear, or out of necessity. But often, when you inquire, there is a solid non-obvious reason for non-trivial diversions from the simplest thing that could possibly work.
I always find it useful to give the benefit of the doubt.
But .. fulltext index done right, including per hashtag, doubles the storage requirements at most. And the ability to call up any tweet from history does not need to be as instantaneous is recent tweets (It's ok to wait 5 seconds for a tweet from 5 years ago).
> The fanout problem turns sharding into a lumpy problem because again, there is a power law distribution that can overwhelm any single machine.
Not really, in the case of twitter (where you only trace edges, not edges-of-edges and edges-of-edges-of-edges like Facebook and Linkedin do). In the simplified model I described, the web frontends all pull from the backend, but for the hot 10-million account followers -- of which there aren't that many --, you can just push them to the webservers so they don't even need to query)
> Robustness at that scale is more than buying a pair of SSDs and setting up RAID.
I most definitely agree. But it is cheaply doable when everything is so perfectly shardable as it is in Twitter.
> Then there's the fun part that Twitter is so reliable these days that when something else on the internet breaks, we check Twitter for updates.
That's possibly an illusion. Yes, Twitter is mostly reliable - but do you have any latency stats? e.g., would you know if, on a daily bases, 10% of tweets take 60 seconds until they appear on a viewer's refresh? The "hot" accounts would be cached and immediately updated, but the long tail might have minutes delay and almost no one would notice.
> But often, when you inquire, there is a solid non-obvious reason for non-trivial diversions from the simplest thing that could possibly work.
> I always find it useful to give the benefit of the doubt.
I do not assume other engineers are morons. In my experience as a consultant, though, those solid non-obvious reasons for non-trivial diversions are much more often than not "historical, do not apply any more", "we didn't have time or resources to do it the right way", "the guy who did the initial design did it wrong for whatever reason, and now we are stuck with it", "there's a legal reason that's not obvious why we can't do it that way", and a few others.
Twitter had the resources to do it better from very early on, and they didn't (I think it was 2012 or so before Twitter became reliable). For all I know, their system now could be the most efficient beast ever, and run on a ZX81 with 1K Ram with ultimate reliability, way better than I could ever hope to build.
I was just pointing out that the user facing side of twitter, technically speaking, is not very impressive. I've been doing it for a while on various technical forums - and not once did anyone offer a reason for why it's much harder than it would seem -- most of the responses were along the lines of "but mysql/pgsql/oracle can't take the firehose load, I tried!". Which is correct, but irrelevant.
By saying it can be done with $12,000 of hardware, you essentially claim it is trivial.
> everything is so perfectly shardable as it is in Twitter.
Shardability, or partitionability, depends on the underlying distribution. It's only "perfectly" partitionable if that distribution is perfectly uniform.
Twitter's traffic patterns are power-laws. Some tags vastly outnumber others. Some users have vastly more followers. Those distributions are unstable across time and space in surprisingly short order.
> Twitter had the resources to do it better from very early on, and they didn't (I think it was 2012 or so before Twitter became reliable).
Twitter wasn't interested in a contest to use the least hardware. They were in a race to keep up with surging load, while migrating from Rails to something that didn't exist yet. Meanwhile they have, under the hood, built infrastructure that could not be bought off the shelf from anyone at the time.
We can't actually come to a conclusion here, because you're arguing a counterfactual. It's always easy to have the ideal solution to a problem you never actually solved yourself, because the visible features of a problem are obvious and the many, many invisible problems are where the bulk of the effort are hiding.
If you want to prove that Twitter doesn't need such large and complex distributed systems, you can probably round up a few hundred thousand in angel funding and sell yourself to Twitter for a nice return.
> By saying it can be done with $12,000 of hardware, you essentially claim it is trivial.
Not at all. What I'm saying is that, with the proper software (which is not trivial to write), you can do it with very little hardware. I know stunt programmers making $500K/year, and they are in some senses infinitely (not just x10) more productive than bad and even average programmers - because they quickly produce working systems that others just can't.
Whether it makes sense to pay $50K/hardware and $500K/programmer or $2M/hardware and $100K/programmers depends on how you run your business, though - and in many cases, the $2M/hardware+$100K/programmers are the more economical choice (because you risk starving for stunt programmers choosing the first)
> Some users have vastly more followers. Those distributions are unstable across time and space in surprisingly short order.
And yet, as a system designer you actually get to choose what the distribution is of - and a good choice makes it uniform. The power laws may or may not favor sharding on uid specifically, but usually there's a simple way to shard uniformly. And if you can't find a way to programmatically shard uniformly, then shard using a lookup table on the userid - 8 billion user ids require all of 8GB of ram if you have 256 shards or less (and if you bundle users in groups of 256, 32MB is suddenly enough). Migrate around to keep balanced. This has been a solved problem for years. Really, even migration patterns. Look at "consistent hashing" literature -- (it's not the fundamental issue that consistent hashing solves, but the peripheral solutions are well known, common, and apply here: migration, redistribution, redirection).
> Twitter wasn't interested in a contest to use the least hardware.
> We can't actually come to a conclusion here, because you're arguing a counterfactual. It's always easy to have the ideal solution to a problem you never actually solved yourself,
First, I basically agree with you, if it wasn't clear. I know not what problems twitter were facing. I suspect that they weren't technical in nature, though -- because the technical issues have been solved before them. It might be management issues, it might be ego issues. I've seen more projects fail or stumble on those than on technical issues.
I actually did solve those problems myself, on a smaller scale (hence my interest, but with perfect sharding that would have scaled to any size, live migration and redistribution and all). But that startup folded because we sucked at getting traction - which only shows to go you that technical prowess matters not in these issues, or at least not much.
> many invisible problems are where the bulk of the effort are hiding.
Again, to be clear - I totally agree with you. I'm just disagreeing with the aura of engineering excellence that Twitter gets in this (and many other threads). They may have it, or may not - I haven't seen evidence that they do. They can keep twitter running well in the last 3-4 years, which means they are reasonably competent. That's all I have evidence for.
> If you want to prove that Twitter doesn't need such large and complex distributed systems, you can probably round up a few hundred thousand in angel funding and sell yourself to Twitter for a nice return.
As I have mentioned several times in other replies (and other discussions) - twitter's problem is not, in fact, engineering, and hasn't been since 2012 at least (it definitely was 2007-2009). They threw enough money/people at the problem, and solved it.
Right now, they are bringing in $2B/year, but spending $2.5B/year. If they are spending more than $200M/year on user facing hardware at this point, I'd be surprised. I'd even be surprised if they are spending $100M/year. But let's assume that I can save them $200M - that's nice, but won't actually save them...
Thanks. I wrote 500B, but I did do the calculation with 500M; I used much larger estimates (as is always recommended in back-of-the-envelope calculation) - namely, 256 bytes per tweet - which, in addition to 140 bytes of text[0], leaves e.g. 8 bytes for user id (2 billion uids for every living person on earth), 8 bytes for microsecond timestamp, and 80 more bytes for pointers/caching/stats/whatever. (Those 16 bytes, incidentally, are enough to have a unique identifier for every tweet)
I assume the average tweet is closer to 80 bytes, and taking advantage of that can let you store twice as much tweets in memory.
[0] Does twitter count utf-8 bytes, code points, graphemes, or something else towards the 140-"char" limit? Regardless, the computation would not yield materially different results.
Ok. I was assuming the average tweet is < 144 chars, and that there would/could be compression applied.
In fact I would guess that the effective average information content in a tweet is very low :) (~10-20 bytes?)
WhatsApp can't tell me how many users were talking about Peanut Butter Sandwiches 30 seconds ago. WhatsApp can't alert me when people suddenly start talking about earthquakes, super bowls, or Elon Musk. WhatsApp doesn't need to fanout a message to millions of consumers, millions of times a day (and with multi-second latency to boot!).
WhatsApp also doesn't provide a CDN, mobile analytics/monetization/debugging platform. WhatsApp also doesn't contribute to open source like Twitter does (Storm, Bootstrap, Aurora).
Describing twitter as a "140-char" mailing list is very reductionist. OTOH WhatsApp problem is very well served by Erlang/Mnesia an open source distributed messaging library developed by telecommunications company who were likely doing WhatApp-scale 10 yeas before WhatsApp came around. While WhatsApp engineers are great, there is very much a "standing on the shoulders of giants" - giants who have well researched the problem of building a system that lets two or three individuals communicate. There wasn't anyone doing Twitter 10 years ago (well, maybe Google).
I don't mean to imply that Twitter is an efficient well oiled machine, but the comparison to WhatsApp is daft.
You're underestimating the difficulty of scaling Mnesia the way WhatsApp did (one source says "In-memory Mnesia database using about 2TB of RAM sharded across 16 partitions to store about 18 billion records." [1]). The prevailing wisdom in the Erlang community was that Mnesia could not be made to do such things. The only well-known user of Mnesia at anywhere near this scale prior to WhatsApp was Klarna. To achieve this, Klarna employed at least one of the original Mnesia developers who wrote proprietary patches for their system [2]. So while you accuse parent of being reductionist, your views of WhatsApp's engineering challenges also seem reductionist. Erlang/OTP software doesn't write and scale itself ... large-scale distributed systems are decidedly non-trivial no matter what tools you've chosen.
When I checked this post, you were below zero - I'm not sure why you are downvoted.
In any case, my main point isn't that that WhatsApp is simple. When I said the parent posted was being reductionist, I was referring to Twitter's product offering. AFAIK (I could be wrong) WhatsApp is a messaging backend and multiple client frontends. The "50" engineers is deceiving becaus, despite the fact that Mnesia could or couldn't have scaled, there is a very non-trivial amount of engineers who worked on it and Mnesia is a good & proven system. In a similar light, Instagram was 13 engineers (or was it employees) and hundreds of django instances - and that maybe all you need for a photo feed, however Instagram didn't have a trending feature until earlier this year and how knows how many engineers that took.
If WhatApp hired another 200 engineers, likely 195 would either be twiddling their thumbs or working on new features outside what was needed in that acquisition to Facebook. Now we can make the argument that Twitter is bloated because they have too many products (Fabric, Vine, Crashlytics, and their numerous other acquisitions) - and the scaling back of those engineers may mean sunsetting those products - and thats fair. To imply they have 4000 engineers building a glorified 140-char mailing list is not.
> 4100 employees for a service dedicated to moving around 140 character messages? Yeah, there might be a little fat to trim there.
Absolutely! You could probably get a single rack to handle storage and broadcast of millions of tweets per minute, perhaps hook it up to a Teradata appliance to handle search and storage at that throughput. Throw it in a nice distributed database indexed by user. Then just another couple of indices for the hashtags and mentions, and the follows/follower graph. Throw in another few tables for handling API permissions, managing schema migrations while ensuring that data from day 1 is still as transparent as yesterday's logs, a highly replicated ad server hooked up to a realtime recommendation engine that analyzes an active user's current stream and recommends the most relevant ad (to maximize revenue of course), set up a few staffed policy teams to comply with the differences in global regulations, handle lawsuits and legal requests, and of course keep competing for more revenue with the likes of Facebook and Google.
Whatsapp is an outlier in many ways and comparing them with any company will make that company seem bloated. As for the comparison, Whatsapp has no monetization model, no marketing and a simpler product.
They used to send me messages about that yearly few but they always ended up extending my membership for free every few months. Then they stopped asking. WhatsApp is de facto free.
In 2014, WhatsApp reportedly made $14 million in revenue. By comparison, Twitter made $479 in revenue in just Q4 2014. I think company size generally scales super-linearly with revenue, so a total staff in the thousands seems reasonable for a post-IPO company.
Are you suggesting that WhatsApp's model of organization is the norm for a sustainable organization? I think it's a clear outlier in many respects.
The company existed to make a return for the owners. The owners exited with $16 billion. That was the motivation of the owners/employees, not $14 million in annual revenue.
I guess that is the argument anyway. FB also recorded it's estimate of the useful lifetime of the WhatsApp users it acquired as a finite intangible asset worth over $2 billion.
My point is that the transaction must be analyzed from the viewpoint of the owners (as that is what the poster I was responding to cared about, viz a viz total profit/revenue to Twitter). When you do that, WhatsApp cashed (well, assuming FB stock can be treated as cash) out more per employee than Twitter.
Sure, lots of other factors are important, but ultimately factors like user base growth and active users are just showing WhatsApp gaining a larger revenue stream.
WhatsApp's revenue that they earned from that user base is what was driving the valuation from the perspective of everyone not in Facebook's camp. Facebook woukdnt have been able to buy WhatsApp if WhatsApp thought they could make more money off that username than Facebook.
> Are you suggesting that WhatsApps business model was based on annual revenue?
Since WhatsApp was a revenue-generation corporation with a board of directors, like any other corporation, before their acquisition, and since I don't like to indulge in conspiracy theories, I'm going to assume that yes -- their business model was based on annual revenue.
> I'm going to assume that yes -- their business model was based on annual revenue.
Then you'd assume wrong. The goal of the board is to maximize return for investors. In WhatsApp case, it's not hard to see that their goal was to grow as fast as possible, revenue be damned, so they could be a very attractive takeover target for a company like Facebook who could eventually hope to monetize that user base.
They're nowhere near relevant. I keep getting local-to-US ads, and I'm in UK (and geolocate as in UK too). Not as bad as showing alcohol ads to 5 year old children, but still pretty bad.
This doesn't mean that there aren't teams of people trying to constantly make those ads more relevant. And since not everyone at a Twitter can statistically be incompetent, I'm going to assume that the problem is actually quite difficult, and perhaps requires a large team.
Did WhatsApp have much in the way of storage needs? I don't think that they did. This is where the cost goes, durability. They were basically just a giant switchboard with some buddy lists (aka glorified MSN Messenger with SMS gateways). It's no wonder they were able to keep costs down and grow. They didn't have a lot of storage and persistence needs, just passing datagrams around the Internet between phones.
Edit: My apologies if I've offended the HN mystique surrounding the god-like status of WhatsApp. Ask Jan yourself, they weren't geniuses. I know Jan. He won't claim to be this exalted persona.
It takes engineers to run and utilize all that stuff. You don't need a back-end scaling team you're only throwing around XMPP in Erlang.
Edit: I know b/c I've done that before in $previousjob. Storage is hard. Storage is where the money goes. Durability is expensive. Storing ephemeral stuff in a non-durable KV store is easy, cheap, and trivial to scale. This is why WhatsApp didn't have so many people, not because they were geniuses.
The need for consistent, highly available persistent storage in a distributed system with a heavy write workload vastly complicates engineering effort. You're fighting fundamental issues like the CAP theorem. When you can't "just use Postgres" everything else gets a lot harder. Companies that have this problem tend to have a lot of engineers: Facebook, Google, Uber, etc.
Disks are slow. If you don't need to wait for disk, then you can handle a great deal more req/s per node, which means the point at which you need to become a distributed system is further away. Without the state synchronization problem, converting to a distributed system consists of little more than adding a load balancer.
Even in an environment that needs synchronized state, the services that can be stateless are much easier to deal with (and you try to make as much of your infrastructure stateless as possible).
Storage costs are not relevant compared to the costs of engineering distributed systems.
I agree with the concept but I don't think Uber falls nearly in the same ballpark as something like Twitter on scalability needs. IIRC Uber's huge headcount is due to it having local recruiting/marketing teams pretty much everywhere
This is just rubbish from developers that don't know what they are doing. I run Sublevel on a $5 VPS from DigitalOcean. Guess what, I optimized it to run smoothly in any situation.
4100 employees is completely out of whack with reality. And for those saying they are incorrectly trying to scale to $20 billion...thinking you will be a $20 billion dollar company in the future (even if it were true) would not justify hiring that many people now.
Without knowing the breakdown I assume that there are a lot of people working in marketing type roles where they are working with big business and celebrities to get them using the platform.
Any company that operates at web scale is a technology company. Besides that look at their opensource contributions. Twitter has/had some impressive engineering talent.
I'm not sure why you're being downvoted; their head count was absolutely insane. Yes Twitter has some insane needs for scaling but 4,100 employees for a 140 character service with some basic ads? Absolutely far too many.
Do you really think it's all just tweets and ads? Spoiler art: it's much, much, much more than that. Massive analytics stack, tons of distributed computing needs, lots of well-maintained OSS projects (Mesos, Aurora, Finagle, Pants, Bootstrap, Zipkin). I could name about 20 other things. The headcount may be a bit excessive but this is a caricature.
I have to jump in and side with the grandparent. Despite everything you just mentioned, if it isn't about "tweets and ads" I would contest that it is exactly about them. Do whatever you want back-end, but that doesn't matter if my front-end experience is not so great. For example, they are terrible with spam bots. I mean when super-model-quality photos suddenly follow me, and the bot is posting shortened URLs, it makes me feel like they aren't on the ball at all.
Those 4,100 people are not just facebooking all day. The question isn't whether they produced anything - it's whether it is actually needed to run twitter. And personally, I believe the answer is "not really".
As an example, they could have gone with Zurb Foundation instead of doing Bootstrap. They could have used (redo, cmake, tup, a hundred of other tools) instead of doing Pants.
If Twitter was non-profit, their breadth of products would have been awesome. But it's for-profit, and most of these projects are negative on the P&L summary, even if you favourably take into account intangibles like "help in recruiting people" and "coolness of brand".
Well sure. I don't think many people think all those people were sitting around twiddling their thumbs all day, but none of those open source projects ultimately benefit Twitter. If those had been OS api clients it would seem prudent (except twitter doesn't like third party integrations any longer).
Cockroaches can survive extremely hostile environments that more "advanced" creatures cannot. They might not be pretty like unicorns, but they are extremely well adapted to resource-poor environments.
sure. There was what seems to be a correction in the PE market where funding become a bit more difficult to get a hold of. As such, an article came out suggesting that unicorns would have problems but cockroaches would survive. While the unicorn portion is a bit silly, the cockroach part is reasonable. Essentially, the companies with low burn rates and actual, real, cashflow will be more likely to survive.
Facebook has 3 properties which have more users than Twitter (Facebook, Whatsapp, Instagram). Facebook itself has more active users than Google and has another property, Whatapp which is also close to or just over that billion user mark.
Nothing lasts forever in tech. How will they monetize Instagram and WhatsApp? Don't show me users. Show me profits from those users if you want to count them.
Is that how it works now? Companies with great products but not a sustainable long-term biz plan exit by getting bought out, and larger companies with a greater product but also no far-long-term biz plan buy those companies and survive by having them as fallback products?
Sounds like the beer industry. Five companies own half the world market. And new breweries are easy to start. If yours gains traction, you get acquired.
And that's in an industry where the products are close to fungible, without depending on network effects.
I guess the one upside about tech is that ideally you're in a field with technology that can tie together disparate products into one ecosystem, so it's not simply about making megacorps that are essentially giant holding companies that own random products and patents.
On the other hand, one binding ecosystem with disparate products can easily become either walled gardens or badly-fragmented loose confederations, so it's not the best situation, either.
Facebook (the service) itself can maintain the company as is, but they have ads on Instagram at least. FB currently makes several dollars per user per quarter in revenue (or profits), with that value depending on what number you go by, but the smallest number you could pick is 1B+. No business is guaranteed to last forever, but there is really no reason to think that they have any real trouble ahead.
The easiest solution would be to expand Whatsapp to support mobile payments. A lot of commerce in countries in Africa and Asia is already conducted over the phone using SMS and other similar technologies, and FB could fairly easily have Whatsapp play that role and earn a decent chunk of money.
Inserting ad images into Instagram would be fairly trivial. Given the large number of celebs on Instagram, shared revenue on product placement is another option. Promotion of high profile Instagram accounts (say, a new movie) etc
If you think profits are still the driving force of what matters, then you will never build an Instagram or WhatsApp at the exit values they achieved.
This sounds like 90's bubble talk, but "eyeballs" do matter. Investors and the market care about the current social value of a product in terms of the value / growth of their user base.
Ask many millenials/teens and they say they more meaningfully interact with Instagram than Facebook. That fact about social trends is worth its weight in gold in terms of its user base.
Was Google cargo culting by turning down efforts by its early investors to have them show banner ads like Yahoo?
Their willingness to hold off for many many years without revenue was now brilliant in hindsight, given they found a much better way to make money on advertising than how Yahoo ever imagined it. But they wouldn't have got there if they caved into short-term thinking obsessed investors who think you need to post immediate revenue.
Twitter's issues are more to do with a lack of product innovation or evolution, not that it has gotten too large. Its problem is that it has gotten too large without expanding its product base to support or justify that growth.
> Ask many millenials/teens and they say they more meaningfully interact with Instagram than Facebook. That fact about social trends is worth its weight in gold in terms of its user base.
Ask how many millenials or teens have bought something or spent money because of an ad on Facebook or Instagram.
No where near as sophisticated as Google's or IBM's. These companies were not founded by rebranding MySpace as Facebook did. Google was founded based on a sophisticated algorithm that had its roots in AI while facebook was copying and pasting text entry boxes.
What on earth does market cap have to do with anything? Their margins are negative 30 percent, and their cash flow is like 150 million dollars. Increasing revenues doesn't mean a whole lot of you're losing money on every sale.
Not necessarily. They have 4000 employees, which probably has a number of non-revenue-generating positions that many SV companies are wont to have. Like Python API WebSockets Diversity Developer Envangelists with full travel budgets to go to 49 conferences a year. (I just made that up, but you get the idea)
Maybe judging a boom by twitter layoff is a bit too soon. However, the underneath mindset rings a bell: people starts to realize winter is coming, but not sure when...
For many businesses, that might be 2 or 3 too many.
Also, that was just an example I pulled out of the air. It could be middle management, or a documentation expert whose role is regulating font size. The point is that for a company with a simple product, an API and clients for that API, and monetization through advertising, that's a lot of employees. If the money's not there, cut the fat.
I used to be a Google Reader/Google Buzz addict. And was interacting with a very living community through those tools.
I now use twitter as my exclusive medium for both mainstream and "niche" news.
I appreciate the wide availability of content, but still think they are much too conservative with their user experience.
The inability to browse tweets by topic is sometimes a pain.
The inability to connect with the right people and build a discussion about topics you discover and find interesting.
Basically twitter should have a chatroom. And propose you conversations about (your) trending topics.
the limitation of which is that most content on a user stream is pulled passively by the users or is a paid advertisement.
as a discovery/exploration tool sucks.
if it had interesting suggestion based on hashtags and followers beyond 'follow this three more person' daily mail it could be a nice networking/news tool, but so far it cannot really help me navigate in the millions streams that might be interesting but are accessible only knowing that exact hashtag
I followed a couple news outlet, I guess there are many more that may interest me but I can't be assed to search them and if I discover them myself they go by preference into feedly, not twitter.
on the other hand, it had been great to follow this and that indie game developer :D
Google Reader got so much right, except maybe feed discovery. Google could have incrementally morphed Reader, something that worked. Twitter is like RSS for regular people.
This what happens when you create a company around a glorified public majordomo public email list manager and virtuous feedback loop used by the media. I'm looking for the Google and companies that invent real tech.
Just because you don't derive utility from a product doesn't mean it's not 'real' tech. Twitter is basically the news medium for me and most of my friends at school. I cannot remember the last time I went to read a news article from a site other than (mostly) twitter.
I would not go that far, but I can certainly say -- seems like half the accounts are re-tweet bots and random products I have no interest in trying to follow me.
@Whitenoice, recruiters hire them mainly because of their skill set. Keep in mind, most layoffs are because a department is being eliminated and the is no where for the employee to go but to take a severance and leave.
This is a good list. Two more reasons:
- Corporations are inefficient, sometimes the HR department is not even aware layoffs are coming
- Corporations are sometimes unethical, they don't care about the unfairness of hiring a new worker rather than trying to re-place a worker about to be laid off
"Moments" tab gives a good indication that Twitter has no idea where to go and is the sum of boring meetings with marketing consultants.
Twitter had potential, but it appears stagnate as all they've done is created a new version of an AOL instant messenger chatroom with hashtags and 128char limit. Limiting the API tokens for 3rd party developers was a really foolish move.
How do you mean? I think of Moments are formalizing something that Twitter does well: conversations/stories around topics of current interest. Sure, you'll engage your connections on these topics. But it's nice to see a broader view of them as well.
Completely agree. "Moments" just has that damn indicator showing stuff is unread on it and it drives me nuts but NOTHING is EVER relevant to any of my interests or even the people I follow. I don't even understand what the point of it is; shouldn't it just be displaying things it thinks I want to look at? At least that makes the most sense to me...
Oh well, I don't think Twitter is going away any time soon but I don't really understand some of the steps they've been taking...and 4,100 employees? Yikes!
it's because you're not their target. Their goal with these features is not to retain existing users but to acquire ones who wouldn't have otherwise used Twitter.
Fair enough. That probably goes for almost everyone on HN as well I would think but obviously we're an edge case.
> Their goal with these features is not to retain existing users but to acquire ones who wouldn't have otherwise used Twitter.
Hmm, how would this attract new folks? Like Facebook tries to show you things you care about but this I'm not sure how it's really sold to perspective users. Since it's a social network I'm guessing much will be word of mouth but that requires support from existing users.
I wonder if things like increasing the character amount and offering a better, maybe threaded conversational way of interacting with folks would bring in more people. Granted those may not be the best choices philosophically but I wonder if they've done any A/B testing where certain areas got different features like more character counts and how it turned out.
If they're really going toward content, I think Moments is a good idea with a questionable name. Something like Super-GoogleNews could eventually evolve, Sprinkle a little recommendation engine over and you got yourself a custom page full of wordstuff.
I agree. Add in the proper personalization and some kind of Twitter-unique algorithm that almost serves as a voting system (perhaps based on favorites, retweets, replies, etc), and you've got yourself a Reddit replacement. Which could be nice, now that Reddit has decided to become a censored safe place and it's core users are leaving.
I'm not much of a user of "Moments" myself, but I'm guessing it is primarily targeted at new users who sign up and go "what now" and don't know who to follow, etc.
One of the interesting parts about Moments, is that I used TweetBot and that feature is nowhere to be found. That being said, I don't even feel like I'm missing anything. It seems like ownership of the product experience is fragmented and the best implementors of the product experience aren't Twitter.
The most innovative technology to have come out of chat programs is Slack's feature that let's you EDIT chats in real time, and leave reactions.
This is such a crazy innovation to the structure of chat programs, that it blows my mind other real time communication platforms haven't incorporated it. Even Facebook Messenger and GChat feel inadequate now that I'm consciously aware that they don't let me edit my chat messages after sending them. It's such a no brainer innovation that nobody has ever done, likely because when you're building a chat program the idea of letting people edit chats after they send them makes the brain of any engineer hurt -- but there's an untapped value to allowing it that many products haven't realized yet.
I'm surprised it took so long. Hopefully the other social-networking giants follow suite and we start getting more companies that have actual sustainable business models.
I see this as a good sign (other than for those laid off of course), Twitter is fantastic but they don't need half as many people as they currently employ for their product. As companies get larger their agility and ability to change decreases. What's important is if they're laying off the right people.
I got an email from a Twtr recruiter two days ago and now am trying to word my polite reply that now doesn't look like the best time to be quitting a good job to join a company that's in the middle of layoffs. I guess they didn't get the word.
Note that it is "if", and not "if and only if". Basically, if you can be tracked or advertised to ("advertracked"), you ARE the product whether you pay or not.
Money has to come from somewhere; if it is not from you, it is by monetizing you. But ... why not monetize the user even when they did pay? It's a whole new revenue stream, that's often orthogonal to the payment stream.
Paid Hulu has ads. Windows Solitaire has ads now. Movie theatres have been showing ads since forever. Cable TV is showing ads.
And now, thanks to the hobby of one developer, my phone blocks ads in /etc/hosts, preventing even in-app ads from appearing. My web browser on my laptop does just as well with a plugin.
All that's left in my browsing experience to hint that sites are trying to maintain themselves on advertising are reels of clickbait stories that come at the side and bottom of every traditional media article. I can't help but hope they are only a technocultural shift away from the abyss of anachronism.
The technology can be used to target us, but in the end machines do the work and they are very predictable. We also have machines, and knowledge, and can stand up against the bullshit.
> Paid Hulu has ads. Windows Solitaire has ads now. Movie theatres have been showing ads since forever. Cable TV is showing ads.
The problem with my idealistic vision of the near future is that we are just as likely to accept this as normal. We habitually pay for stuff and still get ads. We have the tools to block the crap but it's much more important that we decide to.
Millions of people gave up eating gluten because they learned about celiac disease. I'd like to imagine that we are just a few headline-grabbing research papers away from swaying the public about the healthiness of exposing ourselves to psychological bombardment.
Most apps use 3rd-party libraries and servers to show those ads, so they can be blocked at the network connection level (e.g. the hosts file) if you have control over your phone's OS. Having Android helps with that.
The apps need to handle it gracefully because they can't tell whether the ad server is legitimately down or just blocked. I've never had an app work any worse with blocked ads.
Windows Solitaire has ads now. Movie theatres have been showing ads since forever. Cable TV is showing ads.
Although if I were trying to think of business models that seem to be struggling, those would probably have been among the first examples that came to mind. Windows 10 so far appears to be another Windows Vista/8. The movie theatres around here seem to operate under constant fear of closure. We don't have a complete equivalent of US cable TV here, but I get the feeling that on-line, on-demand services are threatening that model as well. In each case, I can't help suspecting that the degree of customer exploitation/price gouging has a lot to do with the waning popularity.
> There's a popular meme "If you're not paying, then you're the product".
Popular and irritating since (a) it's incomplete (if you're paying, you're increasingly product, too, since companies aren't going to leave rifling through your data as a way of making money alone), and (b) it's become a kind of astroturf ad campaign for certain companies.
So it is possible the ceo switch happened now since the board was willing to reduce company size, instead of the other way around? I can imagine markets to take it better if this appears to be the first wise move of a very capable ceo.
So it is possible the ceo switch happened now since the board was willing to reduce company size, instead of the other way around? I can imagine markets to take it better if this appears to be the first wise move of a very capable ceo.
413 comments
[ 31.6 ms ] story [ 6365 ms ] threadCompanies need to have real revenue and produce real profit and then need a valuation based on sane multiples. Twitter got behind that curve badly and ultimately this is what happens.
Layoff are the result of incompetent management, and nobody wants to work for a company staffed with incompetent managers.
Perhaps I missed it but no where did I see any specific mention that this was either surgical or across the board. Care to elaborate? If they didn't specify it certain could go either way.
It's not going anywhere in the next year or two. But in five years? Ten years?
Twitter's main problem is that it has one product, and it's possible to get bored with that product. It's not really possible to get bored of googling things or buying things off amazon, but it's entirely possible to get bored of twitter.
Remember when every man and his dog had a blog? (Or possibly a livejournal?) Remember when the word "blogosphere" was all over the news? What happened to all the blogs?
> What my vague acquaintance had for breakfast
> What my vague acquaintance had for breakfast
> What my vague acquaintance thinks about politics, which I need to grit my teeth and not reply to.
> Slightly amusing post from a well-known celebrity I follow, which someone will post on facebook later in the day anyway.
> Slightly amusing post from a well-known celebrity I quite deliberately don't follow, but which gets retweeted into my feed anyway.
> A politician I am starting to regret following wants my money.
> What some well-known celebrity I don't follow had for breakfast
Tools do something useful. Twitter doesn't -- at least, not for 95% of users.
They became listservs. Ironic huh?
Their problem is that costs are far out of line, which is why they're cutting. Twitter could turn a significant profit if they design for $3 or $4 billion in sales (rather than pretending they're heading toward $20 billion). Their margins should be comparable to Facebook's, their business is no more difficult to optimize.
They have 12% of the sales of Facebook, with ~38% of the employee base. It's an extremely poorly run, bloated business. They're built to scale to Facebook's size, but they will never achieve that size no matter what happens. Long-term they're a $20 or $40 billion market cap business, instead of $250 billion.
Google's original investors were FUMING that the company didn't pursue ads earlier, following with what Yahoo did. In retrospect we now see the insight of that approach.
Twitter's problem is one of product. It's not innovating, it hasn't changed it's core formula in many years. Compare that to Google and Facebook which both made many bold improvements to their platforms after booming into success. (Though facebook one could argue has slowed down in its innovation.)
I wonder how many unicorns lack these criteria. Only Twitter has the discipline of public markets. If it doesn't deliver, public shareholders get burned.
If unicorns don't deliver, private investors get burned, but where do the losses go? VCs, founders, employees, as well as some family offices, banks, university endowments, pension funds...where else? And I sure am curious about the exposure of these last three groups.
Nonetheless, I've had trouble convincing my parents why they should use Twitter.
Companies such as AT& T, Chase, Citi, Bank of America, Amazon etc..
I have ditched Facebook that's a cesspool of humanity and HR magnet.
But since I started really getting into it last year, I've become so much more knowledgeable by following the best entrepreneurs, VCs, anyone with interesting insight. It's a surprisingly pure channel of news/knowledge if you use it correctly and not go on a follow spree.
It's definitely not the place for social networking as far as communicating with friends/family goes.
Most of my non-tech friends in real life broadly do not have Twitter accounts, or if they do, they follow about eight people (usually including Stephen Fry) and have either never tweeted, or only tweeted about four times, usually starting with, "Well, I'm on Twitter now. What do I do next?"
I'm talking about people in their 20s and 30s here. The same set of people do make a lot of use of Facebook (though a few, like me, find it vastly irritating and have deleted their accounts in recent years - but I'm not sure I know anyone female in that age bracket who doesn't have and use a Facebook account).
A bit more anecdata while I'm at it: Instagram seems pretty popular in cities, where people enjoy posting pictures of their pulled pork burgers and cocktails in jam jars - out here in the countryside, I don't hear much about it. The younger generation definitely do use Snapchat and the like, but I think most people over about 25 have no understanding at all of it. WhatsApp seems to have a very broad and universal appeal, and is a real common denominator for anyone who falls under the category of "I want to keep in contact with a friend who lives abroad" (I don't fully understand how it won out over Skype, apart from Skype being a very poor experience on mobile). Lastly, I'm not sure I know anyone who doesn't work in online media who actually uses Pinterest.
Typically when a company announces layoffs its stock goes up. More mature companies have sometimes slashed headcount on the way to a better market situation. But when younger companies announce layoffs, the markets stop assuming they have lots of growth ahead and adjust expectations accordingly -- usually resulting in a significant drop in stock price.
Remember, back in July, Twitter announced earnings beating market estimates. This was at 4pm, its stock surged 4%. About half an hour later, CFO Anthony Noto said Twitter would not see "sustained, meaningful" user growth for a "considerable period of time." TWTR promptly plunged 10%.
http://www.cnbc.com/2015/07/28/twitter-earnings-7-cents-per-...
The truth is that with modern technology you can build a company that has hundreds of millions of high-volume users and yet is not very expensive to operate. Companies, and their revenue, don't necessarily scale the way we're used to from the experience of industrial companies.
So, I'm genuienly curious. How do they make money by "ramping that shit up"? I've seen a lot of companies that have 'get users' as their M.O., but I have no insight into the "profit!!!" phase. Opening their user base seems contary to "profit!!!", from a laymans perspective.
It seems to me like API/User info availability is their leverage w.r.t. profitability, much as it pains me as a potential developer who would like that information for free. How does decentralization make them (and their investors) money? I can't see a way to do both given the incentives.
1. Open API's, good documentation and community engagement goes a long way in establishing developer trust. I don't believe developer trust is correlated significantly with revenue (directly), but it's absolutely correlated with acquisition. I'd argue that Angular or React were partially successful because of the stickiness of their communities, and that's a powerful self-perpetuating force.
2. Monetization is in the queries. Facebook and Google know this. Throwing aside all of their auxiliary services, emerging markets, hardware, et al... between 70-90% of their respective revenue is from direct advertising. If there's a way to scale or "ramp" their revenue model, this would be the portion to focus on.
An alternative: I agree with everyone that they should have turned the APIs back on - they just should have made it mandatory to carry advertising as part of the feeds in order to access the feeds. They could have also layered on federated advertising for anyone consuming the API - meaning, allow those folks to show their own advertising in-stream. They simply pay Twitter a cut. Both of these may have helped them both grow the audience and advertising at the same time.
The desire by some individuals to make insane amounts of money on an idea are directly responsible for a massive SUCK experience for the end users of the software because those users are then forced to use the software in a way that makes the revenue predictable. I used to love to use Twitter because of all the wide variety of apps and things I could do with it with code. Now it's just unusable and dying. I would pay to make that different, but there isn't anything they offer that's worth my time and effort, so I'll go elsewhere.
Why is the data only valuable if someone has fair access to it? Right now, basically anyone can go license the Twitter data and have access to the full firehose. And I don't know how you know if the data is real opportunity here (vs. advertising).
"The desire by some individuals to make insane amounts of money on an idea are directly responsible for a massive SUCK experience for the end users of the software because those users are then forced to use the software in a way that makes the revenue predictable. I used to love to use Twitter because of all the wide variety of apps and things I could do with it with code. Now it's just unusable and dying. I would pay to make that different, but there isn't anything they offer that's worth my time and effort, so I'll go elsewhere."
Are you offering a better path for Twitter and their shareholders, or are you offering a path that feels better, but may have a much lesser outcome for those parties? If so, why would they take it?
And while I agree with you that Twitter is dying, it's not unusable, it's just not usable in the way that you want to use it.
In 2011, 96% of Google's revenue was from AdWords -- in recent years that has lowered slightly due to their diversification and cloud services, but it's still at least a 70% share of their gross revenues.
Facebook is currently ~80%.
The money is in the ads.
The original poster is talking about queries in a difference sense than a Google query - he's referring to it as an API call in a "firehose" sense - so I'm buying access to the Twitter data for my application. I'm suggesting that the number of people who will pay for those API calls is limited.
But no doubt that money is in the ads - that's why I suggested that they should turn the API back on, but require clients to display the ads. Sadly, I think this opportunity is gone as they have killed their developer ecosystem.
[edit] and perhaps that was a poor choice of words. as a developer of high-volume low-latency exchange feeds, "query" has a bit of a different meaning to me.
You are completely right, there is a huge strategic reason for promoting OAuth API's as a large company.
IMO though the API "shutdown" was shortsighted by Twitter. If people are using a competing interface instead of your own, then maybe you should fix your own rather than just shutting down the API. Of course that's easy to say as an outsider. I'm sure the truth is much more nuanced.
Tinder piggybacked off of Facebook OAuth logins in a useful way, and now Facebook has further cemented their permanence as it will now be very difficult for that dating app to sever themselves from Facebook OAuth.
As for directly monetizing the APIs, one thought that springs to mind is charging for the firehose in a federated way. You could, for example, have a query endpoint that looks like this to guarantee standard issuance of data, identity and payment:
Make a payment to the address, then start querying at 50MB/sec for the duration of your valid token (in this example, 2 hours): And yes, I'm proposing Twitter should use Bitcoin to monetize query traffic. Even better, charge people for posting lots of statuses, or following and unfollowing lots of people. Link ask price to signal/noise ratio.Well, maybe, but you can't save your way to $2b in revenue. Stock prices don't generally go up on cost savings, they go up because of future revenue AND profit expectations. Silicon Valley and Wall Street both prize growth in revenue way over profit. But you're correct that they might discover another revenue stream - but now you're betting a lot on the value of federation.
"Make a payment to the address, then start querying at 50MB/sec for the duration of your valid token (in this example, 2 hours):"
I think you are vastly overstating the number of people or companies that would be willing to pay for this firehose. While it may be useful to some companies, overall the data isn't likely to be enough to be a $1b opportunity. More importantly, the number of people willing to buy isn't likely to grow - so a B2B play can be a minor part of their revenue but I don't believe they'd be able to depend on it for massive growth.
"And yes, I'm proposing Twitter should use Bitcoin to monetize query traffic. Even better, charge people for posting lots of statuses, or following and unfollowing lots of people. Link ask price to signal/noise ratio."
Given the current cost structure of other services (Facebook/Instagram/SnapChat), I'd argue you're unlikely to get anyone to pay for it, and you're reducing both the size and any potential growth of your audience dramatically.
This is from their Q1 2015 report, so data licensing and other revenue is probably $200M ARR now. Clearly people are willing to pay for access. If they improve the way they provide that, it just might be bigger than a $1B opp.
I would be willing to pay for on demand abilities to control who I follow and unfollow without being throttled.
- That the opportunity is there - and that the growth will continue. This isn't a given, but it is possible. I think the fact that Twitter is obviously investing more in the advertising business tells us what we need to know about the data business - that it is good, growing, but will always remain a relatively small portion of their overall revenue mix. Now you may argue that that is because they're blocking it - but my assumption is that they are investing based on Total Addressable Market (TAM) and experience. - That the problem is the payment system. There's little to support this - there's little sign of demand or willingness from individuals or small companies to spend on this at scale.
The fact that you're willing to pay for on demand controls is great and fine, but it does not mean that anyone else is willing in the same way.
Investors had to dump more than a billion dollars into Twitter to get it to where it is today.
I'd agree though that at least technically, to deliver the core user product, this money simply wasn't needed. (It might have been needed for advertising or building up contracts, etc.)
2) Twitter reported half a billion in revenue last quarter.
And yet managed to lose 136 million dollars. While revenue is interesting, it just doesn't matter if you can't turn a profit.
Time to go read about their OPEX I suppose. Anyone got a link to a good write-up?
People are expensive, office space in their 30+ offices is expensive, hundreds of thousands of servers are expensive...
1. http://money.cnn.com/2014/03/06/technology/social/twitter-ca...
2. http://www.glassdoor.com/Salary/Twitter-Staff-Software-Engin...
Twitter can easily bring in enough revenue to keep doing what it's doing indefinitely. And that's totally fine. But it's unlikely they're going to grow by 10x in a few years or pivot drastically. Yet those unrealistic expectations were built into the current stock price.
Google's stock price would not be at where it is now if the company "just" stopped at search. It's everything else they kept pursuing that's allowed the company to reach stratospheric valuation heights relative to its real revenue.
But is it? Maybe the public is realizing that tweets were just a cool fad for a while. For twitter to be profitable, I am guessing, there has to be people that want to use it, and (very important) people believing that people want to use it (so they'd pay Twitter for advertising).
Compare with Google. Google provides something momre back (gmail, search, calendar, docs, photo storage, directions + many other things I forgot about).
Twitter provides 140 status messages. There is nothing wrong it, that is very cool. But I think there is a realization that it just isn't as cool as it was in 4 years ago. And it just became 3% less cool.
It seems to me that stock price is correlated with more money: if I have X stock options worth Y dollars each, I effectively have X*Y dollars worth of stock. If stock price increase by 1 dollars, I have X dollars more than I did before. No?
Exceeding expectations with a price drop afterwards is inexplicable.
No. Only if you sell it right now. A lot of problems come from people thinking stocks are more liquid than they are.
Sure, but the "market" price of stock isn't really the value of every* share, its roughly the marginal price of the last/next share purchased or sold, but the more you want to trade, the farther from the market price the value will be.
And changes to the market clearing price don't tell you about changes to the supply and demand curves, just where they intersect. So, for any but a small quantity of stock, you can't tell what the real change in the realizable value of your holdings is just from the change in the market price, since there's no guarantee that a change of $1 in market price equates to a change of $1/share in the price you'll get if you try to sell a particular quantity of a stock.
We tend to ignore that when placing a valuation on a portfolio because if you don't ignore it, you can't get a valuation. But those valuations are approximations, at best.
It probably is. But probably not as much as everything believed it would be.
Coolness is about making money in case, but indirectly. If investors see others not being interested in Twitter and not regarding it as cool, they will pull out. I understand Twitter's primary revenue tool is advertisement. That requires users to advertise to.
I can't tell if you're being ironic or facile, but Twitter has become quite a bit more than "140 status messages"(sic). It's a medium where politicians, dissidents, celebrities, the hoi palloi, brands, and anarchists all interact. Its a place of record quoted in headline news briefs and long-form journalism. It is used to organize rebellion and restricting access to it is viewed by many as a human rights violation.
Twitter as a business has problems, but being a fad isn't one of them.
I'd heavily disagree with that. The fad's not over, but it is a fad. The reason it sticks around is that people have an easy way to communicate with celebrities/popular people, and that was certainly not twitter's goal. The moment something better for this comes along, the fad will quickly fade.
Their entire userbase could do a MySpace overnight.
Restricting access to any website (except in some extreme cases) is viewed by many as a human rights violation. I don't see how Twitter is unique in this respect.
Twitter should never have been a publicly listed company. They don't have a business plan, there was never any prof that they would be profitable. But Silicon Vally types, news media and bankers liked using the platform so: hey presto Twitter is a billion dollar company.
Twitter could be profitable if each user paid just $10 per year, but sadly I don't think people like it enough to pay up.
Any fluctuation within 5% is perfectly normal and totally meaningless.
They haven't announced layoffs. The article is very clear about this.
This isn't a knock on WhatsApp -- I use WhatsApp and don't use Twitter -- but Twitter's scale challenges are vastly more complex than WhatsApp's, which doesn't have many features that can't just very simply horizontally scale.
Back of the envelope calculation: If your messages are 256 char messages, on a 128GB ram server you can fit 500 billion of them in memory; incidentally, that's one day's worth of tweets these days. I recently got a quote for a beefy 256GB ram server from Dell for $6,000. To make things easier, let's assume $10,000 with redundant 1TB SSD and everything else you might need. The data is extremely easy to shard.
With proper caching, you can probably keep 99.9999% of all tweets shown in just one such servers, and serve them within a few ms. I don't have recent experience with web frontends, but a properly configured nginx could probably server on meager hardware can do a few tens-of-thousands requests per second. a Seastar server on beefy hardware can probably handle a significant percentage of Twitter's load.
Search is a little harder, but not by much.
Is this engineering trivial? Far from it. But I've worked with stunt programmers who can build this in a few months with a 3 person team. At least one of them commands a salary of about $500K last I heard.
Is that all Twitter is doing? No, I'm sure they have 10 backend "things" for every user visible "thing". They needed something like Bootstrap for internal use, after all. I don't really believe you can run Twitter with just $1M/year in capex, just $1M/year in opex and just $1M/year in engineering salary - you need a whole lot more than that; In fact, to pull in revenue of $2B/year (which Twitter seems to be on track for), you probably need 50 people just in accounting.
What I am saying, though, is that the technical challenges faced by Twitter (at least those that are visible to users) are highly overrated. If you know what you are doing, Twitter can be run on $50,000 worth of hardware; This has been true since 2007 - Computers were weaker and had less memory then, but Twitter had much less traffic too. I've looked at the numbers every single year since 2007, and at any point in time, the "bare bones essential hardware for interactive response" was in the $50,000 range. Which probably means if they spent more than $1M/year on hardware (or depreciation-equivalent-hosting) at any point in time, they should have hired better engineers.
You will also need space for fulltext indexes, indexes per hashtag (a set which grows constantly and never shrinks, but has a power distribution, meaning that some hot tags will overwhelm any single machine and it constantly changes), the ability to call up any tweet from any point in history (growing by one beefy Dell server per day), the ability to fan out to 1 or 10 or 10 million followers and so on. The fanout problem turns sharding into a lumpy problem because again, there is a power law distribution that can overwhelm any single machine.
Then there's the fun part that Twitter is so reliable these days that when something else on the internet breaks, we check Twitter for updates. Robustness at that scale is more than buying a pair of SSDs and setting up RAID.
One thing I've learnt from being a consulting engineer is to stop assuming other engineers are morons. Sometimes we do things out of ignorance, or out of fear, or out of necessity. But often, when you inquire, there is a solid non-obvious reason for non-trivial diversions from the simplest thing that could possibly work.
I always find it useful to give the benefit of the doubt.
But .. fulltext index done right, including per hashtag, doubles the storage requirements at most. And the ability to call up any tweet from history does not need to be as instantaneous is recent tweets (It's ok to wait 5 seconds for a tweet from 5 years ago).
> The fanout problem turns sharding into a lumpy problem because again, there is a power law distribution that can overwhelm any single machine.
Not really, in the case of twitter (where you only trace edges, not edges-of-edges and edges-of-edges-of-edges like Facebook and Linkedin do). In the simplified model I described, the web frontends all pull from the backend, but for the hot 10-million account followers -- of which there aren't that many --, you can just push them to the webservers so they don't even need to query)
> Robustness at that scale is more than buying a pair of SSDs and setting up RAID.
I most definitely agree. But it is cheaply doable when everything is so perfectly shardable as it is in Twitter.
> Then there's the fun part that Twitter is so reliable these days that when something else on the internet breaks, we check Twitter for updates.
That's possibly an illusion. Yes, Twitter is mostly reliable - but do you have any latency stats? e.g., would you know if, on a daily bases, 10% of tweets take 60 seconds until they appear on a viewer's refresh? The "hot" accounts would be cached and immediately updated, but the long tail might have minutes delay and almost no one would notice.
> But often, when you inquire, there is a solid non-obvious reason for non-trivial diversions from the simplest thing that could possibly work.
> I always find it useful to give the benefit of the doubt.
I do not assume other engineers are morons. In my experience as a consultant, though, those solid non-obvious reasons for non-trivial diversions are much more often than not "historical, do not apply any more", "we didn't have time or resources to do it the right way", "the guy who did the initial design did it wrong for whatever reason, and now we are stuck with it", "there's a legal reason that's not obvious why we can't do it that way", and a few others.
Twitter had the resources to do it better from very early on, and they didn't (I think it was 2012 or so before Twitter became reliable). For all I know, their system now could be the most efficient beast ever, and run on a ZX81 with 1K Ram with ultimate reliability, way better than I could ever hope to build.
I was just pointing out that the user facing side of twitter, technically speaking, is not very impressive. I've been doing it for a while on various technical forums - and not once did anyone offer a reason for why it's much harder than it would seem -- most of the responses were along the lines of "but mysql/pgsql/oracle can't take the firehose load, I tried!". Which is correct, but irrelevant.
By saying it can be done with $12,000 of hardware, you essentially claim it is trivial.
> everything is so perfectly shardable as it is in Twitter.
Shardability, or partitionability, depends on the underlying distribution. It's only "perfectly" partitionable if that distribution is perfectly uniform.
Twitter's traffic patterns are power-laws. Some tags vastly outnumber others. Some users have vastly more followers. Those distributions are unstable across time and space in surprisingly short order.
> Twitter had the resources to do it better from very early on, and they didn't (I think it was 2012 or so before Twitter became reliable).
Twitter wasn't interested in a contest to use the least hardware. They were in a race to keep up with surging load, while migrating from Rails to something that didn't exist yet. Meanwhile they have, under the hood, built infrastructure that could not be bought off the shelf from anyone at the time.
We can't actually come to a conclusion here, because you're arguing a counterfactual. It's always easy to have the ideal solution to a problem you never actually solved yourself, because the visible features of a problem are obvious and the many, many invisible problems are where the bulk of the effort are hiding.
If you want to prove that Twitter doesn't need such large and complex distributed systems, you can probably round up a few hundred thousand in angel funding and sell yourself to Twitter for a nice return.
Not at all. What I'm saying is that, with the proper software (which is not trivial to write), you can do it with very little hardware. I know stunt programmers making $500K/year, and they are in some senses infinitely (not just x10) more productive than bad and even average programmers - because they quickly produce working systems that others just can't.
Whether it makes sense to pay $50K/hardware and $500K/programmer or $2M/hardware and $100K/programmers depends on how you run your business, though - and in many cases, the $2M/hardware+$100K/programmers are the more economical choice (because you risk starving for stunt programmers choosing the first)
> Some users have vastly more followers. Those distributions are unstable across time and space in surprisingly short order.
And yet, as a system designer you actually get to choose what the distribution is of - and a good choice makes it uniform. The power laws may or may not favor sharding on uid specifically, but usually there's a simple way to shard uniformly. And if you can't find a way to programmatically shard uniformly, then shard using a lookup table on the userid - 8 billion user ids require all of 8GB of ram if you have 256 shards or less (and if you bundle users in groups of 256, 32MB is suddenly enough). Migrate around to keep balanced. This has been a solved problem for years. Really, even migration patterns. Look at "consistent hashing" literature -- (it's not the fundamental issue that consistent hashing solves, but the peripheral solutions are well known, common, and apply here: migration, redistribution, redirection).
> Twitter wasn't interested in a contest to use the least hardware.
> We can't actually come to a conclusion here, because you're arguing a counterfactual. It's always easy to have the ideal solution to a problem you never actually solved yourself,
First, I basically agree with you, if it wasn't clear. I know not what problems twitter were facing. I suspect that they weren't technical in nature, though -- because the technical issues have been solved before them. It might be management issues, it might be ego issues. I've seen more projects fail or stumble on those than on technical issues.
I actually did solve those problems myself, on a smaller scale (hence my interest, but with perfect sharding that would have scaled to any size, live migration and redistribution and all). But that startup folded because we sucked at getting traction - which only shows to go you that technical prowess matters not in these issues, or at least not much.
> many invisible problems are where the bulk of the effort are hiding.
Again, to be clear - I totally agree with you. I'm just disagreeing with the aura of engineering excellence that Twitter gets in this (and many other threads). They may have it, or may not - I haven't seen evidence that they do. They can keep twitter running well in the last 3-4 years, which means they are reasonably competent. That's all I have evidence for.
> If you want to prove that Twitter doesn't need such large and complex distributed systems, you can probably round up a few hundred thousand in angel funding and sell yourself to Twitter for a nice return.
As I have mentioned several times in other replies (and other discussions) - twitter's problem is not, in fact, engineering, and hasn't been since 2012 at least (it definitely was 2007-2009). They threw enough money/people at the problem, and solved it.
Right now, they are bringing in $2B/year, but spending $2.5B/year. If they are spending more than $200M/year on user facing hardware at this point, I'd be surprised. I'd even be surprised if they are spending $100M/year. But let's assume that I can save them $200M - that's nice, but won't actually save them...
I assume the average tweet is closer to 80 bytes, and taking advantage of that can let you store twice as much tweets in memory.
[0] Does twitter count utf-8 bytes, code points, graphemes, or something else towards the 140-"char" limit? Regardless, the computation would not yield materially different results.
WhatsApp also doesn't provide a CDN, mobile analytics/monetization/debugging platform. WhatsApp also doesn't contribute to open source like Twitter does (Storm, Bootstrap, Aurora).
Describing twitter as a "140-char" mailing list is very reductionist. OTOH WhatsApp problem is very well served by Erlang/Mnesia an open source distributed messaging library developed by telecommunications company who were likely doing WhatApp-scale 10 yeas before WhatsApp came around. While WhatsApp engineers are great, there is very much a "standing on the shoulders of giants" - giants who have well researched the problem of building a system that lets two or three individuals communicate. There wasn't anyone doing Twitter 10 years ago (well, maybe Google).
I don't mean to imply that Twitter is an efficient well oiled machine, but the comparison to WhatsApp is daft.
[1] http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-...
[2] https://www.erlang-factory.com/upload/presentations/601/EUC2...
In any case, my main point isn't that that WhatsApp is simple. When I said the parent posted was being reductionist, I was referring to Twitter's product offering. AFAIK (I could be wrong) WhatsApp is a messaging backend and multiple client frontends. The "50" engineers is deceiving becaus, despite the fact that Mnesia could or couldn't have scaled, there is a very non-trivial amount of engineers who worked on it and Mnesia is a good & proven system. In a similar light, Instagram was 13 engineers (or was it employees) and hundreds of django instances - and that maybe all you need for a photo feed, however Instagram didn't have a trending feature until earlier this year and how knows how many engineers that took.
If WhatApp hired another 200 engineers, likely 195 would either be twiddling their thumbs or working on new features outside what was needed in that acquisition to Facebook. Now we can make the argument that Twitter is bloated because they have too many products (Fabric, Vine, Crashlytics, and their numerous other acquisitions) - and the scaling back of those engineers may mean sunsetting those products - and thats fair. To imply they have 4000 engineers building a glorified 140-char mailing list is not.
Absolutely! You could probably get a single rack to handle storage and broadcast of millions of tweets per minute, perhaps hook it up to a Teradata appliance to handle search and storage at that throughput. Throw it in a nice distributed database indexed by user. Then just another couple of indices for the hashtags and mentions, and the follows/follower graph. Throw in another few tables for handling API permissions, managing schema migrations while ensuring that data from day 1 is still as transparent as yesterday's logs, a highly replicated ad server hooked up to a realtime recommendation engine that analyzes an active user's current stream and recommends the most relevant ad (to maximize revenue of course), set up a few staffed policy teams to comply with the differences in global regulations, handle lawsuits and legal requests, and of course keep competing for more revenue with the likes of Facebook and Google.
It's pretty trivial.
When you have several hundred million users, that adds up. An effective monetization model, if unusual in SV.
Are you suggesting that WhatsApp's model of organization is the norm for a sustainable organization? I think it's a clear outlier in many respects.
Are you suggesting that WhatsApps business model was based on annual revenue?
Silicon Valley mentality at its finest.
https://www.youtube.com/watch?v=BzAdXyPYKQo&t=15
The company existed to make a return for the owners. The owners exited with $16 billion. That was the motivation of the owners/employees, not $14 million in annual revenue.
My point is that the transaction must be analyzed from the viewpoint of the owners (as that is what the poster I was responding to cared about, viz a viz total profit/revenue to Twitter). When you do that, WhatsApp cashed (well, assuming FB stock can be treated as cash) out more per employee than Twitter.
Since WhatsApp was a revenue-generation corporation with a board of directors, like any other corporation, before their acquisition, and since I don't like to indulge in conspiracy theories, I'm going to assume that yes -- their business model was based on annual revenue.
Then you'd assume wrong. The goal of the board is to maximize return for investors. In WhatsApp case, it's not hard to see that their goal was to grow as fast as possible, revenue be damned, so they could be a very attractive takeover target for a company like Facebook who could eventually hope to monetize that user base.
They're nowhere near relevant. I keep getting local-to-US ads, and I'm in UK (and geolocate as in UK too). Not as bad as showing alcohol ads to 5 year old children, but still pretty bad.
This doesn't mean that there aren't teams of people trying to constantly make those ads more relevant. And since not everyone at a Twitter can statistically be incompetent, I'm going to assume that the problem is actually quite difficult, and perhaps requires a large team.
WhatsApp had ~30 when they were acquired, and were also huge. ("Huuuuuge!", says The Trump)
Those teams are gods of scalability. Twitter is.... kinda the opposite.
Edit: My apologies if I've offended the HN mystique surrounding the god-like status of WhatsApp. Ask Jan yourself, they weren't geniuses. I know Jan. He won't claim to be this exalted persona.
Edit: I know b/c I've done that before in $previousjob. Storage is hard. Storage is where the money goes. Durability is expensive. Storing ephemeral stuff in a non-durable KV store is easy, cheap, and trivial to scale. This is why WhatsApp didn't have so many people, not because they were geniuses.
Disks are slow. If you don't need to wait for disk, then you can handle a great deal more req/s per node, which means the point at which you need to become a distributed system is further away. Without the state synchronization problem, converting to a distributed system consists of little more than adding a load balancer.
Even in an environment that needs synchronized state, the services that can be stateless are much easier to deal with (and you try to make as much of your infrastructure stateless as possible).
Storage costs are not relevant compared to the costs of engineering distributed systems.
Deciding which ad is the most optimal to show benefits from every input (metadata?) you can make available and analyze.
That must require some multiple of effort vs. making sure 140 chars get shown where they're directed to go.
completely incomprehensible why they needed such a large headcount. ten times more than whatsapp would be ok - but not this behemoth.
all these designers do what exactly? ignore ipads, macs, etc.
i love twitter, use it constantly to follow journalists, vcs, basically to get faster news than any site. i hope twitter can be cut back to the bone.
No but that's the majority; I simplified on purpose but obviously they have far, far more people there than necessary.
> lots of well-maintained OSS projects (Mesos, Aurora, Finagle, Pants, Bootstrap, Zipkin)
As much as I love OSS it does not pay the bills.
As an example, they could have gone with Zurb Foundation instead of doing Bootstrap. They could have used (redo, cmake, tup, a hundred of other tools) instead of doing Pants.
If Twitter was non-profit, their breadth of products would have been awesome. But it's for-profit, and most of these projects are negative on the P&L summary, even if you favourably take into account intangibles like "help in recruiting people" and "coolness of brand".
google ads is funding the other google, which behaves like a non-profit. and it works, as their ad business is amazing.
but, it sets a really bad example to follow. twitter thought a long time they could operate like google it seems. time to bring back reality.
And they are already reporting profits.
I doubt they will go the way of Twitter.
And that's in an industry where the products are close to fungible, without depending on network effects.
On the other hand, one binding ecosystem with disparate products can easily become either walled gardens or badly-fragmented loose confederations, so it's not the best situation, either.
In terms of Whatsapp, micro-transaction maybe? That I am not quite sure, it is not going to be as straight-forward as Instagram.
This sounds like 90's bubble talk, but "eyeballs" do matter. Investors and the market care about the current social value of a product in terms of the value / growth of their user base.
Ask many millenials/teens and they say they more meaningfully interact with Instagram than Facebook. That fact about social trends is worth its weight in gold in terms of its user base.
Their willingness to hold off for many many years without revenue was now brilliant in hindsight, given they found a much better way to make money on advertising than how Yahoo ever imagined it. But they wouldn't have got there if they caved into short-term thinking obsessed investors who think you need to post immediate revenue.
Twitter's issues are more to do with a lack of product innovation or evolution, not that it has gotten too large. Its problem is that it has gotten too large without expanding its product base to support or justify that growth.
Ask how many millenials or teens have bought something or spent money because of an ad on Facebook or Instagram.
http://www.wikinvest.com/stock/Facebook_(FB)/Data/EBITDA
Also, that was just an example I pulled out of the air. It could be middle management, or a documentation expert whose role is regulating font size. The point is that for a company with a simple product, an API and clients for that API, and monetization through advertising, that's a lot of employees. If the money's not there, cut the fat.
as a discovery/exploration tool sucks.
if it had interesting suggestion based on hashtags and followers beyond 'follow this three more person' daily mail it could be a nice networking/news tool, but so far it cannot really help me navigate in the millions streams that might be interesting but are accessible only knowing that exact hashtag
I followed a couple news outlet, I guess there are many more that may interest me but I can't be assed to search them and if I discover them myself they go by preference into feedly, not twitter.
on the other hand, it had been great to follow this and that indie game developer :D
That made me giggle. Their entire existence is communication.
Not their fault they are being laid off you know.
- they're hiring for different specialties and departments than what's being cut
- at least in theory, they're hiring people who would be above-average workers while laying off those below average
- they still need to compensate for normal churn, which is probably exacerbated by the layoffs
- they can try giving new candidates weaker compensation to save costs
Ultimately, workers are not fungible and it requires active effort to maintain a team as people come and go. This doesn't change during layoffs.
Or can appear cheaper to bean counters, at least.
Twitter had potential, but it appears stagnate as all they've done is created a new version of an AOL instant messenger chatroom with hashtags and 128char limit. Limiting the API tokens for 3rd party developers was a really foolish move.
Additionally it brought up nonsense click bait.
It looked like they just fed in the front page of yahoo.
Oh well, I don't think Twitter is going away any time soon but I don't really understand some of the steps they've been taking...and 4,100 employees? Yikes!
Fair enough. That probably goes for almost everyone on HN as well I would think but obviously we're an edge case.
> Their goal with these features is not to retain existing users but to acquire ones who wouldn't have otherwise used Twitter.
Hmm, how would this attract new folks? Like Facebook tries to show you things you care about but this I'm not sure how it's really sold to perspective users. Since it's a social network I'm guessing much will be word of mouth but that requires support from existing users.
I wonder if things like increasing the character amount and offering a better, maybe threaded conversational way of interacting with folks would bring in more people. Granted those may not be the best choices philosophically but I wonder if they've done any A/B testing where certain areas got different features like more character counts and how it turned out.
This is such a crazy innovation to the structure of chat programs, that it blows my mind other real time communication platforms haven't incorporated it. Even Facebook Messenger and GChat feel inadequate now that I'm consciously aware that they don't let me edit my chat messages after sending them. It's such a no brainer innovation that nobody has ever done, likely because when you're building a chat program the idea of letting people edit chats after they send them makes the brain of any engineer hurt -- but there's an untapped value to allowing it that many products haven't realized yet.
Money has to come from somewhere; if it is not from you, it is by monetizing you. But ... why not monetize the user even when they did pay? It's a whole new revenue stream, that's often orthogonal to the payment stream.
Paid Hulu has ads. Windows Solitaire has ads now. Movie theatres have been showing ads since forever. Cable TV is showing ads.
All that's left in my browsing experience to hint that sites are trying to maintain themselves on advertising are reels of clickbait stories that come at the side and bottom of every traditional media article. I can't help but hope they are only a technocultural shift away from the abyss of anachronism.
The technology can be used to target us, but in the end machines do the work and they are very predictable. We also have machines, and knowledge, and can stand up against the bullshit.
> Paid Hulu has ads. Windows Solitaire has ads now. Movie theatres have been showing ads since forever. Cable TV is showing ads.
The problem with my idealistic vision of the near future is that we are just as likely to accept this as normal. We habitually pay for stuff and still get ads. We have the tools to block the crap but it's much more important that we decide to.
Millions of people gave up eating gluten because they learned about celiac disease. I'd like to imagine that we are just a few headline-grabbing research papers away from swaying the public about the healthiness of exposing ourselves to psychological bombardment.
The apps need to handle it gracefully because they can't tell whether the ad server is legitimately down or just blocked. I've never had an app work any worse with blocked ads.
Although if I were trying to think of business models that seem to be struggling, those would probably have been among the first examples that came to mind. Windows 10 so far appears to be another Windows Vista/8. The movie theatres around here seem to operate under constant fear of closure. We don't have a complete equivalent of US cable TV here, but I get the feeling that on-line, on-demand services are threatening that model as well. In each case, I can't help suspecting that the degree of customer exploitation/price gouging has a lot to do with the waning popularity.
Popular and irritating since (a) it's incomplete (if you're paying, you're increasingly product, too, since companies aren't going to leave rifling through your data as a way of making money alone), and (b) it's become a kind of astroturf ad campaign for certain companies.