86 comments

[ 3.4 ms ] story [ 179 ms ] thread
This is an excellent summary of the non-obvious realities of building software at scale. Engineers often become so good at building quick, weekend prototypes that they don’t understand the realities of shipping software at scale. Practically speaking, the initial greenfield software development is almost always a tiny fraction of the effort that goes into building a business.

This is a common downfall of engineers who try entrepreneurship before they’ve gained managerial or operational experience. I’ve lost count of how many entrepreneurs I’ve talked to who got something running on an Arduino or Raspberry Pi and think that they’re one step away from a successful business. Really, the hard work is only beginning.

I think this article raises a lot of good points about the weekend prototype, but I also think there is some merit in the question of “what do all those people do?”.

I’ve worked in digitalisation in the public sector of Denmark for quite some time. A few years ago I was part of the group who redefined our national principals for architecture in municipality IT systems. The whole thing is called rammearkitekturen which translates directly into “the framework architecture” and without getting into too much details it was made because we had 98 muniplacities times 300 average IT systems ways or defining what a person was. Which made buying and integrating IT hard. Rammeaekitekturen still hasn’t fixed that, but that’s not what I want to talk about.

One of the things to come out of this mindset was how to design an IT system for eldercare. Where there used to be a physical folder with a printed note containing all the “Moksly may attempt to hit you during showers and this requires two caretakers” and so on as well as hand written notes for day to day things, medicine used to come in little sorted bags clearly labelled “MORNING” and so on, and every morning people would get a printed list of their route that was adjusted for coworkers who were off sick and so on, and that was pretty much it. Today we have multiple different billion dollar systems to handle those basic things, and after two years of being used they still can’t handle differences between day and night-shifts or giving call-in temps the correct sorts of data access.

Instead of having two planners and a subscription to a pharmacy for the sorted medicine bags, we now employ almost 50 people as full time support staff to operate this system as well as throwing endless amounts of resources after project managers, lean consultants, educators and so on to get this system to work.

When polled everything single citizen and caretaking employee replied that the old non-digital systems were better. At no point has anyone asked whether or not it made sense to digitalise this area or if it makes sense to continue trying to implement it.

So I do think that it’s sometimes reasonable to wonder just what all those people are doing in some organisation. Because sometimes systems have a way of making themselves important to the organisation without actually being important to the organisation.

In cases like this, it’s always useful to have an expansive definition of “user”. For example, day to day experiences for caretakers might have degraded under the new patient information system, but maybe it’s now easier to produce charts which bosses want. For better or worse, they are users of the system as well. To whom a change is worthwhile often leads to unexpected beneficiaries with unexpected motivations.
So maybe let the caretakers work in peace and have one additional person draw pretty charts out of thin air which would be just as good?

I've been working on a product which everybody was convinced had Gantt chart functionality. Said functionality was using the venerable Math.random() top to bottom, with nobody being the wiser.

Malicious compliance is certainly one approach, but an org which relies on it is probably messing up in a way that will leak to everything else. I never said the desire for reporting was useful, but it is fundamental to how management works in most places today.
Thinking back on the overall strategy, would it have been better to "just" digitize, OCR and timestamp the coordinating documents for logging and audit purposes? A "document management system" rather than "an eldercare management system"?

That way it's "just" "Pull up the most recent documents relating to or updated for patient Doe, Jane" or "What has changed in the medications prescribed in this district?"

The Document Management System would still manage all the PDFs or caretaker notes, but would (eventually) be searchable, and would still support the primary workflow of "I just want to see all the recent documentation for my patient" or "This is caretaker Alex's most recent route assignment document".

Right. When I've seen these kinds of efforts they seem to take one of two paths:

1. Specialized, monolithic systems. These try to do everything for everyone and usually end up devolving into the 12-month (or longer) release cycles. "Monolith" doesn't mean monolith vs microservice in this context. Just that the system is trying to be all encompassing. This is the norm. It rarely works well because it massively increases the cost for change (money, personnel, and time) even for what should be minor things (Oh, you need a custom report? We'll get it to you next year.).

2. Generalized, flexible systems. A document management system like you discuss would be one potential solution. These tend to be more data-oriented (sharing a common database or a common set of databases) with potentially custom front ends. The front ends tend to permit flexibility for the user, including generating their own queries and workflows. These usually work out better, but are not that common among mega corporation or government digitization efforts.

The former is easier to "sell". You get to manage a massive contract, it looks more productive than it is. The latter tends to be more grassroots, an internal effort. The former is highly visible, the latter is nearly invisible.

Related, people tend to overvalue work that appears difficult and undervalue work that appears easy. The second approach tends to make modifications "trivial", so the managers ask, "What are we paying you for?"

> I also think there is some merit in the question of “what do all those people do?”

There is merit in the question, but I think the answer is less "the core product is a lot more complicated than it seems" and more "a big software company with lots of eyes on it can make even more money by hiring more people to create more products that are ancillary to the core product."

Similar thing with comments whenever a Slack related article ends up on HN. "I don't understand how a shitty IRC clone is worth billions of dollars!" or "Why would anyone use Slack when Matrix is free!" I often wonder if these folks haven't even bothered to actually use the software and realize how much more it does beyond sending messages back and forth and/or how much detail and thought goes into making it usable for the masses, not just software engineers.
I think competency in the domain reduces friction, and without reflection folks generally don't seem to realize that their experiences are usually specialized. So yeah, I think it's a combo of not trying the new thing (because they already have a comfy work flow) and not honestly assessing how the knowledge they take for granted changes their experience. Not that it's that easy of course.
> these folks haven't even bothered to actually use the software and realize how much more it does beyond sending messages back and forth

Are you saying that sending messages back and forth involves a lot more complexity that you don't see or that Slack does a lot more than just messages?

That's kinda a "gotcha" question, because it's not clear what your definitions are. For example, is authentication (single sign-on, etc.) part of what you describe as "sending messages back and forth"? Are files, screen sharing, video calls, etc. still "just messages"?
I didn't intend it to be a gotcha question.

> Are files, screen sharing, video calls, etc. still "just messages"?

I wouldn't think any of those as messages that a basic message-only client would have.

I guess that's the distinction I was after - was the person I replied to talking about all the complexity of a basic message-only client, or were they talking about the complexity of all the stuff beyond that.

I suspect most people in the "Slack is a just a pretty IRC," are discounting the non-text-messaging features. Beyond both the items already mentioned and the ease of adoption, the major benefit to slack is the integration ecosystem.

SlackOps is a really big thing. Making slack the glue across variety of operational systems to coordinate the activity of a team and allowing that team to communicate about it, provides a ton of value. And importantly, allowing this to happen organically with non-IT staff creating their own workflows (in the way that people invent alls sorts of min-apps in spreadsheets) is pretty powerful.

We never used it for anything except messages. Now we're on Teams.

I'd be curious how many users were like us - messages only?

It's not a shitty irc clone, it has a lot of useful features (and some not so useful, looking at you threads).

Now the client on the other hand...

You don't like threads? I find them useful at times, that's part of what I like about Slack, it has enough potential for customization that you can bend it to your culture. For example, you can use bots that suit your workflows, use threads or not use them, use emoji reactions or not use them, use private threads or small channels, or always large channels..
> You don't like threads?

I would love them if it was actually noticeable when someone else creates a thread to which i'm supposed to contribute...

At my workplace, we tag people.
So no voluntary contributions from people you didn't ask ;)
Not exactly I mean people still see the thread root, just like if you didn’t use a thread it would see the message, So if the thread is interesting you look at it
I never see those. They get lost in the main channel noise. They should get pinned somewhere obvious to not get lost with the main chat scrolling.
Then you may not be using threads properly — in a fully threaded channel there is no way to get lost, since there isn’t really much scrolling to do. You just need to be diligent that there can’t be lots of messages between threads.
What, have a "main thread" for stuff concerning everyone/whoever is able to contribute?

No. It's over complicated. They need a better solution.

Perhaps you only use Slack as a followup to meetings/todo items. We use it for all communication and the workflow is about opposite: someone raises an issue, someone or everyone discusses it, someone takes ownership and it turns into their TODO item.

I've seen this so often. Almost every time something about Khan Academy's technology makes it to the front page here, someone brings up some form of this complaint. It got to the point at which I wrote a post [specifically to address why Khan Academy has so much code](https://www.kevindangoor.com/posts/why-khan-academy-has-so-m...). There's just so much complexity involved in making reliable services that provide features used by a variety of types of users.
We were just having this discussion around the Postman fundraise - various people saying "its just a GUI around Curl!!!" and being all offended at the raise's price tag. There's so much more that goes into building an actual business.
Also folks making similar comments are missing that funding is about the future of the product/business, not just scaling growth (though you do see with later stage funding).

Something like Postman is a bet on enterprise tooling for APIs, both building and integrating with them. Looks like Postman has gone beyond their chrome extension, and leveraged that to add more a bunch more features. But even if all that was easy enough to build, the investment is really about the trend and the business ability to capitalize on the trend. VC investment is always more forward-looking than the current product state.

For sure. Valuation of a VC raise !== technical complexity. Easy to conflate from an engineer's seat, I guess.
Well yeah I could do Postman in the weekend. Really I hate Postman and I have it on my todo list. I also try not to use it unless someone sends me his Postman collection.

Thing is, I could do that for myself and I would be the only user and I would never turn it into a company.

Because what devs usually say they could do that product in the weekend - but usable only for themselves. While turning software package into successful product that a lot of people would use is totally different story.

Just like playing piano for your friends on the weekend or guitar by the fire that takes maybe couple weeks of training. To become a pop-star well that is just different pair of shoes.

Yeah, I agree.

It's just that many times these comments are on threads about the fundraise itself. "Why is Postman worth $B when I could build that in a weekend?" is completely missing what it takes to build a business.

One other thing he touched on in passing: people are assuming they'd know what to build from the start. Even in relatively trivial applications a lot of work goes into finding the right way to express what you're trying to build. I recently worked on a small app and spent 40 hours trying out different approaches to the user interface. The solution I went with could be coded up in about 2 hours, but I had to do 38 hours of work that eventually got thrown away before I arrived at it.

This sort of thing happens all the time in professional software development. Sometimes it is a waste of time, e.g. people rewriting things in a new language/framework for negligible benefits, but a lot of it is a necessary part of developing a polished app.

“I didn't have time to write you a simple UI, so I wrote you a complicated one instead.” -- Mark Twain, paraphrased.
Except today more often I see "Our devs put together a basically usable UI, then we ramped up and hired a full time UXer, and it sucks now."
I've seen that happen a thousand times, yes. (Slightly exaggerating, but still.)

I guess we'll just have to wait until people get used to the idea that the Web is an interactive document format, not a replacement for the GUI. I'm hoping the modern SPA trends pass into the void like the Macromedia Flash UI trends did.

At many companies anything is a SPA now, which seems like the worst example of cargo cult mentality. It takes twice as long to develop, but with all these people who only know JavaScript you have to give them something to do.
Totally agree. I always get in trouble when I say that we must build a first version to get a “feel” for the task and then refine for a while. This may involve scrapping the whole thing a few times. This is especially true if something is totally new. Once you have an established UX it’s much easier to plan and make changes.
Ron's second law: The hardest part of getting what you want is figuring out what it is.
I had a similar realization with food recently, bear with me.

You ever hear how in certain cultures, Mom's food is the best? Now, of course there's nostalgia involved. A hint of home can make your taste buds sing just a bit louder. But also, even for a 'simple' meal, there are so many options a chef can make that subtly change the meal.

For example, looking only at one ingredient, you can end up with a dizzying amount of options. For example, do you use chicken thighs or white meat, sear it first or let it braise, brine it or salt it right before going on, use a jar of pre-ground spices or add your own concoction of 5 spices? Each of those is a choice, and only one set of choices ends up with Mom's chicken cacciatore.

In software, any good UX is usually the result of thousands of intentional choices. It takes trial and error and research and actually time spent using the damn thing to figure out the tradeoffs involved in many of those choices. Just like it usually takes new chefs dozens of crappy, burnt, lifeless meals before developing the taste required to make those tradeoffs effectively.

> You ever hear how in certain cultures, Mom's food is the best?

Just had the semi-shocking realization that the cross-section between those cultures and the cultures where moms are stay at home is quite large.

What are the cultures where people don't have nostalgia for the food their parents fed them?

Like, people will get weepy over canned veggies. Personally I think nostalgia is the most significant ingredient here.

In a world where children are sent to daycare, Headstart, Pre-K[indergarten] and then Kindergarten all before entering the public/private school system for the next 18 or so years, I have increasingly seen nostalgia for school cafeteria meals from those now in their 20s. Doesn't matter if it's awful quality from the same source as hospital and prison food; they grew up with it as their primary meal source.
Exactly. I objectively hate it, but I would eat that square pizza with square pepperoni bits again without a second thought if offered.
The thing that hit me many years ago was when I was very young (less than five, I think), and I was at my great grandmothers house, and she fixed dinner. And I LOVED the creamed corn she made! I made such a fuss about it that she had to get another can off the shelf and make a second helping.

It was just plain creamed corn with condensed milk. But it was also the best thing I had ever eaten.

She was very confused. As was my grandmother. And my mom. None of them understood just how important that “granny corn” was, despite the fact that it was straight from a can into the pan to be heated a little, and then put into a bowl for me.

I heard that story time and time again, for decades.

Association is a huge part of impression and memories. That’s why smell is such a good trigger for emotions, because it is very primal and connected so closely to those parts of your brain that are involved in association.

Even with simple projects.. I just finished working on redoing a static webpage for a client. Really simple, started at 17 pages, cut it down to 6 pages, only text and images with a contact form and a few links. I ended up having to build 3 different versions just to get a layout and design that worked for the client's business and another version I built initially for my own working environment the customer never even seen and wasn't charged for.
When I was working on my masters degree project I thought I created a pretty good UI. At least I couldn’t think of a way to make it better. Then I did a 1 on 1 evaluation with a user who knew nothing about the subject matter. She had some recommendations that significantly improved the UI. I changed the UI and then I did a 1 on 1 evaluation with a knowledgeable user. He had more recommendations. I implemented them and then did a small group evaluation and then a large group evaluation. At each step suggestions were made that greatly enhanced the UI. By the time I was finished the UI was much different than what I came up with on my own, but tremendously improved to my great astonishment.
yes, it's akin to when people describe discovery / invention as cutting away the cruft to isolate the gem. when you get the gem, everything fits, it's small, solid, shiny.. but it took hundreds of cuts.
I don't see nearly as many people claiming Google could be built in a weekend, but I do see that sentiment a lot in threads about Uber. It's easy to think it's a simple app, but there was a great response from a former EM highlighting just why that isn't so[0].

> This reminds me of a common fallacy we see in unreliable systems, where people build the happy path with the idea that the happy path is the “real” work, and that error handling can be tacked on later. For reliable systems, error handling is more work than the happy path.

Is there a name for this fallacy? I think this isn't merely a familiar anecdote, but something that points to the root of why armchair architects think things are easier than they actually are.

[0] https://news.ycombinator.com/item?id=25376346

On the one hand I agree but on the other hand it’s really much easier to be a follower and avoid the mistakes the pioneers have made. I am pretty sure there are a ton of legacy things in the Uber platform that are obsolete now and you wouldn’t have to implement if you stated from scratch. But you would also probably run into the same issues that Uber took many years to solve and you probably also would need years. So many be the advantage of being second isn’t as much as thought initially.

I often think SpaceX was able to move that quickly because they could learn from decades of experience of NASA and others what works and what doesn’t.

People make this observation about Uber mostly because they can replicate the thing in a few months (it's never a weekend). And proof of that is that many people did replicate Uber software in a few months, some times even with some commercial success.

Differently from Google search, the hard thing about Uber is not the software. What makes a lot of sense, but leaves the question of what are all those people doing there, what the article answers, but the answer is not trivial in any way.

Any chance you could add, that wide text is really hard to read body > *:not(pre) { max-width: 600px; margin-left: auto; display: block; margin-right: auto; } pre { max-width: 1200px; margin: 40px auto; background: #f6f6f6; padding: 16px; }
I could do that in a weekend.
There is a firefox extension that I use, called Stylus[1], that lets you add custom CSS to websites. The thing that helps most is that Stylus lets you browse other people's styles for website, and there are already themes for Dan Luu's website.

[1] https://addons.mozilla.org/en-US/firefox/addon/styl-us/

Safari and Firefox has reader view. I'm not sure about Chrome or IE since I don't use those. I'm sure they'll have something similar.
(comment deleted)
First thing I did was hitting the shortcut for Safari's Reader. Would love to see that in other browsers.
Sounds like the Pareto Principle at play: 80% of the product is created with 20% of the effort. It is getting the next 19%, then 0.9%, then 0.09%, and so on, that is the challenge.
That 0.9% then 0.09% are also where the real success comes from in many fields.
Also sounds a bit like Dunning Kruger. It looks easy because people are ignorant of how hard the problem is.

Many years ago I built a search engine for a very big discussion board website, which had hundreds of millions of messages. Keep in mind back then a powerful server had 2GB of RAM and a pretty slow CPU and HDD, so it had to be distributed across multiple shards to work fast enough, and there was no Elastic Search back then, so it really was hard work, plus getting the ranking and quality tuned was a huge effort. Of course when we launched it people responded with "I don't get what the big deal is, why can't you just run a SELECT WHERE LIKE query and get it over with over the weekend?". It was hurtful at first but then I realized how ignorant the person making such a statement is about the problem.

One could build a self driving prototype in a weekend of hacking (using the libraries, tooling, etc available) and train it. I think it's a perfect example of the weekend prototype vs production reality difference.
(comment deleted)
"I could do that in a weekend" is obviously idiotic but why choose Google as the example (possibly the hardest piece of software to replicate)?
If you follow the citation trail:

0. Dan writes an article about marginal employment decisions, which cites 1. Alex Clemmer wrote (http://blog.nullspace.io/building-search-engines.html) about building a search engine to compete with Google, who cites 2. John Peebles, who wrote (https://peebs.org/2014/01/04/we-need-viable-search-engine-co...) about the urgent need for a viable competitor to Google, saying:

> I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype, and have missed the enormous opportunity that exists right in front of our noses. Does the next search engine have to be as big, involved in as many things, employ as many people, and fight on the same footing to be accomplish the goal of providing a counterpoint to Google?

This article was upvoted fairly well on on HN (https://news.ycombinator.com/item?id=7011472), though the top comments are all in agreement this is not a simple task. This zeitgeist, along with Dan's personal experience on the subject, is why he wrote the article using Google as the example.

> A company that could eliminate organizational inefficiency would be a larger innovation than any tech startup, ever.

Fully agree, and would extend to any social organization. One of the most important aspects of organizational efficiency is managing the signal/noise ratio of internal communications. We now have the ability to turn effective bureaucratic protocols into material reality with automated information systems.

The organizations, whether private or public, that best tackle this problem will come out on top on the 21st century, regardless of ideology and historical advantage.

That is why you want to have one pizza size teams. Those teams should also be as independent as possible.

Unfortunately it takes quite a bit of orchestration on higher level of abstraction and having well organized management.

maybe part of the whole conversations is survivorship bias? id be interested to see data on the past positions and companies people on both sides of the argument have worked on. ie: if you have seen how teams of hundreds are justified, it makes sense. likewise, if you have only worked on small teams, its hard to understand what 100s of engineers would do.

its been a few years but i knew of multiple alexa-100 sites that were all running on just two 4 core servers(this is back when alexa was a useful measure of size, mind you). managed essentially by the hosting company. so "scale" is not always directly linked to traffic, i would think complexity dictates the need for more engineers rather then size.

Fun article, but exceptionally difficult to read. I know some people like using plain ol html, but just a few lines of CSS would vastly improve the reading experience. I know its from 2016, but still.

Eyes read best when content is a few hundred pixels long. Content should be centered etc.

It's pretty terrible as it is

One of the nice things about HTML/CSS is that there's such a thing as "user styles", which can apply on top (or instead of) the "author styles".

Opera pioneered this interface, AFAIK, and even had one-click toggle buttons for both author and user styles (as well as images), but most of today's browsers don't make this functionality transparent and available. I suppose you could do via dev tools?

For a simple change like what you're asking, perhaps even changing browser's font settings would suffice?

get stylish then
Yep it was pretty horrible, so I hit reader mode that narrowed everything down and increased the font size.

I find it's quite a bother to have to do all the CSS necessary for good reflow on a number of devices, so I too just have everything just spill to the ends, and just do reader mode on devices. But I don't really know much CSS apart from the really basic stuff, and haven't ever sat down to see how it would be done.

Yeah, for a site like this, it could literally be just adding

max-width: 700px; margin: 0 auto;

This would make it look great on pretty much all screens.

A 700px max-width would actually make it quite annoying to read for me (as a lot of content on the web is, unless I switch to reader mode)....
Why? Is it not wide enough for you? or something else?
Yes, it's not wide enough.

Just holding up a book to my screen, 700px is about the width of the text in a mass-market paperback. But the reading distance for a screen is much larger than the reading distance for a book, of course. And hardcovers or trade paperbacks have wider text columns than that, of obviously, and are generally somewhat nicer to read than mass-market paperbacks... though also often get held at slightly larger reading distances.

Right.

You hold a book closer to your eyes though than your screen.

Both have about the same line length, as you point out, but the book's font is smaller. The website font is much larger - to make up for the greater distance between the screen and your eyes.

I think 700px would be wide enough, 800px would also work.

From what I've read, going wider than about this at around a 14px size font and it becomes increasingly difficult for the eye to track between lines.

On a phone, a max with of 700 or 800 px obviously wouldn't apply since the phone screen is smaller.

On this site, I usually zoom in to 150% to 200% on my computer to make the font size large enough to comfortably read.

> but the book's font is smaller.

If the book's font is smaller, but the lines are the same width, that means more words per line in the book, yes?

And each line is taking a larger fraction of the visual field, since the book is closer to my eyes.

Basically, if we assume books are somewhat optimized for ease of reading (and to a large extent they are), websites should be aiming for the same angular sizes as books. The reading distance is larger, so the physical sizes should be larger: larger fonts and wider lines...

This post describes the entire web browser peanut gallery in a nutshell.
Sometimes, though, you do need to scale back to a weekend prototype. Understanding what it takes to scale the system is good, but the analysis paralysis from imagining that every problem must have a Facebook-scale solution is silly too.

I mean, those Volvo XC90s are damn slick, but somehow my local pizza place delivers hot tasty stuff on time in an old Fiat Seicento. I think the pizza quality would suffer if the owner invested in a more reliable and better car.

I suspect this is connected to the happy path coding phenomenon. The core issue probably IS that easy. But you need to cover all the edge cases reasonably well too to have a business and those you can’t necessarily see until you start coding
If you're going to make this criticism, I feel you also need to be capable of recognizing when such criticism actually is valid [1]. That is, under which circumstances would someone be justified in saying "okay this is not an optimal problem"

(A famous example from the other direction was when Uber re-engineered a querying service to better fit the use case, where several HNers and a Medium article showed why their re-invented wheel was worse than an off-the-shelf solution that they just weren't using correctly [2].)

If you can't satisfy that, then this feels like a fully-general rejection of any claim that an org has over-complicated a problem or has unnecessary bloat, and/or that no one can criticize until they've personally worked (for n months) at the org. Which itself would be hard to defend.

[1] Based on my "Scylla-Charybdis"/Goldilocks Heuristic: http://blog.tyrannyofthemouse.com/2015/12/the-scylla-charybd...

[2] https://news.ycombinator.com/item?id=21537008

One other thing is, many of you could write something like a mouse driver in a weekend. That is not enough for the business however. The UI needs to be Electron based for easy maintenance, and you need to add telemetry, because how else will you know if people actually use your product? So you need backend and API teams too for each platform.
"Considering how much of Google's $500B market cap comes from search" Time flies
The rule seems to be that the less a software developer knows about how to build X, the easier it seems to be for him/her/it to build X.