I completely agree with the idea that scalability is often overrated. While it's certainly important for large organizations and enterprises, for smaller companies and side projects, focusing too much on scalability can actually hinder growth and creativity.
It's often more beneficial to focus on solving a problem effectively and efficiently, rather than trying to scale it before it's even proven to be successful. By starting small and iterating based on customer feedback, you can create a better product that solves a real problem, which will naturally attract more users and drive growth.
That being said, scalability should definitely not be ignored altogether. As a project grows, it's important to plan for future scalability, but this should only be done once the product has proven its worth and a solid foundation has been established.
It's kind of along the lines of "premature optimization is the root of all evil." A true quote, that doesn't mean optimizing isn't important! The whole trick is in being able to tell what's core and what's premature.
Another way of looking at it -- scalability is pretty much the entire value proposition of software. It allows people to do the same, or more, amount of work without having to linearly scale out the number of people working on something. When you're first starting out, demonstrating even being able to _do_ the work is the first hurdle, but past that you're definitely going to need to show you can scale that out efficiently.
> Another way of looking at it -- scalability is pretty much the entire value proposition of software
That sounds like a good way to look. People are Turing-complete, although that doesn't mean that they are very good at scale without instructions, and "programming" clusters of people has its own set of quirks. :)
> It allows people to do the same, or more, amount of work without having to linearly scale out the number of people working on something.
But this is not the entire story, but only the throughput part of it. And there are entire classes of software where at least part of the value lies in latency. That's, for example, most of Internet as we know it, from the hypertext browsing to videoconferencing, and most software we would call "embedded", from watches to spaceships.
> But this is not the entire story, but only the throughput part of it. And there are entire classes of software where at least part of the value lies in latency.
> (...) scalability is often overrated. (...) focusing too much on scalability can actually hinder growth and creativity.
I find this blend of comment unproductive and intellectually dishonest. "Often" and "too much" are weasel words that remove any objective meaning from the verbs they qualify, to the point where stating the exact opposite is also valid.
The truth of the matter is that scalability is only not important when you don't need it, but it's already disastrously too late if you failed to scale when you need to.
The problem is if the commenter says “doing X works” instead of doing “doing X often works” there will be a HN avalanche of “yabut so-and-so didn’t do X” and allegations of survivorship bias.
> (...) there will be a HN avalanche of “yabut so-and-so didn’t do X” and allegations of survivorship bias.
Isn't it also survivorship bias to claim that scalability is overrated because a non-scalable system didn't broke in a particular case of a local deployment with residual demand?
Meanwhile hikes in demand hitting an unscalable system can and do break businesses, and when it happens is already too late to rearchitect things.
That being said, scalability should definitely not be ignored altogether.
Yeah. You have to think about what's possible to scale.
You're planning to scale to 1,000,000 or 1,000,000,000 users eventually? OK. Well, what's your service? Are you planning the next Facebook where everybody can talk to anybody? Then you have some challenges you better start figuring out.
Or, is it a SaaS where nobody needs to access anybody else's data? OK, then you can probably simply scale vertically for a while up to $SOME_MILESTONE, and then scale horizontally via some fairly simple sharding or maybe even just database partitioning or something. No need to do that now.
> Are you planning the next Facebook where everybody can talk to anybody? Then you have some challenges you better start figuring out.
Even Facebook didn’t start trying to figure that out until they had to. Famously you needed an invite for a very long time. I’d imagine infrastructure limitations factored into this.
You’d be surprised how many successful companies push off dealing with scaling problems until they absolutely have to.
> Famously you needed an invite for a very long time.
And then it opened to needing a college/university email address. Then it opened to everyone.
I got in with my college email address, and at the time it was considered a feature, not a limitation: a combination of higher quality since people couldn't just make new spam accounts whenever, plus a bit of a status symbol compared to just being on MySpace.
> You're planning to scale to 1,000,000 or 1,000,000,000 users eventually? OK. Well, what's your service? Are you planning the next Facebook where everybody can talk to anybody? Then you have some challenges you better start figuring out.
Thinking this way actually leads to one of the most common failure modes. Specifically the day-dreamer death where the principals become so enamored with a fantasy of what the end-state could look like that they lose focus on the next step to improve their odds of success. This is under-reported because day-dreamers are so common that when they try a startup and fail, it's not really an interesting pattern to report on. Such founders will tend to cite specific reasons that sound better in context (eg. wrong market, product flaws, sales funnel flaws, etc). But as someone who's been in the web startup space 25 years, I've seen it over and over again where founders fail to fully engage with the hard problems right in front of them in favor of working on what's fun, easy or tied to some dopamine hit of imagined future success.
In practice, you should not spend one second thinking about the requirements of 1B users until you have at least 10M. You will need to completely rewrite everything multiple times along the way to that scale anyway, and more importantly, you won't know the product requirements to actually get there until you achieve the earlier milestones and get the necessary user feedback about what's working at each size of user base.
Precisely. It'd be like trying to make an F22 fighter jet right off the bat when you're still in the era of biplanes.
I think people often underestimate the strata of learning, and the essential feedback that gets you to the next level. Or at least that's how it works for mere mortals like myself :)
I'm advocating for thinking about if you'll need an F22 fighter jet someday.
If the answer truly is "yes", that's something you'll need to figure out early or at least plan for.
For the other 99.99% of situations where the answer is "no" or "almost certainly not" then absolutely, you should not be wasting time building an F22 or even thinking about it.
There's a misunderstanding. I may have miscommunicated.
I'm advocating for understanding your business, not for prematurely building things.
If you're in the extremely rare class of businesses trying to build something on the scale of FB or YouTube where that sort of massive scaling is intrinsic to your mission then yes, you need to start thinking about those problems early.
For everybody else, yeah, absolutely focus on product and getting to 100 or 1000 or 100K or whatever users first.
That's exactly right: it depends so much on the context and (expected) access patterns. Sometimes you really do need to plan ahead, but even that's more about not painting yourself into architectural corners rather than specific tech.
What I see also is not just scalability over-engineering, but the same in the solution domain. When you don't really understand your customers needs sometimes there's a desire to create something with ultimate flexibility.
In the days of Xp we used to say YAGNI when people would be overly general. I'm trying to bring that back into fashion.
IFIFY: focusing too much on scalability WILL actually hinder growth and creativity.
It will hinder growth and creativity because you are solving the scaling problem, which is a hard problem. Especially if you're early on and don't really have a product or customers.
I've seen multiple projects tank because people working on it were solving problems they didn't have and wouldn't have for the next 2-3 years at least. But they only had a few months of funding.
Scalability "requirements" are often misused by the developers so they get to play with "cool" and often very slow technologies at the owner's expense. I've seen it happen over and over.
Many experienced devs have ptsd from working themselves into a corner on unscalable stack which required death marches to resolve. Not surprisingly they often do a complete 180 after that. A strong cto / senior eng core would prevent the company from going to either extremes but unfortunately not many of them have that.
I am sure there is truth to this, but from my personal experience the death marches I have seen have come from overly complex designs which try to handle all future problems but then fail miserably. Whereas evolving a simple design to be more scalable often works out well.
I am sure there are many situations this isn't true, but I have not seen them in my experience.
Yeah. Resume-driven development. "Hey, we need k8s and/or huge AWS clusters because scalability" when often something simpler would suffice.
At a previous position our devops guy had provisioned a cluster of god knows how many AWS Redis instances. I pointed out this was a lot more than we needed. He swore up and down we needed them because scalability.
He had no idea how much data was in those things. LESS THAN A MEGABYTE of data. In the whole cluster. We had no plans to grow it, either. And we were only accessing it a few times per day. We didn't even need Redis. We could have just stored it in the database directly.
He didn't know any of that, and he didn't care. Because it didn't matter. Resume-driven development.
He wasn't stupid. He was very smart. He knew what he was doing. "Managed a Redis cluster" looks good on a resume. "Just used the database instead" does not.
> He wasn't stupid. He was very smart. He knew what he was doing. "Managed a Redis cluster" looks good on a resume. "Just used the database instead" does not.
Did your company fire him and replace him with a DBA instead? If not, then the resume driven development is working!
I hate it when people look down on resume-driven development while simultaneously also doing things that promote it by letting HR people unrelated to technology screen resumes for recruiting.
Here's a hint -- if your company is screening, even subconsciously for buzzword (and believe me, most of them do) then you are enabling resume-driven development whether you like it or not.
You're almost exactyl describing what went down at my previous employer.
The company already had a bunch of poorly written legacy systems which were used by about 50 clerks and written in PHP, Node.js and some .Net. They had problems with poor code and data quality, but none with performance.
So they recruited a new CTO and a new Head of DevOps which planned the new infrastructure together.
Of course now everything has to be split up in Microservices (or Nanoservices, because each one should basically have the least functionality possible) and thrown in the cloud.
The only thing where a separate service actually makes sense are the machine learning processes that should extract information out of received documents.
The rest could basically run in a basic PHP application on a Raspberry Pi without performance issues.
When I quit that circus, development had basically stopped altogether and they were already paying five figure numbers each months just for hosting.
I moved to a large corporation with the expectation that things should be better there.
Turns out, that basically the same is happening on a larger scale. There they have a team of four guys having to manage more than a dozen microservices which were cut in strange ways.
I am only a senior developer and no architect myself, but the ones I know are everywhere between crippling depressions and just laughing their asses of because of the stupidity that's going on in the tech world right now.
There are companies where it's done properly, but either I have extremely bad luck in my surroundings, or those are an absolute minority.
Then these guys dance off from that to a company that actually needs to store tens of billions of k/v pairs, HA, and can make a big mess there before anyone figures out they're actually clueless beyond `sed s/replicas: 3/replicas: 69/ && docker-compose up`.
>He wasn't stupid. He was very smart. He knew what he was doing. "Managed a Redis cluster" looks good on a resume. "Just used the database instead" does not.
After nearly two decades in early stage startups I couldn’t agree more. Looking back, I know we often built too much too soon, and had too much confidence that we were building the right thing.
These days I often advise would be founders to start with doing their idea manually with just the minimum amount of tech.
Maybe just a google sheet and answering emails and a handful of “customers.” If you can’t give people a good experience with a totally custom personal experience, they’re not going to like it any more when it’s automated and scalable.
too much confidence that we were building the right thing
SW is weird in that way, we can basically build houses that nobody wants to live in. Thing is you probably were building the right thing, structurally speaking.
My rule over time has become "structure follows complexity" i.e. Anticipate complexity where possible but don't apply the engineering structure until the complexity really exists. If you built an MVP house, don't add the second bathroom until you've got customers complaining about deadlocks on the first.
It's tough because that basically runs counter to the Spirit of an Engineer, which is just to imagine the grandest well-thought-out + organized thing and march toward creating it.
The bonus of having two decades+ in building SW though is you start to see the patterns and realize that the same fundamental needs drive everything, so the intersection of building something great with customer needs becomes less of a mystery and more of a "how fast can we juice this thing". At that point I think scalability prep is like compound interest.
There’s a ninety ten rule with structure that can be hard to communicate in code reviews. There’s a subtle difference between not adding unnecessary structure and writing code that is antagonistic to it being added later. Opening a PR where someone added a feature or fixed a bug exactly where I would have done so is one of the best feelings. Especially when they’re changing my code. Either they’re a very good coworker or I’ve laid the groundwork well. Either of these are good news.
Very recently I’ve come to think of my architecture style as Jazz. Like the old trope about listening to the notes he isn’t playing. But also the improvisational nature and the flexibility to adapt to new information.
> There’s a subtle difference between not adding unnecessary structure and writing code that is antagonistic to it being added later
This is such a wonderfully concise way of describing something that I try to teach people.
I have come to use the phrase "no regrets". Leave things out, but do it in a way that you won't regret it, if you need to add it later.
Another phrase that I read somewhere else was, "Weeks of programming can save you hours of planning" (pretty sure I read it on HN, and I wish I had kept the link). The point being that when you decide to leave something out, or when you decide to intentionally do a half-assed job of something, you should still think about how someone in the future would go about efficiently fixing it.
If you do it right, there's missing functionality, but little or no technical debt. If you do it wrong, you miss out on functionality and also take on technical debt. And I think that's the subtlety that you speak of.
Most people who push for early design understand that they have more control of the project at the beginning, before deadlines accelerate, before a bunch of bad hires and competing priorities. And they are definitely speaking from a place of regrets. This hurt so much last time I vowed never again.
There’s a list of things I won’t try to add on later. It’s smaller than it used to be, but the shiv I brandish at you for suggesting we cut those corners is also much scarier.
Some things, when you cut the corners off you also cut your fingers off.
We have all promised that we will “fix it later”. But whatever we do now is what’s done. Fixing it later is a promise whose resolution is often not in our control.
I suspect this dynamic is a big part of why Joel Spolsky jokes about transforming from a technical leader into an amateur psychologist. We are all lying to ourselves first and to others second.
Or as Feynman said, the trick is not to fool yourself, and you’re the easiest person to fool.
Or a personal favorite, unattributed: I know the guy who made this plan [me] and he’s full of shit.
Jim Highsmith has the best theory I’ve heard in the subject. His opinion is that the reason we keeps having the same unwinnable arguments is because we think in terms of solving problems but the real issue in front of us is resolving paradoxes. There are no neat answers that fit in a short essay box on the test everything depends, and everything depends on everything, and something is always changing.
What the PTSD reactions come from is other people making decisions on behalf of the rest of the team without consent. But like most things in software it’s a 2 dimensional problem not a 1 dimensional one. The trick is to delay irreversible decisions as long as possible and get all responsible parties involved.
The flip side of the coin is that reversible decisions should be made as cheaply as possible. For instance my “Kobayashi Maru” solution to the bike shedding problem is that the team should have budgeted three colors of paint into the bike shed budget, picked green for the first coat, and moved on immediately. That $200 is far cheaper than the time and importantly energy that would go into discussing this at a meeting with senior staff members present. If you start thinking about all of the $5000 conversations you have to solve $2000 problems it’ll drive you mad. I almost recommend you don’t look at it, for your own safety. Almost.
> The trick is to delay irreversible decisions as long as possible and get all responsible parties involved
This is exactly right, but the irreversible decisions are often horizontal slices of software design that underlie other decisions - thing like authz, IPC, API design etc. If these questions can be identified and resolved up front, the cost of building vertical slices - features - goes way down. You're not tempted to reinvent the wheel, you can focus on the business logic.
What I'm saying is: it's worth spending time planning the foundations of a product. That includes the technology choices, but also the high level logical layout of the whole product. If you approach this foundational stuff like any other project, then if you're doing it right, you'll "get all responsible parties involved", like you should.
But defer the detailed planning of domain logic until you're actually ready to write it, and remain flexible with the plans you do make. "Plans are nothing; planning is everything".
Because like the GP said,
> the later you design something the more info at hand you have to make the design fit better
I find it remarkable how often engineers just leap in and start coding, and worry about the foundations later. Or never. I suspect this is one reason why rewrites are so common. If you start with bad foundations, it's often easier to blow up the whole building and start again.
> I find it remarkable how often engineers just leap in and start coding, and worry about the foundations later. Or never. I suspect this is one reason why rewrites are so common. If you start with bad foundations, it's often easier to blow up the whole building and start again.
To be fair, leaping in and starting coding allows you to discover the problem better. The problems start where you get stuck with your initial design and don't adjust/remake it once you explored some ideas, maybe found some new along the way and found out what doesn't fit.
That is of course problematic if you have deadline driven development because rewriting last 2 weeks of code might seem like total waste even if then-better design gonna save you much more even in near feature.
Not rewriting code when design flaws appear will lead to rewriting it eventually later and more expensive.
Of course, all of that depends on how much you know about domain and the problem. Exploring existing solutions and their flaws or having really detailed requirements might be much better approach that doesn't require starting to write code.
I believe that quote is thought to have come from Frank Westheimer or at least he is associated with an earlier version of it.
“Why spend a day in the library when you can learn the same thing by working in the laboratory for a month?”
Jazz is an excellent analogy. Been in software dev for 30 years. It's hard to explain to hirers that one can only play jazz well (i.e. know what matters in a software project as you go) after a long time, so trust me :)
I realized at some point I got put in charge of things to shut me up. Someone would ask me why and I would launch into a treatise on all of the confounding factors and cognitive science reasons and and and OMG shut up already. Clearly you care about this more than I do, why don’t you do it?
Don’t put a hyperkinetic person in charge of something thinking that you’ll call their bluff.
"keeping an escape hatch". I like that. A former colleague said something similar that also stuck with me. Good software architecture allows you to delay decisions.
We talking Dandong, city at the border of China and North Korea? They built everything including a giant bridge, high-rise apartments, shopping centers, parks, and a ferris wheel. But there's no people and it's been ~10 years now. It is, however, still fully maintained. [0]
Yup. Literally first sentence of my citation "Dandong New City, a district of Dandong which was built from scratch to be a China-North Korean trade hub, now stands mostly empty." [0]
A large number of them being filled and another large number of them YET not being filled still makes dozens of millions of cities coming to being out of nowhere and providing people with affordable, well-planned urban environments instead of using the demand to skyrocket the real estate prices at the cost of causing homelessness and suffering.
>It's tough because that basically runs counter to the Spirit of an Engineer, which is just to imagine the grandest well-thought-out + organized thing and march toward creating it.
I'm not sure if Spirit of an Engineer is a book, essay, or some prior set of principles I'm unaware of but if it's not (and even if it is), I tend to disagree here.
The spirit of of an engineer, in my opinion, is to solve problems and problems for people--that's what I and most engineers I've spoken with have been drawn to (ignoring those purely seeking a high paying profession here since were talking about "spirit"). When I was youthful I sought out technical complexity and grandiose solutions (I admittedly still admire it when it's done correctly). None the less, to me, most of it is wasteful and at the end of the day I want to build the most useful solution for people to make their lives easier.
Some problems require high complexity and glass cathedrals in terms of technical solutions, most really don't, at least most of the problems I've been exposed to.
Actual engineers generally do try to solve problems for people. But the engineers understanding of the people that they are solving it for is generally rather flawed. And therefore their solutions frequently are difficult for the intended users for reasons that the engineer can't see.
And yes, engineers really do err on the side of designing for future problems and future users that often will never exist. This effect is worst on a rewrite of a system - look up the "second system effect". We tend to only truly get pragmatic tradeoffs the third and later times.
Maybe “spirit of an engineer” could be better phrased as “visionary mindset”. Lots of engineers I’ve worked with imagine potential problems and treat them as requirements to solve even though the problem may never actually materialize.
Figuring out which problems to solve before they happen and ignoring the ones that probably won't happen is the cosmic dance of software development. The Serenity Prayer states "... accept the things I cannot change, courage to change the things I can, and wisdom to know the difference." Except change "cannot"/"can" to "don't need to at least not yet"/"need to now because it's about to be an issue!".
Make a bare bones version and see if it sticks, and go from there. Both the Progressive Enhancement philosophy and the Lean Startup methodology demand the minimal before building further. Ex: Will anyone click this survey link? Who knows but you'll learn real quick if you offer them $50. If they still don't, then it probably ain't gonna get clicked ever. If they do, then you found at least _something_ viable.
It's always a balance. You need some structure to avoid spaghetti, maybe a bit more to allow for plausible extensibility (microscopic pre-abstraction) but not too much you're adding fat.
I'm not sure if Spirit of an Engineer is a book, essay, or some prior set of principles I'm unaware of
Good idea, you are welcome to help me hone my later chapters. For me the engineering spirit - especially in software - is being a creator/builder. At the core, someone who takes the Legos in front of them and creates with those pieces.
A couple people took my "grandest" comment as a nod toward scale/size/narcissism, which is reasonable, and I meant it more in terms of total architectural completeness - i.e. a system perfectly fulfilling all the requirements that necessitated its creation. A blade of grass is not "grand" in the primary sense, but it still is grand + magnificent by its own perfection in design.
Yours is an incredibly good comment. The comparison software/architecture is something that I have encountered here and there over the years, but you brought it to a new level. "don't add a second bathroom until you ve got customers complaing about deadlocks on the first". Beautiful. Thanks
There is however a pitfall that this approach seems to lead to unless you actively prevent it. Roughly the process is 1. You hire someone to take care of things manually. 2. As you grow, your devs are worrying about other things and so instead of automating you hire more and more people to handle the toil that should have been automated along the way. 3. Because these jobs are handled by one group of people and developers are working on customer facing features, the communication and prioritization necessary to fix the issues rarely come up.
Steve Blank has talked about this for decades. A startup is about validating you have a product that customers want. You want to connect with customers and prove you have a market before scaling.
https://steveblank.com/tag/customer-validation/
I think this kind of concierge / person behind the curtain approach only works for a certain type of business though, usually services where aspects of the service pipeline can either be automated or not.
For purer technology / product businesses, how do you do this, fundamentally? How would Google have manually mocked up their early product? How would Facebook? Github? Tesla for that matter?
Sometimes you really do just unavoidably have to build the product out before testing the market, and if it doesn't work, just accept the sunk cost and throw it away - and sometimes fail completely as a result.
I don't see this as a fundamentally solvable inefficiency, just part of how tech product startups work, and the very tradeoff that must be made to allow for innovation to happen.
there are still smaller pieces you can MVP to a smaller audience before launching it to the world.
> Google have manually mocked up their early product? How would
Crawl an intentional community (remember webrings?) or other small directed subset of the web and see if you're able to get better search results using terms you know exist in the corpus, rather than all of the Internet.
> Facebook?
They had Myspace as an example so the idea wasn't exactly unproven.
> Github?
Kernel developers were already using the software, successfully, all (for large values of "all") they did was bring it to a wider market with a better UI.
> Tesla for that matter?
People don't get this, but Tesla's biggest achievement to date, isn't the cars themselves, but the factory that they're built in. There's no way to MVP an entire factory, but building a car in a bespoke, pre-assembly fashion is totally possible and totally doesn't scale.
If you're asking if electric cars were known to work, the first one came out in 1832. If you're asking about product-market fit, they keep selling cars they haven't made yet, just to gauge demand. Aka where's my Cybertruck!?
> > Google have manually mocked up their early product? How would
> Crawl an intentional community (remember webrings?) or other small directed subset of the web and see if you're able to get better search results using terms you know exist in the corpus, rather than all of the Internet.
But that isn't a mock up, it's the real thing but on a smaller dataset. If you're going to do the real thing anyway, why not run it on all the data you can?
After all, the throttling factor to release is in the engine, not in the dataset. If you're going to write the full engine anyway, there's nothing to be saved by limiting it to a subset of the data.
Google was manually editing and merging Perl scripts to get web scraping data almost every day early on. Yahoo manually verified content and added it to lists with hand typed summaries fir years, even up to the time Google came in the scene.
You are right that some businesses scale better doing high touch customer service like this. In the case of Pilot, you have the lead sales guy and accounting domain expert (CEO) asking SaaS customers to schedule enterprise service sales calls with him.
Which makes total sense. What he’s not automating is setting up Marketo or Hubspot and Drip or Constant Contact and committing to some CRM system that is both impersonal and adds friction when you’re going for quality over quantity.
He could hire BDRs and CSMs or outsource and spam and set up AI and knowledge bases, and possibly scale up faster.
But not only would that take away the personal touch and competitive advantage, he’d lose the opportunity to educate himself on what real customers really need.
Not to mention all the time spent evaluating tools and setting up automation and negotiating contracts that lock you in to a specific process that might not be what you want 2 years down the line when your business changes.
It really depends on the situation. No one at a successful startup says, we spent too much time on scalability in the beginning. They probably say the opposite. I worked at one company that we couldn’t rack servers fast enough to keep up with our growth. Our competitor who was the market leader had to turn off the ability for new customers to signup to prevent their site from crashing under load. We ended up surpassing them.
Getting a million customers isn’t that challenging and really a function of money. With $50 CPA, you just need $50m. 1m customers will take down most sites unless you do some optimizations.
I think you’re both right, even if I’d guesstimate the situation you encountered is very, very rare.
You should make architectural choices that CAN be scaled within reason, on demand and ideally incrementally. That doesn’t mean you need to build your app on Kubernetes and automate welcome emails, surveys, metrics, A/B testing etc. But you probably shouldn’t build on top of fundamentally non-scalable systems either.
This is provided you’re in an environment that can reasonably scale to 1M users or whatever, which isn’t true for all domains. Say B2B where that last B is only 10000 people world wide.
It’s difficult to explain to people what you said without the experience of living in the failures of early optimization. K8s from day one, a-b testing, programmatic quality control without any paying clients on the product, scrum master and a PM and a product owner for 5 devs! Whole thing feel like a Ponzi scheme to keep overseas staff employed, perpetrated by the overseas development shop.
Unless you built something absolutely atrocious, scaling it vs getting it ‘right’ or ‘right enough’, is trivial. You couldn’t sign up to Facebook if you weren’t a college student.
Chatgpt overloads daily so what? When your app becomes necessary to enough people, bring in your k8s evangelists and shove into x cloud provider. But by god it doesn’t take 30 people to write a marketable mvp.
Maybe I had the fortunate luck that we never had issues of finding market fit. We also had lots of investment outside of software dev. Fintech for example requires significant investment in compliance(eg: kyc).
Chatgpt doesn’t have competitors. Customers can’t go elsewhere. Twitter had the same protection when they launched, there wasn’t a short form social network so they could have full day outages. Friendster would be the opposite, when they closed signups, customers just went to MySpace.
Scale goes beyond the ability to support traffic. It also means efficiently handle data/traffic. Heap is a good example, great product but they haven’t figured out to provide that level of analytics without significant costs which are passed on to the customer.
Kunerbetes works just fine on a single node. And it provides a lot of good things over docker run scripts or docker-compose. Kubernetes basically is an OS for OCI images. Even if you don't need orchestration, it still provides great experience. And when you would need orchestration, you will grow organically.
Right now I'm using cluster of 2 nodes (master, worker). No reliability, but convenience is good and we have very simple path to more nodes when we would need it.
The claim was that the unnecessary complexity (of which Kubernetes was just one) from day 1 consumes valuable time that could be spent on making the product better.
Kubernetes brings very little unnecessary complexity and does not consume valuable time (if there's at least one person in the team who knows it well enough).
Kubernetes might start consume valuable time if you would start to bring niceties. Like GitOps, CI, and so on. Those things are good but not necessarily justify time spent from the start. So you can start with simple deployment yaml which you can write with 10 minutes and apply it manually whenever you need to update it.
Kubernetes almost out of the box brings to you:
ingress. You don't need to set up your own nginx reverse router. This task alone could consume days of setting up and maintaining. I work with organization which spends 2-3 hours to update their nginx if they need new route. It must be done in off-hours because they break it every single time they touch it and then they need to fix it. Kubernetes ingress is a breeze of fresh air after this nightmare.
cert-manager. Again, I work with organization which failed at setting up certbot. Their certs break all the time. cert-manager just works.
zero-downtime rollout restart. This is absolutely nice thing which works from the box and hard to achieve manually.
kubernetes-dashboard. There're other dashboards as well. I think that lens is very popular although I didn't use it. It allows for people who know very little about kubernetes to interact with it productively. For example in my team we have developers who use kube-dashboard to view logs, to exec into pod and check things, to update deployment, change envs. They just push buttons in GUI, they have no idea about any yamls or kubectl. And it works.
And those things in my experience so far do not bring any unnecessary complexities. Quite the opposite. They solve lots of inevitable complexities for very little cost.
I strongly agree. Having Docker images of your application is very convenient gor deployment and fairly easy to set up for anyone who has done so before. Then once you have these images anyway, hosting in (managed) k8s in the cloud is a breeze.
I'm not sure why you are getting downvoted. It would be great to hear of some of the bad experiences of those that disagree. Were they running their own cluster instead of a managed one? Or doing k8s with a team where nobody had any experience? Or run into some helm-chart spaghetti from hell?
Post product market fit scalability is critical. But a lot of effort is spent on things that will never get traction in startup land. Before you get traction the times at bat is the more important metric.
In you're case, you clearly already had Product-Market Fit. Most startups do not. It's far more valuable to build quick than it is to build for scale until you know PMF.
> It really depends on the situation. No one at a successful startup says, we spent too much time on scalability in the beginning. They probably say the opposite.
Depends. No one at a unicorn says that, but I've seen plenty of startups that.
The unicorn is the outlier, not the rule.
> I worked at one company that we couldn’t rack servers fast enough to keep up with our growth. Our competitor who was the market leader had to turn off the ability for new customers to signup to prevent their site from crashing under load. We ended up surpassing them.
Were they still in business at the end of all this? Because it turned out you made a bet, and it came through. The odds are against this, though.
> Getting a million customers isn’t that challenging and really a function of money. With $50 CPA, you just need $50m. 1m customers will take down most sites unless you do some optimizations.
A million in what time period, and what are the usage patterns? Because from what I've seen, as long as the backend isn't in Python, Ruby, PHP or similar, a single hosted server easily vertically scaled to handle more than 2m million users, because it a) didn't need to handle 2m concurrent connections, and b) usage patterns were infrequent enough (users interacted only every few minutes at most).
This seems like a big reason why taking on investors too early can be harmful. You can't be sitting on runway money and taking baby steps with it, even though sometimes that's the right thing to do.
They seem to be discussing the more complex, business- and customer-driven aspects of technology, not the simple process of building a CRUD application. DoNotPay.com is one of these examples. It's a straightforward CRUD software where one enters parameters to establish a contract; however, despite claims to the contrary, a human being actually creates the contract form and sends it back 24 to 72 hours later. The entire enterprise embodies the idea of a Mechanical Turk.
as an engineer this part I entirely agree with. Far too often CEO driven development leads to pivoting 1Month into 9 month projects, 9 months in a row. such that you have nothing deployed when you at least could have had 1 thing, but instead you have none.
This can be solved as much by discipline to stay on track as by choosing to reduce scope to 1 month ideas only.
The HN effect is overrated. I had two articles on the front page at the same time and never exceeded 1000 requests per minute. And very very few of those visits turn into repeat users.
Edit: (The load was easily handled by a single Heroku dyno. I turned it up to two dynos overnight, just in case, but it turned out to be unnecessary.)
Not being able to deal with low quality traffic spikes is one of those things that engineers see as a failure, but aren’t necessarily so from a business perspective when compared with the potential cost of a scalable infrastructure.
Not that all spikes are low quality, or even that one on HN ones are. I reckon for some products they are.
Hire an experienced (0-1) Product Designer and have them work out the features, design, system, interaction flow, and research. Test and validate you have the right thing before you start building. This can be done well in 6 weeks and it'll save years and years of debt.
Anywhere you'd find any other employee. Product designers aren't rare, it's just that immature development organizations usually use designers wrong. Bringing designers in at the end to add polish to the user's experience nearly guarantees they'll be polishing a turd. A big part of a designers value is figuring out what problem users need to solve with your software and what tools they need to do so.
I do at least 2 of these calls per day. Interacting with end users is the only way to maintain PMF. IMO anyone too big, cool, or fast-growing… sadly mistaken.
The bit that stuck with me (re-reading for the first time since 2008, I forgot how aggressive it was): "Scalability is not your problem, getting people to give a shit is."
There's also the situation where you've passed your scalability limits, you have users, the system is falling down and the business has blinders on and simply won't accept that they need to invest in a better way.
Being overly speculative and ambitious early on is a great way to fail. Building on top of something past its limits and ignoring the (developer, customer, velocity) pain is yet another.
One great way to guard against this is to force teams to design for scaling up and down. I regularly give a "plan for success, and plan for failure" speech that covers this area.
If your service opex breaks even at 1k users because you designed for capacity up to 100k users, and you actually end up with 100 dedicated super-sticky users, you'll resent the users you have as they bleed you to death.
If, instead, you build a system that less efficiently, but still reasonably, scales to 100k users by ramping up cost to support 100k users by 4-8x... but has an opex break even at 50 users... Congrats! you've planned for success and planned for failure.
Go get those 100k customers, then optimize. But don't wait to optimize an unscalable system until you already have customers. That's a recipe for lost customer trust and "ahead of their time" epitaphs.
> And, funny enough, the people that didn't think about scalability, if they're smart, have typically moved on to something else.
OTOH, the smart people who thought about scalability and designed a highly complex system to handle the load of 5 requests per minute have also moved on.
I've been working on and off in supercomputing since the 80s.
every system I've ever worked on has needed work to get to a certain scale. and that carries you for a while and if you need to get past the next plateau you need to rearchitect again. rinse and repeat.
so its not a binary thing.
so unless you have some specific goal in mind, invest to get a little past where you need to be today.
premature scaling is the root of a lot of wasted time
That said if you know where boundaries are in your design you should keep those obvious and clear so that scaling will not be hard if you are lucky enough to need to scale.
It's easy to say "scalability is overrated" if you've dealt with unnecessary k8s deployments.
It's easy to say "scalability is underrated" if you've dealt with businesses built on hundreds of standalone PHP/Perl/Python/JS scripts with zero unifying architecture, where migrating a data schema is virtually impossible because of the state of technical anarchy.
Looking at the article title and comments I wasn't able to clearly tell. The second heading "do things that don’t scale" is clearly recognizable and meaningful, but then in agreement, I'd have no reason to click.
You damned kids never heard campfire stories of being brought in to consult on “scaling” a bespoke system for a growing company that has been limping along with VBScript in an Excel spreadsheet for five years past when it was still tenable to do so. The amount of feature parity required to get out of that blind alley often killed the project and injured the company. Some lived it, the rest of us knew someone who had.
There was a brief moment after Oracle purchased Sun where I thought Oracle might try to engineer am Excel competitor on Open Office that had an easier migration to a real database (Oracle), but that dream died early.
FWIW I have a friend who works with market analysis and his excel scripts save an enormous amount of manual labor, today. It was the best tool for the job, for them. There are even international competitions in excel automation, which is kinda funny but also points to how far ahead Excel is for actual business uses.
Are there scaling issues? Version control issues? Absolutely! But again, that doesn’t mean that it’s not the best tool for the job.
It’s easy to mount the highest horse from a technical perspective, but as engineers it’s also our responsibility to be curious about what people are using voluntarily, and not just what we think they should be using.
I was this many years old when I learned that tar and zip on linux have an rsync compatibility mode that tries to do some cleverness with compression blocks to make it easier to diff two archives.
XML files in a zip file (plus images and other things). There are many individual xml files in a single zip file, each file (part) is generally responsible for different area, e.g. one file for one sheet cells, one file for style definition, one for comments, one for workbook structure and so on.
The whole structure is called Open Packaging Conventions and it is implemented in `System.IO.Packaging`.
At my last job, my work was basically to read data from proprietary APIs and shit an excel table.
I think I was hit by everything. Easy stuff at first: Crlf issues, XML Apis, weird rpc apis.
Then, halfway through the project, the results had to change. Not only the order, datatype and headers (I actually overengineered the first version so those were configurable), but the format, duplicate on multiple columns (and empty fields counted as duplicate...). Worst job I've ever done. I'm also disappointed in myself tbh.
But now I'm a bit of an expert on excel format issue and limitations, and that already helped me.
As recently as in 2014 my college friend was asked to help a company which was running on Excel and reached the point you mentioned.
IIRC he just optimized their sheets and set up a CRM for the things that had no business being stored in an Excel file and that was already very helpful.
Attempts at writing actual software to deal with the problem would have failed and everyone was acutely aware of that.
FWIW IRC ANAL, a couple jobs ago there was a policy about only hiring copy/paste from Stack Overflow headcounts. Since good code was literally not allowed, scaling meant more servers with more RAM.
They eventually replaced me with a handful of their relatives, or at least it seemed. It was a lot of fun watching how many LOC one could "write" to request some trivial data from an endpoint. Luckily I was only golfing the latter half of that tenure so I have no regrets.
If you're doing any computation or highly concurrent workloads then you will discover the performance issues with Ruby well before you outgrow your persistence layer.
I have done both in Ruby, and addressing it was not a big problem. E.g. my MSc. involved doing a lot of statistics and image processing in Ruby, and solving the performance bottlenecks meant rewriting about ~30 lines or so in C using Ruby Inline. Later I did map tile rendering on demand handling thousands of layers uploaded by customers in Ruby. Both using 1.8.x, btw. - far slower than current versions.
It took more CPU than if we'd rewritten the core of it in something else, but it let us iterate the rendering engine itself much faster, and most of the expensive computations were done by extensions in C anyway (e.g. GDAL and the like).
Of course you can find areas where it's still not viable, but my experience is that if you start by prototyping in whichever high level language - no matter how slow - that is most productive for you, you'll inevitably find you need to rewrite far less than you might think to get it fast enough. But more importantly: The parts you end up rewriting will very often be entirely different parts than what you expected, because being able to iterate your architecture quickly tends to make you end up in a very different place.
I know the argument, I don't buy into the argument (actually not a rubyist). Ruby just imposes some factor x on the performance as compared to Java roughly speaking. Which means you'll start a little earlier to look into queuing requests, distributing stuff in ruby as opposed to a java or go application. Nevertheless if we talk about scale, this is just a constant multiplier.
With a kafka queue, and worker nodes, compute-heavy jobs are easily distributed across many worker nodes.
For many parallel requests, you load-balance.
If the PostgreSQL or mysql table is the bottleneck (persistence layer), well this is actually a design decision thats orthogonal to the programming language (PostgreSQL won't scale better with a JDBC ORM).
Usually web applications fork and runs processes to do the computational work you describe (ffmpeg, imagemagick, git, etc). These are usually written in variety of fast(er) languages like C, C++, Java, etc. Plus now you get multicore scaling for free.
But the question is, are you trying to make your life miserable by scaling before exhausting other options?
Most applications can be optimised for performance by orders of magnitude, much easier than trying to scale them by orders of magnitude. Any problem is much easier to solve and faster to deliver when you can fit it on a single server.
Some people just don’t know how many users can be served from one server.
Usually it is simple thinking that goes wrong. System is slow - add more hardware. But then it turns out developers did bad job and you could still run it from single small server only that someone would have to write code understanding big O notation.
Main point of big O notation is that there are things that implemented incorrectly will be slow regardless of how much hardware you throw at it.
I don't know if knowledge of big O notation is that big a deal - many of the issues I've seen at least in recent years have come about in O(n) code where the value of n in production use cases was several orders of magnitude higher than anyone had bothered to test (classic example, all our test files were ~50k in size. But recent customers expected to be able to work with files over 1Gb - which of course took 20000x longer to process). And the user perception between something taking 100ms and taking well over 30 minutes is rather a lot. In fact in this particular case realistically there's no way we could process a 1gb file in the sort of time it's reasonable to have a user sit there and wait for it, so it really requires a rethink of the whole UX. In other cases it turned out some basic DB lookup consolidation was sufficient, even if it did require writing new DB queries and accepting significantly higher memory usage (as previously the data was read and discarded per item). If I have found the occasional bit of O(n^2) code that didn't need to be it was usually just a simple mistake.
Notation alone maybe not but O(n) just like you write needs to be addressed. Users or stakeholders expect that they can get "all data" loaded with one click and it should always be instant. With N getting bigger just like you write UX or workflow often has to be changed to have data partitioning even if it O(n) - like adding pagination, moving crunching statistics to OLAP. It quickly gets worse when you have database joins and you might have to understand what database does because you can also have O(n^2) queries then even if db engines are insanely fast and optimized on their own knowing what query does like full table scans also can kill performance.
Until it was bought by AOL, ICQ presence scaled as the largest Digital 64-bit Unix box they sold. (Messages were peer to peer.) it worked remarkably well (solid hardware platform, so not HA but pretty available nonetheless). Network communication was UDP, which was quite a lot cheaper at the time.
You'd think... I worked at one that tried. I joined the team after the project had been running for quite some time and they said their goal was 10,000 transactions per second. I said, OK, what can we accomplish in 100 microseconds? They laughed at me and said they were going to horizontally scale.
There are some big blunders that you can commit to that are incredibly difficult to fix. I think the advice should be on avoiding nailing down closed doors, just keep them locked instead and put the key under the door mat.
I have a Java codebase that calls more constructors inside their constructor. So you have this massive god object that instantiates the whole project in side the constructor. If you want to run parts of it in separate threads you can't just take it apart. You first have to rewrite all the constructors.
“… Because the problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle. “ —Joe Armstrong, creator of Erlang progamming language
I don't think this is a problem with object oriented languages. You could certainly do the same thing in any programming language. All you have to remember is to keep your constructors as simple as possible. Passing in dependencies through the constructor is often an easy solution so I don't get the hate for dependency injection frameworks. The original XML driven iteration of Spring was pointlessly overcomplicating it but nowadays you can just define your "beans" in code which basically means the DI Framework just helps you set up a dependency graph and nothing more.
I have worked with both and 100% disagree. Give me a mess of bad PHP (my by far least favorite language) any day of the week. It is usually trivial to scale, unlike when I get handed some complex Kubernetes mess. Fixing failed attempts at scaling is usually much harder than making naive bad PHP code scale. It is amazing how much harm "clever" engineers who do premature optimizations can do.
Your manager was correct. People often overengineeer or overarchitect what needs to be done. In reality, you need to meet the current needs of your business. Not every single action has to account for what might happen - or what you hope will happen - if absolutely everything happens perfectly.
Cash when you're growing so fast you're running into scalability issues is also likely to be far cheaper than cash before you have any customers to speak about.
This is more general than scalability: The more of any work you can afford to defer without hampering growth, the more of your initial capital you can spend on growing, and the more you manage to grow the cheaper it will be to raise money to build out the rest when it's time for your next investment.
The hard part is to determine which aspects you can safely defer.
Before startups rush to cargo-cult this model, I want to point out one important thing – Adjust according to your market.
Pilot's customer base probably looks very different than yours. The author says in the article that they are a "startup-focused accounting firm", so for their customer base which is likely startup founders or similar, the experience of talking to a fellow CEO probably "helps to 10x the customer experience" as the article puts it.
On the other hand if you are selling me some software for a boring Enterprise integration, please don't send me calendar invites for a conversation with your CEO, I'd rather get great API docs and integration guides.
Wow, very good point. Basically, this article is basically marketing for them, since they know startup founders read stuff like this, and they want to sell to founders. Blergh.
Yes, I run small customer oriented project, having meetings with people who spend couple of bucks is kinda ridiculous(to both, I think only super chatty people who like to was time would come), but I noticed discord is super powerful - people often share ideas or problems since its very effortless. (Also very rarely when I am not on discord other people jump in with explanations&help "this is not how you do it... you should ...")
Scalability is unnecessary when you're not growing rapidly. But when you are growing rapidly, it's suddenly very important. And if you didn't think about it before, it's already too late.
You don't have to build to scale right now. But you should have a plan of when you're going to need to scale, and how you'll do it. Or don't, and wait until your pants are on fire and then run around screaming.
I agree with the premise, but would rephrase it as “know what you’re going to need, when you’re going to need it, and have a rough idea what it’ll take yo get there”.
And as the article implies, not just systems but also your processes. I recently saw a moderately big startup strangling themselves on processes that they won’t need for several more years. Mid level leadership (in itself a problem) agonized over scale, and didn’t realize that the products were being built at a snails pace and money was being burned with abandon. Late 2022 was a harsh wake up call.
In huge text: "We partner with the best financial tools in the business"
and a lot of logos from famous companies! I think, oh wow I haven't heard of this company before. But then see the text below the large text in small, gray text:
"We're fluent in your finance tech stack and seamlessly integrate with the tools you use."
In addition to having deep integrations with these tools, we actually do have real partnerships with almost all of them.
"What's a real partnership?" is a great question, but I'd mostly think of it principally as "We know them, they know us, we have a red phone we can pick up to get things solved for each other" (which is good for the customer), and then also "And we occasionally do some comarketing together."
I do agree but I hope people don't take the wrong lesson from this.
For example, there's a "right way" to not do scalability and a wrong way. The right way would be avoiding patterns like microservices early on, while still minimizing human intervention and brittle integrations. The wrong way is to just have some kludgy solution that causes headaches for your customers.
With our budgets tightening we axed startup vendor we were paying $90K/yr to because their shit was slow and kept breaking, pissing off our customers. We were able to spin up replacement functionality for their product in 6 weeks.
Also, scaling issues are not a problem, they're milestones that ought to be celebrated. The things you are building are being used. You get to solve known unknowns, which are so much better than unknown unknowns. The reward is clear. What's not to love?
The time you waste on building an unnecessary scalable system is nothing compared to the time and energy you spend on maintaining that overly complex system.
242 comments
[ 4.8 ms ] story [ 318 ms ] threadIt's often more beneficial to focus on solving a problem effectively and efficiently, rather than trying to scale it before it's even proven to be successful. By starting small and iterating based on customer feedback, you can create a better product that solves a real problem, which will naturally attract more users and drive growth.
That being said, scalability should definitely not be ignored altogether. As a project grows, it's important to plan for future scalability, but this should only be done once the product has proven its worth and a solid foundation has been established.
Another way of looking at it -- scalability is pretty much the entire value proposition of software. It allows people to do the same, or more, amount of work without having to linearly scale out the number of people working on something. When you're first starting out, demonstrating even being able to _do_ the work is the first hurdle, but past that you're definitely going to need to show you can scale that out efficiently.
That sounds like a good way to look. People are Turing-complete, although that doesn't mean that they are very good at scale without instructions, and "programming" clusters of people has its own set of quirks. :)
> It allows people to do the same, or more, amount of work without having to linearly scale out the number of people working on something.
But this is not the entire story, but only the throughput part of it. And there are entire classes of software where at least part of the value lies in latency. That's, for example, most of Internet as we know it, from the hypertext browsing to videoconferencing, and most software we would call "embedded", from watches to spaceships.
Yes, that's a good point!
I find this blend of comment unproductive and intellectually dishonest. "Often" and "too much" are weasel words that remove any objective meaning from the verbs they qualify, to the point where stating the exact opposite is also valid.
The truth of the matter is that scalability is only not important when you don't need it, but it's already disastrously too late if you failed to scale when you need to.
Isn't it also survivorship bias to claim that scalability is overrated because a non-scalable system didn't broke in a particular case of a local deployment with residual demand?
Meanwhile hikes in demand hitting an unscalable system can and do break businesses, and when it happens is already too late to rearchitect things.
Doesn’t have the same ring to it.
You're planning to scale to 1,000,000 or 1,000,000,000 users eventually? OK. Well, what's your service? Are you planning the next Facebook where everybody can talk to anybody? Then you have some challenges you better start figuring out.
Or, is it a SaaS where nobody needs to access anybody else's data? OK, then you can probably simply scale vertically for a while up to $SOME_MILESTONE, and then scale horizontally via some fairly simple sharding or maybe even just database partitioning or something. No need to do that now.
Even Facebook didn’t start trying to figure that out until they had to. Famously you needed an invite for a very long time. I’d imagine infrastructure limitations factored into this.
You’d be surprised how many successful companies push off dealing with scaling problems until they absolutely have to.
And then it opened to needing a college/university email address. Then it opened to everyone.
I got in with my college email address, and at the time it was considered a feature, not a limitation: a combination of higher quality since people couldn't just make new spam accounts whenever, plus a bit of a status symbol compared to just being on MySpace.
Thinking this way actually leads to one of the most common failure modes. Specifically the day-dreamer death where the principals become so enamored with a fantasy of what the end-state could look like that they lose focus on the next step to improve their odds of success. This is under-reported because day-dreamers are so common that when they try a startup and fail, it's not really an interesting pattern to report on. Such founders will tend to cite specific reasons that sound better in context (eg. wrong market, product flaws, sales funnel flaws, etc). But as someone who's been in the web startup space 25 years, I've seen it over and over again where founders fail to fully engage with the hard problems right in front of them in favor of working on what's fun, easy or tied to some dopamine hit of imagined future success.
In practice, you should not spend one second thinking about the requirements of 1B users until you have at least 10M. You will need to completely rewrite everything multiple times along the way to that scale anyway, and more importantly, you won't know the product requirements to actually get there until you achieve the earlier milestones and get the necessary user feedback about what's working at each size of user base.
I think people often underestimate the strata of learning, and the essential feedback that gets you to the next level. Or at least that's how it works for mere mortals like myself :)
If the answer truly is "yes", that's something you'll need to figure out early or at least plan for.
For the other 99.99% of situations where the answer is "no" or "almost certainly not" then absolutely, you should not be wasting time building an F22 or even thinking about it.
I'm advocating for understanding your business, not for prematurely building things.
If you're in the extremely rare class of businesses trying to build something on the scale of FB or YouTube where that sort of massive scaling is intrinsic to your mission then yes, you need to start thinking about those problems early.
For everybody else, yeah, absolutely focus on product and getting to 100 or 1000 or 100K or whatever users first.
What I see also is not just scalability over-engineering, but the same in the solution domain. When you don't really understand your customers needs sometimes there's a desire to create something with ultimate flexibility.
In the days of Xp we used to say YAGNI when people would be overly general. I'm trying to bring that back into fashion.
It will hinder growth and creativity because you are solving the scaling problem, which is a hard problem. Especially if you're early on and don't really have a product or customers.
I've seen multiple projects tank because people working on it were solving problems they didn't have and wouldn't have for the next 2-3 years at least. But they only had a few months of funding.
I am sure there are many situations this isn't true, but I have not seen them in my experience.
At a previous position our devops guy had provisioned a cluster of god knows how many AWS Redis instances. I pointed out this was a lot more than we needed. He swore up and down we needed them because scalability.
He had no idea how much data was in those things. LESS THAN A MEGABYTE of data. In the whole cluster. We had no plans to grow it, either. And we were only accessing it a few times per day. We didn't even need Redis. We could have just stored it in the database directly.
He didn't know any of that, and he didn't care. Because it didn't matter. Resume-driven development.
He wasn't stupid. He was very smart. He knew what he was doing. "Managed a Redis cluster" looks good on a resume. "Just used the database instead" does not.
Did your company fire him and replace him with a DBA instead? If not, then the resume driven development is working!
I hate it when people look down on resume-driven development while simultaneously also doing things that promote it by letting HR people unrelated to technology screen resumes for recruiting.
Here's a hint -- if your company is screening, even subconsciously for buzzword (and believe me, most of them do) then you are enabling resume-driven development whether you like it or not.
Our CTO actually thought he was doing a good job.
The company already had a bunch of poorly written legacy systems which were used by about 50 clerks and written in PHP, Node.js and some .Net. They had problems with poor code and data quality, but none with performance.
So they recruited a new CTO and a new Head of DevOps which planned the new infrastructure together. Of course now everything has to be split up in Microservices (or Nanoservices, because each one should basically have the least functionality possible) and thrown in the cloud.
The only thing where a separate service actually makes sense are the machine learning processes that should extract information out of received documents. The rest could basically run in a basic PHP application on a Raspberry Pi without performance issues.
When I quit that circus, development had basically stopped altogether and they were already paying five figure numbers each months just for hosting.
I moved to a large corporation with the expectation that things should be better there. Turns out, that basically the same is happening on a larger scale. There they have a team of four guys having to manage more than a dozen microservices which were cut in strange ways.
I am only a senior developer and no architect myself, but the ones I know are everywhere between crippling depressions and just laughing their asses of because of the stupidity that's going on in the tech world right now. There are companies where it's done properly, but either I have extremely bad luck in my surroundings, or those are an absolute minority.
Why not just lie that you did that?
How employer would check that, lol.
(I know... that's a big "if")
So it is beneficial to have actual experience instead of just lying about it.
Sometimes.
then he'll probably have harder time anyway if the interviewer knows their stuff
These days I often advise would be founders to start with doing their idea manually with just the minimum amount of tech.
Maybe just a google sheet and answering emails and a handful of “customers.” If you can’t give people a good experience with a totally custom personal experience, they’re not going to like it any more when it’s automated and scalable.
SW is weird in that way, we can basically build houses that nobody wants to live in. Thing is you probably were building the right thing, structurally speaking.
My rule over time has become "structure follows complexity" i.e. Anticipate complexity where possible but don't apply the engineering structure until the complexity really exists. If you built an MVP house, don't add the second bathroom until you've got customers complaining about deadlocks on the first.
It's tough because that basically runs counter to the Spirit of an Engineer, which is just to imagine the grandest well-thought-out + organized thing and march toward creating it.
The bonus of having two decades+ in building SW though is you start to see the patterns and realize that the same fundamental needs drive everything, so the intersection of building something great with customer needs becomes less of a mystery and more of a "how fast can we juice this thing". At that point I think scalability prep is like compound interest.
Very recently I’ve come to think of my architecture style as Jazz. Like the old trope about listening to the notes he isn’t playing. But also the improvisational nature and the flexibility to adapt to new information.
This is such a wonderfully concise way of describing something that I try to teach people.
I have come to use the phrase "no regrets". Leave things out, but do it in a way that you won't regret it, if you need to add it later.
Another phrase that I read somewhere else was, "Weeks of programming can save you hours of planning" (pretty sure I read it on HN, and I wish I had kept the link). The point being that when you decide to leave something out, or when you decide to intentionally do a half-assed job of something, you should still think about how someone in the future would go about efficiently fixing it.
If you do it right, there's missing functionality, but little or no technical debt. If you do it wrong, you miss out on functionality and also take on technical debt. And I think that's the subtlety that you speak of.
There’s a list of things I won’t try to add on later. It’s smaller than it used to be, but the shiv I brandish at you for suggesting we cut those corners is also much scarier.
Some things you can cut corners and save time.
Some things, when you cut the corners off you also cut your fingers off.
We have all promised that we will “fix it later”. But whatever we do now is what’s done. Fixing it later is a promise whose resolution is often not in our control.
Or as Feynman said, the trick is not to fool yourself, and you’re the easiest person to fool.
Or a personal favorite, unattributed: I know the guy who made this plan [me] and he’s full of shit.
Jim Highsmith has the best theory I’ve heard in the subject. His opinion is that the reason we keeps having the same unwinnable arguments is because we think in terms of solving problems but the real issue in front of us is resolving paradoxes. There are no neat answers that fit in a short essay box on the test everything depends, and everything depends on everything, and something is always changing.
What the PTSD reactions come from is other people making decisions on behalf of the rest of the team without consent. But like most things in software it’s a 2 dimensional problem not a 1 dimensional one. The trick is to delay irreversible decisions as long as possible and get all responsible parties involved.
The flip side of the coin is that reversible decisions should be made as cheaply as possible. For instance my “Kobayashi Maru” solution to the bike shedding problem is that the team should have budgeted three colors of paint into the bike shed budget, picked green for the first coat, and moved on immediately. That $200 is far cheaper than the time and importantly energy that would go into discussing this at a meeting with senior staff members present. If you start thinking about all of the $5000 conversations you have to solve $2000 problems it’ll drive you mad. I almost recommend you don’t look at it, for your own safety. Almost.
This is exactly right, but the irreversible decisions are often horizontal slices of software design that underlie other decisions - thing like authz, IPC, API design etc. If these questions can be identified and resolved up front, the cost of building vertical slices - features - goes way down. You're not tempted to reinvent the wheel, you can focus on the business logic.
What I'm saying is: it's worth spending time planning the foundations of a product. That includes the technology choices, but also the high level logical layout of the whole product. If you approach this foundational stuff like any other project, then if you're doing it right, you'll "get all responsible parties involved", like you should.
But defer the detailed planning of domain logic until you're actually ready to write it, and remain flexible with the plans you do make. "Plans are nothing; planning is everything".
Because like the GP said,
> the later you design something the more info at hand you have to make the design fit better
I find it remarkable how often engineers just leap in and start coding, and worry about the foundations later. Or never. I suspect this is one reason why rewrites are so common. If you start with bad foundations, it's often easier to blow up the whole building and start again.
To be fair, leaping in and starting coding allows you to discover the problem better. The problems start where you get stuck with your initial design and don't adjust/remake it once you explored some ideas, maybe found some new along the way and found out what doesn't fit.
That is of course problematic if you have deadline driven development because rewriting last 2 weeks of code might seem like total waste even if then-better design gonna save you much more even in near feature.
Not rewriting code when design flaws appear will lead to rewriting it eventually later and more expensive.
Of course, all of that depends on how much you know about domain and the problem. Exploring existing solutions and their flaws or having really detailed requirements might be much better approach that doesn't require starting to write code.
https://news.harvard.edu/gazette/story/2007/04/frank-h-westh...
Not sure what the actual source is.
I tried to bottle it in this document: https://pcblues.com/assets/pdf/approaching_software_projects...
Don’t put a hyperkinetic person in charge of something thinking that you’ll call their bluff.
The potential need to scale and add features will inform my design decisions, but they will not direct them.
[0] https://www.globaltimes.cn/content/974210.shtml
[0] https://www.globaltimes.cn/content/974210.shtml
https://youtu.be/XopSDJq6w8E
That's responsible, future-oriented development.
I'm not sure if Spirit of an Engineer is a book, essay, or some prior set of principles I'm unaware of but if it's not (and even if it is), I tend to disagree here.
The spirit of of an engineer, in my opinion, is to solve problems and problems for people--that's what I and most engineers I've spoken with have been drawn to (ignoring those purely seeking a high paying profession here since were talking about "spirit"). When I was youthful I sought out technical complexity and grandiose solutions (I admittedly still admire it when it's done correctly). None the less, to me, most of it is wasteful and at the end of the day I want to build the most useful solution for people to make their lives easier.
Some problems require high complexity and glass cathedrals in terms of technical solutions, most really don't, at least most of the problems I've been exposed to.
And yes, engineers really do err on the side of designing for future problems and future users that often will never exist. This effect is worst on a rewrite of a system - look up the "second system effect". We tend to only truly get pragmatic tradeoffs the third and later times.
Make a bare bones version and see if it sticks, and go from there. Both the Progressive Enhancement philosophy and the Lean Startup methodology demand the minimal before building further. Ex: Will anyone click this survey link? Who knows but you'll learn real quick if you offer them $50. If they still don't, then it probably ain't gonna get clicked ever. If they do, then you found at least _something_ viable.
Good idea, you are welcome to help me hone my later chapters. For me the engineering spirit - especially in software - is being a creator/builder. At the core, someone who takes the Legos in front of them and creates with those pieces.
A couple people took my "grandest" comment as a nod toward scale/size/narcissism, which is reasonable, and I meant it more in terms of total architectural completeness - i.e. a system perfectly fulfilling all the requirements that necessitated its creation. A blade of grass is not "grand" in the primary sense, but it still is grand + magnificent by its own perfection in design.
I’d vote for SWE if we are taking up acronym namespace
i think that's just delusions of grandeur. Not everybody would be (or should be) building the sistine chapel.
There is however a pitfall that this approach seems to lead to unless you actively prevent it. Roughly the process is 1. You hire someone to take care of things manually. 2. As you grow, your devs are worrying about other things and so instead of automating you hire more and more people to handle the toil that should have been automated along the way. 3. Because these jobs are handled by one group of people and developers are working on customer facing features, the communication and prioritization necessary to fix the issues rarely come up.
Ignoring history is a guaranteed path towards repeating it.
For purer technology / product businesses, how do you do this, fundamentally? How would Google have manually mocked up their early product? How would Facebook? Github? Tesla for that matter?
Sometimes you really do just unavoidably have to build the product out before testing the market, and if it doesn't work, just accept the sunk cost and throw it away - and sometimes fail completely as a result.
I don't see this as a fundamentally solvable inefficiency, just part of how tech product startups work, and the very tradeoff that must be made to allow for innovation to happen.
there are still smaller pieces you can MVP to a smaller audience before launching it to the world.
> Google have manually mocked up their early product? How would
Crawl an intentional community (remember webrings?) or other small directed subset of the web and see if you're able to get better search results using terms you know exist in the corpus, rather than all of the Internet.
> Facebook?
They had Myspace as an example so the idea wasn't exactly unproven.
> Github?
Kernel developers were already using the software, successfully, all (for large values of "all") they did was bring it to a wider market with a better UI.
> Tesla for that matter?
People don't get this, but Tesla's biggest achievement to date, isn't the cars themselves, but the factory that they're built in. There's no way to MVP an entire factory, but building a car in a bespoke, pre-assembly fashion is totally possible and totally doesn't scale.
If you're asking if electric cars were known to work, the first one came out in 1832. If you're asking about product-market fit, they keep selling cars they haven't made yet, just to gauge demand. Aka where's my Cybertruck!?
The hundreds of millions of USD as an interest free loan seemed more important than anything else.
> Crawl an intentional community (remember webrings?) or other small directed subset of the web and see if you're able to get better search results using terms you know exist in the corpus, rather than all of the Internet.
But that isn't a mock up, it's the real thing but on a smaller dataset. If you're going to do the real thing anyway, why not run it on all the data you can?
After all, the throttling factor to release is in the engine, not in the dataset. If you're going to write the full engine anyway, there's nothing to be saved by limiting it to a subset of the data.
Because more data requires more cleaning and standardization (with more edge cases). It also requires a bigger scale to obtain and process.
You are right that some businesses scale better doing high touch customer service like this. In the case of Pilot, you have the lead sales guy and accounting domain expert (CEO) asking SaaS customers to schedule enterprise service sales calls with him.
Which makes total sense. What he’s not automating is setting up Marketo or Hubspot and Drip or Constant Contact and committing to some CRM system that is both impersonal and adds friction when you’re going for quality over quantity.
He could hire BDRs and CSMs or outsource and spam and set up AI and knowledge bases, and possibly scale up faster.
But not only would that take away the personal touch and competitive advantage, he’d lose the opportunity to educate himself on what real customers really need.
Not to mention all the time spent evaluating tools and setting up automation and negotiating contracts that lock you in to a specific process that might not be what you want 2 years down the line when your business changes.
Getting a million customers isn’t that challenging and really a function of money. With $50 CPA, you just need $50m. 1m customers will take down most sites unless you do some optimizations.
You should make architectural choices that CAN be scaled within reason, on demand and ideally incrementally. That doesn’t mean you need to build your app on Kubernetes and automate welcome emails, surveys, metrics, A/B testing etc. But you probably shouldn’t build on top of fundamentally non-scalable systems either.
This is provided you’re in an environment that can reasonably scale to 1M users or whatever, which isn’t true for all domains. Say B2B where that last B is only 10000 people world wide.
Unless you built something absolutely atrocious, scaling it vs getting it ‘right’ or ‘right enough’, is trivial. You couldn’t sign up to Facebook if you weren’t a college student.
Chatgpt overloads daily so what? When your app becomes necessary to enough people, bring in your k8s evangelists and shove into x cloud provider. But by god it doesn’t take 30 people to write a marketable mvp.
Chatgpt doesn’t have competitors. Customers can’t go elsewhere. Twitter had the same protection when they launched, there wasn’t a short form social network so they could have full day outages. Friendster would be the opposite, when they closed signups, customers just went to MySpace.
Scale goes beyond the ability to support traffic. It also means efficiently handle data/traffic. Heap is a good example, great product but they haven’t figured out to provide that level of analytics without significant costs which are passed on to the customer.
Everywhere? Sounds like you're cargo culting.
There's a use-case for it, and that use-case is when your load is so high you need the orchestration.
Right now I'm using cluster of 2 nodes (master, worker). No reliability, but convenience is good and we have very simple path to more nodes when we would need it.
But ... no one claimed it didn't, did they?
The claim was that the unnecessary complexity (of which Kubernetes was just one) from day 1 consumes valuable time that could be spent on making the product better.
Kubernetes might start consume valuable time if you would start to bring niceties. Like GitOps, CI, and so on. Those things are good but not necessarily justify time spent from the start. So you can start with simple deployment yaml which you can write with 10 minutes and apply it manually whenever you need to update it.
Kubernetes almost out of the box brings to you:
ingress. You don't need to set up your own nginx reverse router. This task alone could consume days of setting up and maintaining. I work with organization which spends 2-3 hours to update their nginx if they need new route. It must be done in off-hours because they break it every single time they touch it and then they need to fix it. Kubernetes ingress is a breeze of fresh air after this nightmare.
cert-manager. Again, I work with organization which failed at setting up certbot. Their certs break all the time. cert-manager just works.
zero-downtime rollout restart. This is absolutely nice thing which works from the box and hard to achieve manually.
kubernetes-dashboard. There're other dashboards as well. I think that lens is very popular although I didn't use it. It allows for people who know very little about kubernetes to interact with it productively. For example in my team we have developers who use kube-dashboard to view logs, to exec into pod and check things, to update deployment, change envs. They just push buttons in GUI, they have no idea about any yamls or kubectl. And it works.
And those things in my experience so far do not bring any unnecessary complexities. Quite the opposite. They solve lots of inevitable complexities for very little cost.
I'm not sure why you are getting downvoted. It would be great to hear of some of the bad experiences of those that disagree. Were they running their own cluster instead of a managed one? Or doing k8s with a team where nobody had any experience? Or run into some helm-chart spaghetti from hell?
Depends. No one at a unicorn says that, but I've seen plenty of startups that. The unicorn is the outlier, not the rule.
> I worked at one company that we couldn’t rack servers fast enough to keep up with our growth. Our competitor who was the market leader had to turn off the ability for new customers to signup to prevent their site from crashing under load. We ended up surpassing them.
Were they still in business at the end of all this? Because it turned out you made a bet, and it came through. The odds are against this, though.
> Getting a million customers isn’t that challenging and really a function of money. With $50 CPA, you just need $50m. 1m customers will take down most sites unless you do some optimizations.
A million in what time period, and what are the usage patterns? Because from what I've seen, as long as the backend isn't in Python, Ruby, PHP or similar, a single hosted server easily vertically scaled to handle more than 2m million users, because it a) didn't need to handle 2m concurrent connections, and b) usage patterns were infrequent enough (users interacted only every few minutes at most).
as an engineer this part I entirely agree with. Far too often CEO driven development leads to pivoting 1Month into 9 month projects, 9 months in a row. such that you have nothing deployed when you at least could have had 1 thing, but instead you have none.
This can be solved as much by discipline to stay on track as by choosing to reduce scope to 1 month ideas only.
Edit: (The load was easily handled by a single Heroku dyno. I turned it up to two dynos overnight, just in case, but it turned out to be unnecessary.)
Not that all spikes are low quality, or even that one on HN ones are. I reckon for some products they are.
The bit that stuck with me (re-reading for the first time since 2008, I forgot how aggressive it was): "Scalability is not your problem, getting people to give a shit is."
And, funny enough, the people that didn't think about scalability, if they're smart, have typically moved on to something else.
Being overly speculative and ambitious early on is a great way to fail. Building on top of something past its limits and ignoring the (developer, customer, velocity) pain is yet another.
If your service opex breaks even at 1k users because you designed for capacity up to 100k users, and you actually end up with 100 dedicated super-sticky users, you'll resent the users you have as they bleed you to death.
If, instead, you build a system that less efficiently, but still reasonably, scales to 100k users by ramping up cost to support 100k users by 4-8x... but has an opex break even at 50 users... Congrats! you've planned for success and planned for failure.
Go get those 100k customers, then optimize. But don't wait to optimize an unscalable system until you already have customers. That's a recipe for lost customer trust and "ahead of their time" epitaphs.
OTOH, the smart people who thought about scalability and designed a highly complex system to handle the load of 5 requests per minute have also moved on.
every system I've ever worked on has needed work to get to a certain scale. and that carries you for a while and if you need to get past the next plateau you need to rearchitect again. rinse and repeat.
so its not a binary thing.
so unless you have some specific goal in mind, invest to get a little past where you need to be today.
That said if you know where boundaries are in your design you should keep those obvious and clear so that scaling will not be hard if you are lucky enough to need to scale.
Slack still answers every customer feedback personally which is endearing and encourages more ideas.
It's easy to say "scalability is underrated" if you've dealt with businesses built on hundreds of standalone PHP/Perl/Python/JS scripts with zero unifying architecture, where migrating a data schema is virtually impossible because of the state of technical anarchy.
Scaling is hard.
There was a brief moment after Oracle purchased Sun where I thought Oracle might try to engineer am Excel competitor on Open Office that had an easier migration to a real database (Oracle), but that dream died early.
Are there scaling issues? Version control issues? Absolutely! But again, that doesn’t mean that it’s not the best tool for the job.
It’s easy to mount the highest horse from a technical perspective, but as engineers it’s also our responsibility to be curious about what people are using voluntarily, and not just what we think they should be using.
A git integration in the editor, and a decent test runner. Some API hooks do Jetbrains can do something with it maybe.
The whole structure is called Open Packaging Conventions and it is implemented in `System.IO.Packaging`.
I think I was hit by everything. Easy stuff at first: Crlf issues, XML Apis, weird rpc apis.
Then, halfway through the project, the results had to change. Not only the order, datatype and headers (I actually overengineered the first version so those were configurable), but the format, duplicate on multiple columns (and empty fields counted as duplicate...). Worst job I've ever done. I'm also disappointed in myself tbh.
But now I'm a bit of an expert on excel format issue and limitations, and that already helped me.
IIRC he just optimized their sheets and set up a CRM for the things that had no business being stored in an Excel file and that was already very helpful.
Attempts at writing actual software to deal with the problem would have failed and everyone was acutely aware of that.
They eventually replaced me with a handful of their relatives, or at least it seemed. It was a lot of fun watching how many LOC one could "write" to request some trivial data from an endpoint. Luckily I was only golfing the latter half of that tenure so I have no regrets.
That's why you use something like a Rails monolith....but oh wait, Ruby doesn't scale well!
Look at Github, for instance.
If you're doing any computation or highly concurrent workloads then you will discover the performance issues with Ruby well before you outgrow your persistence layer.
It took more CPU than if we'd rewritten the core of it in something else, but it let us iterate the rendering engine itself much faster, and most of the expensive computations were done by extensions in C anyway (e.g. GDAL and the like).
Of course you can find areas where it's still not viable, but my experience is that if you start by prototyping in whichever high level language - no matter how slow - that is most productive for you, you'll inevitably find you need to rewrite far less than you might think to get it fast enough. But more importantly: The parts you end up rewriting will very often be entirely different parts than what you expected, because being able to iterate your architecture quickly tends to make you end up in a very different place.
With a kafka queue, and worker nodes, compute-heavy jobs are easily distributed across many worker nodes.
For many parallel requests, you load-balance.
If the PostgreSQL or mysql table is the bottleneck (persistence layer), well this is actually a design decision thats orthogonal to the programming language (PostgreSQL won't scale better with a JDBC ORM).
But the question is, are you trying to make your life miserable by scaling before exhausting other options?
Most applications can be optimised for performance by orders of magnitude, much easier than trying to scale them by orders of magnitude. Any problem is much easier to solve and faster to deliver when you can fit it on a single server.
Usually it is simple thinking that goes wrong. System is slow - add more hardware. But then it turns out developers did bad job and you could still run it from single small server only that someone would have to write code understanding big O notation.
Main point of big O notation is that there are things that implemented incorrectly will be slow regardless of how much hardware you throw at it.
The Silicon Valley world would do well to learn some things from finance. You’d never write a horizontally scaled order matching engine.
It wasn't successful....
Exactly. The author forgot the put the word "premature" before the word scalability.
I have a Java codebase that calls more constructors inside their constructor. So you have this massive god object that instantiates the whole project in side the constructor. If you want to run parts of it in separate threads you can't just take it apart. You first have to rewrite all the constructors.
“… Because the problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle. “ —Joe Armstrong, creator of Erlang progamming language
I don't think this is a problem with object oriented languages. You could certainly do the same thing in any programming language. All you have to remember is to keep your constructors as simple as possible. Passing in dependencies through the constructor is often an easy solution so I don't get the hate for dependency injection frameworks. The original XML driven iteration of Spring was pointlessly overcomplicating it but nowadays you can just define your "beans" in code which basically means the DI Framework just helps you set up a dependency graph and nothing more.
Obviously with some caveats. That you don't fall over at a small amount of users and you are at an early stage.
This is more general than scalability: The more of any work you can afford to defer without hampering growth, the more of your initial capital you can spend on growing, and the more you manage to grow the cheaper it will be to raise money to build out the rest when it's time for your next investment.
The hard part is to determine which aspects you can safely defer.
Pilot's customer base probably looks very different than yours. The author says in the article that they are a "startup-focused accounting firm", so for their customer base which is likely startup founders or similar, the experience of talking to a fellow CEO probably "helps to 10x the customer experience" as the article puts it.
On the other hand if you are selling me some software for a boring Enterprise integration, please don't send me calendar invites for a conversation with your CEO, I'd rather get great API docs and integration guides.
You don't have to build to scale right now. But you should have a plan of when you're going to need to scale, and how you'll do it. Or don't, and wait until your pants are on fire and then run around screaming.
And as the article implies, not just systems but also your processes. I recently saw a moderately big startup strangling themselves on processes that they won’t need for several more years. Mid level leadership (in itself a problem) agonized over scale, and didn’t realize that the products were being built at a snails pace and money was being burned with abandon. Late 2022 was a harsh wake up call.
In huge text: "We partner with the best financial tools in the business" and a lot of logos from famous companies! I think, oh wow I haven't heard of this company before. But then see the text below the large text in small, gray text: "We're fluent in your finance tech stack and seamlessly integrate with the tools you use."
oh...
In addition to having deep integrations with these tools, we actually do have real partnerships with almost all of them.
"What's a real partnership?" is a great question, but I'd mostly think of it principally as "We know them, they know us, we have a red phone we can pick up to get things solved for each other" (which is good for the customer), and then also "And we occasionally do some comarketing together."
For example, there's a "right way" to not do scalability and a wrong way. The right way would be avoiding patterns like microservices early on, while still minimizing human intervention and brittle integrations. The wrong way is to just have some kludgy solution that causes headaches for your customers.
With our budgets tightening we axed startup vendor we were paying $90K/yr to because their shit was slow and kept breaking, pissing off our customers. We were able to spin up replacement functionality for their product in 6 weeks.
Other kinds of scalability (financial, development) are underrated.
Does it work well enough in the geo you want? Yes? You're good to go .