Ask HN: Have we screwed ourselves as software engineers?

409 points by tejinderss ↗ HN
I cannot help but wonder, where is our software industry heading? There are overly complicated solutions to simple problems, huge push for moving to fancy stacks just for the sake of moving. Distributed systems? Kubernetes? Rust for CRUD apps? Blockchain, NoSql, crypto, micro-frontends and the list goes on and on. Its gone too extreme to the point where no one is exempt from these things anymore. Couple of years ago, I thought, its fine as long as I am not involved in this complexity, I can turn a blind eye towards it. But now, this unnecessary complexity has seeped in my day job as well. Managers start talking about "micro services", "writing" kubernetes operators in Go, moving away from python (because its too "slow"); someone recently gave a talk in my company, how to make a 500 line python script (which heavily involves in-efficient handling of IO) go faster with Rust. Someone else talks about that we need to move our poly repos into mono-repo because that where the leaders of the industry are moving to. Even recruiters started asking questions like "have u looked at modern languages like Go?"

I cannot help but wonder, that we have possibly screwed ourselves pretty bad, and there is no escape from it. The vocal minority tries to push these overly complex solutions down everyone's throats, and management loves this, because it creates "work" for the sake of it, but it doesn't add any real business value.

What are your thoughts on this? Will industry move towards simple solutions after experiencing this churn down the line or are we doomed forever?

429 comments

[ 3.6 ms ] story [ 231 ms ] thread
The pendulum always swings back the other way.

Perhaps after a downturn, things will revert to the mean.

To what degree is this valid logic?

Under what contexts does “reversion to the mean” apply?

Over what time frame?

The “mean” implies one dimension. What quantity are you referring to?

(comment deleted)
From Wolfram Math World

> Reversion to the mean, also called regression to the mean, is the statistical phenomenon stating that the greater the deviation of a random variate from its mean, the greater the probability that the next measured variate will deviate less far. In other words, an extreme event is likely to be followed by a less extreme event.

It is a stretch (and invalid, generally) to extrapolate this well-defined phenomenon to a complex system. There are many complex systems that maintain or increase their complexity.

> There are many complex systems that maintain or increase their complexity.

Increased complexity is only achieved with additional energy input. im suggesting that if energy (read: money) going into the system decreases rather than increases, there will be a reduction in complexity.

The software industry has significant energy input.
Consider a counterexample: an open source project that loses favor / funding / contributors. Does it become less complex? Probably not.
when it comes to software, it is easier to engineer a complex solution over a simple one. so in that sense, i would agree with you.
https://www.econlib.org/is-there-a-swing-of-the-pendulum/

> Is There a Swing of the Pendulum?

> By Pierre Lemieux

> People are often tempted to see social (including economic and political) phenomena in terms of a “swing of the pendulum.” In this perspective, problems such as wokism (just to give an example) will be corrected when the pendulum swings back. I suggest that this approach is easily misleading and seldom useful.

> What quantity are you referring to

How about global IT spending (measured in trillions). Or perhaps spending in "new technologies." If there is a downturn, there would probably be a reduction in IT spending, and so many of these overly complex application solutions could get cut, delayed, complexity downgraded, etc.

For me it's a miracle each time my PC boots. We often forget the sheer marvel of modern computers and need to appreciate what we have. Remember Steve Jobs on stage showcasing how you could send an email with an iPhone. Back then it was amazing, but now it's common and we're all jaded about it. We need to recapture the joy of computing, not build large overarching abstractions.
The iPhone is very pretty but the software was not technically impressive. I have much older handheld PCs that could send emails and can still do more than the current iPhone.
Can you do augmented reality on your handheld PC? How about edit HD video?
You completely missed the point.
First software ate the world. Now software is eating software too. My point is, what you write makes sense if the target isn't moving. Writing software will inevitably become more complex because the envelope is always being pushed.

What you say about needless complexity is a very valid point, but it's just growing pains imo.

I wonder if "growing pains" fully captures what's going on here. It might just be natural for people to grab more layers and tools when the run into problems. Everyone loves to demo how the new thing works with "just a simple YAML file."

Honestly everyone would be better off doing everything in code (python, bash, go, rust, c it doesn't matter) directly. They're easy to debug, flexible, and everyone already knows how to work with them.

I suppose grabbing more layers and tools instead of thinking deeply about why you're having the problem and resolving it at its core is the growing pain I'm referring to.

On the other hand perhaps the industry is being ever-increasingly led by a younger workforce who already came into this ecosystem, and there's less chance of a full retro/introspective about why things are they way they are.

I don't disagree with you and your examples are definitely over-engineering / busy work. In my experience a lot of it is driven by the desire for young engineers to learn a new language. If someone paid me to move something to Rust, I would do it. I heard good things about Rust and I would love to get paid to learn it.

But has being a software engineer become easier or harder over the last 30, 20, 10, 5 years? I wasn't an engineer for that long but my impression is that programming today is a lot easier. Dev tools, compilers and linters are very good. There's also a lot more community documentation on stack overflow. Some of the complexity is hidden from the developer, which is good and bad. It can bite you in the ass later, but in 95% of cases its a good trade off in my experience. For instance, my preferred stack is Serverless on AWS. I can set up a single config and have cloud storage, an api, a database, logging, auth all permissioned and in a file I can check in. And with a generous free tier, it's pretty much free. I'll admit if something goes wrong its not fun to debug, but it's remarkably fast and simple for me to spin up a CRUD api.

Some things are easier, but then we've made it so quick we become the lone typist implementing the vision of the all-important UX consultant.
I’ve been working in software engineering for 30+ years so I can say that yes, things are definitely much easier. Debugger in the 80s/90s were finicky beasts that were shrouded in esoterica and as a result, it was usually much easier to try to debug code by adding print statements than it was to actually use a debugger. I’m still somewhat amazed by the capabilities of contemporary debuggers.

Libraries outside those provided by the OS/compiler tended to be hard to come by. Certainly the universe of freely available library code that we have now was nonexistent (I’d argue that CPAN was a big factor in the spread of Perl in the 90s—well, that an the assumption that was widespread back then that CGI scripts had to be written in Perl).

As a community, we’ve collectively learned a lot of important lessons as well. Legacy systems like Perl and LaTeX tend to install common code in a single universal directory for everyone rather than having this be application/document specific as became the case with Java and the repositories will only give you the latest version of an artifact¹ which has tended to lead to stagnation since backwards compatibility becomes non-negotiable, although some lessons haven’t stuck (like the fact that Rust’s crates.io only has a single-level namespace for artifacts).

1. Not sure if this is absolutely the case with CPAN. CTAN, which was quite possibly the first centralized repo² does not manage multiple versions of artifacts.

2. I remember when it was first being set up, since there was no guarantee that any tooling beyond TeX would be available to consumers, there being TeX scripts to do things like encode/decode binary files into emailable ASCII format. The original CTAN FTP server in the UK also had the remarkable feature that it would generate directory.zip for any directory that you requested.

No, it definitely has got worse, I've been doing this 15 years, the sweet spot was the Rails revolution. Before that a lot of frameworks were a bit too much magic, and not enough understanding of how browsers, http and html worked.

Simple MVC stacks went to all languages, jQuery front-ends doing enough but not a lot. JavaScript enhanced easy to reason about server-side stacks.

You used to spend a couple of days a year, yes YEAR, mucking around with tooling. Now it wouldn't be too much of an exaggeration to say you spend a day or so a week fighting some part of your stack because it's been over-engineered.

IDEs can't keep up so you have to run inscrutable command line tools that fail if you have deviated even slightly from whatever error-prone rain-dance some moron claiming 'best practice' has forced into the build process.

Programming used to be about writing code to solve business problems. The shift to DevOps has been a massive productivity drain and most stacks are now incredibly brittle.

The worse part has been debugging, which you touch upon. Native calls, simple call stacks, easy error logging. All gone.

Moving everything into hard to debug http calls has been a disastrous productivity sink.

The irony has been that as languages have got significantly better our productivity has actually dropped massively because of the ridiculous amounts of over-engineering in "modern" code bases.

I recently worked on a project with 2 devs that took 3 months with a modern stack. The prototype in a standard MVC stack I'd made to demo to the client took 2 days.

It's utterly ridiculous and sometimes I feel like the boy in the story about the emperor with no clothes.

> I recently worked on a project with 2 devs that took 3 months with a modern stack. The prototype in a standard MVC stack I'd made to demo to the client took 2 days.

Can you expand on the modern stack and that standard stack? Is the "standard stack" you propose easier to work with now than it was 15 years ago? I guess the "standard stack" is just out of fashion?

My preferred stack (aws serverless BE, nextjs front end or just aws app sync) keeps me in one language (typescript), with some stuff like GraphQL that you have to know. But the tooling around that helps keep my errors in compile time for the most part.

Is being in one language really that much of a benefit? I've worked on several large production apps with node backends and I'd rather use almost any other backend language.

And I take issue with saying you "have to know" GraphQL, that's a pretty specialized tool for pretty specific problems. Most things are not graph structures.

Maybe I'm biased because I've been using node.js for 10+ years but node.js has a decent API.

Simple, slightly opinionated. You just need to be careful of dependencies and what kool kids are doing these days. GraphQL is a complete nightmare.

If you don't transpile anything and are conservative with your dependency you're golden.

In that case it was a simple .Net core MVC stack, Vs an angular/Web API stack.

For some inscrutable reason you can no longer do a simple 3 page form with a results page without making a SPA these days without getting someone claiming you're doing it wrong, it's nuts.

Worse still, the angular app has all sorts of weird bugs. Auto-complete somehow screws with the validation of inputs, back button doesn't work properly as you lose all your data, URLs dont work, etc., etc., etc. There's also all sorts of craziness like the whole admin frontend is bundled with the client-facing part as the front-end developer didn't know how to split them up.

Utterly preposterous.

The problem is often that very skilled Devs give advice about incredibly specific stacks that only someone of their abnormally high-skill level can maintain.

Throw in a few juniors or a few mediocre developers and the whole project turns into a complete and utter mess.

And don't get me started on your stack that's mainly inappropriate for most applications.

GraphQL. Talk about the next NoSQL fiasco in the making. Perhaps you missed the whole saga of everyone doing new development in NoSQL and then a few years later we had the flurry of blog posts about "Why are we losing data? I didn't realize ACID was so important..."

What is it you don't like about GraphQL? It seems to do what it was designed to do pretty well. I take it the issue is more with the pseudo-databases it's used as a front-end for?
Because it's now the 'right' way to do data.

Even though it's not 90% of the time.

Huh interesting. I'm not that plugged in to tech fashions, but was looking at GraphQL just the other day. My conclusion was that it didn't fit though, mostly because I wanted something that supported truly ad-hoc querying and mutation. With GraphQL it's really more like an RPC protocol, so you have to figure out your queries ahead of time, there was nothing like SQL's ability to do ad-hoc querying.
Serious question - if it was so much better then, why are approximately zero new companies building a Laravel/Rails/whatever app and using jQuery for the front end? If it's that much of an advantage I would expect at least someone who is trying that (because surely some are) to succeed with their lean, mean tech stack. Why wouldn't you have just written the project in that standard MVC stack instead of a modern one?

You won't get any objection from me on the debugging point, it's much harder, especially when you're crossing environment - e.g. running the front end locally but maybe hitting a remote dev or QA backend. I will point out though that there are logging tools that support the pretty standard practice of having correlation/transaction/trace IDs on your requests, such that you put in a GUID from an error and it shows you the entire request and anything that request spawned.

Indie makers famously do just that. Pieter Levels is making millions with PHP and jQuery.

I consult primarily in node and react with all sorts of transpilers mess and shitty packers. The stuff of nightmares - we waste so much time making things work - but simplifying the stack will never get traction among the not-anymore-technical principal engineers or even amongst the other developers who need a fancy cv for their next gig.

My side businesses use python, django, jQuery, old node.js without modules, rust, svelte.

Engineers hired in big companies want to work on shiny technologies and build their cv.

> Pieter Levels is making millions with PHP and jQuery

no, the reason Pieter Levels makes millions is not PHP and jQuery, but because he's also a brilliant sales/marketing/business person

OP didn't say PHP and jQuery were why Pieter Levels is making millions - just that they're tools he's using to do it.

Ergo, they are at least fit for that purpose. They aren't bad enough to prevent him from making millions.

Yea people absolutely still build companies with these tools. If you just want to start a Saas company as a side project you would do yourself a disservice if you didn't use something like Rails, Django or Laravel.

The problem is in larger companies, developers stopped caring about just solving the business problems and moved on to solving non existent technical issues to build resumes to go the next job to get more salary. They go to company to company like a parasitic infection leaving them to rot by introducing Kubernetes React, Go microservices with Rust cli tools.

Also it might be an issue of not being fulfilled in life outside of work.

APIs plus frontend can be simple too and I find SPAs better from a UX perspective.

The real biggie when switching to microservices is that you suddenly have to reinvent all the goodies a relational database gives you for free.

The majority are simply going to follow popular opinion regardless of the merits, and developer efficiency is often not that important. I also think the efficiency gains are bigger for smaller and inexperienced teams.

Also, people are getting used to app-like experiences and designers are designing for it. Building an app-like experience is more natural as a Single Page Application, which basically means taking on the modern frontend stack. There are places that push against this, but to do so requires buy-in to the engineering side over product and design. Even then, the engineering side has to be knowledgeable enough to not follow popular opinion and come to the determination that Laravel/Rails/Django is actually the right tool, which isn't always the case.

People still do deploy production ready systems using RoR, Django/Python or whatever "sweet spot" framework you want to mention. Some run quite successful businesses.

You can't generalise from your experience over the last few months.

> Programming used to be about writing code to solve business problems. The shift to DevOps has been a massive productivity drain and most stacks are now incredibly brittle.

Some businesses _have_ to "shift to DevOps" in order to operate at the scale and resilience required.

Some businesses have unnecessarily migrated to over engineered infrastructure because monkey see monkey do.

Saying "everything is ruined" completely misses the dynamic.

As an earlier poster explained there are a lot more tools in the box now. Making the right choice requires experience, a good understanding of the problem to be solved and discipline in implementation.

Get it wrong one way and you end up with an over-engineered mess that takes forever to get work done with.

Get it wrong another way and you end up overwhelmed by traffic, unable to scale in response and forever fighting fires.

God I am sick of these apologetics every time someone expresses skepticism.

> Get it wrong another way and you end up overwhelmed by traffic, unable to scale in response and forever fighting fires.

To nitpick this specifically, over my 12-year career toiling over this stuff there has never been a scenario where this has required a radical rework to solve. Boring-ass B2B shit rarely requires that level of engineering and, at least in my case, the workloads were fairly predictable and increased in a linear fashion. The one time I did accidentally end of DDoSing ourselves, I temporarily stood up nginx instead of our aging Apache install and was able to serve enough requests to fix the problem. (We then transitioned our app to run in nginx)

It was one fire, and it took a little bit of brainpower to fix. Then the DO droplets were humming along perfectly, and last I heard they continue to do so to this day.

The operational aspects of this double-digit-millions-per-year business ran on a postgres database that compressed to 6gb.

The next business I worked for did billions per year in business value and, until HQ mandated migrating everything to GCP, was humming along perfectly fine on Heroku for a monthly spend well under five digits. Ironically, I think they initially wanted us to be on-prem but couldn't support our stack and would have left all of that to our devops guys. GCP was the compromise (oof!).

> God I am sick of these apologetics every time someone expresses skepticism

That's a highly obnoxious response to a measured and reasonable comment.

Because it is repeating theme on these boards: someone expresses a nonconventional or old-school way of doing things, and someone else always jumps on with why overcomplicated cloud shit should be the solution to everything.

Most businesses are never going to be a Netflix or a TikTok or whatever. Yet their businesspeople are absolutely infected with this mindset. And don't get me wrong, it is a disease and its primary symptoms are exactly what OP is complaining about.

So instead of building practical, easy-to-maintain systems we cater to the dreams of excessively optimistic (can you even be anything else as management?) nontechnicals who think every idea is going to require the kind of scale and tooling that someone like Facebook has at their disposal. And they're backed up magpie engineers more fascinated with interesting than functional infrastructure.

Ironically, that very tooling and scale ends up demanding even more resources which demands even more tooling to manage. What a virtuously (hah!) profitable phenomenon for the vendors of these tools.

Good grief.

> and someone else always jumps on with why overcomplicated cloud shit should be the solution to everything.

I didn't say anything like that. the opposite in fact.

Next time try reading and understanding a comment before going off like an obnoxious jerk

You didn't say it, I did. Pot, meet kettle. This "obnoxious jerk" has an axe to grind.
well stop bloody venting on me.

jerk

What's with the name calling? Are we five years old? And you call me a jerk?
Sounds like you’re a bit too emotionally wrapped up in this, there’s no reason to get so mad at other commenters with different experiences. Maybe take a walk. It also seems like your convinced that your experience can be generalized to the entire massive industry, which is a little presumptuous.
I was looking for a new job recently. Almost every single job advert listed "microservices".

So yes, it is affecting our entire industry. Every aspect of it.

There are very small number of organisations that actually have any sort of need of a microservice architecture.

Worse still, actually talk to these orgs, they'll say they actually have a "hybrid" microservice architecture. Which is basically the worst of both worlds, all the pains of managing microservices, without any of the benefits you get in a normally built application (derisively called 'monolith' with all the negative connotations that word has). Half your calls disappear into the black hole of HTTP calls. No pressing F12 on a method call and going straight to the code. No easy stepping through code in the debugger. No simple download the code and just press play and it all works.

I like solving business domain problems. Not tooling problems. Tooling problems are incredibly boring and frustrating to me. To a certain type of programmer, rather than actually doing their actual job, they absolutely love introducing tooling problems as busy work. Because the actual business domain problems don't interest them. Then switch role as they've got the new hotness on their CV before they have to maintain the craziness they've introduced to the code stack.

Case in point on a project I helped get over the line recently. I joined 1.5 years already done on the project, development has slowed to a crawl. Lead architect designed a system of DDD, event-sourcing, message queues, microservices. Just to add a new field I had to edit 10 files. To add a new form I had a PR which edited/added 40 different files. How it actually worked completely flummoxed juniors + mid-level devs, it was beyond them.

All for a 10 page form that would have at most 150,000 uniques in one month per year. Roughly 1 request per second, assuming a ten hour day and 1 request per form page. Child's play.

A standard stack would have easily handled that load, probably even on a VM. A dedicated server would never have gone over 10% CPU. It would have been massively easier to develop, and cost 1/10th in dev time.

At one point I had a quick go at re-writing a section without the trendiness. Just to see, as I'd never have got it through the politics involved in that PR. I switched the event-sourcing, microservices, 5-tier craziness for a simple, easy to understand, service. Took me 1/2 a day, tests passed, reduced DB calls. Over a thousand lines removed, 100 added. Absolute nuts.

Millions wasted on trendy architecture. Of course the architect left a year into the project for greener pastures.

Your comment seems to imply that computer programming == web development. Would you say your comments also apply to embedded, mobile, games, data science/ML and scientific computing?
> The worse part has been debugging, which you touch upon. Native calls, simple call stacks, easy error logging. All gone.

> Moving everything into hard to debug http calls has been a disastrous productivity sink.

There are a lot of good things coming out of the latest big experiments but this has been a major blow. I have worked on software where the intended debugging approach was to write some code, manually push it out to a shared dev environment, read CloudWatch logs for debugging. It is by far the worst way to debug code that I have ever seen. Things that would take me minutes to debug in a normal setup can take hours or days. Projects like LocalStack aim to improve this a little bit but it's completely counter to the ethos of many "cloud-first" developers.

> Moving everything into hard to debug http calls has been a disastrous productivity sink.

I'll admit for the vast majority of web apps out there, this is not needed, but there is definitely a scalability concern if your entire stack is a single ruby on rails mvc application.

This is the misconception that will just never die.

I've worked on multiple products with 1 million+ users running on stock Rails MVC backends. Everything scales just fine. Until you hit a DB hardware limit you can just keep adding more web instances behind your load balancer. And DB Hardware limits these days are astronomical.

If you're actually hitting the limit of scaling our web backends horizontally and you don't have the money to deal with the problem, you might need to take a hard look at your product.

And I'm confident this is true for whatever your backend is: Django, Rails, Laravel, whatever.

I've been in the industry since 1998. Things have gotten worse. Part of it is that in certain ways more is expected of software today but that's not entirely true.

Currently I'm involved in working on a "data intensive" app which munches a few dozen gigabytes of data and presents summarizations through a web based UI. Usually in the form of charts, time series plots, heatmaps etc. Adding small, incremental features to it is an ordeal. The "backend team" needs to mock and then provide the API endpoints, then the frontend team needs to "break down the work on the React components", then DBAs need to oversee the new queries. The security team needs to review risk impacts etc. One change is at least a few weeks of work.

Contrast that with the work that I did in 1998. It was a "data heavy" desktop application that visualized in 3d seismic data collected by ultrasound probes. The data volumes were in the order of hundreds of megabytes up to a gigabyte. The app was built as a desktop app using FLTK and OpenGL. It took about a day or two to roll out an incremental feature once we knew what was being asked. By two weeks it was in all of our customers' hands. While there was no Stackoverflow.com and we relied primarily on offical documentation with the help of "Effective C++" and "More Effective C++" the build/debug/test cycle was much shorter, the tool chain was much simpler and more transparent, the UI toolkit was much easier to grok and even though it didn't have an "elegant state framework" somehow we made it work well without turning it into a mess.

As an industry, we gave up on localized desktop processing in the name of "consumer preference" (or so we were told) while we assumed an immense amount of complexity in order to server the same computation from a centralized place.

Well, some things got easier and other things got harder. Some of the harder things are just due to higher consumer expectations or more competition, but, harder they are.

The easier things are mostly obvious. Better languages, open source libs etc. Things that are harder now than before:

1. UI. The web is not a good UI platform, sorry. Designing UI in the 90s was easier, except for the need to do manual memory management if you weren't using Visual Basic. Partly because there was little expectation of branded UI, so you could easily re-use large control libraries that came with the OS which were/are pretty feature complete and well documented.

2. Cross language interop. Microsoft had this nailed. COM was a beast, but it worked and there was an actual real market of cross-language, auto-bound objects and GUI controls (COM objects, OCX controls). There was actually a thriving ecosystem of languages on Windows which are now mostly forgotten (Delphi, FoxPro, VB6, Paradox, Visual Prolog etc). Nowadays cross-language interop is a joke. Transpile to JavaScript or go via a C FFI, maybe, if you're lucky.

3. Expectation of supporting multiple platforms. This causes a lot of dysfunction because the lowest common denominator is really low, especially if you treat the browser as a "platform". In the 90s Microsoft had a monopoly. That had its own problems, but, it meant you could write a Windows app, once, and everyone would accept it. Nowadays you want web+mobile, and "web" is not really a platform in the sense Windows was. If you're doing anything serious you still need a desktop app.

4. Over-specialization/staffing. Back then everyone was "full stack". The developer was also a DBA, at least to some extent, and you could just throw up a quick GUI Windows app that connected directly to the database. The DBA would then manage security and backup. It wasn't really a full time job, at least not on a per app basis. Nowadays even simple projects feel absurdly over-staffed. Do you really need a backend guy, a frontend guy and a devops guy for an ordinary LOB app? Probably not.

5. Process overkill. Waterfall is underrated. It got a bad rap because it requires you to understand your customer, and for your customer to understand what they actually want, and for them to not change their mind every five minutes. Not always possible. Nonetheless, agile has become some kind of monster. Half the words in the average agile methodology are made up and a lot of it is really questionable. If your business domain weren't totally unstable and your users weren't totally incompetent (often the case!), then you could sit down and write an actual spec, which people would read and sign off on, and then you could build it. And it'd work, and after the initial debugging / shakeout period, people would be happy. Many of those apps never broke! Just imagine the output of the typical agile web AWS-based web stack teams today lasting 20 or 30 years without a dozen rewrites along the way. Very hard to imagine that.

> I wasn't an engineer for that long but my impression is that programming today is a lot easier. Dev tools, compilers and linters are very good.

What sticks out to me when I'm looking at some older code editors/IDEs, is how crazily spartan they are - to the point where they are just inconvenient to use.

There's a big leap from "a few people at work are overengineering things" to "we have screwed ourselves as software engineers". I don't think that you can generalize to the entire industry based on your experience.
If you believe that's all there is to those ideas, maybe you need to step away and think about them for a while. Sure, there's going to be some resume padding happening in larger orgs. But all those ideas solve real problems too.

I think you're just in a very negative space if you start with "Distributed systems" as something overly complicated. At some scale getting a bigger machine either doesn't make financial sense or is just not possible to implement efficiently. Some ideas are taken too far or implemented where not needed. But I'd rather recommend you to learn where each one of them started and why. Criticize for valid reasons, but don't become a curmudgeon.

>> If you believe that's all there is to those ideas, maybe you need to step away and think about them for a while. Sure, there's going to be some resume padding happening in larger orgs. But all those ideas solve real problems too.

They do solve real problems. The question is whether or not they solve the problem at hand, and if they create other issues in doing so.

I was re-decking a back yard bridge with a friend and he brought a framing hammer. I'd never used one before and I always had a finishing hammer and didn't know the difference. I learned a new tool and even ran out and bought my own for the project, which worked fantastically well. It's still in my toolbox and hasn't been used since. You just don't use that thing to hang pictures on the wall because it may well f-- up the wall a bit. Using the right tool for a job is way more important than using a particular tool for any other reason.

I’ve never liked this best-tool-for-the-job mental model with software engineering. A given project has multiple needs, and unlike more physical tools there is a very high marginal cost for each incremental tool you use. So there is a huge balance between many well suited tools or a few but more generalized, less fit tools. This isn’t to say there’s an obvious place where to strike the balance, but the “best tool for the job” metaphor undermines recognizing it, and I’ve found this balance to be at the core of good tool picking for a given project.
I agree with you. But dont add a new tool (even a more general one) unless it fills a gap, or has other specific benefits.
I also dislike the BTftJ (best tool for the job) cliche.

Best according to who?

Based on what priorities?

Relative to what other options?

Based on what kind of experience and expertise with the various options?

My takeaway: BTftJ must only be a starting point for dialog and discussion. Shortly therafter, it is time to get real about each of our experience and biases. Otherwise, BTftJ is a only a thin veneer.

Thank you for calling out BTftJ as a mental model. Recognizing the very idea of mental models is a huge step forward. Arguably, it could be used as a metric for how advanced an organization (or even society) has become.
>> I’ve never liked this best-tool-for-the-job mental model with software engineering.

I don't either. As photographers say, the best camera is the one you have with you. Same thing, the best tool for the software job is the one you already have and everyone knows how to use. Why anyone would want to introduce these high-end fads to an organization without proper reasoning is beyond me.

That’s definitely part of it. Another part is just the various multipliers you get from having a small number of tools. Team mastery, more mobility of personnel (no/low ramp up times), unexpected opportunities for re-use, high quality onboarding due to narrow scope, etc.
I think the metaphor is more apt than you’re giving it credit for. You just don’t have enough experience with the trade offs involved with “the best tphysical ool for the job”. :)
I use a framing hammer to hang pictures on my wall, and I've never messed up my wall.
Well that's because you don't drive the nail in all the way to hang a picture. Would not recommend it for molding for example - any time you want to drive the nail flush without scuffing the material around it. My bridge had plenty of cross-hatch marks around the nails when done ;-)
> But all those ideas solve real problems too.

All of them, except for blockchain. That one can go die on the trash heap of history.

Well, it's a good tool for money laundering and purchasing drugs.
That's not fair, it makes a great Ponzi scheme, too.
(comment deleted)
It's not.
How many "darknet markets" accept anything else these days? What's the alternative, cash in the mail? Venmo?
Keeping a permanent log for one's illegal activities seems... suboptimal.
> That one can go die on the trash heap of history.

As long as there are Rust jobs or blockchain related jobs there, you're going to have to cope and wait for a very very long time until that is ever going to 'die on the trash heap of history' to happen.

I'm not sure you realize just how quickly that can all go away (and does). Blockchain companies are a dime a dozen. They don't all last forever.
> I'm not sure you realize just how quickly that can all go away (and does).

So why hasn't it died quicker than it should have years ago as many have incorrectly predicted then?

> Blockchain companies are a dime a dozen. They don't all last forever.

And who said that the 'companies' did last forever? Why do you think I said as long as?

I'm just wondering if the whole thing is guaranteed to totally go away 100% and to be absolutely used by no-one since clearly someone also thinks so. That is my question.

Access to the dollar is a real problem. Stablecoins help resolve that.

I am not a crypto stan but it has at least one usage.

e: Sorry, forgot it was verboten to say anything contrary to the "crypto has no uses whatsoever" line.

It’s a good store of value. Like gold used to be. As a currency I’m not sure it will ever work.
I'll never understand the bitter hatred toward blockchain at hackernews. Are you guys just mad you knew about bitcoin when it first started but didn't get any? You can't possibly truly believe blockchain solves zero problems can you?
I know blockchain enthusiasts claim that it solves some problems, but in reality it fails to solve them because they are not fundamentally tech problems. You could think of it as a performance art documentary of a group of people learning to recreate the world's financial regulations in a bottle, but the price of admission is a trail of fraud victims. Furthermore, it is wasting the world's energy supplies, and eating up our attempts to move to renewable resources.
Blockchain itself solves trust problems. Decentralized database/ledger solves issues where trust matters.

As for bitcoin itself, which I assume the latter part of your comment is talking about:

It’s predictable trustless money. It’s not perfect. But where you totally Lose me is:

> it is wasting the world's energy supplies

This is the latest POLITICAL attack vector. When you parrot messages like this, you expose yourself as a political victim that is not educated about the topic. Literal parrot.

If you have an interest beyond being a political foot soldier, do your own OBJECTIVE research. Learn about bitcoin miners using renewables, how much energy the industry actually uses, the carbon footprint of payment networks and fiat etc.

It’s an effective political attack vector because in all likeliness you won’t do your homework. You’ll read my comment, experience cognitive dissonance, judge me as a bitcoin cultist, and move on with your busy life.

If you will point me to sources on those numbers you consider reliable, I'll take a look at them.
What’s crazy is the scale a modern computer can operate at. There _are_ problems that need more scale than that but they are the minority. Meanwhile it’s out of fashion to spend time improving application performance and instead people go horizontal early, with devastating complexity issues.
That's why I mentioned "doesn't make financial sense" too. If you can throw another $X/mth box at it, or have engineers spend weeks improving the system's performance for the same effect... there's a point where scaling out makes way more sense. Whether that leads to complexity issues is really case-by-case.
> At some scale getting a bigger machine either doesn't make financial sense or is just not possible to implement efficiently.

You don't even need to get scaling into the picture.

You do not get any form of fault tolerance by deploying stuff in a single lonely box. If you care about reliability and resilience, you have to have multiple deployments up and running at the same time.

Also, you already have a distributed system if you have a browser calling your server.

And lastly, if you happen to manage an service used globally or regionally and perceived performance matters then you have no good alternative to have regional deployments.

> Rust for CRUD apps?

Are all CRUD API’s insensitive to performance? Correctness?

No, but Rust is a low productivity, high formality language; one needs to make a tradeoff whether it's really that important. Given that most CRUD services are just a layer on top of a database.

This is the reality of software development; you have a budget and you have a goal. If all your budget goes up on making it correct without finishing it, you have a problem.

Anyway, it's just a CRUD app, it doesn't have to be as formal.

>No, but Rust is a low productivity, high formality language;

Setting aside performance (Python is slow as molasses) that still requires some qualification because there is the 'maintenance' dimension. I would take Rust over Python for CRUD any day, because the 'formality' is a feature not a bug for maintenance. I would take Java/C# over Rust because the former balance performance and formality very well. In fact, I wouldn't use a dynamically typed language like Python or Ruby for backend infrastructure code if there were any performance or long-term maintenance requirements.

I'm far from being proficient in rust but my productivity writing crud apps in it is pretty good (just throw some Rocket + SQLX and you're flying).

Rust is a very high level language, it's not like writing assembly. You rarely run into complicated situations with the borrow checker when writing trivial crud apps.

The ecosystem is not as mature for web apps compared to other languages, so it may be more efficient to pick something else, but ultimately the most important factor (when building trivial stuff) is picking a language you and your team are comfortable with. Whether that's Rust or node.js, it won't matter much.

If I were to make something else which is better supported in Rust, eg. a videogame, using bevy + ecs would run circles around a bunch of other languages and their game frameworks.

Claiming that Rust is a low productivity language, generally, is too broad of a claim to be even be plausible.

The parent commenter may have some narrow or situational definition of productivity.

I asked the question rhetorically. The answer is "no". Characterizing an application as CRUD does not exempt it from some mix of performance goals.
One hypothesis (but not a single answer) is that complexity creates jobs. Engineering something complex and clever creates job security and consulting hours. Fads, trends, and ideas come and go like the tide, and it makes the big wheel go around. Kubernetes, which 99% of developers have no use case for, is indeed “job security” for what could equally be achieving achieve with Unix and a few she’ll scripts. (I’m being deliberately provocative here).
This is an important point. Looking at open job reqs can give the impression that X is very successful, when the reality is that any organization using X has an explosion in how many developers they need, and also a high burnout rate in the ones they have. Plenty of other developers quietly and more-or-less-happily working with non-X, and you never see their job in the job boards.
This question is getting a lot of attention. As such, it is consuming our time. I’d suggest it can be improved and revised. Currently, it is rather vague. I suspect the author could take more time and make it clearer. There are a lot of interesting themes — it deserves to be unpacked and clarified.
True that, but perhaps asking the question and getting these answers is part of how that happens, and the end result is a longer-form blog post somewhere.
I think about it more from the perspective of "building stuff that is useful and interesting". I can very quickly build a lot of cool, useful stuff with JS + Node + React + Postgres.

Yeah there is a lot of overbuilding and BS in our industry, but I don't think we're unique in that regard. It is safe to block out the noise and focus on what excites you.

Agreed.

You should know that in the past when OOP was not common, we had to work a little harder doing things like managing our own memory or building a LAMP server to publish our web pages.

There was a thriving market for language and UI add ons. The result was that each company had their own internal dev tools and recruiting people outside of the company who had experience with those tools was nearly impossible.

All that said, we were at a point where entry to programming was easy (think Visual Basic in the 90s). The quality generally went down as everyone was pushing their "first project" as if it was a polished product. Finding actual good programs on PC is close to the situation on mobile where most of the apps are trash.

OOP to me would be classified as the industry blindly moving towards unnecessary complexity. It is indeed the definitive example of how the industry over-engineers things.

While still prevalent today, there's a huge sentiment against this paradigm. Modern languages and frameworks such as React, Golang and Rust show how the tide is turning against this.

Three object oriented languages prove how there's sentiment against OOP?

Or are you saying that Rust/Golang/React are simplistic in a world of over complexity. React I would generally agree with, the other two not really.

React is extremely complex internally
Very true, but unlike backend languages you can generally get away without knowing much about the internals. Unless you're making dev tools of course.
I think react was an attempt at simplification. OOP isn't the only thing that influences complexity. Simply switching a project from OOP to a more FP like paradigm doesn't necessarily make the framework simpler because of other confounding factors.
All three are moving away from OOP.

Functional components in react. Zero classes in golang or Rust.

It's actually quite an obvious paradigm shift. No classes with these languages yet every language prior to this has classes.

Yet I don't understand why people still insist on calling go and rust object oriented or why they try to argue this point when there is such an obvious change. I mean sure you can twist the language into something that looks like it's oop but come on man. Haskell is OOP if that's the case. Let's not argue about whether these languages are OOP or not. The point is there is an obvious movement AWAY from OOP with substraction of popular and critical syntax and features.

For React , the trends are harder to see. See here: https://hackernoon.com/react-functional-components-are-the-f...

The react team recommends functional components over classes, and react itself was derived from functional languages and concepts. I believe the team ideally would like to see ReasonML as the future and react with js syntax is more of a necessary in evolutionary inbetweener thing.

Functional can be tricky to debug, as the frowned-on "intermediate state" of imperative programming makes for great x-ray examination points during debugging.

The backlash against OOP is partly because existing OOP engines are limiting in many languages. Passing, grouping, and custom-scoping of "blocks of code" (BOC) should be more flexible so that we are not forced into hierarchies or spaghetti scoping. We need new languages that make managing and scoping BOC more flexible. I want to define the scope, not let Nadella or Bezos do it. The distinction between a lambda and method would then be fuzzier. BOC's would no longer be forced to be one or the other. (One these days I may make a proof-of-concept language.)

Imperative and OOP are not the same though.

None of the languages I mentioned are FP. Though FP has benefits and many of the languages or frameworks mentioned above are either moving towards FP or borrowing concepts from FP.

You're not wrong about intermediate state. I shouldn't have emphasized FP here that wasn't my point. An imperative program that is just imperative without OOP or FP tends to be simpler then OOP itself.

I would argue with notion that this complexity is unnecessary.

But most importantly — if it really is, as you say, you can profit of that. Open a consultancy, and solve client's problems without using these over-engineered solutions. If your competition truly wastes a lot of time, then you would be able to solve the same problems faster and in a more effective manner.

That is a nice theory, but then you need to do sales, and if the people in charge are swayed by buzzwords like microservices and monorepos, and your marketing language only goes as far as, say, "reliable, proven technologies", you'll be passed by.
Highly depends on where you work. In my company we stick with "use boring stuff" and have a limited amount of "innovation spending". I look at the complexity of these other things mostly as a computer science theory and ways that language and solving problems could be done.
It's another way to favor incumbents with huge resources: it's an anti-competitive psyop to raise barriers to entry and stymie startups in endless shiny thing chasing, because the incumbents are scared of the disruption, so those big players that move a bit slower, seek ways to slow down the scrappy players that can move fast, while those "fast movers" get drunk on the koolaid of buzzwords manufactured on blogs from those incumbents, and trends like being agile, microservices, etc, while willingly injecting themselves with massive technical debt from overcomplicated frameworks, and nobody's allowed to question it otherwise it's civil flame war.

Even if you don't believe it's a conspiracy, you have to admit that: that dynamic favors incumbents, the big frameworks often come from the big incumbents, and if you are a big incumbent even if you weren't manufacturing this conspiracy but you saw its possibility I mean why not take advantage of it?

The outlier is the scrappy indie developer like Peter levels who runs his multi-million dollar a year business on his own basically on a single machine using PHP and only recently started using git. That may be an extreme example but it paints a picture of what radical effectiveness and efficiency looks like and it's vastly different to the Kool-Aid but don't mention it otherwise the mob will come for you.

May the no-code & indie scene save us all. Amen

Did you learn the word 'incumbent' yesterday?
Aaw, so cute. My smart piece threatened you and you want to prove you're smarter than me. Nice try but you failed.... I learn all the time, you should try it. XD
I saw your original post and your cringe edited one. I won't even comment.
(comment deleted)
Oh don't cringe, it was made for you, Haha. I guess you're cringing at your own daftness in the face of my brilliant comment. You were meant to see it. You mean this one? https://en.m.wikipedia.org/wiki/Pine_Gap

As for you not commenting, it seems that would be a good idea on your part. You keep trying, and keep failing. Do you often do things you say you won't? You lack self discipline? Anyway, thanks for obsessively checking my edits.

Take meds, get vax, and touch grass.
Is that what you do? How come it's not working for you?
> nobody's allowed to question it otherwise it's civil flame war.

With enough years in this industry I'm starting to see how much of the culture is just "cargo cult Silicone[sic] Valley!" -- People are doing things just because the latest/greatest unicorn did it that way, and frequently there is a form of Zen/Chan disregard of meaning or learnings from the past. "Move fast and break things" often plays out as "Disregard knowledge and act like there wont be consequences" . I frequently see threads/posts on HN by experienced people laughing that "we've known this since the 70s" in reference to The Mythical Man Month, or other bits from Computer Science, and yet most managers think they know better than decades of industry experience.

In general, people only move their career by pushing for new things which inevitably become increasingly complex overtime as all the low hanging fruit is gone.
Nim is easy to use but performance.

I'm releasing a (web) development platform for it this month. Just getting the code ready.

I don't believe the issue being addressed is a lack of web development platforms here.
The author mentioned CRUD with Rust, which sounds like they think Rust is overkill for CRUD.
Software ate the world, but didn't digest it quite properly, and now the world is in many ways broken.

I think some of the complexity stems from trying to make digital things that simply aren't or shouldn't be.

So, you have valid points, but...

1) some of these things (e.g. node, microservices) already peaked a few years back, being overapplied and now the pendulum is swinging the other way

2) others (e.g. Kubernetes, React, monorepo) were developed at large, profitable companies that others wish to emulate (or work at someday), so they find excuses to use them. This case takes longer to reach a point where things swing against it, because everyone wants to pretend their company is the size of FAANG or will be soon, but the same process of overapplication and backlash happens eventually

3) in the midst of all that noise, there are some new things which are in fact a good idea for most developers. I don't know Rust or Go, but perhaps they are examples of that.

The key for us as developers is, unless we wish to work at FAANG, try to spot (3) in the forest of (1) and (2), and don't let (justified) annoyance at (1) and (2) blind us to the fact that (3) is out there as well.

Yes, but it has very little to do with any of the things you mentioned. Software became so prevalent that it became a mantra, "Software is eating the world" but people in IT have little to no choice in what's put on the plate. They've been convinced that positions of responsibility and authority are bad and should be left to the MBA's. Eschewed most industry groups that provide some semblance of protection that nearly every other professional organization has adopted. Doctors, lawyers, actuaries, accountants, all have professional organizations that are powerful and provide some protections. I believe that most of the pathology that you see is a result of a push and pull between management and IT as people in IT seek those protections in other ways.

Just one example, Scala. First, I'm not criticizing the language itself. It has it's place, but what I saw was programmers trying to create a protected space that would provide higher bill rates. Java was everywhere and hiring a Java developer was easy. Scala was new and had steep enough learning curve that you could drastically shrink the candidate pool while at the same time selling the shiny new toy to management. They could create complex, arcane code that kept new developers from getting up to speed while providing the excuse that they were inferior developers and weren't smart enough to keep up. It didn't work for very long as management caught on that they weren't getting much other than higher labor costs. Go seems to be the latest incarnation of that while Rust is a bridge too far to sell to management.

So it's this back and forth, provide something to management that they can sell to their superiors something new. Management buys into it as long as they can get promoted before it inevitably blows up and the developers who sold it move on to new projects, rinse and repeat.

> They've been convinced that positions of responsibility and authority are bad and should be left to the MBA's.

Meh. From my experience, many developers actively don't want to go into management, because usually your whole day is filled with management crap and you can't go and actually code any more. And developers who do switch to management often end up as miserable bosses because their bosses don't care about "leadership trainings".

Additionally, many companies have the non-management track end at senior level, which means zero career progression for those who do not wish to transition to management.

>> your whole day is filled with management crap and you can't go and actually code any more.

These are clearly the wrong people to be pushing into management. Good management (it does exist) includes people who have coded, but are willing to give that up to enable others to do that. They get satisfaction from being enablers and making space for their underlings to be creative and make decisions.

Further, there are many excellent managers who don't have a typical developer background, but can recognize what success means for their team within an organization and how to achieve it. I've been managed by many excellent managers with backgrounds in chemical engineering and the classics.

The developers you describe should decline these positions and find a better fit where they can make better use of their time. Choosing to accept positions like this hurts them, as well as others.

The way I look at it is: there are more tools in the toolbox than ever before. Which makes our judgement (the thing they really pay us for) even more important. Kubernetes, for example, is a specific solution to a specific problem. The solution is complex but so is the problem. If k8s give you the right trade-offs for your situation, then it’s not busy work.

Of course, there are plenty of project where judgement Is thrown out the window in favor of adding a buzzword to everyone’s resume. I’ve heard it called “promotion-based architecture”, as in you pick the technology most likely to get you promoted. (If that works, it says all sorts of not great things about your organization).

Regardless, I don’t think the availability of tools is the root problem. It’s and industry-wide lack of emphasis on identifying and understanding the problem first.

"promotion-based architecture" aka "CV-driven architecture" :)
No, it's called "résumé-driven design".
I call it Resume Driven Development
Yes, this has been my observation as well. I also see this happening concurrently with what I call the "shiny object" problem. Developers have their eye on a shiny new library, framework, language, etc. and have a seated desire to use it at the expense of it being an inappropriate choice for the company.

I believe these two phenomenons are producing a positive feedback loop in the industry. Selected technologies address one challenge, but introduce complexity. The complexity becomes difficult to manage. So other technologies are incorporated to manage the complexity. In the midst of all this, core technologies are replaced, swapped out, or transitioned to under the desires of the dev(s). For example, switching from one js rendering library to another, while preserving the old legacy code. The complexity footprint keeps growing and it doesn't stop because the engineers themselves aren't entirely committed to the project. They can incorporate the new technology to the project, pad their resume, and bail to a new employer if things grow out of wack.

I worked with someone who did this. I'm glad he has moved on. He would not do basic tasks correctly. Had an attention span of a toddler. He was DBA and couldn't google simple problems. I asked him to do something he would say give me 20 minutes. 2 weeks later he would say "IT CANT BE DONE". Then, I would do it myself in a few minutes after googling the error.

Instead of applying knowledge, he would bring up all these buzzwords to meetings and not really understand what he was talking about. I get angry just thinking about it.

Hmm, incompetent folks exist and need to be let go. However, I think it is a slightly different category than RDD.
> I worked with someone who did this.

Nothing in your comment points to resume-driven development. You even failed to mention any project or design decision. You just decided to get angry at a coworker because of something.

Promotion based architecture is self fulfilling prophecy at least in BI/ data world.

I see everybody around me moving to cloud, without really good explanation why. Only reasonable thing I can see as an pattern is that cloud experience on top of data things gets paid 30% more. It made me consider cloud a lot.

I was considering switching to cloud, just so I can put in my CV "experience with migration to cloud".

For next person commenting that it makes sense: it doesn't with 200gb database and super predictable workload, growth and usage.

There are non-IT perspectives on this as well.

My company offers two deployment scenarios: host it yourself and cloud hosted (SaaS). Many of our customers choose the latter because their internal IT systems require more process, and they just want to get something up and running.

Just to add, at my last company it could take up to 6 months to get a VM provisioned on prem. We could provision what we needed in Azure on demand.
>For next person commenting that it makes sense: it doesn't with 200gb database and super predictable workload, growth and usage.

Depends on the company. I've been working for marketing agencies for the last 15 years and they're generally staffed by, at most, 1 IT person who is in charge of a third-party vendor relationship that offers managed IT services. Those IT resources (internal or vendor) don't specialize in data and often don't know how to deal with it well, predictable workloads or not, and often offer up solutions which are not appropriate (cost or otherwise) whereas there are managed solutions from cloud vendors that do. BigQuery, for example, handles compute and storage for you and rolls it into one reasonable query price. No need to worry about any management of anything there and just sit Data Studio (included) or any other BI tool (Tableau, etc) on top and you're good to go.

I get your skepticism and welcome it, but you're being a little rough. We're cloud native at my current company (I did a partial cloud migration at my last which was completed after I left) and it makes my life leading a Data Engineering and Data Science team MUCH easier without upfront hardware/software costs or long-term contracts, which were STAGGERING and left us with much more difficult to maintain, upgrade, etc hardware that took up most of my job as opposed to almost none of it today.

YMMV.

Partially depends but company with 1 IT person that is not data focused, there is low probability that they will have that amount of data.

For anything above what we have I think it weirdly depends on country/ salaries.

In US, 100k per year is no brainer for cloud as even 1 FTE would cost much more.

In non-western Europe, 100k is deal killer as you can hire 2 senior DBAs and you still have enough money for quite a reasonable server.

With extensive experience in this vertical, (marketing agency and analytics), 200GB is a _partial_ day for one datasource, let alone table or total DB size.
but does team have 1 IT/ data person?

Edit: sorry I see that you are the same person I was replying to. Then to understand it better, who is consumer of so vast amount of data?

The company has 1 IT person and uses a third-party vendor for IT support. We have several data engineers and scientists, but we are not specialists as DBA or cloud infrastructure; it's just one very small part of our work.
Interesting, thats correct, its segment of the companies that I didn't take into consideration. Thanks
(comment deleted)
> 1 IT person who is in charge of a third-party vendor relationship that offers managed IT services

That is something completely different from migrating your inhouse applications to the cloud by introducing kubernetes. I think your example show nicely that there is not one solution for everything. As an industry we have to put way more emphasis on deciding tradeoffs by understanding requirements and risks.

I see everybody around me moving to cloud, without really good explanation why.

People just buy into the "cloud" marketing. They don't have the ability to think and reason, and so don't understand that "cloud" just means "renting someone else's computer."

I built a complex in-house medical system. Quick, reliable, and liked by the users.

I was in the middle of adding a major new feature when all of the management in the IT department quit. The new people immediately decided that the whole thing had to be "in the cloud." I was removed from the project, and they hired three people full-time to rebuild it.

That was three years ago. The new system is still not online. The users are still using my old system, and the feature I was working on never got added because my presence and input was not welcome because I "don't understand the cloud." So I got moved to other projects.

People talk about "the cloud" with the same fervor and language as members of a cult. And you will be an outcast if you dare challenge their way of thinking.

Given what you explained, it seems to me that the cloud was an excuse given by the new overlords to just get you out off the way. Cloud is fashionable now, but any excuse would work for them. This seems to be a power grab.
I don't think it's about power. Because of the way the organization is structured, I am isolated from the IT department.

Moving me out brings no advantages to either part of the org. In fact, it's probably a disadvantage to IT because it has to burn three people from its allocated headcount. Where previously, it had a bonus person that didn't come out of the IT budget.

It's always about power and politics when bone-headed decisions like that are made. You correctly identified that _you_ did not report to IT, therefore _you_ were a threat to IT's power. Maybe you could have been re-orged under IT, but they probably saw you as a troublemaker who wouldn't kotow.
At the risk of sounding cultish: I would argue “the cloud” is a bit more than just renting someone else’s computer. Hosting companies existed long before the “cloud”.

The difference is the abstraction layer. I would define “the cloud” as a layer that abstracts away the physical infrastructure. (Which works until it doesn’t but that’s a longer comment.)

You can even apply “the cloud” to your very own fleet of computers with the right software.

Like I said in the original comment, it’s a specific solution to a specific problem. Whether it’s right for your system I have no idea.

You can even apply “the cloud” to your very own fleet of computers with the right software.

If it's on-prem, it's not a cloud, it's a fog.

At least, that's what I've been calling some of my deployments.

(comment deleted)
> People just buy into the "cloud" marketing. They don't have the ability to think and reason, and so don't understand that "cloud" just means "renting someone else's computer."

It sounds you're trying to dismiss and downplay solutions to problems you're oblivious to.

Even if you want to simplisticly frame "the cloud" as "renting someone else's computer", keep in mind that:

* It's not one computer but as many as you'd like, and get them at the click of a button,

* These are computers which are managed 24/7 by a team of highly trained specialists,

* These computers can be located anywhere in the world and simultaneously in multiple regions,

* These computers are designed to be fault-tolerant and handle (and survive) way more stuff that can conceivably be thrown at them.

> People talk about "the cloud" with the same fervor and language as members of a cult.

I'm sorry to say but you sound like you're militantly opposed to a solution to problems you either don't understand or refuse to understand.

"The cloud", even when following the simplistic and clueless belief that it's just "renting someone else's computer", solves whole classes of technical and business problema that your box under your desk cannot solve.

> Only reasonable thing I can see as an pattern is that cloud experience on top of data things gets paid 30% more.

You say that like a pay rise of 30% is not a good enough reason all by itself for many people.

> For next person commenting that it makes sense: it doesn't with 200gb database and super predictable workload, growth and usage.

Perhaps not for technical or business financial reasons, but those are not the only possible reasons someone might do something. As mentioned before it can make a lot of sense to migrate to cloud if it means you can get a 30% pay rise.

There's a finance angle behind cloud stuff too that's irrestible for the bean counters: cloud stuff is operations expense, on-prem is a capital expense. Unfortunately these folks are heavily incentivized to favor OpEx

I'm not a bean counter, all I know is those guys at my last job would rattle off about it like zombies. IIRC its a tax thing

Opex is fully deductible each year the expense occurs, wheras capex requires amortized depreciation over the life of the purchased item. Makes it harder to calculate taxes.
Should we call it Amortisation as a Service (AaaS) then?

Is that really the only benefit?

I mean, I'm not a bean counter. I'm just guessing that's why they like it.
Wrong. People has been leasing on-prem hardware for decades before the cloud existed.

Depending on taxation rules you might want to buy or lease.

At this employer it seems most of the stuff in their datacenter was purchased.
It's not a tax thing, long-term leases or colocation is a liability for acquisitions. (You could also build a revenue-positive company, but that's the loser way out, apparently.)
It makes sense when you aren't a datacenter hosting company as your core business. Once you're done with your DRP and BCP it's highly unlikely that your server-under-the-desk (as it usually is) is worth the risk.

By the way, moving 'to the cloud' never means a specific thing because people have made words (intentionally?) vague to the point where you have to explicitly specify all the factors you take into account with your work in order to figure out which 'cloud' they had in mind.

Running a static workload doesn't require elasticity, but a 'cloud' isn't just elasticity. If you want "a program on a server with some storage that is always there" without having to deal with hardware, software, networking, storage, backup, maintenance, upgrades, service contracts etc. then an arbitrary virtual machine where someone else takes care of the rest makes total sense.

And its easy to compare the cost of the cloud running your hardware to the cost of physical hardware. However, its much more difficult to compare the indirect costs between the two, and that's where I think many people go sideways.
Hell, I'm working with ~200TB and it still doesn't make sense if you work out the costs.

As far as I can tell it only makes sense if you have a ton of data, and only ever use a fraction of it at a time, for one-off jobs, infrequently.

It's a bit dismissive to just focus on scaling as a cloud advantage. You also have lower maintenance, easier redundancy, lower chance of outages, easier backup and more such non-functionals. A 200GB DB costs about $50/month on AWS (other providers are less), that's not an interesting amount of money for most businesses.
Thats way off. With 64gb ram we have around $50 month on hetzner yes. But from asking around the other BU with very similar size and usage to ours, somehow they are paying around $2600-5000/ month on azure
Are they using Oracle or MSSQL where you're paying insane software licensing costs?

With Postgres[1], you would need to be using an absolute beast of a server to be paying that much, even if you're paying the 2x premium to go month-to-month, and a low-powered machine could be around $50/month for a 200GB DB.

[1] https://azure.microsoft.com/en-us/pricing/details/postgresql...

Yeah, its true that they are using MSSQL.

Based on link you provided, I calculated that 40gb memory would be $290 and 80gb $570 per month (in UK).

So price seems still quite steep, considering that same specs on-site would be ~$2000 one off. (8 months return on investment).

Then for this server, I honestly don't believe its easier to do maintenance in cloud, as you need to do lots of initial configuration there (be it IAM, permissions, firewall etc. ) compared to have it just in local network

> With 64gb ram we have around $50 month on hetzner yes.

A single box on Hetzner is by far not comparable to, say, using a managed service as AWS RDS. It's not even apples to oranges, and more of apples to freshly-squeezed orange juice served by a buttler. Think about it: do you get any form of fault tolerance with your single box?

Thats why I am saying that $50/ month is way off. The price for that would be much higher
> Thats why I am saying that $50/ month is way off. The price for that would be much higher

Ah yes I agree, sorry for not being clearer. I wanted to add on to what you said.

Even though I'm a big fan of Hetzner and a happy customer for years, it's important to know what we're buying and be mindful of what we're not having as a tradeoff.

No one has given you a good explanation for why to move to the cloud?

Let me fix that for you! :)

I work for a Fortune 200 company who has their own data centers and has for decades. We just completed the construction of two new data centers about four years ago at a cost of $110 million dollars. Those new data centers are now at 70% of their capacity for power requirements. You should take a look at the specs on Intel's latest server-class processors. It's insane! We're talking half horse power and higher for each processor - and you want blades full of these things and then a rack of those blades and a row of those racks! Data centers now consume more power than industrial manufacturing! Most maddening of all - most of that power is going to go to heat, and guess what? You need giant chiller units to cool the place down. The power costs alone are insane.

You also lose agility. Need a new server? Go to your favorite cloud console and provision it. Better yet, use a serverless architecture and don't even worry about servers! Your own data center? Right now there's a 4-6 month wait time for new servers and storage equipment, and an 8-12 month wait time for networking equipment. Not exactly agile, is it?

You already know you need to staff to manage and patch your servers but you're also going to need staff to procure and manage your software licenses - and those licenses aren't cheap! Don't forget the ongoing tail - you pay for the license and then you get to pay 20% per annum forever after for that license. Those licenses also restrict your agility - you're pressured to use the software that's licensed in order to get better value. Try working with a procurement-based architecture!

You think you're going to avoid the software licensing hell by using open source? Good luck with that! That means you are responsible for the packaging, distribution, and support of whatever it is you're utilizing. That's more staff you need. They will inevitably miss patching some critical vulnerability that's going to land you in the news!

I can go on, and on, and on. But here's one thing I can say about applications that we've moved to the cloud: they cost 30% as much to host in the cloud as they do on-premise, and that's not even accounting for the entirety of all the costs I enumerated above! We have ultimate flexibility and agility, and we can quickly utilize managed open source platforms to solve business problems. On-prem? You lose all of that.

Faster, cheaper, better - that's why people are moving to the cloud.

Open source clearly does not mean packaging your own software. This casts a shadow of doubt on the rest of your valid points.
It does if you're not paying for it. There are plenty of companies that will gladly handle the packaging, assure all the interdependencies work well together, and even support the effort. They also do release management, ensuring that everything is on the most recent version that all works together. That has a cost. You can either pay someone to handle that work for you (smart - you and many other companies are sharing that cost) or you can do that work yourself (not so smart - now you shoulder all the costs).

Just as an example, consider Hadoop and its ecosystem. Another example is Elasticsearch. There's work involved in making these platforms production-ready and keeping them and their dependencies up-to-date and ensuring all your applications in your portfolio are using the same set of software. Most organizations are not prepared to take that on. So they turn to a vendor who will do it for them for much cheaper than they could do it themselves.

That's why the saying is "free as in freedom, not free as in beer" because open source software is not free as in beer. At least not when factoring in the total costs.

> Open source clearly does not mean packaging your own software.

The packaging argument was made in the context of complying with software licenses. Responsible companies which perform due diligence on the software they run have to track the provenance of all software that ships as part of their dependency closure. If you want to ensure you're not vulnerable to lawsuits then you can't simply apt-get stuff from a PPA. You need to build it yourself, and track exactly what goes into that build.

Meanwhile, if you opt to run managed service from a cloud provider, you don't have to bother with that because that's not your problem (or liability) anymore.

It is your problem though, your customers are affected.

Do people successfully lawsuit their cloud providers for downtime?

But that should apply to all readily available container images on docker hub as well, right? Conceptually that is not different from some guys PPA.
> But that should apply to all readily available container images on docker hub as well, right?

Yes, and it does indeed apply to all readily available container images.

In fact, it applies to any and all software packages put together by third-parties.

I mean, who in their right mind downloads random stuff from the internet and expects to just drop it in production software which you build your business upon?

Go to your favorite cloud console and provision it.Better yet, use a serverless architecture and don't even worry about servers! Your own data center? Right now there's a 4-6 month wait time for new servers and storage equipment, and an 8-12 month wait time for networking equipment. Not exactly agile, is it?

Sure, that all works, as long as you have infinite money. My last job was at a company that was heavily invested into a serverless architecture, using the full suite of AWS tooling. They did an audit of one of their codebases and they found that generating a particular piece of data cost $20, almost entirely in fees to AWS. The company's roadmap involved scaling up the generation of this data, so they embarked on a huge process of optimizing and refactoring, to try to get the cost down. This effort tied up probably 40-45 engineers for at least six months. Probably cost them roughly 3 million dollars in salary alone, let alone opportunity cost in terms of other projects not delivered. No one amongst the management ever seriously considered moving to an on-prem solution, even though it was blatantly clear that the only value the company's massive usage of AWS Lambda, SQS, RDS and other managed services was delivering was additional profit margin for Amazon's balance sheet, and resume line items for the engineers who now got to check off the "used Terraform" box when applying to their next job.

You must have missed the part where I said But here's one thing I can say about applications that we've moved to the cloud: they cost 30% as much to host in the cloud as they do on-premise, and that's not even accounting for the entirety of all the costs I enumerated above!

Every app I've migrated to the cloud so far has resulted in a 70% cost reduction, but I'm also not trying to take my entire application portfolio to the cloud. To wit, we're not planning on abandoning our data centers anytime soon and that's not just because we can't move our applications to the cloud fast enough. At this point in time we recognize there are some applications best kept on-premise - but they're a minority, maybe 20% tops of my application portfolio. That means 80%+ of my application portfolio can be moved to the cloud, makes sense to move to the cloud, and I can save 70% on costs by doing so. That's a fantastic deal!

Have you done an audit in the other direction? How many "cloud native" apps do you have, which would realize similar cost reductions from moving to on-prem? The point I was trying to make is that many companies, especially newer startups, don't even discuss on-prem as an option. They build everything cloud-native from day 1, which then leads to them painting themselves into a corner like I described above. They're having to totally redesign and rearchitect their application, not because their business requirements dictate it, but to conform with AWS' limitations.
It's May 6, 2022. You're launching your startup next month. Are you going to host on-prem? What, with server equipment backlogs now 4-6 months, and that's if you're a large enterprise customer? Network equipment backlogs are now in the 8-12 month range. Not to mention the cost of needing conditioned power, cooling, staff, space, etc. You're going to utilize the cloud. It'd be insane to do otherwise.

You do raise a fair point - what about the applications that are more economical to host on-prem? In my own portfolio I estimate 20% or so of my applications make more sense to host on-prem. What to do about those?

The most practical option is to mandate every application you build in the cloud have a plan for how to run on-prem. Note those that would be very difficult to run on-prem - those are your key cloud dependencies. For example, in my own portfolio I have a couple of applications depending on Lex and therefore would be very difficult to host on-prem.

Pay special attention to your so-called "crown jewels" - your apps that distinguish you from your competitors. If those were dependent on a key technology such as Lex and would be difficult to migrate on-prem then you need to identify other cloud providers you could migrate to. In this particular example I know Microsoft's and Google's clouds both offer similar services.

If you design and build your applications with no thought of the future then you're already in a bad position.

It's May 6, 2022. You're launching your startup next month. Are you going to host on-prem? What, with server equipment backlogs now 4-6 months, and that's if you're a large enterprise customer?

On-prem or cloud, why would I be waiting until 1 month before launch to provision anything?

You're pretending like on-prem is full of all these physical limitations, but Amazon, Microsoft, Google, Oracle, et. al. can just magically make hardware appear out of nowhere. It's the same hardware, subject to the same supply limitations, and I'm going to be paying either way. Either I pay more per hour for my cloud instances, or I pay a bunch up front to provision my on-prem servers.

Cloud makes sense if the load on your service is intermittent or unpredictable enough that you're okay paying AWS's premiums in order to have the ability to scale on demand. But if the load on the services is knowable, then hosting that service on the cloud is paying a 25% markup for no reason whatsoever. AWS is by far the most profitable portion of Amazon's business units. Those profits are coming from somewhere. They're coming from your business.

>>I see everybody around me moving to cloud, without really good explanation why.

In that case you certainly aren't looking very hard for that explanation.

The rationale of moving to "the cloud" is crisp and very well-defined, to the point where even AWS makes it their point to explain in very clear terms in their Architecting With AWS courses.

Basically it's all about being able to scale without bothering with procurement, not having to manage everything down to power bills and rentals, turning big capex into small opex, and being able to deploy globally and reliably with a click of a button.

That, however, comes at a steep cost. The more serverless you get, the higher the price tag.

Also, beyond a certain scale you're better off going back to managing your own infrastructure.

> Only reasonable thing I can see as an pattern is that cloud experience on top of data things gets paid 30% more. It made me consider cloud a lot.

People with cloud experience are paid more because they can deliver more value. A random guy with, say, experience in EC2 and CloudFormation and CloudWatch, can single-handedly put together a robust fault-tolerant webservice that's deployed globally and automatically scales to meet any demand fluctuation.

There are 10, probably more, cloud consultants per developer. Hard to argue against that.

There is nothing to say against a cloud version of your office suit but otherwise it is quite underwhelming. Short lived applications with a 90% API because the developer want to keep the door open to imprison the users once the application has captured a critical mass of users.

I host on AWS because I don't pay the fairly high invoices. Less domain and TLS maintenance for me. Personally I rent a server and host my personal stuff there including software repositories.

edit: Guess that technically would also qualify as cloud, but I think a differentiation is necessary. Cloud mostly means product x or y instead of just another hosted server.

(comment deleted)
Hired onto a project as a specialist. Funders had hired a project manager also. Project manager wanted to use technology X ( which has long since faded ) instead of well entrenched and dev-hireable technology Y because technology Y had not yet entered the trough of disillusionment and anyone who could honestly say they had managed a technology X project could 10x their hourly. Money was ignorant about this level of detail. PM won, as they do. Whole thing a money pyre.
the auld mortgage driven development strategy
> It’s and industry-wide lack of emphasis on identifying and understanding the problem first.

Or just no real incentives of the people involved to do so. As a dev, I don't get any real credit for biz outcomes.

I’d argue the quality of tools is the problem. I am starting to view every new tool as just a new set of log messages I have to sort through to figure out why things aren’t working.

I spend more time trawling GitHub issues to find workarounds than I do actually using the tools.

> “promotion-based architecture”

That's a great way to describe the phenomenon.

I don’t think you’re wrong in your observation (though perhaps a bit hyperbolic in the doom) but I’m perplexed why you think there’s “no way out”.

Software is malleable, people are generally smart. It may take longer than you hope it does but things will shake out just fine as teams/companies are forced to look critically at their infra spend vs utilization and adjust accordingly.

In an inflationary environment, this is fine. Just wait when we have interest rates at 12%.
This comment is irrelevant to the point of being a non-sequitur