Ask HN: Have we screwed ourselves as software engineers?
I cannot help but wonder, where is our software industry heading? There are overly complicated solutions to simple problems, huge push for moving to fancy stacks just for the sake of moving. Distributed systems? Kubernetes? Rust for CRUD apps? Blockchain, NoSql, crypto, micro-frontends and the list goes on and on. Its gone too extreme to the point where no one is exempt from these things anymore. Couple of years ago, I thought, its fine as long as I am not involved in this complexity, I can turn a blind eye towards it. But now, this unnecessary complexity has seeped in my day job as well. Managers start talking about "micro services", "writing" kubernetes operators in Go, moving away from python (because its too "slow"); someone recently gave a talk in my company, how to make a 500 line python script (which heavily involves in-efficient handling of IO) go faster with Rust.
Someone else talks about that we need to move our poly repos into mono-repo because that where the leaders of the industry are moving to. Even recruiters started asking questions like "have u looked at modern languages like Go?"
I cannot help but wonder, that we have possibly screwed ourselves pretty bad, and there is no escape from it. The vocal minority tries to push these overly complex solutions down everyone's throats, and management loves this, because it creates "work" for the sake of it, but it doesn't add any real business value.
What are your thoughts on this? Will industry move towards simple solutions after experiencing this churn down the line or are we doomed forever?
429 comments
[ 3.6 ms ] story [ 231 ms ] threadPerhaps after a downturn, things will revert to the mean.
Under what contexts does “reversion to the mean” apply?
Over what time frame?
The “mean” implies one dimension. What quantity are you referring to?
> Reversion to the mean, also called regression to the mean, is the statistical phenomenon stating that the greater the deviation of a random variate from its mean, the greater the probability that the next measured variate will deviate less far. In other words, an extreme event is likely to be followed by a less extreme event.
It is a stretch (and invalid, generally) to extrapolate this well-defined phenomenon to a complex system. There are many complex systems that maintain or increase their complexity.
Increased complexity is only achieved with additional energy input. im suggesting that if energy (read: money) going into the system decreases rather than increases, there will be a reduction in complexity.
> Is There a Swing of the Pendulum?
> By Pierre Lemieux
> People are often tempted to see social (including economic and political) phenomena in terms of a “swing of the pendulum.” In this perspective, problems such as wokism (just to give an example) will be corrected when the pendulum swings back. I suggest that this approach is easily misleading and seldom useful.
How about global IT spending (measured in trillions). Or perhaps spending in "new technologies." If there is a downturn, there would probably be a reduction in IT spending, and so many of these overly complex application solutions could get cut, delayed, complexity downgraded, etc.
What you say about needless complexity is a very valid point, but it's just growing pains imo.
Honestly everyone would be better off doing everything in code (python, bash, go, rust, c it doesn't matter) directly. They're easy to debug, flexible, and everyone already knows how to work with them.
On the other hand perhaps the industry is being ever-increasingly led by a younger workforce who already came into this ecosystem, and there's less chance of a full retro/introspective about why things are they way they are.
But has being a software engineer become easier or harder over the last 30, 20, 10, 5 years? I wasn't an engineer for that long but my impression is that programming today is a lot easier. Dev tools, compilers and linters are very good. There's also a lot more community documentation on stack overflow. Some of the complexity is hidden from the developer, which is good and bad. It can bite you in the ass later, but in 95% of cases its a good trade off in my experience. For instance, my preferred stack is Serverless on AWS. I can set up a single config and have cloud storage, an api, a database, logging, auth all permissioned and in a file I can check in. And with a generous free tier, it's pretty much free. I'll admit if something goes wrong its not fun to debug, but it's remarkably fast and simple for me to spin up a CRUD api.
Libraries outside those provided by the OS/compiler tended to be hard to come by. Certainly the universe of freely available library code that we have now was nonexistent (I’d argue that CPAN was a big factor in the spread of Perl in the 90s—well, that an the assumption that was widespread back then that CGI scripts had to be written in Perl).
As a community, we’ve collectively learned a lot of important lessons as well. Legacy systems like Perl and LaTeX tend to install common code in a single universal directory for everyone rather than having this be application/document specific as became the case with Java and the repositories will only give you the latest version of an artifact¹ which has tended to lead to stagnation since backwards compatibility becomes non-negotiable, although some lessons haven’t stuck (like the fact that Rust’s crates.io only has a single-level namespace for artifacts).
⸻
1. Not sure if this is absolutely the case with CPAN. CTAN, which was quite possibly the first centralized repo² does not manage multiple versions of artifacts.
2. I remember when it was first being set up, since there was no guarantee that any tooling beyond TeX would be available to consumers, there being TeX scripts to do things like encode/decode binary files into emailable ASCII format. The original CTAN FTP server in the UK also had the remarkable feature that it would generate directory.zip for any directory that you requested.
Simple MVC stacks went to all languages, jQuery front-ends doing enough but not a lot. JavaScript enhanced easy to reason about server-side stacks.
You used to spend a couple of days a year, yes YEAR, mucking around with tooling. Now it wouldn't be too much of an exaggeration to say you spend a day or so a week fighting some part of your stack because it's been over-engineered.
IDEs can't keep up so you have to run inscrutable command line tools that fail if you have deviated even slightly from whatever error-prone rain-dance some moron claiming 'best practice' has forced into the build process.
Programming used to be about writing code to solve business problems. The shift to DevOps has been a massive productivity drain and most stacks are now incredibly brittle.
The worse part has been debugging, which you touch upon. Native calls, simple call stacks, easy error logging. All gone.
Moving everything into hard to debug http calls has been a disastrous productivity sink.
The irony has been that as languages have got significantly better our productivity has actually dropped massively because of the ridiculous amounts of over-engineering in "modern" code bases.
I recently worked on a project with 2 devs that took 3 months with a modern stack. The prototype in a standard MVC stack I'd made to demo to the client took 2 days.
It's utterly ridiculous and sometimes I feel like the boy in the story about the emperor with no clothes.
Can you expand on the modern stack and that standard stack? Is the "standard stack" you propose easier to work with now than it was 15 years ago? I guess the "standard stack" is just out of fashion?
My preferred stack (aws serverless BE, nextjs front end or just aws app sync) keeps me in one language (typescript), with some stuff like GraphQL that you have to know. But the tooling around that helps keep my errors in compile time for the most part.
And I take issue with saying you "have to know" GraphQL, that's a pretty specialized tool for pretty specific problems. Most things are not graph structures.
Simple, slightly opinionated. You just need to be careful of dependencies and what kool kids are doing these days. GraphQL is a complete nightmare.
If you don't transpile anything and are conservative with your dependency you're golden.
For some inscrutable reason you can no longer do a simple 3 page form with a results page without making a SPA these days without getting someone claiming you're doing it wrong, it's nuts.
Worse still, the angular app has all sorts of weird bugs. Auto-complete somehow screws with the validation of inputs, back button doesn't work properly as you lose all your data, URLs dont work, etc., etc., etc. There's also all sorts of craziness like the whole admin frontend is bundled with the client-facing part as the front-end developer didn't know how to split them up.
Utterly preposterous.
The problem is often that very skilled Devs give advice about incredibly specific stacks that only someone of their abnormally high-skill level can maintain.
Throw in a few juniors or a few mediocre developers and the whole project turns into a complete and utter mess.
And don't get me started on your stack that's mainly inappropriate for most applications.
GraphQL. Talk about the next NoSQL fiasco in the making. Perhaps you missed the whole saga of everyone doing new development in NoSQL and then a few years later we had the flurry of blog posts about "Why are we losing data? I didn't realize ACID was so important..."
Even though it's not 90% of the time.
You won't get any objection from me on the debugging point, it's much harder, especially when you're crossing environment - e.g. running the front end locally but maybe hitting a remote dev or QA backend. I will point out though that there are logging tools that support the pretty standard practice of having correlation/transaction/trace IDs on your requests, such that you put in a GUID from an error and it shows you the entire request and anything that request spawned.
I consult primarily in node and react with all sorts of transpilers mess and shitty packers. The stuff of nightmares - we waste so much time making things work - but simplifying the stack will never get traction among the not-anymore-technical principal engineers or even amongst the other developers who need a fancy cv for their next gig.
My side businesses use python, django, jQuery, old node.js without modules, rust, svelte.
Engineers hired in big companies want to work on shiny technologies and build their cv.
no, the reason Pieter Levels makes millions is not PHP and jQuery, but because he's also a brilliant sales/marketing/business person
Ergo, they are at least fit for that purpose. They aren't bad enough to prevent him from making millions.
The problem is in larger companies, developers stopped caring about just solving the business problems and moved on to solving non existent technical issues to build resumes to go the next job to get more salary. They go to company to company like a parasitic infection leaving them to rot by introducing Kubernetes React, Go microservices with Rust cli tools.
Also it might be an issue of not being fulfilled in life outside of work.
The real biggie when switching to microservices is that you suddenly have to reinvent all the goodies a relational database gives you for free.
Also, people are getting used to app-like experiences and designers are designing for it. Building an app-like experience is more natural as a Single Page Application, which basically means taking on the modern frontend stack. There are places that push against this, but to do so requires buy-in to the engineering side over product and design. Even then, the engineering side has to be knowledgeable enough to not follow popular opinion and come to the determination that Laravel/Rails/Django is actually the right tool, which isn't always the case.
You can't generalise from your experience over the last few months.
> Programming used to be about writing code to solve business problems. The shift to DevOps has been a massive productivity drain and most stacks are now incredibly brittle.
Some businesses _have_ to "shift to DevOps" in order to operate at the scale and resilience required.
Some businesses have unnecessarily migrated to over engineered infrastructure because monkey see monkey do.
Saying "everything is ruined" completely misses the dynamic.
As an earlier poster explained there are a lot more tools in the box now. Making the right choice requires experience, a good understanding of the problem to be solved and discipline in implementation.
Get it wrong one way and you end up with an over-engineered mess that takes forever to get work done with.
Get it wrong another way and you end up overwhelmed by traffic, unable to scale in response and forever fighting fires.
> Get it wrong another way and you end up overwhelmed by traffic, unable to scale in response and forever fighting fires.
To nitpick this specifically, over my 12-year career toiling over this stuff there has never been a scenario where this has required a radical rework to solve. Boring-ass B2B shit rarely requires that level of engineering and, at least in my case, the workloads were fairly predictable and increased in a linear fashion. The one time I did accidentally end of DDoSing ourselves, I temporarily stood up nginx instead of our aging Apache install and was able to serve enough requests to fix the problem. (We then transitioned our app to run in nginx)
It was one fire, and it took a little bit of brainpower to fix. Then the DO droplets were humming along perfectly, and last I heard they continue to do so to this day.
The operational aspects of this double-digit-millions-per-year business ran on a postgres database that compressed to 6gb.
The next business I worked for did billions per year in business value and, until HQ mandated migrating everything to GCP, was humming along perfectly fine on Heroku for a monthly spend well under five digits. Ironically, I think they initially wanted us to be on-prem but couldn't support our stack and would have left all of that to our devops guys. GCP was the compromise (oof!).
That's a highly obnoxious response to a measured and reasonable comment.
Most businesses are never going to be a Netflix or a TikTok or whatever. Yet their businesspeople are absolutely infected with this mindset. And don't get me wrong, it is a disease and its primary symptoms are exactly what OP is complaining about.
So instead of building practical, easy-to-maintain systems we cater to the dreams of excessively optimistic (can you even be anything else as management?) nontechnicals who think every idea is going to require the kind of scale and tooling that someone like Facebook has at their disposal. And they're backed up magpie engineers more fascinated with interesting than functional infrastructure.
Ironically, that very tooling and scale ends up demanding even more resources which demands even more tooling to manage. What a virtuously (hah!) profitable phenomenon for the vendors of these tools.
Good grief.
I didn't say anything like that. the opposite in fact.
Next time try reading and understanding a comment before going off like an obnoxious jerk
jerk
So yes, it is affecting our entire industry. Every aspect of it.
There are very small number of organisations that actually have any sort of need of a microservice architecture.
Worse still, actually talk to these orgs, they'll say they actually have a "hybrid" microservice architecture. Which is basically the worst of both worlds, all the pains of managing microservices, without any of the benefits you get in a normally built application (derisively called 'monolith' with all the negative connotations that word has). Half your calls disappear into the black hole of HTTP calls. No pressing F12 on a method call and going straight to the code. No easy stepping through code in the debugger. No simple download the code and just press play and it all works.
I like solving business domain problems. Not tooling problems. Tooling problems are incredibly boring and frustrating to me. To a certain type of programmer, rather than actually doing their actual job, they absolutely love introducing tooling problems as busy work. Because the actual business domain problems don't interest them. Then switch role as they've got the new hotness on their CV before they have to maintain the craziness they've introduced to the code stack.
Case in point on a project I helped get over the line recently. I joined 1.5 years already done on the project, development has slowed to a crawl. Lead architect designed a system of DDD, event-sourcing, message queues, microservices. Just to add a new field I had to edit 10 files. To add a new form I had a PR which edited/added 40 different files. How it actually worked completely flummoxed juniors + mid-level devs, it was beyond them.
All for a 10 page form that would have at most 150,000 uniques in one month per year. Roughly 1 request per second, assuming a ten hour day and 1 request per form page. Child's play.
A standard stack would have easily handled that load, probably even on a VM. A dedicated server would never have gone over 10% CPU. It would have been massively easier to develop, and cost 1/10th in dev time.
At one point I had a quick go at re-writing a section without the trendiness. Just to see, as I'd never have got it through the politics involved in that PR. I switched the event-sourcing, microservices, 5-tier craziness for a simple, easy to understand, service. Took me 1/2 a day, tests passed, reduced DB calls. Over a thousand lines removed, 100 added. Absolute nuts.
Millions wasted on trendy architecture. Of course the architect left a year into the project for greener pastures.
> Moving everything into hard to debug http calls has been a disastrous productivity sink.
There are a lot of good things coming out of the latest big experiments but this has been a major blow. I have worked on software where the intended debugging approach was to write some code, manually push it out to a shared dev environment, read CloudWatch logs for debugging. It is by far the worst way to debug code that I have ever seen. Things that would take me minutes to debug in a normal setup can take hours or days. Projects like LocalStack aim to improve this a little bit but it's completely counter to the ethos of many "cloud-first" developers.
I'll admit for the vast majority of web apps out there, this is not needed, but there is definitely a scalability concern if your entire stack is a single ruby on rails mvc application.
I've worked on multiple products with 1 million+ users running on stock Rails MVC backends. Everything scales just fine. Until you hit a DB hardware limit you can just keep adding more web instances behind your load balancer. And DB Hardware limits these days are astronomical.
If you're actually hitting the limit of scaling our web backends horizontally and you don't have the money to deal with the problem, you might need to take a hard look at your product.
And I'm confident this is true for whatever your backend is: Django, Rails, Laravel, whatever.
Currently I'm involved in working on a "data intensive" app which munches a few dozen gigabytes of data and presents summarizations through a web based UI. Usually in the form of charts, time series plots, heatmaps etc. Adding small, incremental features to it is an ordeal. The "backend team" needs to mock and then provide the API endpoints, then the frontend team needs to "break down the work on the React components", then DBAs need to oversee the new queries. The security team needs to review risk impacts etc. One change is at least a few weeks of work.
Contrast that with the work that I did in 1998. It was a "data heavy" desktop application that visualized in 3d seismic data collected by ultrasound probes. The data volumes were in the order of hundreds of megabytes up to a gigabyte. The app was built as a desktop app using FLTK and OpenGL. It took about a day or two to roll out an incremental feature once we knew what was being asked. By two weeks it was in all of our customers' hands. While there was no Stackoverflow.com and we relied primarily on offical documentation with the help of "Effective C++" and "More Effective C++" the build/debug/test cycle was much shorter, the tool chain was much simpler and more transparent, the UI toolkit was much easier to grok and even though it didn't have an "elegant state framework" somehow we made it work well without turning it into a mess.
As an industry, we gave up on localized desktop processing in the name of "consumer preference" (or so we were told) while we assumed an immense amount of complexity in order to server the same computation from a centralized place.
The easier things are mostly obvious. Better languages, open source libs etc. Things that are harder now than before:
1. UI. The web is not a good UI platform, sorry. Designing UI in the 90s was easier, except for the need to do manual memory management if you weren't using Visual Basic. Partly because there was little expectation of branded UI, so you could easily re-use large control libraries that came with the OS which were/are pretty feature complete and well documented.
2. Cross language interop. Microsoft had this nailed. COM was a beast, but it worked and there was an actual real market of cross-language, auto-bound objects and GUI controls (COM objects, OCX controls). There was actually a thriving ecosystem of languages on Windows which are now mostly forgotten (Delphi, FoxPro, VB6, Paradox, Visual Prolog etc). Nowadays cross-language interop is a joke. Transpile to JavaScript or go via a C FFI, maybe, if you're lucky.
3. Expectation of supporting multiple platforms. This causes a lot of dysfunction because the lowest common denominator is really low, especially if you treat the browser as a "platform". In the 90s Microsoft had a monopoly. That had its own problems, but, it meant you could write a Windows app, once, and everyone would accept it. Nowadays you want web+mobile, and "web" is not really a platform in the sense Windows was. If you're doing anything serious you still need a desktop app.
4. Over-specialization/staffing. Back then everyone was "full stack". The developer was also a DBA, at least to some extent, and you could just throw up a quick GUI Windows app that connected directly to the database. The DBA would then manage security and backup. It wasn't really a full time job, at least not on a per app basis. Nowadays even simple projects feel absurdly over-staffed. Do you really need a backend guy, a frontend guy and a devops guy for an ordinary LOB app? Probably not.
5. Process overkill. Waterfall is underrated. It got a bad rap because it requires you to understand your customer, and for your customer to understand what they actually want, and for them to not change their mind every five minutes. Not always possible. Nonetheless, agile has become some kind of monster. Half the words in the average agile methodology are made up and a lot of it is really questionable. If your business domain weren't totally unstable and your users weren't totally incompetent (often the case!), then you could sit down and write an actual spec, which people would read and sign off on, and then you could build it. And it'd work, and after the initial debugging / shakeout period, people would be happy. Many of those apps never broke! Just imagine the output of the typical agile web AWS-based web stack teams today lasting 20 or 30 years without a dozen rewrites along the way. Very hard to imagine that.
What sticks out to me when I'm looking at some older code editors/IDEs, is how crazily spartan they are - to the point where they are just inconvenient to use.
I think you're just in a very negative space if you start with "Distributed systems" as something overly complicated. At some scale getting a bigger machine either doesn't make financial sense or is just not possible to implement efficiently. Some ideas are taken too far or implemented where not needed. But I'd rather recommend you to learn where each one of them started and why. Criticize for valid reasons, but don't become a curmudgeon.
They do solve real problems. The question is whether or not they solve the problem at hand, and if they create other issues in doing so.
I was re-decking a back yard bridge with a friend and he brought a framing hammer. I'd never used one before and I always had a finishing hammer and didn't know the difference. I learned a new tool and even ran out and bought my own for the project, which worked fantastically well. It's still in my toolbox and hasn't been used since. You just don't use that thing to hang pictures on the wall because it may well f-- up the wall a bit. Using the right tool for a job is way more important than using a particular tool for any other reason.
Best according to who?
Based on what priorities?
Relative to what other options?
Based on what kind of experience and expertise with the various options?
My takeaway: BTftJ must only be a starting point for dialog and discussion. Shortly therafter, it is time to get real about each of our experience and biases. Otherwise, BTftJ is a only a thin veneer.
I don't either. As photographers say, the best camera is the one you have with you. Same thing, the best tool for the software job is the one you already have and everyone knows how to use. Why anyone would want to introduce these high-end fads to an organization without proper reasoning is beyond me.
All of them, except for blockchain. That one can go die on the trash heap of history.
As long as there are Rust jobs or blockchain related jobs there, you're going to have to cope and wait for a very very long time until that is ever going to 'die on the trash heap of history' to happen.
So why hasn't it died quicker than it should have years ago as many have incorrectly predicted then?
> Blockchain companies are a dime a dozen. They don't all last forever.
And who said that the 'companies' did last forever? Why do you think I said as long as?
I'm just wondering if the whole thing is guaranteed to totally go away 100% and to be absolutely used by no-one since clearly someone also thinks so. That is my question.
I am not a crypto stan but it has at least one usage.
e: Sorry, forgot it was verboten to say anything contrary to the "crypto has no uses whatsoever" line.
As for bitcoin itself, which I assume the latter part of your comment is talking about:
It’s predictable trustless money. It’s not perfect. But where you totally Lose me is:
> it is wasting the world's energy supplies
This is the latest POLITICAL attack vector. When you parrot messages like this, you expose yourself as a political victim that is not educated about the topic. Literal parrot.
If you have an interest beyond being a political foot soldier, do your own OBJECTIVE research. Learn about bitcoin miners using renewables, how much energy the industry actually uses, the carbon footprint of payment networks and fiat etc.
It’s an effective political attack vector because in all likeliness you won’t do your homework. You’ll read my comment, experience cognitive dissonance, judge me as a bitcoin cultist, and move on with your busy life.
You don't even need to get scaling into the picture.
You do not get any form of fault tolerance by deploying stuff in a single lonely box. If you care about reliability and resilience, you have to have multiple deployments up and running at the same time.
Also, you already have a distributed system if you have a browser calling your server.
And lastly, if you happen to manage an service used globally or regionally and perceived performance matters then you have no good alternative to have regional deployments.
Are all CRUD API’s insensitive to performance? Correctness?
This is the reality of software development; you have a budget and you have a goal. If all your budget goes up on making it correct without finishing it, you have a problem.
Anyway, it's just a CRUD app, it doesn't have to be as formal.
Setting aside performance (Python is slow as molasses) that still requires some qualification because there is the 'maintenance' dimension. I would take Rust over Python for CRUD any day, because the 'formality' is a feature not a bug for maintenance. I would take Java/C# over Rust because the former balance performance and formality very well. In fact, I wouldn't use a dynamically typed language like Python or Ruby for backend infrastructure code if there were any performance or long-term maintenance requirements.
Rust is a very high level language, it's not like writing assembly. You rarely run into complicated situations with the borrow checker when writing trivial crud apps.
The ecosystem is not as mature for web apps compared to other languages, so it may be more efficient to pick something else, but ultimately the most important factor (when building trivial stuff) is picking a language you and your team are comfortable with. Whether that's Rust or node.js, it won't matter much.
If I were to make something else which is better supported in Rust, eg. a videogame, using bevy + ecs would run circles around a bunch of other languages and their game frameworks.
The parent commenter may have some narrow or situational definition of productivity.
Yeah there is a lot of overbuilding and BS in our industry, but I don't think we're unique in that regard. It is safe to block out the noise and focus on what excites you.
You should know that in the past when OOP was not common, we had to work a little harder doing things like managing our own memory or building a LAMP server to publish our web pages.
There was a thriving market for language and UI add ons. The result was that each company had their own internal dev tools and recruiting people outside of the company who had experience with those tools was nearly impossible.
All that said, we were at a point where entry to programming was easy (think Visual Basic in the 90s). The quality generally went down as everyone was pushing their "first project" as if it was a polished product. Finding actual good programs on PC is close to the situation on mobile where most of the apps are trash.
While still prevalent today, there's a huge sentiment against this paradigm. Modern languages and frameworks such as React, Golang and Rust show how the tide is turning against this.
Or are you saying that Rust/Golang/React are simplistic in a world of over complexity. React I would generally agree with, the other two not really.
Functional components in react. Zero classes in golang or Rust.
It's actually quite an obvious paradigm shift. No classes with these languages yet every language prior to this has classes.
Yet I don't understand why people still insist on calling go and rust object oriented or why they try to argue this point when there is such an obvious change. I mean sure you can twist the language into something that looks like it's oop but come on man. Haskell is OOP if that's the case. Let's not argue about whether these languages are OOP or not. The point is there is an obvious movement AWAY from OOP with substraction of popular and critical syntax and features.
For React , the trends are harder to see. See here: https://hackernoon.com/react-functional-components-are-the-f...
The react team recommends functional components over classes, and react itself was derived from functional languages and concepts. I believe the team ideally would like to see ReasonML as the future and react with js syntax is more of a necessary in evolutionary inbetweener thing.
The backlash against OOP is partly because existing OOP engines are limiting in many languages. Passing, grouping, and custom-scoping of "blocks of code" (BOC) should be more flexible so that we are not forced into hierarchies or spaghetti scoping. We need new languages that make managing and scoping BOC more flexible. I want to define the scope, not let Nadella or Bezos do it. The distinction between a lambda and method would then be fuzzier. BOC's would no longer be forced to be one or the other. (One these days I may make a proof-of-concept language.)
None of the languages I mentioned are FP. Though FP has benefits and many of the languages or frameworks mentioned above are either moving towards FP or borrowing concepts from FP.
You're not wrong about intermediate state. I shouldn't have emphasized FP here that wasn't my point. An imperative program that is just imperative without OOP or FP tends to be simpler then OOP itself.
But most importantly — if it really is, as you say, you can profit of that. Open a consultancy, and solve client's problems without using these over-engineered solutions. If your competition truly wastes a lot of time, then you would be able to solve the same problems faster and in a more effective manner.
I am:
https://htmx.org
Even if you don't believe it's a conspiracy, you have to admit that: that dynamic favors incumbents, the big frameworks often come from the big incumbents, and if you are a big incumbent even if you weren't manufacturing this conspiracy but you saw its possibility I mean why not take advantage of it?
The outlier is the scrappy indie developer like Peter levels who runs his multi-million dollar a year business on his own basically on a single machine using PHP and only recently started using git. That may be an extreme example but it paints a picture of what radical effectiveness and efficiency looks like and it's vastly different to the Kool-Aid but don't mention it otherwise the mob will come for you.
May the no-code & indie scene save us all. Amen
As for you not commenting, it seems that would be a good idea on your part. You keep trying, and keep failing. Do you often do things you say you won't? You lack self discipline? Anyway, thanks for obsessively checking my edits.
With enough years in this industry I'm starting to see how much of the culture is just "cargo cult Silicone[sic] Valley!" -- People are doing things just because the latest/greatest unicorn did it that way, and frequently there is a form of Zen/Chan disregard of meaning or learnings from the past. "Move fast and break things" often plays out as "Disregard knowledge and act like there wont be consequences" . I frequently see threads/posts on HN by experienced people laughing that "we've known this since the 70s" in reference to The Mythical Man Month, or other bits from Computer Science, and yet most managers think they know better than decades of industry experience.
I'm releasing a (web) development platform for it this month. Just getting the code ready.
I think some of the complexity stems from trying to make digital things that simply aren't or shouldn't be.
1) some of these things (e.g. node, microservices) already peaked a few years back, being overapplied and now the pendulum is swinging the other way
2) others (e.g. Kubernetes, React, monorepo) were developed at large, profitable companies that others wish to emulate (or work at someday), so they find excuses to use them. This case takes longer to reach a point where things swing against it, because everyone wants to pretend their company is the size of FAANG or will be soon, but the same process of overapplication and backlash happens eventually
3) in the midst of all that noise, there are some new things which are in fact a good idea for most developers. I don't know Rust or Go, but perhaps they are examples of that.
The key for us as developers is, unless we wish to work at FAANG, try to spot (3) in the forest of (1) and (2), and don't let (justified) annoyance at (1) and (2) blind us to the fact that (3) is out there as well.
Just one example, Scala. First, I'm not criticizing the language itself. It has it's place, but what I saw was programmers trying to create a protected space that would provide higher bill rates. Java was everywhere and hiring a Java developer was easy. Scala was new and had steep enough learning curve that you could drastically shrink the candidate pool while at the same time selling the shiny new toy to management. They could create complex, arcane code that kept new developers from getting up to speed while providing the excuse that they were inferior developers and weren't smart enough to keep up. It didn't work for very long as management caught on that they weren't getting much other than higher labor costs. Go seems to be the latest incarnation of that while Rust is a bridge too far to sell to management.
So it's this back and forth, provide something to management that they can sell to their superiors something new. Management buys into it as long as they can get promoted before it inevitably blows up and the developers who sold it move on to new projects, rinse and repeat.
Meh. From my experience, many developers actively don't want to go into management, because usually your whole day is filled with management crap and you can't go and actually code any more. And developers who do switch to management often end up as miserable bosses because their bosses don't care about "leadership trainings".
Additionally, many companies have the non-management track end at senior level, which means zero career progression for those who do not wish to transition to management.
These are clearly the wrong people to be pushing into management. Good management (it does exist) includes people who have coded, but are willing to give that up to enable others to do that. They get satisfaction from being enablers and making space for their underlings to be creative and make decisions.
Further, there are many excellent managers who don't have a typical developer background, but can recognize what success means for their team within an organization and how to achieve it. I've been managed by many excellent managers with backgrounds in chemical engineering and the classics.
The developers you describe should decline these positions and find a better fit where they can make better use of their time. Choosing to accept positions like this hurts them, as well as others.
Of course, there are plenty of project where judgement Is thrown out the window in favor of adding a buzzword to everyone’s resume. I’ve heard it called “promotion-based architecture”, as in you pick the technology most likely to get you promoted. (If that works, it says all sorts of not great things about your organization).
Regardless, I don’t think the availability of tools is the root problem. It’s and industry-wide lack of emphasis on identifying and understanding the problem first.
I believe these two phenomenons are producing a positive feedback loop in the industry. Selected technologies address one challenge, but introduce complexity. The complexity becomes difficult to manage. So other technologies are incorporated to manage the complexity. In the midst of all this, core technologies are replaced, swapped out, or transitioned to under the desires of the dev(s). For example, switching from one js rendering library to another, while preserving the old legacy code. The complexity footprint keeps growing and it doesn't stop because the engineers themselves aren't entirely committed to the project. They can incorporate the new technology to the project, pad their resume, and bail to a new employer if things grow out of wack.
Instead of applying knowledge, he would bring up all these buzzwords to meetings and not really understand what he was talking about. I get angry just thinking about it.
Nothing in your comment points to resume-driven development. You even failed to mention any project or design decision. You just decided to get angry at a coworker because of something.
I see everybody around me moving to cloud, without really good explanation why. Only reasonable thing I can see as an pattern is that cloud experience on top of data things gets paid 30% more. It made me consider cloud a lot.
I was considering switching to cloud, just so I can put in my CV "experience with migration to cloud".
For next person commenting that it makes sense: it doesn't with 200gb database and super predictable workload, growth and usage.
My company offers two deployment scenarios: host it yourself and cloud hosted (SaaS). Many of our customers choose the latter because their internal IT systems require more process, and they just want to get something up and running.
Depends on the company. I've been working for marketing agencies for the last 15 years and they're generally staffed by, at most, 1 IT person who is in charge of a third-party vendor relationship that offers managed IT services. Those IT resources (internal or vendor) don't specialize in data and often don't know how to deal with it well, predictable workloads or not, and often offer up solutions which are not appropriate (cost or otherwise) whereas there are managed solutions from cloud vendors that do. BigQuery, for example, handles compute and storage for you and rolls it into one reasonable query price. No need to worry about any management of anything there and just sit Data Studio (included) or any other BI tool (Tableau, etc) on top and you're good to go.
I get your skepticism and welcome it, but you're being a little rough. We're cloud native at my current company (I did a partial cloud migration at my last which was completed after I left) and it makes my life leading a Data Engineering and Data Science team MUCH easier without upfront hardware/software costs or long-term contracts, which were STAGGERING and left us with much more difficult to maintain, upgrade, etc hardware that took up most of my job as opposed to almost none of it today.
YMMV.
For anything above what we have I think it weirdly depends on country/ salaries.
In US, 100k per year is no brainer for cloud as even 1 FTE would cost much more.
In non-western Europe, 100k is deal killer as you can hire 2 senior DBAs and you still have enough money for quite a reasonable server.
Edit: sorry I see that you are the same person I was replying to. Then to understand it better, who is consumer of so vast amount of data?
That is something completely different from migrating your inhouse applications to the cloud by introducing kubernetes. I think your example show nicely that there is not one solution for everything. As an industry we have to put way more emphasis on deciding tradeoffs by understanding requirements and risks.
People just buy into the "cloud" marketing. They don't have the ability to think and reason, and so don't understand that "cloud" just means "renting someone else's computer."
I built a complex in-house medical system. Quick, reliable, and liked by the users.
I was in the middle of adding a major new feature when all of the management in the IT department quit. The new people immediately decided that the whole thing had to be "in the cloud." I was removed from the project, and they hired three people full-time to rebuild it.
That was three years ago. The new system is still not online. The users are still using my old system, and the feature I was working on never got added because my presence and input was not welcome because I "don't understand the cloud." So I got moved to other projects.
People talk about "the cloud" with the same fervor and language as members of a cult. And you will be an outcast if you dare challenge their way of thinking.
Moving me out brings no advantages to either part of the org. In fact, it's probably a disadvantage to IT because it has to burn three people from its allocated headcount. Where previously, it had a bonus person that didn't come out of the IT budget.
The difference is the abstraction layer. I would define “the cloud” as a layer that abstracts away the physical infrastructure. (Which works until it doesn’t but that’s a longer comment.)
You can even apply “the cloud” to your very own fleet of computers with the right software.
Like I said in the original comment, it’s a specific solution to a specific problem. Whether it’s right for your system I have no idea.
If it's on-prem, it's not a cloud, it's a fog.
At least, that's what I've been calling some of my deployments.
It sounds you're trying to dismiss and downplay solutions to problems you're oblivious to.
Even if you want to simplisticly frame "the cloud" as "renting someone else's computer", keep in mind that:
* It's not one computer but as many as you'd like, and get them at the click of a button,
* These are computers which are managed 24/7 by a team of highly trained specialists,
* These computers can be located anywhere in the world and simultaneously in multiple regions,
* These computers are designed to be fault-tolerant and handle (and survive) way more stuff that can conceivably be thrown at them.
> People talk about "the cloud" with the same fervor and language as members of a cult.
I'm sorry to say but you sound like you're militantly opposed to a solution to problems you either don't understand or refuse to understand.
"The cloud", even when following the simplistic and clueless belief that it's just "renting someone else's computer", solves whole classes of technical and business problema that your box under your desk cannot solve.
You say that like a pay rise of 30% is not a good enough reason all by itself for many people.
> For next person commenting that it makes sense: it doesn't with 200gb database and super predictable workload, growth and usage.
Perhaps not for technical or business financial reasons, but those are not the only possible reasons someone might do something. As mentioned before it can make a lot of sense to migrate to cloud if it means you can get a 30% pay rise.
I'm not a bean counter, all I know is those guys at my last job would rattle off about it like zombies. IIRC its a tax thing
Is that really the only benefit?
Depending on taxation rules you might want to buy or lease.
By the way, moving 'to the cloud' never means a specific thing because people have made words (intentionally?) vague to the point where you have to explicitly specify all the factors you take into account with your work in order to figure out which 'cloud' they had in mind.
Running a static workload doesn't require elasticity, but a 'cloud' isn't just elasticity. If you want "a program on a server with some storage that is always there" without having to deal with hardware, software, networking, storage, backup, maintenance, upgrades, service contracts etc. then an arbitrary virtual machine where someone else takes care of the rest makes total sense.
As far as I can tell it only makes sense if you have a ton of data, and only ever use a fraction of it at a time, for one-off jobs, infrequently.
With Postgres[1], you would need to be using an absolute beast of a server to be paying that much, even if you're paying the 2x premium to go month-to-month, and a low-powered machine could be around $50/month for a 200GB DB.
[1] https://azure.microsoft.com/en-us/pricing/details/postgresql...
Based on link you provided, I calculated that 40gb memory would be $290 and 80gb $570 per month (in UK).
So price seems still quite steep, considering that same specs on-site would be ~$2000 one off. (8 months return on investment).
Then for this server, I honestly don't believe its easier to do maintenance in cloud, as you need to do lots of initial configuration there (be it IAM, permissions, firewall etc. ) compared to have it just in local network
A single box on Hetzner is by far not comparable to, say, using a managed service as AWS RDS. It's not even apples to oranges, and more of apples to freshly-squeezed orange juice served by a buttler. Think about it: do you get any form of fault tolerance with your single box?
Ah yes I agree, sorry for not being clearer. I wanted to add on to what you said.
Even though I'm a big fan of Hetzner and a happy customer for years, it's important to know what we're buying and be mindful of what we're not having as a tradeoff.
Let me fix that for you! :)
I work for a Fortune 200 company who has their own data centers and has for decades. We just completed the construction of two new data centers about four years ago at a cost of $110 million dollars. Those new data centers are now at 70% of their capacity for power requirements. You should take a look at the specs on Intel's latest server-class processors. It's insane! We're talking half horse power and higher for each processor - and you want blades full of these things and then a rack of those blades and a row of those racks! Data centers now consume more power than industrial manufacturing! Most maddening of all - most of that power is going to go to heat, and guess what? You need giant chiller units to cool the place down. The power costs alone are insane.
You also lose agility. Need a new server? Go to your favorite cloud console and provision it. Better yet, use a serverless architecture and don't even worry about servers! Your own data center? Right now there's a 4-6 month wait time for new servers and storage equipment, and an 8-12 month wait time for networking equipment. Not exactly agile, is it?
You already know you need to staff to manage and patch your servers but you're also going to need staff to procure and manage your software licenses - and those licenses aren't cheap! Don't forget the ongoing tail - you pay for the license and then you get to pay 20% per annum forever after for that license. Those licenses also restrict your agility - you're pressured to use the software that's licensed in order to get better value. Try working with a procurement-based architecture!
You think you're going to avoid the software licensing hell by using open source? Good luck with that! That means you are responsible for the packaging, distribution, and support of whatever it is you're utilizing. That's more staff you need. They will inevitably miss patching some critical vulnerability that's going to land you in the news!
I can go on, and on, and on. But here's one thing I can say about applications that we've moved to the cloud: they cost 30% as much to host in the cloud as they do on-premise, and that's not even accounting for the entirety of all the costs I enumerated above! We have ultimate flexibility and agility, and we can quickly utilize managed open source platforms to solve business problems. On-prem? You lose all of that.
Faster, cheaper, better - that's why people are moving to the cloud.
Just as an example, consider Hadoop and its ecosystem. Another example is Elasticsearch. There's work involved in making these platforms production-ready and keeping them and their dependencies up-to-date and ensuring all your applications in your portfolio are using the same set of software. Most organizations are not prepared to take that on. So they turn to a vendor who will do it for them for much cheaper than they could do it themselves.
That's why the saying is "free as in freedom, not free as in beer" because open source software is not free as in beer. At least not when factoring in the total costs.
The packaging argument was made in the context of complying with software licenses. Responsible companies which perform due diligence on the software they run have to track the provenance of all software that ships as part of their dependency closure. If you want to ensure you're not vulnerable to lawsuits then you can't simply apt-get stuff from a PPA. You need to build it yourself, and track exactly what goes into that build.
Meanwhile, if you opt to run managed service from a cloud provider, you don't have to bother with that because that's not your problem (or liability) anymore.
Do people successfully lawsuit their cloud providers for downtime?
Yes, and it does indeed apply to all readily available container images.
In fact, it applies to any and all software packages put together by third-parties.
I mean, who in their right mind downloads random stuff from the internet and expects to just drop it in production software which you build your business upon?
Sure, that all works, as long as you have infinite money. My last job was at a company that was heavily invested into a serverless architecture, using the full suite of AWS tooling. They did an audit of one of their codebases and they found that generating a particular piece of data cost $20, almost entirely in fees to AWS. The company's roadmap involved scaling up the generation of this data, so they embarked on a huge process of optimizing and refactoring, to try to get the cost down. This effort tied up probably 40-45 engineers for at least six months. Probably cost them roughly 3 million dollars in salary alone, let alone opportunity cost in terms of other projects not delivered. No one amongst the management ever seriously considered moving to an on-prem solution, even though it was blatantly clear that the only value the company's massive usage of AWS Lambda, SQS, RDS and other managed services was delivering was additional profit margin for Amazon's balance sheet, and resume line items for the engineers who now got to check off the "used Terraform" box when applying to their next job.
Every app I've migrated to the cloud so far has resulted in a 70% cost reduction, but I'm also not trying to take my entire application portfolio to the cloud. To wit, we're not planning on abandoning our data centers anytime soon and that's not just because we can't move our applications to the cloud fast enough. At this point in time we recognize there are some applications best kept on-premise - but they're a minority, maybe 20% tops of my application portfolio. That means 80%+ of my application portfolio can be moved to the cloud, makes sense to move to the cloud, and I can save 70% on costs by doing so. That's a fantastic deal!
You do raise a fair point - what about the applications that are more economical to host on-prem? In my own portfolio I estimate 20% or so of my applications make more sense to host on-prem. What to do about those?
The most practical option is to mandate every application you build in the cloud have a plan for how to run on-prem. Note those that would be very difficult to run on-prem - those are your key cloud dependencies. For example, in my own portfolio I have a couple of applications depending on Lex and therefore would be very difficult to host on-prem.
Pay special attention to your so-called "crown jewels" - your apps that distinguish you from your competitors. If those were dependent on a key technology such as Lex and would be difficult to migrate on-prem then you need to identify other cloud providers you could migrate to. In this particular example I know Microsoft's and Google's clouds both offer similar services.
If you design and build your applications with no thought of the future then you're already in a bad position.
On-prem or cloud, why would I be waiting until 1 month before launch to provision anything?
You're pretending like on-prem is full of all these physical limitations, but Amazon, Microsoft, Google, Oracle, et. al. can just magically make hardware appear out of nowhere. It's the same hardware, subject to the same supply limitations, and I'm going to be paying either way. Either I pay more per hour for my cloud instances, or I pay a bunch up front to provision my on-prem servers.
Cloud makes sense if the load on your service is intermittent or unpredictable enough that you're okay paying AWS's premiums in order to have the ability to scale on demand. But if the load on the services is knowable, then hosting that service on the cloud is paying a 25% markup for no reason whatsoever. AWS is by far the most profitable portion of Amazon's business units. Those profits are coming from somewhere. They're coming from your business.
In that case you certainly aren't looking very hard for that explanation.
The rationale of moving to "the cloud" is crisp and very well-defined, to the point where even AWS makes it their point to explain in very clear terms in their Architecting With AWS courses.
Basically it's all about being able to scale without bothering with procurement, not having to manage everything down to power bills and rentals, turning big capex into small opex, and being able to deploy globally and reliably with a click of a button.
That, however, comes at a steep cost. The more serverless you get, the higher the price tag.
Also, beyond a certain scale you're better off going back to managing your own infrastructure.
> Only reasonable thing I can see as an pattern is that cloud experience on top of data things gets paid 30% more. It made me consider cloud a lot.
People with cloud experience are paid more because they can deliver more value. A random guy with, say, experience in EC2 and CloudFormation and CloudWatch, can single-handedly put together a robust fault-tolerant webservice that's deployed globally and automatically scales to meet any demand fluctuation.
There is nothing to say against a cloud version of your office suit but otherwise it is quite underwhelming. Short lived applications with a 90% API because the developer want to keep the door open to imprison the users once the application has captured a critical mass of users.
I host on AWS because I don't pay the fairly high invoices. Less domain and TLS maintenance for me. Personally I rent a server and host my personal stuff there including software repositories.
edit: Guess that technically would also qualify as cloud, but I think a differentiation is necessary. Cloud mostly means product x or y instead of just another hosted server.
Or just no real incentives of the people involved to do so. As a dev, I don't get any real credit for biz outcomes.
I spend more time trawling GitHub issues to find workarounds than I do actually using the tools.
That's a great way to describe the phenomenon.
Software is malleable, people are generally smart. It may take longer than you hope it does but things will shake out just fine as teams/companies are forced to look critically at their infra spend vs utilization and adjust accordingly.