Ask HN: Has web development become insane or am I stuck in my ways?
At the digital agency I worked at, we moved to frameworks like Laravel and the frontend side of things was pretty static with custom CSS and some custom Javascript. Everything was hosted on a big dedicated server that would eventually become a VPS per client. The web was really simple then but understandly so were users' expectations of what the web could do and so there wasn't this mismatch that I feel I'm finding now.
Fast forward to today and I'm doing primarily product development for early stage startups.
CSS is a problem that is solved with Tailwind (for me at least anyway, I know it's a contentious topic). Backend is a problem solved with Laravel which truly does provide everything I've ever needed and in cases where it hasn't (for compute heavy services), I've pulled those areas out into a mini Go service and whacked it on a separate server. Javascript is just a problem...
Anecdote 1: I've worked at startups that had Laravel Monoliths where they tried to decouple it into microservices of their own, but this failed; it introduced so much complexity, mental concentration overhead of switching between code-bases and reduced developer autonomy: unless you knew how to deploy these new things and how code libraries should be shared amongst the new services you might as well pack up your toys and go home; you're not smart enough for this. Anyway, the services were bought back into the main repository because it was getting out-of-hand and frustrating to work with.
Anecdote 2: I recently deployed a Laravel app for a startup on an EC2 server. It worked great. I used Laravel Forge to do it and it took me half a day at most (once the dependancies I needed in PHP extensions were sorted). The startup wanted to move to Docker and have a CI / CD system. Previously I could just push changes to `master` and it would deploy using Forge. I watched as a CI / CD system was built that when production moved to it, it kept falling over and not really working very well. Additionally my autonomy to fix things was then taken away because there are layers or technologies stacks some of which I know but some of which I don't in the way. The whole thing became so frustrating to see. What was a perfectly working server that took me half a day to setup has now taken over 3 - 4 engineers full time work to end up with something that gives 0 benefit (except the idea that the server can never die, but that doesn't seem to be the case because it kept dying on any deployment and ended up with more downtime than the EC2 server ever had).
So why are companies with technology stacks that really arne't getting huge amounts of traffic or require huge amounts of complexity, choosing to use microservices based architectures or build their entire application in something like next.js which requires wiring together so much stuff (Prisma, Auth0) etc. Why do we introduce so much complexity that needn't be to this entire thing?
At some point, even though my speed of delivery with my framework and stack setup is frankly very fast and I can provide real value to people, it seems the perceived value of the above shennanigans is greater than the value of actually shipping stuff that matters if you look at the job boards, technologies used and pay given. I would love to be Bilbo Baggins in The Shire minding his own business, but I worry that if I don't jump on the bandwagon of overengineering everything to make myself feel smart then I'll quickly become irellevant in the job market. Am I completely missing the point of why these things are being done or is modern web development choices with regards to language and architecture just certifiably insane?
74 comments
[ 2.4 ms ] story [ 146 ms ] threadAfter all if you’ve got such a perfectly simple way to handle all those concerns then your approach should win any day now. Right?
Other people would tell a story similar to yours except it revolves around Ruby on Rails, React, Docker, CSS, npm, emotion, or some other tools or standards.
You either are in a state of peace with your tools or not, and that is something that can change over time. For my job I work on a React application that was started by other people that has some good ideas in it but has plenty of accidents built in.
I was doing a side project and was tempted to make a little application in Svelte, which would have been a perfect fit. I used React instead because I wanted to build on and extend my experience. I had a 60kb bundle which might have been 20kb in Svelte but the graphics files are way bigger than that and it loads faster than a speeding bullet as it is.
That's the way I attain peace.
To get a new job, signal that you will find that peace wherever you go with whatever tools they use. It might take a while (about a year), and sometimes it doesn't happen, but if you believe it you can usually make it so.
i fucking wonder everyday how he attained bliss in chaos, hahaha.
There are two main types of tool selection failures I see:
1. Teams that try to choose the “best” tools according to what they perceive as the state of the art without regard to the problem their trying to solve. This tends to become a game of picking whatever they think FAANG might be using, even if their problem is orders of magnitude smaller and simpler. At the extremes, these are the teams that end up building a Kubernetes-deployed custom React-based CMS with server-side rendering when really they could have set up a Wordpress install with a custom theme in a couple of weeks.
2. Teams that embody the phrase “When all you have is a hammer, everything looks like a nail.” These teams have been using the same tech stack for a long time and they’re so good at it that they refuse to consider anything else. Instead of trying new tools, they try to brute-force their current stack to get the job done. Some times this works well if the problem is a good fit for the stack. Other times, they waste a ton of time and money doing things that could have been solved quickly by trying a new tool.
None of us are perfect, but most failures fall into one of the two categories above. Eager startups often struggle with #1 when they have a pseudo-CTO cofounder who wants to pick all of the technologies but hire someone else to do all of the work.
My approach is to draw up multiple proposals with different stacks and present them with an overview of possible risks, benefits, and timelines. Usually once it’s all laid out, reasonable engineers might recognize that the right tool for the job isn’t always the trendiest tool. It takes some good presentation skills to navigate this but it can be done.
At the same time, I see some opportunities for you to grow your experience a bit. A CI/CD system has real benefits for delivery, but it shouldn’t require 3-4 full-time engineers to get going. It’s worth learning how to do this right so you can actually evaluate it, rather than dismissing it as useless complication. Frankly, setting up a basic CI/CD workflow with modern tools like GitHub or Gitlab isn’t a big chore. You should be able to come up with something usable and a good understanding of the basics in a workweek by yourself.
2. the delusion that the way google manages 1-3% of global computing power is appropriate for your company which bursts up to < 10 servers under "high" load
2a. though you can quickly get to a place where you do need more control over migrations once your db has a nontrivial size
3. early engineers and bad engineers love complexity. Love it.
ps -- lots of these choices come down to the choices of senior people. There are good shops out there! This may resonate with you. https://mcfunley.com/choose-boring-technology
How would you think about a simple approach to microservices based on http requests? Say we have an application that resembles Twitter and want to split the "messages" functionality off the main codebase. We could run all "messages" functionality on a different server. Served by a completely different codebase. When the user asks for a url below /messages/ we simply do a "GET messageserver.com/messages/...?uid=<userid>" get the result, put it into the template we are rendering at the {messages} placeholder and output the page to the user.
Looking around on the Twitter website, I would expect there to be rather little entanglement of the messages functionality with the rest of the site.
But as soon as you do, which is probably the case, you end up doing method calls over HTTP and you need to have a strategy for refactoring two services at the same time that can be online at the same time but with different versions.
Then you have the extremely low barrier of entry, resulting in `is-odd` packages polluting the shit out of the ecosystem.
Then you have everyone and their mother becoming a "JS coder" and bombarding twitter with dev.to articles.
You end up in JS hell.
The solution is to eject and avoid javascript as much as you can. At this point it's already too late to reverse course, or worse the majority don't even want to reverse course.
I've know many people, including my younger brother who said "fuck this" and went full backend to avoid working with npm/node/js.
Then you realize your resume is getting screened out at half the jobs look look at because you "missed out". FWIW it happens on the backend too, see docker, k8s, and all the related container infra I “missed out on”
Also, how easy (or hard) is it to get a backend only job? I'd like to avoid CSS/JS completely, but unable to find such jobs.
Javascript provides a decent API for interacting with the DOM dynamically. I understand you're frustrated with culture, community, and tooling, but you can still use Javascript without these to make web pages interactive in a way you can't do pure server-side. The MDN web docs are a great place to start.
After grasping HTML and CSS basics, prototyping my heavily interactive tool seemed impossible with JS alone, until I learned React. React solved real problems for me. Create-react-app with a firebase backend got me VERY far with zero formal education.
But I struggled with SEO, load times, and needed a sprinkle of server-side logic. Next.js solved those problems, and more.
As my prototypes grew into products, complexity was hard to manage. Refactors were impossible, even with small, modular components and well tested functions. Enter TypeScript. The single most important tool in my toolbelt. I refuse to work in plain JS now.
I'm several years in now, at my 2nd full-time dev gig and working on my 3rd side-business. My stack is Typescript, Next, vercel, firebase. Both at work and at home. Hardly any third-party dependencies needed beyond these (stripe, coingate, mapbox). I write most my CSS by hand with css grid because its the least verbose way I've found so far. I auto-format with Prettier.
I can use types and tests across my stack, I can enforce bespoke auth/permissions behavior, all kinds of queries, media upload, pretty much everything I'll ever need outside of real-time collaboration. I can build and maintain profitable products by myself. I've done it.
This also happens to be a very popular stack. What's insane about it? Every alternative I've looked at looks way less sane. Multiple languages and repos for one app? No thanks. Managing server instances and databases migrations myself? I hope I never have to. Software is fun, but I like making products more. JavaScript just happens to be the best way I've found so far.
It's a real tangible accomplishment to find a tech stack you're productive with and don't have to change or update frequently. So, congratulations!
The last major real revolution on the backend in my eyes was Sinatra, not for Sinatra itself, but how it changed how everyone wrote web servers. The last real revolution on the front end was Angular.js (v1.x) and then shortly after, React.
There's a lot of change year after year in the real world, but a majority of it you can ignore.
Our industry is pretty crazy in the amount of tech it uses to move shit from A to B according to "best practices".
Not saying we shouldnt but I am saying that if we did start making things as simple as possible again, it would be extreamly productive and fun.
FWIW, I have never worked in a team where CI was not needed.
One weekend not too long ago, I started a Rails app with Hotwire and found myself wonderfully, nostalgically productive again. Maybe I'm old, maybe I'm burnt out but I can tell you one thing: I'm having fun and building neat, performant, and fast web apps. I hope that speaks for itself when I look for my next job.
https://unpoly.com
https://mitchel.me/slippers/
One dimension of this is that it isn't just about your own ability to be productive as a solo developer (unless you limit yourself to solo-developer jobs, which can be an option!), it's about the org's ability to be productive as a (constantly changing, mixed-experience-levels) group. It's frustrating to see one's own productivity suffer for the sake of aggregate productivity, but it's a reality of our industry.
I can't speak to the particular situations you described but I can speak to this being a real trade-off that often has to be made.
The vast majority of engineers have never built things to "scale", and don't need to.
Even in large companies the proportion of things that needs scale is very small.
We have docker now but that is only to manage legacy dependencies.
You don't always need complex infrastructure for scale.
Getting your indexes correct, keeping the number of calls to the db low, and tuning your db can do magic.
Ignorance, fear, and our urge to pick The Best option.
If something is complex enough, at some point everyone is ignorant to some degree. That makes it easy to let fear get a hold of you. Still: whenever you invest, you want to invest in The Best Option since your resources are limited.
You're a professional photographer, and you need a new camera. Your old one lasted a decade, so anything you'll buy now will be better and cheaper than the old one. Guaranteed! It's unlikely you know everything about cameras, though. But since you're going to invest in one, you want to get The Best option for the price. You're reading blog posts, reviews, and random comments by people on the internet, and it's becoming harder and harder to decide. There are so many factors involved that you become paralized, you don't know how to decide. Also, you don't know exactly if you're going to be making pictures in the jungle or in the ocean, how many pictures you're going to have to be able to take and how long your battery should last.
Then you read that the world's most successful photographer uses a camera that is rumoured to be able to handle all those situations, and has auto-renewing battery and infinite storage. The pricing is a bit unclear, but at this point you're so fatigued that you decide to just get one and find out along the way. At least you know it has the potential to be The Best.
If what you're doing works for you, just keep it up. I don't think that you have an obligation to add complexity to what you're doing just so you can stay relevant.
I think this might be a recipe for being a person with 13-years of experience and yet unhirable at the same time.
I don't know if it is, but your advice sounds a little bit dangerous to me.
If you look at OP's examples, I don't think that any company would disqualify an otherwise competent developer for having only worked on monoliths instead of microservices or for using an easy deploy set up instead of a something with Docker and a CI/CD system.
And of course OP still has tonnes of experience with their specific stack which is still quite popular - that's the opposite of being unhireable.
The other part is that you might be overestimating how difficult it is to pick up something new when it's necessary. Most of these aren't that big of a deal to learn for someone with experience.
You're dropping the ability to do trivial joins, trivial reporting, trivial SQL aggregates, over... what, exactly?
If you can perform a join to get the data yourself, nothing is stopping anyone from circumventing your API that is intended for fetching the data or by making breaking changes to the single database that your code relies on.
But yes, trivial joins is effectively the trade off for this encapsulation and can mean other services have to collect their own versions of data from services which means you now have eventual consistency.
For example, I've been on the operational side of a leading browser game in germany some 5 - 10 years ago. Obviously, it must have been an amazing CI/CD workflow with great automation. No. The global deployment was me. With SCP. And we made money. I'm not necessarily proud of that on a tech level, but it worked. The job after that... we made money by using chef to copy war-files into tomcats and then we had some application to sell. Again, it's not technologically amazing, but it has a huge advantage: It's simple to get right, and hard to get seriously wrong.
Now, still at the job, but with a changed scope, yes, we are embracing docker and a container orchestration. But we do so, because we have to scale from 10 developers to 100 - 150 developers and we cannot handle the complexity in a central config management. We cannot support the 3-5 languages and frameworks necessary centrally anymore, so it makes sense to introduce a framework that's easier to cooperate on, in the layer that needs cooperation. It increases the total tech stack complexity, but it simplifies the things the individual team has to deal with, so it's a net benefit for most people.
But all that is driven by necessity. If you tell me to deploy half a dozen of same-blueprint php/spring boot/.NET Core apps on 1-2 servers without any infrastructure, I'd look at some ansible and some cheap VMs. Or if there is a natural thing like capistrano or whatever Laravel Forge is, that. I like not doing work and making mistakes hard to make.
However, all of that said, in your particular case, a CI/CD pipeline using docker / EC2 doesn't really sound that mad. The patterns are super well understood there. Maybe it's just a case of getting used to it? Fwiw, we do pretty much that and we have fully automated deployments without much hassle. We use ECS / Fargate too which might sound like a other level of black magic but honestly, serverless is pretty amazing too. I don't even have to think about EC2 now. I'm not saying this to showoff, the point is that this new tech maybe takes some getting used to, but maybe it's just a lack of anyone experienced leading the way on it and helping you out?
As for microservices, that can be a nightmare when done wrong but it totally depends. If you just split things up without really understanding why, you're now going to have two problems as the old adage goes. The distributed monolith is the worst of all worlds, especially when the boundaries between services are poorly thought out. If in doubt, and unless you can articulate exactly why you're creating another service, build a well structured monolith imho.
13 years is a long time for technologies to build on top of each other, and while more things are possible now than ever, a trade off of that is that complexity is higher than ever, too.
Also, the boring middle road doesn’t get the press. No one is writing stories about all the little apps and services chugging along with their solid rails or laravel or whatever backends, serving basic html or API requests all day.
Sure, you're fine with running and managing these servers, but what about the new hire who replaces you when you move on?
Specifically it forces most of the DevOps tribal knowledge to be coded/documented. Those "Oh, that's right I needed to recompile the mod_wsgi extension manually because this version of the CentOS didn't support python 3" moments go away. (This is a real example from the last server I needed to setup by hand.)
It conveniently it also makes setting it up on a new machine a piece of cake. Clone the repo, run the setup command/script and run the
It took me a while to grok, all of the pieces, but I'm glad to have it.
That said it can get extremely complex (and by extension fragile) for little benefit. My entire setup and deployment are controlled with two scripts one for creating a cluster (CDN, DB, message queue, app server, monitoring) and one GitHub action script for CI/CD.
It's all very understandable even for people who've never worked with these technologies before. Writing the scripts on the other hand take a lot of knowledge and it took me a month to really grok what I was doing.