> "So in the 70s you could pay per request billed to millisecond precision?"
It's been a while since I encountered it but yes, mainframe time was so blisteringly expensive that usage was accounted for on a per-job basis and billed to individual user accounts. I don't recall the resolution or pricing anymore but students would be issued an hour or so (?) of processing time to complete their coursework per semester. Perhaps somebody with better recall than I can fill in the details.
I know on the Control Data Cybers the time was measured by the second with 3 digits after the decimal. Might have been measured with even finer granularity, but that's what they displayed.
I remember seeing somewhere a Linux or Unix program or system that shows how much money it costs for running the program. I think it was from an online course but really couldn't remember where, but the cost is very expensive...
Mainframes were (and still are) billed by fractional second of CPU usage, total storage, bandwidth between storage and compute, bandwidth off box, total number of jobs scheduled, and per-use fees for third party tools such as sort, RPC servers, and such. So times literally have not changed in this regard.
> So what have we now? A “mono repo” codebase, because clearly a Git repository per function in your system would be too much, a large deployment descriptor per fine-grained component, which Spring maybe called “Controller” but is now called “Function”, and instead of combining them all on your desktop, you send them off to someone else’s mega-mainframe [AKA "the cloud"]. You deploy, get an error message, and login to CloudWatch to see what actually happened - it’s all batch-driven, just like the bad old days, so progress is slow. At least we’re not having to walk to the printer on every try, but that pretty much sums up the progress of the last half century. Oh, and a “Function” will be able to handle a single request concurrently, so here’s your AWS hosting bill, we needed a lot of instances, I hope you won’t have a heart attack. Yes, we run workloads that your nephew can run on his Raspberry Pi 4, but this is the future of enterprise.
I've mentioned as much to several C-suite executives and professional venture investors I know who tend to wax philosophical about "exponential growth in cloud services and big data" and whatnot. Invariably, their response has been to tell me that I just don't get it, or that I make no sense. They give me puzzled looks and maybe even get a bit exasperated with me.
Surely others here on HN have run into similarly minded individuals too. They're not hard to find.
Everyone's chasing "scalable", mostly for no reason at all.
There's also this enduring myth you see repeated even here of how much saving there is by not having "devops", when only very big companies needed them in the first place. A few minutes per month is the most devops any startup would need to do in reality running their own VMs or servers, after an hour or two of setup. And that hour or two is often less than faffing around getting all the labyrinth moving parts of AWS/Azure/etc. working.
And the whole thing becomes a transparent farce when you look at providers like Azure, where their own admin interface is so slow and prone to random breaks you have to ask:
How is this scalable?
It's clearly not scalable for themselves, so how is it "scalable" for everyone else?
There are benefits, mainly for me I'll use them for hassle free deploys[1], but the whole thing sometimes feels like a massive con to sell the performance of a potato by advertising it as a jet engine.
[1] Like a month back I deployed a demo prototype web app I wrote in 3 hours onto Azure straight from github with like two clicks, and got the contract, can't really beat that.
You have all that DIY too, except maybe shock bills, but it's your problem to fix. Like with everything, extremes are bad and all managed or all unmanaged are both bad. My personal guideline is use off-the-shelf services if they are cheaper than DIY, including labor, or if DIY provides no competitive advantage and off-the-shelf is cheap enough. But if DIY is much cheaper or gives a competitive advantage, do it yourself.
24? I guess it's that long because it is rural? We use Dell servers and I sleep a bit better having seen their 4hr support replace a motherboard with a bad lifecycle controller within 2 hours of the initial call.
Try calling Amazon and getting that kind of response for your problems.
But it is a few minutes per month. A properly architected application won’t depend on a single server so a hardware failure will be as unimportant as it is the serverless world.
Architecting a system properly is one of the key functions of devops. You don’t get it by investing a few minutes per month, whether public cloud or bare metal.
I always have standby copy deployed elsewhere. And I make it a point not deploying production until a single script could rebuild new clean system by using backed up data, configs and software packages. So for me it is no a big deal at all. Well if the hardware died I have to pay for replacement which is not fun but it is a very rare occurrence.
You do not notice that either with the right setup. I also rather hire a good SA and pay 10-100x less monthly (for which I can hire an army of devops here in the eu by the way) with the risk of all hardware dying at the same time (which does not happen, not now, not in the 90s when I started hosting).
Sure serverless etc has it's upsides and in that case I do use it, I just encounter close to 0 companies per year that need it or benefit from it. Aws ways benefits.
The choice is not only serverless or hosting on bare metal - there are a lot of options in between. You can hire VMs and have the vendor transparently deal with hardware faults for example without downtime. Running your own VMs doesn’t mean running your own hardware.
The original claim is not at all unrealistic, I’ve been running a small shop with about 20 VMs without issues for years, uptime seems to me about comparable with serverless or hosted services from big players who regularly have small outages and there really is very little work involved.
Its like a well stocked grocery store with thousands of brands, I think the biggest draw of "the cloud" is the illusion of infinity plenty, all you need to do is press a button (even though the reality really doesn't measure up, at least not in ways people naively assume)
> There's also this enduring myth you see repeated even here of how much saving there is by not having "devops", when only very big companies needed them in the first place.
I dunno, the first company I worked at was very small, under 30 people, sales included, and I was hired on as an SA. If they were using serverless, they would not have needed to hire me.
Deployments and system upgrades were very toilsome. The dev lead would spend a few hours deploying new code and we’d only do it around midnight in case there was a problem (our customers were almost entirely US based). I spent a lot of my time automating OS upgrades on the web servers behind our proxy. All of this work would be unnecessary for that small company if they were using serverless.
Additionally, as mentioned before, the majority of their traffic was during the day so they could have scaled down automatically at night, cutting costs. Would this be cheaper than running machines 24/7? I’m not sure, but I could see why you’d be interested in at least looking into it
> Deployments and system upgrades were very toilsome. The dev lead would spend a few hours deploying new code and we’d only do it around midnight in case there was a problem (our customers were almost entirely US based). I spent a lot of my time automating OS upgrades on the web servers behind our proxy. All of this work would be unnecessary for that small company if they were using serverless.
Not using serverless doesn’t mean there can’t be a CICD pipeline.
Yeah of course. I’m not saying that serverless is the only ci-cd solution, many businesses build ci-cd pipelines on top of serverless apis. However, most serverless platforms take care of a lot of the deployment process out of the box, which would have been enough for the business I was describing in my comment
You don't even need serverless for those things. Any of the automated deployment/orchestration systems in the last ten years solve those problem (ansible, terraform, etc).
Yeah it’s true but it requires a lot more config. The job I mentioned in my comment involved thousands of lines of anisble to perform operations you wouldn’t even need to consider/could perform in a few clicks when using a serverless platform.
Don’t get me wrong—serverless platforms are not a ci-cd silver bullet. Many organizations build pipelines on top of serverless deployment apis, but it certainly does take care of a lot of things for you—not to mention other features mentioned elsewhere in my comments
And since we have containers it's become really much easier actually ... Deploy a Traefik container which will watch dockerd socket and self-configure HTTPS and all when your other script starts container. Really, it's much easier than back in the days: https://github.com/jpic/bashworks/tree/master/vps
Lines of config are like lines of code, refactoring them regularity is basic hygiene like brushing your teeth. You can "have no config lines" if you go ahead and click your config but then you don't have "infra as code" anymore so that's also "going back to the 70's" I guess.
All you needed was PaaS, which has existed for a long time. Or the even older name: managed services, offered by pretty much any decent hosting provider for decades.
I remember using companies like MediaTemplate and FullControl 10 years ago for clients and not having to worry about OS updates.
I don’t think your past experience reflects current practice.
My reality at a 20 person shop with a few devs is much closer to the OP - a few hours setup (we save the cloud config and can remake servers in a few minutes), servers run for months without intervention, uptime is on par with cloud services, deploys take seconds. Managing servers or deploys just are not major problems for us.
Serverless is not a solution to deploy problems - mostly because they’re just not significant problems for us, if anything the dev deploy story is worse than renting VMs. It’s an interesting take on serving requests and I do believe some variant of it might be the future (a variant more like cloudflare workers IMO), but at present there are serious drawbacks and it is not clearly better, particularly when provided by a predatory company like Amazon.
When your infra is in AWS, you need an "AWS Certified DevOps"[0] instead of a "typical DevOps", I mean, the typical OSS hacker - which is kind of sad really, and makes me wonder: is this how we're going to build an Idiocracy[1] ? Anyway, things like backups, monitoring, alerting, debugging, is going to be a requirement anyway, and for me it's harder when locked in proprietary stuff such as AWS, I'd rather have OSS I have freedom to hack ;)
"The cloud" is a leasing program except even more optimized for OpEx - pay-as-you-go monthly billing and you now don't even need to hire ops people because it's "serverless".
(Of course you have to force your devs to do the ops stuff now, but that's not accounting and HR's problem now.)
Well, if you mean CapEx because one needs fewer programmers and a higher AWS bill, then yes I agree with you. But if you mean CapEx because one needs fewer physical computers that they own and a higher AWS bill for computers that they rent, then no I do not agree with you.
First, the markup in just sheer ops terms is enormous. Minimum 4x, and in some cases 20x. Sure, you get a couple more nines of reliability, but 99% of startups don't need that much reliability.
Second, in countries like Canada where capital expenditure on computer systems can be heavily marked down[0] people still use AWS and DigitalOcean by default.
The main reason people don't do colocating is that it is a waste of time better spent. DigitalOcean is cheap enough. AWS is cheap enough and if the team doesn't have experience running Postgres, fine, they use RDS. Sometimes it's just sheer laziness at first that later blossomed into "holy shit, we're growing too fast to even think about migrating off of these giant AWS bills" which was essentially Dropbox's story if I recall correctly.
[0] 55% per year, and in practice they allow full expensing in the first year for petty amounts of outlay. For example, if your business is a data centre, sure it's 55% per year, but if you run a flower farm and fully expense a new laptop for sales the CRA is generally fine with it.
I'd guess that the vast majority of "cloud" customers are boring enterprises doing boring enterprise stuff, probably not even Internet-related stuff at all.
Does anyone know of a resource showing me some parameters for estimating how far different managed servers reach? How many concurrent users, bandwidth, etc.? Have a hard time getting a mapping for expected concurrent users -> hardware requirements. How quickly can you adjust the HW resources in a non-cloud setup if your startup product is more successful than expected?
Impossible question, it depends what your app does and what your users do. Even within the same company, 1000 concurrent users of Google Maps will use completely different resources than 1000 concurrent users of gmail. It'll even change over time for the same app and same users, as their behaviour or the world around them changes.
Scalability is just one reason for using serverless. There are many others. Most of which have to do with being able to focus on the part of your code that actually does something instead of on infrastructure and integration. For example, not having to configure and maintain VMs, containers, or Kubernetes, off-loading a part of security onto the cloud provider, and built-in integration with supporting services like logging and metrics.
Serverless, like every design decision, comes with its own set of trade-offs. Whether those trade-offs make sense for your environment is something you have to determine. At my company we have seen a productivity increase and shorter development times using serverless. Of course, that was after the learning curve which is certainly not trivial.
Apparently, you don't even know what that word means. You can not "have" DevOps. It's a methodology and not a thing. Besides, DevOps means exactly the opposite of what you are implying: it is the practice of not having a dedicated operations team and letting devs handle as much of the operations for their own applications as they can.
> few minutes per month
I can tell you my team spends a good 30% of its time on operations. Some of that (for example adminstrating K8S clusters) could be offloaded to cloud providers for money.
I sometimes think of how far we still are from utility-quality computing services. Every company needs electricity with near-perfect uptime, but almost no company has a full-time electrician in-house.
> Everyone's chasing "scalable", mostly for no reason at all.
No technical reason, maybe. There is a social reason though: if you point out that there is no need to build scale (yet), you are pointing out that the business will not achieve overnight success. Even if it's true, it kills excitement and is therefore not something that people want to hear. Who wants to realize that what they're spending large amounts of time on (and possibly even neglecting other things like health, sleep and social contacts) will not even make all their financial dreams come true? Much easier to not think too hard about it and keep on doing what everyone is doing.
So, building out scalable infra is signaling that you too believe in the company and are preparing for the moment (any time now!) that "webscale" is needed. It's showing that you are part of the team.
I think what’s important to remember is that these kinds of “truth” are not the full story.
Yes, a technology probably won’t live up to its marketing.
Yes, what works is probably good enough.
Yes, changing has switching costs and sacrifices institutional knowledge.
Yes, we’ve seen these technologies before and switched away from them from various reasons.
That’s all fine and sometimes frustrating as an engineer. But it’s not the only reason decisions are made.
Sometimes pitching your exec team on The New Hotness gives cover for changes that actually do need to happen.
Sometimes adopting The New Hotness is marketing to recruit engineers who want to try something new.
Sometimes adopting The New Hotness is because some people are bored and uninspired in their jobs and just want to believe in something good again.
The success of things like MongoDB and blockchain are not because they are better or even very good at most things people use them for. They’ll never live up to the hype. But people need things to get excited about. It’s psychological. Attacking them purely on technical points can lead one to miss the actual point.
I’ve wondered whether these good engineers are really a net benefit. They crank out something spiffy, move your big systems to it, and then disappear to work on the next hotness, leaving an already burdened tech team to support yet one more spiffy stack that is not quite as reliable or easy to use as we would like.
I think you need to either talk to your system architecture function or get a system architecture function in your organisation.
Managing this stuff is a job, and it requires discipline and organisation. It's one of the few functions of an IT department that is left once you go to cloud (as dealing with Oracle's bullshit is then gone). The younglin's doing development in the business functions have to play by the rules - and the business functions need to agree that there are rules or they will :
a) spend much monies on the issue you describe
b) get cracked open like fresh eggs by any passing hacker who is keen on sports cars
I'm not sure I would call people that would rewrite something just because it gets boring otherwise as "engineers". I think the proper term would be "children".
Yet the best engineers in the field aren’t working on legacy banking apps. The problem is that once a product is put into production and the hard problems are solved, the engineering becomes significantly less challenging (unless you’re in an ultra rare scenario of exponential growth).
The problem we have in software is that we have engineers build bridges and then somehow expect them to stay around to paint the thing, change light bulbs, etc and stay fulfilled. A person who is an expert at making new things won’t like being in maintenance. It’s that simple.
Sure, that's absolutely true. But it doesn't mean that you need to rebuild already working stuff over and over.
It's much more fun to improve the status quo by inventing something nice as to repeat the work of the past generation (only painting things in a different color).
Thanks for putting this into words. People want a reason to log on, a reason to show up to work.
I wrote a blog post on Monday about a relatively old and not-hot language. The post nevertheless trended on Medium and received 50k views. A lot of the feedback was from people who were just happy to see someone senior say something positive about the not-hot language.
Anyway, working with The New Hotness also gives people something to put on their resumé: “I built system X on top of The New Hotness. Nobody had built such a system before for The New Hotness”.
Not when that becomes table stakes, no. If you have five different companies offering you that kind of money, eventually you'll start thinking about more than just the money.
Well the funny thing is that once you crack FAANG interviews you can usually get offers from multiple of them and then money no longer becomes the decision factor. The choice becomes, “work on X and make $400k or work on Y and make $400k”. You bet the tie breaker often ends up being the prospects for working on cool shit.
I suspect the majority of readers are being paid less than $100k with no stock or bonuses, and correctly intuit that new-hotness things on their résumé will help them climb the salary ladder towards that magic $100k threshold.
I think it's sad that the vast majority of people in our field (at least in my experience) is so damn excited about building new stuff all the time. There's no appreciation for maintaining, optimizing, and improving existing systems. There's only talk of "adding value" but very little of "keeping value."
So people keep moving on to the next project, leaving the old ones running live but to rot, and at best every once in a while someone will be assigned the dreadful task of having to figure out why the "legacy" (even though it's still fully in operation and servicing customers) system had an outage and can't recover.
Our field suffers from a serious lack of focus. People don't learn to work under constraints. Instead, they prefer to build a whole new thing, full of complexity and hidden issues, just to get around a few nuisances that, with the right mindset, actually force one to be creative (which should make work fun).
I've recently commented here on HN about how I feel like I'm outside the norm on these things [0]. I was happy to learn I'm not alone. I do think those of us who don't mind working on "legacy" systems and perhaps with limited tools are the minority though.
Sometimes it's not even about working on old systems. You work with current tech, build something, and as soon as it's barely deployable, you're expected to move on to the next thing. "It's a great opportunity for your career," they say, "you're gonna get to build something new and make key decisions."
Why does it have to be decided for me? It's my current predicament. I'm working on a project that took me two months to get in the right mindset for and get all the context I needed. I had to try things out, sort out ideas, until I got a clear picture of what to build. Now I've built the first iteration of it. It works, and I'm super excited about it. But it's not done by any means. It needs more features. It needs to be polished so it doesn't make operations miserable running the thing. And yet... I have to move on, and hand it over to someone who doesn't have the same excitement for it that I have, and has maybe 10% of the context.
All because building new things (and adopting The New Hotness, as you say) is what "everyone" wants.
It doesn't help if your career path in a company is tied towards 'visibility,' where certain roles will never see a promotion no matter how critical they are.
I've chosen to take on more than one heavy maintenance job, no client facing visibility in terms of fancy new features. Feedback in performance reviews would always be "it's hard for people to see what you're doing." In the end you have to find your own way to game the system to get appropriately recognised, but of course that kind of office politics isn't really what you want to spend time either.
I'm doing more 'new feature' style work now but I'm happy to not be tumbling through the waves of hype. We get a little bit of new hotness, because of course you need to embrace some aspect of that to avoid stagnating and getting too stuck in your ways, but not so much of it that the engineering team is incentivised to churn through half-assed features.
It's great for those engineers that they are so far up Maslow's hierarchy that they're willing to bring significant risk to their employers in the pursuit of self actualization. However, there are a lot of engineers in the world (outside SF and NYC) that are still concerned about lower levels like physiological and safety needs. I wonder if the world getting more accustomed to remote work will change the cost benefit analysis on hiring those engineers instead.
>"Invariably, their response has been to tell me that I just don't get it"
I cured this one when my native server running on not so expensive multicore CPU with enough RAM left their cloudy setup in a dust performing more than a 100 times faster and still having huge potential reserve in vertical scalability. They were just astonished at what a single piece of modern hardware in combination with the native code could do.
Congratulations, from now on you are on call 24/7 in case anything happens to that machine. It is now your own pet. When RAM fails in unpredictable random ways, you'll spend hours reading the syslogs. When your SSDs fail, you'll spend hours ordering new ones, setting raid up and migrating the backups you've had to provision, automate, supervise and secure yourself.
When the server gets even remotely suspected of having been hacked into, you must trash it overnight and never use any of it again.
You talk like using a physical server somehow prevents building a highly available system or from using automation. Why would the hardware be any more of a pet than any random VM in the cloud? You could also just run a hypervisor on the host and get the benefits of virtualization along with full control of hardware.
Sure, if a drive breaks you'd probably want to replace it instead of just replacing the entire machine; it's a thing you don't need to do with VMs, but that doesn't mean it's difficult. It definitely doesn't take "hours".
Managing physical setups is different from managing fleets of cloud instances, but it's not necessarily automatically inferior.
>"You talk like using a physical server somehow prevents building a highly available system or from using automation. Why would the hardware be any more of a pet than any random VM in the cloud?"
It is pure scaremongering and making not very nice assumptions about people's abilities without any real substance. If this is the idea of attracting customers or advocating the approach it seems like a very poor job.
I always have up to date standby for a fraction of money they spend on cloud. In the last 10 years or so I do not remember RAM failing in an unpredictable ways. When/if my SSD fails It'll take me a whole 10 minutes ordering another one from amazon. Oh and I always have spares lying around. My backups are automated.
Anyways nice job advertising cloud and trying to scare people into using it. You happy with it - good for you. No need to convince me as I have my own reasons and do not have to buy into yours.
> "...I do not remember RAM failing in an unpredictable ways."
I'm about halfway through my career and I have seen something like that happen exactly once; the erratic behavior of the machine had my team scratching our heads for a fair bit until we ran memtest86 on it after running out of ideas. So I would say that it's not impossible, just extraordinarily unlikely.
Things did happen with my stuff as well. I had graphics card semi failed in an unpredictable way so I just replaced it. I also had one system randomly crashing because thermal paste dried up on one ancient CPU. It is just a little annoyance that happens extremely rarely. Anything is possible and I'll deal with it when/if. Nothing world shattering here. Cloud can and does fail as well. The list of reported outages affecting large chunks of customer beats my accidental cases with the healthy margin.
I find your quote here and the follow up seriously unconvincing. You know what a miserable workflow is? Preparing all my build scripts according to the half baked build tool that our in-house Deployment team cooked up to use fabric to spray my deployment onto a bunch of nodes in our data center, all the tests and script linting pass, so then I go push it through our CI/CD pipeline (which can’t do the above tests for me because again this is all threadbare maintained by in-house tooling teams who could not be less aligned to end product deliverables), and lo and behold it breaks. Good thing I’m not paying so much for CloudWatch so I get to literally go ssh to the worker node of the CI/CD system where my task failed and cross my fingers that it wasn’t reclaimed for the next task and that the log files weren’t rotated off, grep my way to tracebacks that probably involve things no one on my team has ever heard of, spend 1/2 a day chasing them all down (if I’m lucky), and finally kicking the box to restart a build a few times and it goes. Then it goes to the in house homebrewed kubernetes deployment scripts where it promptly chokes on a secret that wasn’t configured right (thanks home made script linters), so then I have to go down a rabbit hole of eye melting Helm config of secrets for an hour but thank god I’m not paying for a simple cloud UI for this. So we get the secret cleared up and everything is great, until the kubernetes scheduler can’t schedule my pods because our data center is at capacity and now I literally can’t get work done until I chase down some probably incorrect and certainly unmaintained Grafana chart of who is using how much excess memory on which nodes, and wrangle someone with authority to force another team to stop eating lunch and downsize pods right now. After all this, 6-7 hours later and 20-30 labor hours later, my workload is finally deployed, but it will die overnight for an uncaught reason that doesn’t alert anyone, and even if it did, the team managing compute resources is in a permanent state of PTSD from never ending pages so they wouldn’t bother caring about why my workload failed and will just assume it’s application logic bugs or Docker bugs on my side for as long as they can just to avoid yet another un-debuggable failure landing on their plate, so that’ll be great for morale in the morning when I get to derail yet another day and spend time doing every type of ops work that is not my job and prevents us from delivering product goals.
This is every company I’ve ever worked for or ever heard any colleagues, friends or associates work for, that tries to build their own SRE practice on top of data centers.
Large cloud platforms alleviate so so much of this pain. The “back to the 70s” workflow sarcastically described in the quoted passage you shared sounds like an absolute dream compared to the nightmare of shipping code on internally created developer tool & deployment platforms interfacing to bare metal or space leased from data centers.
I can’t give the cloud providers my money fast enough and I have only ever felt happy about that for years and years now.
This is the type of cultural accuracy I've been looking for.
Are all the commenters in this thread just cynical engineers who really can't see past their own narrow world view? I get snake oil is snake oil, but there is such a strong brand of "anything new must be snake oil" that runs through the HN community it boggles my mind.
The world is actually full of complex systems. And only once these systems are in place does the software engineer get to do his work! People on this site act as if they have all the understanding and knowledgeable necessary to decry something like cloud vendors because clearly these same people know better than everyone else and these are actually simple problems if only we had their perspective and knowledge.
The ego and arrogance of people in tech really shines through here.
People on this site act as if they have all the understanding and knowledgeable necessary to decry something like cloud vendors because clearly these same people know better than everyone else and these are actually simple problems if only we had their perspective and knowledge.
A 1000 times this. These are solved problems. Anything new has to be better to be taken seriously.
> there is such a strong brand of "anything new must be snake oil" that runs through the HN community.
I would say there's at least as strong a "brand" of "if it is a new way it is probably a better way" on HN as well.
I think making blanket assumptions in either direction is a terrible idea. Best to be aware of your own biases, and try to be as objective as you can in your decisions.
Experience matters, and there are new solutions that are inferior to older ones in many cases (yes, there is plenty of "snake oil" for sale). However, sometimes new people look at an old problem with fresh eyes and come up with better ideas on how to solve it.
This really isn't anything new. When I was fresh out of college I remember older programmers talking about "such-and-such" is just the old way of doing it repackaged. There is lots of oscillation in the computer industry.
> The ego and arrogance of people in tech really shines through here.
You mean, "the ego and arrogance required to bend code and machines to their will and not the other way around". Maybe try to pass that and learn a thing or two.
Cool story, but the majority of that is not related to serverless. You still need a CI/CD pipeline with serverless and I’ve dealt with just as many miserable setups there.
> I can’t give the cloud providers my money fast enough
People run VMs and k8s on the cloud as well and run into every one of the problems you just described (aside from literal hardware failures).
Your rant seems to mainly be about having a good CICD pipeline and that’s not really related to cloud or serverless.
I think you completely miss the point. With cloud tooling, my team is unblocked to spin up the resources we need, manage CI / CD in our unique way that solves our problems, quickly try out serverless if it’s a good model for a given use case, change logging or alerting, get new compute resources like GPUs, etc., all without being blocked by a choke point of in house engineering limitations and staffing limitations and political in-fighting over who owns what and who gets to define how abstract vs purpose-specific the tooling will be.
You toss out “build a good CI/CD pipeline” like that’s just some localized, solvable problem, and it’s exactly that lack of imagination and lack of understanding about the true multitude of use cases that causes this problem in the first place.
I agree “serverless” isn’t a panopoly solution to everything, but externalizing and democratizing all infra tools to cloud vendors is a major transformative solution that is worth a lot of money and solves a lot of problems.
For example, if someone like you had any leadership influence in my company’s infra org, I’d be trying to spend money on cloud tools just to get away from your authority as fast as I could, and I would be 1000% correct to feel that way and would end up with much healthier tool solutions and less blockages on use-case-specific delivery requirements by doing a giant spin move around your myopic perspective.
This has nothing to do with technology or tooling, this is your team having the autonomy to do what works for you.
At my current client, my team is dependent on the "Platform Team" for _everything_ that isn't directly related to the code we're writing. Do we need a new Azure queue? That's a Platform ticket in JIRA. Do we need a new tool in our CI/CD pipeline? That's a Platform ticket in JIRA. Do we have a build failing for something other than tests? That's a Platform ticket in JIRA. Do we have an issue in production? We need to engage with Cloud Ops because we don't have access to anything in prod.
Plus, it takes six weeks to get a feature from development to production because horrible workflow and manual processes.
Plus, development teams are prevented from implementing anything without the approval of Solution Architects who are too far removed from the business to understand what's needed and too far removed from the technology to be able to design a working solution.
It's all fucking horrible, and yet we're using all the modern cloud tooling: Spring Boot, Azure, containers, K8S, Gitlab pipelines.
In contrast, a couple clients ago my team had full autonomy to do what we believed was the right solution. Stories would often be groomed, implemented, and delivered into production all within a 2-week sprint. No muss no fuss. When they weren't, it was normally due to us having to wait on another team to deliver their work.
I'm sure you will be able to setup proper pipelines on new projects within a 24h time frame and be brilliant at finding root causes by reading the stack trace and associated sources in the blink of an eye when you will be a senior hacker, I'm not worried about that!
Anyway, I think your comment proves that once we decide to invest in something, our brain does everything it can to justify that choice, Human nature really :)
I don’t understand your comment. It seems like you think it carries a rhetorical punch to (I guess) suggest that in the cloud version we’d still be slow and make mistakes? The writing is so unclear it’s hard to tell. But if that is what you’re saying, I think you deeply missed the point.
Yes of course my team won’t solve bugs instantly just because of cloud tools and of course we will make mistakes implementing bad infra designs, especially early on as we are still gaining experience with efficient and cheap cloud patterns. Nobody said otherwise and my comment before has no connection to anything like that.
Rather, if we have control over our own infra, we can adaptively fix those issues for ourselves without getting delays and philosophical or gatekeeper arguments from central data center admins. We would have autonomy to change our own deployment system, provision more compute resources, generalize something to work with GPUs, test out different delivery and runtime frameworks, create our own ingress failure robustness patterns, all without blockages and mandated central infra bottle necks.
To create such a configurable and adaptable API-driven portal into bare metal or a data center yourself is way too slow and costly, only the largest companies can do it, and then they turned it around and made it into cloud vendor options.
When medium sized companies try to do it, the most common outcome is a ton of exceedingly arrogant SRE types will create piss poor tooling that barely works for maybe the top 2-3 use cases and then will whine and complain that they can’t support generalizing it for the long tail of other use cases, and because they have political power as the gorilla in the room sitting on top of the only interface to the data center, nobody can argue with them, and you get a burnout mill with constant turnover on all the other teams who have to do SRE’s job for them while listening to the arrogance about what a rip off cloud tools are.
I don’t understand your comment. It seems like you think "anyone talking about the rip off cloud tools" are "exceedingly arrogant SRE types creating piss poor tooling that barely works, whining and complaining that they can’t support generalizing it, sitting in actual data centers, with a burnout mill with constant turnover on all the other teams". But apart from that, I'm sure you're a lovely person to discuss with ;) Anyway, I was just saying that senior hackers have more options than you seem to have, also that we tend to find excuses to justify our investments once we are locked in them, it's perfectly natural and nothing to be ashamed of.
>"...compared to the nightmare of shipping code on internally created developer tool & deployment platforms..."
I feel sorry that it worked for you this way (well rather it does not work). I have totally opposite experience. It takes single click of a button for me to build, test, deploy a code change using on premises and rented hardware.
I did cloud as well when it was mandated by client and it did not feel any simpler, faster etc. More complex and way more expensive it sure was but since it was not me paying the money I could safely ignore that part (not for my own projects and servers though as I do count my money).
The fun thing is that many C-people are fed the VC kool-aid about scalability. They want everything to br scalable because they want to cash in. Except that not many companies reach the level where you need to scale your tech stack "exponentially". And you usually don't want to scale the dumb prototype that ended up in prod anyway. The trap here is that those systems will allow you just that in a totally unauditable way... Good luck replacing parts of those systems as well when everything is interconnected...
"You deploy, get an error message, and login to CloudWatch to see what actually happened - it’s all batch-driven, just like the bad old days, so progress is slow."
Even without cloud APIs and containers and whatever, I find there's a creeping thing that happens here which has been helpful to become consciously aware of and intentional about fighting back on.
Basically, the thing is that wrappers become interfaces. You write tool A, and then tool B, and then some kind of script C which calls the two tools and produces a result. Then someone says "dang, so convenient", and sets up a Jenkins/Rundeck/Airflow/whatever job that runs script C against some source of inputs automatically— maybe there's even a parameterized build capability where arbitrary inputs may be submitted to it on demand.
Before you know it, the fact that the original tools and script exist has been forgotten, because everyone's interface to this functionality is a handy webpage. And when someone wants to add more features, they don't even go back and change the script, they end up just adding it directly into the job definition. Before you know it, the full scope of what is being done on "the system" isn't even really available any more to be installed, invoked, and tinkered with locally, not in a meaningful way, anyway.
It's not even necessarily all bad, but when I see this happening in a domain where I have the awareness of the underlying tools (or I may even have written or contributed to them), I do try to help demystify them by writing up docs on how to do the local setup, helping others get going with it, etc.
You can solve anything by adding a layer of abstraction - except having too many layers of abstraction.
"Serverless" always reminds me of CGI/fastcgi, which was still a thing the last time I wrote web apps for money. It had the advantage of "scaling to zero" and could run a huge number of low-usage apps on a single 2000-era server.
> I've mentioned as much to several C-suite executives and professional venture investors I know who tend to wax philosophical about "exponential growth in cloud services and big data" and whatnot. Invariably, their response has been to tell me that I just don't get it, or that I make no sense. They give me puzzled looks and maybe even get a bit exasperated with me.
One word: Worldcom
I still remember coming up to 2000 sitting in a meeting room of an equipment provider I recently joined looking at projections of market growth, competitors and our own sales and profitability. Shortly afterwards that part of the business was sold, I remained with the other non internet part and we sailed through the .com crash unscathed.
A lot of what AWS offers makes no sense. A lot what some AWS customer's teams are building is overly complex and a horror show to maintain. But there is also value: Maintained images, backups, on demand resources and specialized resources. We hit one of the inflection points in Moore's Law and that drives us into horizontal scaling and specialized hardware.
Like with Worldcom and the fantastic internet growth story back then there is a kernel of truth and value. Find people that understand it and be glad others are playing around and subsidizing the investments in the core.
I love the scalability "argument". The capabilities of a single thread of processing in a modern x86 CPU are underestimated by several orders of magnitude in most shops. The unfortunate part is that the more you chase the scalability with the cloud bullshit, the more elusive it becomes. If you want to push tens of millions of transactions per second, you need a fast core being fed data in the most optimal way possible. Going inwards and reducing your latency per transaction is the best option for complex, synchronous problem domains. Going outwards is only going to speed up if you can truly decouple everything in the time domain.
Judging by the occasional post on HN, even a lot of experienced software engineers (the "experienced" being my take) seem to have no good handle on what a single thread and a single machine should be capable of handling, what the latency should look like and what it should cost as a result. The posts that I mean are about "look how we handled the load for a moderately active web-app which is 90% cacheable with only five nodes in our k8s cluster".
I really don't know why that is, my guess would be that few people have really built something beginning to end with the most boring tech possible (start out with PHP+MySQL like everyone did 15 years ago). Or they always operate at levels of abstraction where everything is slow, so they have simply gotten used to it, like their text-editor not running smooth unless it's on an i9, because the text-editor now is a pile of abstractions running on top of Electron when vim was able to run smooth decades ago. It's sad and both an opportunity at the same time, because you can be the one with the correct gut feeling of "This should really be doable with a single machine, if not we're doing something fundamentally wrong".
I think you're right on the money with people having gotten used
to it. Once I truly started harnessing the power of Vim combined
with shell scripts and terminal multiplexers, my patience for
many other programs and tasks decreased even further.
We have the computing power to run complex physical simulations
or AI training sequences on a normal home computer, but for some
reason use programs that take 100 times longer to start than
old software despite not having more features, and websites that
sometimes even come with loading screens. Electron isn't even as
bad or slow as many people think, but somehow developers still
manage to throttle it so hard that I might as well just use a
website instead.
As someone who is just starting with nodejs and web development
I find a lot of the tech feels nice but sometimes also
unnecessarily abstract. Sure, it tends to make a lot of the code
very elegant and simple, but every additional framework adds
another guide and documentation to look at, another config file
that has to be correctly referenced by an already existing one so
npm knows what to do, another couple of seconds of build time
and another source of bugs and build problems. Then of course
you need that trendy package for easier imports maintenance -
which IDEs automatically handled in the past, but now we gotta
use an editor running in a hidden web browser which started from
scratch in terms of features but is just as slow in return.
Once I get our current product in a good spot WRT maintainability and sales pipeline, I am planning to spend some time (~6 months) looking at developing ultra-low latency developer tooling. I feel like I can deliver these experiences through a browser using some clever tricks.
Going through my current project has really worn me out with regard to tolerating UX slowdowns in developer tools. I am getting so incredibly tired of how shitty visual studio performs on a threadripper with effectively infinity ram. I've had to turn off all of the nice features to get our codebase to be even remotely tolerable. And, dont get me wrong. This is not a plea to Microsoft to fix VS. I think everyone involved can agree that fundamentally VS will be unfixable WRT delivering what I would consider an "ultra-low" latency user experience (mouse/keyboard inputs show on display in <5ms). Any "fixing" is basically a complete rewrite with this new objective held in the highest regard.
Current developer experiences are like bad ergonomics for your brain. This madness needs to end. Our computers are more than capable. If you don't believe me, go play Overwatch or CS:GO. At least 2 independently-wealthy development organizations have figured it out. Why stop there?
That's not possible, at least if you measure the full time from
actuating the key to the screen showing the updated content. The
record holder in that regard is still the Apple II with 30ms[0].
But I agree, modern software should react much faster. It's kind
of weird how on one hand we have hardware that can run VR with
120Hz and extremely low input latency, and at the same time some
programs that only need to display and change a couple bytes in
a text file need 4 seconds to even start.
I am referring mostly to the latency that I do have control over in software. I.e. time between user input event available for processing and view state response to user.
Yup. At the last place I worked me and a colleague would joke that given the traffic our ecommerce platform actually had, we should be able to run the whole thing on a single Raspberry PI. I think he even ran some numbers on it. What we actually had was two containerized app servers and a separate large RDS instance and SES, SQS, SNS, ELB and all the other bits and pieces AWS buy in gets you. The cloud bills for the traffic we handled were ridiculous.
> Or they always operate at levels of abstraction where everything is slow, so they have simply gotten used to it
That's a lot of it, I believe. One startup I dealt with was honestly proud of how they were scaling up to hundreds of requests per second with only a few dozen AWS VMs to handle it.
I'm old enough to know how ludicrous those numbers are, having built systems handling more load on a single desktop machine with 90s hardware.
But I've come to realize some of the younger developers today have actually never built any server implementation outside of AWS! So the perspective isn't there.
While I think this article is a great discussion about the increasing complexity of software and how the trends driving that complexity are marketed, but I think any article discussing these trends without also talking about the organizational structures of the teams delivering and running the code miss the point a bit.
When you have an organization with hundreds or thousands of developers delivering code that works together to support a business there is some types of complexity that are inescapable. So the question is “where do you want the interfaces between people and teams?”
SOA and microservices push that interface to the API layer which drastically increases the complexity of the code, but is a logical place to put the complexity if you consider the organization, people, and code a single system.
I think you're correct but I also think the article is correct.
At scale there are levels of inescapable complexity. There are a lot of tools that help manage this complexity. However too many in smaller teams emulate bigger companies and inject wasted complexity by using tools that solve problems they don't have.
Most organizations don't have hundreds or thousands of developers working on some coherent product.
In fact this is quite seldom. So all the tech that solves problems that occur in such constellation should be quite exotic.
But it isn't. The reason is that computing "on a mainframe" (the cloud) can be sold orders of magnitude more expensive. The rest is marketing of the mainframe owners.
Depends on how you define product, there are a large number of tech enabled Fortune 500s that do not sell a tech product but have built an integrated internal product for their operations that do need that scale. Airlines for example.
But yes, I agree that most companies jump to this level of complexity too quickly.
Airlines? How many billions of flights and passengers do you have to manage daily as an airline? Don't forget, computers are fast.
Also I think airlines do just fine on core system form the 80'ies until now. They also don't tend to have software development teams with "hundreds or thousands" of people to my knowledge. So maybe not a good example.
There is only a hand full of companies globally really in need of all that "serverless" stuff as I see it.
I am intimately familiar with the software and IT organizations of multiple airlines. You are wrong, they have large distributed software development teams building very complex software to run every part of the airline’s operations.
Some buy off the shelf, others buy components off the shelf and focus on custom where they feel it gives them an advantage, and others have built the whole thing themselves.
Airlines have been running their own data centers and fully understand the costs associated with managing their own infrastructure and many have decided to migrate to the cloud because they have decided it is worth outsourcing that function, it is not a key competency and they cannot keep up with the pace of innovation in the IaaS market. If they look at their IT as a complete system, the extra costs of the cloud are offset in a number of key areas, including increasing their ability to scale.
You mention of the shelf software, what do you think the organizations that build the COTS stuff (like Sabre) in this industry look like? Where do you think it is run? They’re all moving to microservices and cloud hosting too, including serverless where they think it makes sense (and a lot of times when it doesn’t, just like everyone else).
The smart ones (who understand Conway’s Law) are making these moves because they are enablers of smaller, more independent teams which we know are better at delivering software. The really smart ones know they’re paying the price in increased complexity the comes with a distributed system, but are willing to pay it because that cost is lower than the cost of the collaboration necessary to build a monolith of that size.
Logistics companies are one of the most interesting intersections of software and the real world. Most organizations are making a mistake by focusing on scaling their software too early or unnecessarily. Airlines and logistics are not, they have business problems that their existing software cannot solve in areas the COTS options do not fit in their operations.
TFA is talking about serverless, not cloud computing in general (e.g. rent a VM, run stuff in it).
I don't love that I pay aws to run my open source domain(s), but the extent of their job is: keep my VM up, provide me with outbound email that works, offer backups. For now, I can live with that.
> Cloud is an expensive, overly complicated, lock-in boondoggle.
Only if you go with companies that make it such.
Renting web hosting on someone's collection of rack-mounted systems running some well-known OS is Cloud, and there's nothing inherently bad about that, unless you've defined Cloud to mean "Only The Bad Stuff About Using Computers Other People Own". You can run things like they were run in the 1990s, complete with Apache and mod_perl CGI and MariaDB swapped in for MySQL and everything held together with shell scripts and run via ssh and ftp. No magic, no lock-in, no complexity beyond what every other serious general-purpose computer system is these days.
I agree that Big Cloud Providers layer on pricing models and whatever middle-management-ware they can come up with to obfuscate the underlying mechanics, but if you're against that, say so. Don't catch everything in your blast radius because it reduces the overall impact.
> I hear this a lot in interviews and it's a clear signal that you haven't worked on anything big enough yet.
Quite a bold wager, and totally incorrect. I hope you're not presumptive and rude to all of your candidates.
FWIW, I'm a senior engineer at a large tech company where I started pre-IPO and have had pretty great career growth. I've been on a variety of teams dealing with most of the levels of our stack: eventually consistent data models that are active-active, k8s, frontend, tearing down monoliths, etc.
I'm fine with microservices for medium to large orgs, and I love Kubernetes.
Lambdas and cloud are expensive lock-in, and holding this view doesn't mean you hate distributed systems or complexity. You can staff your own platform teams and manage your own capabilities. Imaging machines, upgrading, failing over - it's all automatable.
The excuse I hear is that companies don't want to build things that aren't related to their core competency. But I'd argue your software stack fundamentally relies on the environment it lives within and that ceding control poses a non-zero risk.
I'm incredibly fearful of the future where all software runs within AWS or Azure. We'll all be paying taxes to these companies and we won't be able to run or replicate similar environments on our own. It'll be difficult for new engineers to learn as our field will effectively become a "trade".
Can you imagine the world where it's expected that everything is composed of SQS and Lambda and other bullshit?
"I'd argue your software stack fundamentally relies on the environment it lives within and that ceding control poses a non-zero risk"
This is factually correct, but more nuanced than "just build it in house". Many large orgs externalise this risk by relying on AWS / Azure / Google. Nobody ever got fired for picking IBM is more relevant now than ever, and it is because the largest companjes on the planet have invested huge amounts of time and money solving these problems.
I think you misunderstood me at some point, because nowhere did I advocate that everything should be a lambda. Or that everything should be in the cloud.
The point I am making is that a lot of people I speak to are afraid of complexity and lock in, when they should be afraid of never being able to grow past what they are comfortable with.
To your point about this becoming a trade... Brace yourself, I would wager most development positions will be obsolete in 10-20 years as AI assisted infrastructure and development hits full stride. Especially dev ops and front end.
> You deploy individual stateless functions. But not inside a Java monolith, that is old, but on top of a distributed system. You would have been laughed out of the door if you had proposed that in 2000, and you should be laughed out of the door right now, but such is the power of marketing.
I’m sorry but this absurd and blatantly ignores the reasons why people choose serverless. No mention of scale to zero, virtually infinite auto scaling, automated deploys, arbitrary runtimes, ability to spin up code in multiple regions around the world with zero operational overhead, the list goes on...
These are non-trivial things. You’d have to pay engineers to build the same systems provided by serverless if you want smooth deployments and high availability. Also, imagine if you wanted to start operations outside of the continental United States. You’d need to find a data center provider, buy machines, pay someone to install them (and hope they do it correctly), and then, when ever you have a technical problem, you need to email the datacenter team to have someone plug in a KVM with 5 seconds of lag so your engineers can debug, and then you have to pay admins to constantly maintain the servers, and consistently pay for their rack space even if they aren’t being fully utilized. None of this is a problem on a fully managed serverless platform.
This article is just click bait capitalizing on FUD in the software community. Serverless will be a part of the future of software infra whether luddites are mad about it or not
Disclaimer: I work on serverless products so I am biased
Why is the alternative to serverless, colocating servers in a DC?
The alternative is renting VMs from those same clouds. It works fine and is cheaper when you work on software that actually has users. It also doesn't result in the cloud provider cramming a crappy under-baked programming model down your throat.
Per request pricing and "scaling to zero" is just old fashioned unbundling. You are selling a smaller quantity of a product at a higher unit price and putting a bit of lipstick on that pig to convince people they are not getting hosed.
> Why is the alternative to serverless, colocating servers in a DC?
Because that’s what the author was comparing it to. Yes you can rent vms, and that works for many workloads. However serverless platforms provide other benefits like deployments, load balancing, and auto scaling that you’d need to setup yourself on raw vms. Plus, with services like AppEngine Flex you can run serverless on vms to cut costs if you have consistent traffic.
> Per request pricing and "scaling to zero" is just old fashioned unbundling. You are selling a smaller quantity of a product at a higher unit price and putting a bit of lipstick on that pig to convince people they are not getting hosed.
It really depends on your problem. Sometimes you might have a very low QPS service (like maybe a few QPH) and running a whole vm would be overkill. Some business have really bursty traffic and maybe some regions won’t get any traffic at all during the night but they still want to deliver low latency service to the people living on that side of the globe. Additionally, these are just reasons to serverless for scale to zero and ignores other benefits, like reduced operational costs, that business can use to justify paying more for compute time.
There are lots of reasons to use and not use serverless for compute. I don’t think it is the solution for everyone and everything, but I do think the hate and FUD directed towards it is unwarranted and uniformed.
Disclaimer: I work for Google, opinions are my own
I agree. I am working on a devops team that has assembled a half decent CI/CD infrastructure for widespread use at our company of 600 devs and thousands of apps. We are on our second iteration, which is even leaner and more invisible to the devs. However, we are seeing a move to serverless would be a net gain for all involved, specifically the scale to zero and not caring about the container and kubernetes configuration.
I think "scaling from zero" is actually the more interesting part of server-less. With AWS for example, it's super trivial to get a lambda running: no EC2 instance size to choose or elastic IP address to set up. You don't even have to choose a back-end framework, or set up monitoring to make sure your system stays available. You can just start writing code directly in the console the very first hour you start your project if you want, and already have a functioning API.
When you're working on your MVP, you can skip dev ops completely and get your infrastructure cost very close to zero. Then if you actually do end up building a product which is fortunate enough to have to worry about scale, you can work on cost optimization. I think this is a very relevant and valid use case for server-less.
I think you are miss understanding my comment. I was responding to the author’s comparison of serverless and the cloud to collocated servers in a DC, I never argued that other services in the cloud are bad/shouldn’t be used/inferior to serverless in all cases.
I suppose this wasn’t clear because another commenter had a similar question/response
> You’d have to pay engineers to build the same systems provided by serverless if you want smooth deployments and high availability. Also, imagine if you wanted to start operations outside of the continental United States. You’d need to find a data center provider, buy machines, pay someone to install them (and hope they do it correctly), and then, when ever you have a technical problem, you need to email the datacenter team to have someone plug in a KVM with 5 seconds of lag so your engineers can debug, and then you have to pay admins to constantly maintain the servers, and consistently pay for their rack space even if they aren’t being fully utilized. None of this is a problem on a fully managed serverless platform.
What are you even talking about dude? You don't have to build a rack in a colo to run a non-Serverless app. There's this thing called The Cloud now, and Virtual Machines, and other fancy new technologies from 15 years ago.
"Serverless" as in Lambda is a joke. "Serverless" as in Fargate is legitimately great at lowering ops overhead. But "zero operational overhead" is absolute bullshit. Even Lambda has operational overhead.
Serverless is just the abstraction of a function into a compute unit. There's like 20 other things you need for a functioning product, many of which literally cannot be replaced by Serverless-anything. And the severe limitations of many Serverless technologies make it very unwieldy to use.
Give it 10 years and it may morph into something more like a monolith whose functions become "Serverless", like a Beowulf cluster where forks are executed on remote nodes and there's some ugly glue to tie it all together. That seems a lot more seamless and useful than trying to architect 100 tiny pieces together with crappy custom integrations everywhere.
> What are you even talking about dude? You don't have to build a rack in a colo to run a non-Serverless app. There's this thing called The Cloud now, and Virtual Machines, and other fancy new technologies from 15 years ago
I think you are miss understanding my comment. I was comparing serverless to colo because that’s essentially what the author was doing. I agree that there are plenty of other cloud compute solutions that work for many workloads.
> There's like 20 other things you need for a functioning product, many of which literally cannot be replaced by Serverless-anything
I totally agree. Serverless is usually just a piece of infrastructure that is combined with other resources like databases, batch clusters, blob storage, vms, etc. I don’t think you’d be able to create a very interesting app using stateless functions alone, though I could be proven wrong ;)
From my anecdotal evidence, people are attracted to serverless because they spend way to much on aws and they hope that by turning things off they'll spend less.
Not many people have the problems solved by a serverless architecture (or they need more freedom), which is why it's not as common as just running your own long running instances.
Not denying your experience, as it's probably true that the folks you've seen pursuing serverless may be misguided, but I want to contest your last point where you say "not many people" would benefit from serverless.
I would say that "literally any single person" benefits from serverless if they don't have the skills, knowledge, or time to handle the server/infrastructure responsibilities that come from running software.
If I'm a one man development shop with a sufficient budget, I'll probably find it far more productive when using serverless architecture to host my business logic than not.
Serverless comes with its own problems though - a less seamless dev experience and cold starts plus you have to host your data on their choice of data store etc.
I totally see the advantages, but how many benefit from that? Most companies don't need to deploy on servers world-wide, and don't need infinite scaling.
And cloud debugging has its issues too: I remember endless fudging with print statements, deploying once more and waiting for the changes to propagate in a AWS lambda handler, because the bloody thing simply wouldn't run locally.
> I totally see the advantages, but how many benefit from that? Most companies don't need to deploy on servers world-wide, and don't need infinite scaling.
That’s a good question. I think serverless has a lot to offer, global compute, and infinite scaling are a subset of features. I think it really comes down to your specific workload—it’s definitely not a be all end all silver bullet
One example I can think of I mentioned in another comment in this article which I have pasted below:
the first company I worked at was very small, under 30 people, sales included, and I was hired on as an SA. If they were using serverless, they would not have needed to hire me.
Deployments and system upgrades were very toilsome. The dev lead would spend a few hours deploying new code and we’d only do it around midnight in case there was a problem (our customers were almost entirely US based). I spent a lot of my time automating OS upgrades on the web servers behind our proxy. All of this work would be unnecessary for that small company if they were using serverless.
Additionally, as mentioned before, the majority of their traffic was during the day so they could have scaled down automatically at night, cutting costs. Would this be cheaper than running machines 24/7? I’m not sure, but I could see why you’d be interested in at least looking into it
Ah, the bullshit. When we showed GCS engineers our hadoop workload, their eyes went really big and told us that they don't have that kind of capacity in a single AZ.
Then you are working at hyperscale yourselves already, and there are maybe 50 organisations like you out there.
Maybe.
Right now I don't think that there is a single telco that has the scale to break GCP, or a single manufacturing company (exempt Tesla) or a single energy company. Or bank.
I agree that pharma, social media and streaming may do this. But since Netflix is on AWS I am doubtful of the last.
I worked for one of the top-10 ecommerce shops (by transaction value) in the world. It was interesting for me to find out that the aggregate CPU power of our hadoop cluster seems comparable to what an F1 teams uses to simulate air dynamics to tune their cars.
There's a big world out there, there are probably tons of big computer users that we are largely unaware of.
Plenty of scientific computing applications fall over on the "cloud", too.
The LHC produces somewhere between 20-30 petabytes/year[0][1] (at a rate of about 200 Megabyte/second[2] after ignoring many "uninteresting" events)
The LHC is a big name, but there are many other applications in disciplines like physics, genomics, meteorology etc for whom "just use the cloud" is a terrible suggestion.
I think that alphafold may indicate that biotech isn't as disjoint from cloud solutions as many suppose. Super computing is also getting eaten as grid solutions go into hyperscale, but yes ..I find you LHC and the mega telescopes.
I did big data at one of the biggest European telcos. The only thing that kept any of it on prem was capex Vs opex (and we had sunk capex so that wasn't coming back into the budget) AT&T claimed their Hadoop installation was ten times bigger than ours, but it was still will within the parameters of any of the hyperscalers.
> or a single manufacturing company (exempt Tesla)
Tesla is TINY.
Try things like Siemens.
EDIT Besides, the point is not the whole of AWS of GCS; it's the about availability zones vs singular datacentres. Distributed computing doesn't really like to throw data even within the same AZ - in-DC will always be faster.
AZ and the modern idea of distribution suit certain things - streaming, for example, benefits from it, but not everything.
This is stupid. If you don’t recognize the benefits of serverless of the capabilities of this you shouldn’t be writing about it. Go back to retirement if you remember the 70s. Maybe you would have laughed people out the door in 2000 but that doesn’t mean you by any means are right. Serverless doesn’t add complexity but provides a service that manages everything for you without having to manage scaling manually. This provides ease of use across teams and also consistency which is not found in self managed servers across any size of organization.
There might be a useful rebuttal buried in your reply, but it's going to be killed by downvotes because you obscured it with ageism and arrogant dismissal.
I’ll admit I was three sheets to the wind when I originally commented on this. Definitely could have left the ignorant low blows out.
That being said, I think what gets me most about this is it seems to overlook the fact that serverless exists to remove the effort and time and cost spent on maintaining servers and aims to allow developers to focus purely on providing value. There’s no doubt you pay more to have it managed by someone else, but at the same time costs are optimized as best as possible by only charging on a per use basis rather than having servers running all or most of the time to host services.
Massive monoliths are very difficult to maintain and often lack resiliency based on the often coupled nature of them. Microservices aim to solve this and also provide more ownership for teams. They also allow for flexibility in hosting as well.
My company is migrating a huge monolith to microservices. One of the proposed benefits is that it makes end to end testing easier because each service only interacts with a couple other services, so each service will be easier to write integration tests for. But in the monolith, those services communicate over interfaces, and are currently tested as unit tests! So we're replacing unit tests with integration tests and calling that a win because our integration test coverage is higher.
If one part of the application needs to be changed frequently and/or is used by other applications, then it makes sense to separate it as a microservice. However, for other parts, I think it might be better to stay as a monolith since when you have a network in your calls, things get messier.
Networks are unreliable even if it is your local area network. In my career I saw some unrelated network equipment was sending a RST packet to load balancer with application servers IP address and resulting in service unavaiable errors. It took us weeks to find that out since it was happening very rarely and there were many layers between load balancer and application servers.
Network means slowness, connection errors, unknown results, etc all of which should be handled carefully.
Not sure if rate of change truly matters. In my opinion, microservices make most sense when dealing with multiple team. It's much easier to update core services when everyone depends on it through the network, rather than requiring people to redeploy their stuff. If you're a single team operation, it's not really a problem if all your stuff runs in a single process. It may even benefit you in terms of performance and scalability, to be honest.
Rate of change may matter when your monolith application takes too long to deploy (i.e a big war application with lots of initializations at start), although rolling updates may help here. It will also be easier to rollback if things go wrong.
On the other hand, I agree with your point of multiple teams. I tried to mean same thing with "used by other applications"
How long are you talking? I can't think of a process I've ever worked on that took more than a few seconds to start. As you say, if you have any sort of rolling deployment mechanism the start time of your application is somewhat irrelevant. Unless your application takes a massive time to start, it seems like the wrong thing to optimise for. Surely you only deploy things a few times a day, if that, and hopefully most deployments don't require rollbacks.
Also, with the current network speeds and memory availability, it's hard to imagine the war size to be a bottleneck in web development.
Depending on other services via a well defined network interface is good for decoupling teams. So it depends on the number of employees you have coding for the same platform, rather than on clients or anything like that.
> How long are you talking?
There were some Spring applications that took more than 5 minutes to start in my old job. However, things are most probably improved now with lazy initializations of beans and maybe developers of that applications could have done better at that time. I don't know it is still the case now.
> Also, with the current network speeds and memory availability, it's hard to imagine the war size to be a bottleneck in web development.
Network speed was not the problem, decompression / expansion of the application was. There were war files about 100-150 MB size. Also I saw one with size of 1.5GB however it was an exception. These may or may not be a concern depending on your CPU utilization, disk speed, memory, etc.
Most decisions depends on environment. For some environments, as you said, deployments are not a concern for microservices, for some they are.
If networking is not a problem, why not deploy it already decompressed? I'm assuming the 5 minutes time included decompressing it? It seems a bit weird to take so much time to allocate some objects in memory and possibly establish a few connections.
Anyway, can you share which kind of environment even a 5 minute deployment would be a problem? Most places I worked at it took a lot more than 5 minutes to actually implement the change that's going to production, so deployment times were pretty much irrelevant overall.
OP here. The move is definitely needed in our case as it's to the point where IntelliJ won't even load anymore. But I had to laugh when improved testability was brought up as one of the benefits.
> XML, however, was universally rejected in favour of things like JSON, Yaml, HCL, Toml - all free of structure, with zero indication whether a computer would find your prose gibberish or the next Shakespeare play until you actually pushed your code to some test cluster.
There are a few fundamental differences between these languages, but it's just plain wrong to say that any of them are not "structured." Did OP mean "schema-less" instead? Because the only fundamental difference at that level is that XML has a built-in mechanism for metadata referring to a file's own schema, while in JSON (and I believe the others, even though I've used them less) the schema is not linked from within the file itself. You can still verify the schema of these languages, though, just as you could with XML. And once you've verified the schema, local code can do stricter validation to make sure things hang together properly, that values are within limits, and so on. After that, I expect modern tools like Terraform and CDK to check with the target platform that the scheduled change can be applied before trying to change anything.
That said, OP has several valid points:
- Many projects try to make a microservices architecture long before it's needed, even in projects where it might never be needed. I suspect some people think microservices are actually easier to maintain, since they are smaller pieces, but only extremely trivial or shoddy microservices will be easier to maintain than a corresponding monolith, because many things which are well-understood in a local setting are much harder to do in a highly distributed setting, simply because the infrastructure is now a bazillion separately managed pieces.
- The feedback loop can be abysmal. Without any easy alternatives to do "local cloud" deployments, having to wait for several minutes for feedback about a tiny change is frustrating in the extreme.
I'd say that JSON is "structured", and probably HCL as far as I know, but the others have problems.
I haven't worked with an API that uses XML in a long time, but does it express the difference between "0123" and 123 well? I think that it either doesn't, or you add some really awkward schema language with XML namespaces ...
I just translated my .travis.yml to JSON, and I learned that
deploy:
on:
branch: master
gets translated to {"deploy": {"true": {"branch": "master"}}} because "on" is a synonym for true. I'd call that unstructured. It also has problems with keys that start with -, etc.
Also, all of the languages are often used with say Go templates, which makes them more unstructured. They're no longer statically parsable, which is "table stakes" for doing anything else (validation or static analysis).
I took some pains to make shell statically parsable [2], and I think it's now a better foundation than these config languages, per my other comment here: https://news.ycombinator.com/item?id=25483270
I think HCL is probably a lot better, but it also seems to be the least used! It's structured, and has its own mechanism for expressing duplication that's not the equivalent of a 70's-style #include.
> I haven't worked with an API that uses XML in a long time, but does it express the difference between "0123" and 123 well? I think that it either doesn't, or you add some really awkward schema language with XML namespaces ...
> There are a few fundamental differences between these languages, but it's just plain wrong to say that any of them are not "structured." Did OP mean "schema-less" instead?
I read him as referencing what people would have called "structured programming" in the 1980s, and pointing out that we have these settings/metadata/control files in languages that have no capacity for "structure" in the sense that, say, Pascal or C would have over BASIC or assembly, that is, thin-to-no factoring or abstraction capacity. Hence the contrast with "Configuration as Code."
Now, depending on your personality and experience, you might find the idea of using a complete language as config an opportunity for someone to do something non-obvious or horrifying, and the closer to the "settings" end of the setting/metadata/control spectrum you're on the more plausible that position is. When you get to a level of complexity that's the other end I'm not so sure and horizontally scaling systems might well be arriving there.
> the only fundamental difference at that level is that XML has a built-in mechanism for metadata referring to a file's own schema, while in JSON (and I believe the others, even though I've used them less) the schema is not linked from within the file itself
As long as you're using plain DTDs for your schema, yes. Like early XML-based config languages such as idk JSPs/web-app and Jakarta (née J2EE) deployment descriptors used and still use. But later XML configs use XML Schema and namespaces, which are similarly detached from an instance document as eg JSON Schema and the others you mention [1].
(1) Languages have gotten worse as distributed back ends have gotten more powerful. The IBM JCL and XML references in this post were good!
(2) Workloads that currently run on such back ends could run on a single computer, or at least with many fewer resources. This is one reason I got into shell in the first place! I wrote some shell scripts that saturated 32 cores instead of using distributed systems. In other words I try to avoid the "COST" or "parallelizing your overhead" problem.
We're paying a huge productivity tax and in many cases not reaping the rewards. I think a better better UNIX SHELL can help in the following ways:
(1) We need to bring the interactivity of Unix back to distributed systems. We're still in the days of "IBM Job Control Language" with Kubernetes config and similar kinds of "declarative cloud configuration". We need a flexible and efficient analogue of Bourne shell.
(2) Unix shell is already how you set up local development environments: Docker embeds shell; virtualenv changes your shell state, Ruby's bundler, OCaml's opam switch, etc. We need to evolve this into first class and polyglot environments specified in shell.
Debugging distributed systems locally could be the norm, but it's not.
The local topology should simply be a configured variant of the distributed topology, but it's not. I used to do this at Google with a trick of generating a shell script with BCL (Borg Config Language).
(3) A better shell should be able express configurations that evaluate to JSON, and also statically validate them before pushing, to solve this problem.
XML, however, was universally rejected in favour of things like JSON, Yaml, HCL, Toml - all free of structure, with zero indication whether a computer would find your prose gibberish or the next Shakespeare play until you actually pushed your code to some test cluster.
I want https://www.oilshell.org/ to go in this direction, and there is already some progress. (Feel free to contact me if this resonates with you.) And I have a draft of a blog post about shell, distributed systems, and languages that I need to publish, based on these comments from a couple weeks ago:
Those comments talk about the other problem with the cloud: that it locks you in to APIs! We need the POSIX of distributed systems. Kubernetes was trying to do that, but it's not good enough.
Shell can solve this problem more economically: it expresses UNPORTABLE glue code to leave your application PORTABLE. I did that with Oil's continuous builds, and the Unix-y gg FaaS framework also appears to do that in a pretty nice way.
Yes, it is one of the most entertaining and insightful articles I have read on computing and programming language for a long time.
All the best for your new Oil shell, and I agree that we need to have similar to POSIX standard for distributed systems but it must put local-first software as the first class citizen [1].
Personally I think the local-first software cannot happen soon enough given that what we have endured for the past 30 years with regard to inefficiency and drudgery of the web based applications.
Just wondering about your thoughts on SDLang [2] as the type alternative to JSON, YAML etc, and lightweight alternative to XML?
So I think Oil will be very familiar to SDLang users, BUT it has programmability, which you need in cluster configuration and build configuration, e.g. to express duplication.
So I would divide the space into a few categories:
Very important: With categories 1 and 2, people add template languages like Go templates on top to solve the programmability problem, which we don't want!!! We don't want to move typed data back to the stringly typed realm!
Also, I don't think the "expression style" of HCL, Nix, and BCL is great. It's good for small things, but it starts to get hard to read for large configs. I'd prefer simple for loops and if statements, basically like the Starlark language (a subset of Python used in Bazel).
Serverless functions are usually free. This is a huge boon for learning: I'm a college student with limited resources, and even spinning up a $5/mo VPS takes some thought and budgeting.
I learned web dev with free web hosts & free VPS services back when I was 13-15. Further democratization will only help newer generations explore & tinker with confidence (scale-wise and tech wise).
I'd be curious how thing would look if the cloud providers broken up horizontally. Like, if the companies that provided VM's were different from the ones who provided higher-level services. That's kind of how it was originally with Heroku and its add-ins, and AWS/Rackspace. I liked that so much better and wish it stayed that way. If you want a VM, go to Rackspace and knock yourself out. If you want a hosted thing then go to Heroku, or whichever service hosts things in the language of your choice. But the point is, you make a choice, when you sign up for whichever service, what level of the stack you want to operate in.
I can also imagine this approach having benefits as a consumer of services, but it's hard to imagine a world where vertical integration would not win over time. There's just too much of an advantage in terms of large cloud providers being able to optimize their hardware setups at data-centers to make their services cheaper, more performant, and more tightly integrated than 3rd party competitors.
I guess. But then Anthos, Arc, EKS Anywhere, they're all horizontal, and so it's odd that they are also all run by the cloud providers themselves. I'd think those services would be the next layer, run by different companies.
Still, what you said is right. I mean, heroku still exists, but the ever present complaints are that it's too expensive at scale and that it is too constraining. So, maybe the vertical is the best approach.
"Yes, we run workloads that your nephew can run on his Raspberry Pi 4, but this is the future of enterprise."
This hit home... the amount of money I've seen businesses pour down the drain in my past few years of experience is insane. Offices with less than 10 people being talked into server setups costing tens of thousands of dollars for nothing more than a NAS, recoridng camera footage, Door access, etc.
Lately, the main reason for using “serverless” (aws lambda in particular and all related wax) has primarily to do with security “compliance” at enterprises, cost of it in particular, and nothing with developer productivity or technology... aws just makes it super trivial and cheap to meet all the compliance requirements if using only this dilapidated stack.
It’s no wonder any startup with half the brain would still pick rails/ruby for actually creating something of value, while enterprise middle managers and bean counters argue about how to not write code.
> Split your codebase, split your teams, create a lot of opportunities for mediocre coders to grow into mediocre engineering managers, everybody was happy.
This is the best description of modern software development I have ever seen.
> It seems that, contrary to what everybody is saying, software developers are in plentiful and cheap supply so wasting a lot of their talent makes economic sense over the perceived cost of adapting more powerful programming systems.
> but we just throw more bodies at the problem because Gartner tells us this is the future.
This last part is the real reason for it: executives are blind morons who have no idea how to run technology-dependent companies, and their poor leadership bleeds down because hierarchical organization.
I used to blame executives because it seemed like they were making terrible decisions that everybody knew were terrible. But now I realize that the reason they're making the terrible decisions is they just have absolutely no concept of how the organization should run. They don't understand the customer, the employee, or the product, but they have generic ideas of how to make generic things, so they just do that and hope the people under them will fix everything.
So of course we're managing everything as tiny little components that get glued together. They're "Fordisming". And "Globalizing" is already here too. Many companies already have one microservice made in India, another in Romania, a third in Spain. Who cares if they barely talk? Who cares if they reinvent the wheel and use conflicting technology? It's all just a game of "reduce cost and increase profit".
The next big hotness is going to be the API-gateway-service-mesh-distributed-tracing monolith that every service will be plugged into. Developers will be given more power to turn on and off things that are connected to this monolithic service frontend. They will thus be able to have faster feedback, but nobody will be coordinating the changes they are making, so actually things will break more often due to conflicting changes.
Finally, some time after that, the developers will figure out that it would be better if, instead of depending on all the external Infrastructure/Configuration-as-Code, they should be using frameworks that have everything built into them, and have their application tell the framework what to do, so a dedicated person won't need to write it all using external tools and weird DSLs. This will persist as a sort of "Java framework for Kubernetes-native Applications", although it'll extend into all the thousands of shitty extension apps that plug all the holes in K8s. And then we'll truly have a new monolith, but a distributed one.
For all its flaws in design or philosophy, cloud computing has a sweet spot.
As an indie, lone dev, I can run my site, email and a basic web app for about 10 USD a month and, if my app kicks off, I can scale it to a million users without breaking a sweat.
To me, that is the true power of the modern cloud computing paradigm. An influx of great apps and services enabled by cheap scalable and sweet competitive cloud services.
Properly scaling a site is more than just clicking a button. Your architecture needs to have been built from the ground up in a way that actually can scale horizontally, for starters.
True. I made an assumption that the app is built that way. Of course, if te app architecture inherently unscalable, no amount of computing power or private servers is going to help.
> True. I made an assumption that the app is built that way. Of course, if te app architecture inherently unscalable, no amount of computing power or private servers is going to help.
Isn't it usually a mistake to design your app to be scalable to 10,000 users when all you may ever have is 10?
I was running multiple sites by myself. There was less encapsulation (unless you were working with someone on bsd), for sure, but I'd say the advantages are more visible in terms of security (hacking was much easier back then) than ease to deploy.
Doing your own ops has never been that hard (YMMV) and I don't think it improved much nowadays.
Scaling to n machines didn't change as well.
Deploy n instances, load balancers, scaling vertically then horizontally, db sharding.
You still have the same fundamentals.
One pro of today is having a single artifact to deploy (tons of cowboys deployments and FTP uploads in the past).
One big cons you have nowadays is learning all the latest crap $currentCompany is using and having to peel the layers of abstractions to see why things work so badly.
I kind of disagree. It's doable, but still requires maintenance. For instance, even running a simple blog off of a VPS requires you to worry about setting up an SSL certificate and keeping it renewed, staying on top of security updates, and putting together some kind of scaling solution assuming you ever want to write a post which gets to the front page of hacker news.
I think there's definitely an argument at a certain scale to just outsourcing that complexity so you can actually work on what contributes to your unique value proposition.
Your examples are poor. LetsEncrypt and certbot automated cert renewals a long time ago. If you’re still having to manually renew your SSL certs, you’ve picked a crappy vendor. Security updates are also as simple as a cron job that runs your favorite package manager update command. You want to run those commands on multiple machines? Use the several different FOSS orchestration platforms that make it easy for you to deploy and configure your distributed services. Sure, these things take some initial set up, but it’s nothing that can’t be accomplished in a few business days for a start-up.
Wait until you see the "data transfer" costs for what is basically or literally free from the bare metal/colo hosts. $10 is too much for what you're describing. You could be paying $0-$3.
Most users of the top public clouds are paying 2-10x more than they should just because using the public cloud "is what you do".
This is clearly a surface opinion without a great deal of experience in leveraging the numerous benefits of cloud and serverless.
Complexity can be managed if you start out thinking about. The biggest problem with the cloud is ad-hoc. If you go in just building shit without a strategy and guardrails, it's going to cost a lot of money and end up in pretty bad shape.
> Split your codebase, split your teams, create a lot of opportunities for mediocre coders to grow into mediocre engineering managers, everybody was happy.
Too many companies have to hope of hiring the level of competence required to not suck. It's painful but true. AWS is cheap and serverless is popular when setting up a linux image security and correctly can take weeks or months.
Which is why it's great that server-less exists. I would much rather live in a world where a single developer can test out an idea in a way which will scale than where you need to hire from an expensive and scarce pool of dev ops engineers to get a product off the ground.
You can still get "scale to 0" with cgi-bin applications. Yes, it's old (but it works). Yes, it's slow (so is Lambda).
Unlike Lambda, CGI works on any web server and any OS. It's also extremely easy to mock because the input and output is stdin/stdout, not AWS API invocations that cost money every time you run.
What you miss out on is "scale to infinity" (read: your bill goes to infinity too).
I recall when AWS first released Lambdas I thought of cgi-bin, especially with PHP where the deployment model is just pushing files. As you say, it has pretty much the same upsides and downsides.
That model does scale well for the web layer at least. Just lots of cheap VPS nodes. Of course, the bottleneck then is the database or whatever layer provides state and persistence.
I think the main selling point of Lambda (and most SaaS) is you don't have to worry about the machine running your code. Personally I haven't used it so correct me if I'm wrong but you don't have to worry about things like monitoring, network administration and security (beyond your code and general opsec).
Yeah, FastCGI is a little closer since it fixes the startup time problem.
It's weird to me that PHP uses FastCGI but Python support for it is really limited. I'm using FastCGI on Dreamhost and it was a pain to get it to work.
Don't think the general notion that serverless is either a return to the past or a negative development is true.
Long feedback loops seem to be more of an issue with a lack of engineering discipline where I've worked. Implementing a comprehensive, fast test suite is up to the team to build, not AWS. Serverless is a tool like any other that needs to be used in a responsible manner.
If you're using cloud features that geniunely cannot be tested locally or quickly I would reconsider using more stable technologies that are perhaps enhanced by your cloud provider instead of totally replaced by them. _You_ are responsible for creating a short feedback loop regardless of your deployment environment.
I've been using serverless in both GCP and AWS for many years now and have helped migrate mid sized startups onto those platforms for event processing and API build. In that time I've seen bills go from thousands of dollars to a few dollars a month.
Mid sized startups are now able to compete in data heavy fields precisely because of technologies like serverless that have enabled small teams to build scalable systems. When you work in transaction processing or analytics I think you'll understand that scalability directly limits how many customers your platform can support.
Not everyone just makes simple json http endpoints.
309 comments
[ 4.5 ms ] story [ 292 ms ] threadWow times haven’t changed at all.
He has some good points on poor configuration languages and bad architecture, but it’s lost in the rant.
It's been a while since I encountered it but yes, mainframe time was so blisteringly expensive that usage was accounted for on a per-job basis and billed to individual user accounts. I don't recall the resolution or pricing anymore but students would be issued an hour or so (?) of processing time to complete their coursework per semester. Perhaps somebody with better recall than I can fill in the details.
https://www.rigacci.org/docs/biblio/online/sysadmin/ch21.htm
https://faq.nearlyfreespeech.net/q/ResourceAccountUnit
So basically if your website doesn't get any hits, you don't get charged.
I've mentioned as much to several C-suite executives and professional venture investors I know who tend to wax philosophical about "exponential growth in cloud services and big data" and whatnot. Invariably, their response has been to tell me that I just don't get it, or that I make no sense. They give me puzzled looks and maybe even get a bit exasperated with me.
Surely others here on HN have run into similarly minded individuals too. They're not hard to find.
Plus ça change, plus c'est la même chose.
There's also this enduring myth you see repeated even here of how much saving there is by not having "devops", when only very big companies needed them in the first place. A few minutes per month is the most devops any startup would need to do in reality running their own VMs or servers, after an hour or two of setup. And that hour or two is often less than faffing around getting all the labyrinth moving parts of AWS/Azure/etc. working.
And the whole thing becomes a transparent farce when you look at providers like Azure, where their own admin interface is so slow and prone to random breaks you have to ask:
How is this scalable?
It's clearly not scalable for themselves, so how is it "scalable" for everyone else?
There are benefits, mainly for me I'll use them for hassle free deploys[1], but the whole thing sometimes feels like a massive con to sell the performance of a potato by advertising it as a jet engine.
[1] Like a month back I deployed a demo prototype web app I wrote in 3 hours onto Azure straight from github with like two clicks, and got the contract, can't really beat that.
https://www.youtube.com/watch?v=bzkRVzciAZg
Right up until something crashes or some hardware fails.
Unlike the public cloud where Nothing Ever Goes Wrong (tm).
There are never brownouts.
There are never latency spikes.
No undocumented quota limits.
Documented but confusing quota limits.
Unexpected performance issues caused by other customers.
Forced upgrades.
Forced patching.
Shock bills.
No siree Bob, that kind of thing just doesn't go on in the public cloud! It's all rainbows and unicorns....
Your original claim is quite unrealistic and you'd do well to retract it. It's bad advice both for public cloud and for self-managed efforts alike.
We supported disaster recovery and high availability deployments.
Any failed component, Dell would send us a replacement in less than 24 hours.
Try calling Amazon and getting that kind of response for your problems.
But, yeah, that is worse-case scenario.
Sure serverless etc has it's upsides and in that case I do use it, I just encounter close to 0 companies per year that need it or benefit from it. Aws ways benefits.
The original claim is not at all unrealistic, I’ve been running a small shop with about 20 VMs without issues for years, uptime seems to me about comparable with serverless or hosted services from big players who regularly have small outages and there really is very little work involved.
I dunno, the first company I worked at was very small, under 30 people, sales included, and I was hired on as an SA. If they were using serverless, they would not have needed to hire me.
Deployments and system upgrades were very toilsome. The dev lead would spend a few hours deploying new code and we’d only do it around midnight in case there was a problem (our customers were almost entirely US based). I spent a lot of my time automating OS upgrades on the web servers behind our proxy. All of this work would be unnecessary for that small company if they were using serverless.
Additionally, as mentioned before, the majority of their traffic was during the day so they could have scaled down automatically at night, cutting costs. Would this be cheaper than running machines 24/7? I’m not sure, but I could see why you’d be interested in at least looking into it
Not using serverless doesn’t mean there can’t be a CICD pipeline.
Don’t get me wrong—serverless platforms are not a ci-cd silver bullet. Many organizations build pipelines on top of serverless deployment apis, but it certainly does take care of a lot of things for you—not to mention other features mentioned elsewhere in my comments
And since we have containers it's become really much easier actually ... Deploy a Traefik container which will watch dockerd socket and self-configure HTTPS and all when your other script starts container. Really, it's much easier than back in the days: https://github.com/jpic/bashworks/tree/master/vps
I remember using companies like MediaTemplate and FullControl 10 years ago for clients and not having to worry about OS updates.
My reality at a 20 person shop with a few devs is much closer to the OP - a few hours setup (we save the cloud config and can remake servers in a few minutes), servers run for months without intervention, uptime is on par with cloud services, deploys take seconds. Managing servers or deploys just are not major problems for us.
Serverless is not a solution to deploy problems - mostly because they’re just not significant problems for us, if anything the dev deploy story is worse than renting VMs. It’s an interesting take on serving requests and I do believe some variant of it might be the future (a variant more like cloudflare workers IMO), but at present there are serious drawbacks and it is not clearly better, particularly when provided by a predatory company like Amazon.
[0] https://aws.amazon.com/certification/certified-devops-engine... [1] https://en.wikipedia.org/wiki/Idiocracy#Plot
It exists to shift CapEx to OpEx for accounting. Any technical considerations are a distant third place on the list.
(Of course you have to force your devs to do the ops stuff now, but that's not accounting and HR's problem now.)
First, the markup in just sheer ops terms is enormous. Minimum 4x, and in some cases 20x. Sure, you get a couple more nines of reliability, but 99% of startups don't need that much reliability.
Second, in countries like Canada where capital expenditure on computer systems can be heavily marked down[0] people still use AWS and DigitalOcean by default.
The main reason people don't do colocating is that it is a waste of time better spent. DigitalOcean is cheap enough. AWS is cheap enough and if the team doesn't have experience running Postgres, fine, they use RDS. Sometimes it's just sheer laziness at first that later blossomed into "holy shit, we're growing too fast to even think about migrating off of these giant AWS bills" which was essentially Dropbox's story if I recall correctly.
[0] 55% per year, and in practice they allow full expensing in the first year for petty amounts of outlay. For example, if your business is a data centre, sure it's 55% per year, but if you run a flower farm and fully expense a new laptop for sales the CRA is generally fine with it.
Serverless, like every design decision, comes with its own set of trade-offs. Whether those trade-offs make sense for your environment is something you have to determine. At my company we have seen a productivity increase and shorter development times using serverless. Of course, that was after the learning curve which is certainly not trivial.
Apparently, you don't even know what that word means. You can not "have" DevOps. It's a methodology and not a thing. Besides, DevOps means exactly the opposite of what you are implying: it is the practice of not having a dedicated operations team and letting devs handle as much of the operations for their own applications as they can.
> few minutes per month
I can tell you my team spends a good 30% of its time on operations. Some of that (for example adminstrating K8S clusters) could be offloaded to cloud providers for money.
No technical reason, maybe. There is a social reason though: if you point out that there is no need to build scale (yet), you are pointing out that the business will not achieve overnight success. Even if it's true, it kills excitement and is therefore not something that people want to hear. Who wants to realize that what they're spending large amounts of time on (and possibly even neglecting other things like health, sleep and social contacts) will not even make all their financial dreams come true? Much easier to not think too hard about it and keep on doing what everyone is doing.
So, building out scalable infra is signaling that you too believe in the company and are preparing for the moment (any time now!) that "webscale" is needed. It's showing that you are part of the team.
Yes, a technology probably won’t live up to its marketing.
Yes, what works is probably good enough.
Yes, changing has switching costs and sacrifices institutional knowledge.
Yes, we’ve seen these technologies before and switched away from them from various reasons.
That’s all fine and sometimes frustrating as an engineer. But it’s not the only reason decisions are made.
Sometimes pitching your exec team on The New Hotness gives cover for changes that actually do need to happen.
Sometimes adopting The New Hotness is marketing to recruit engineers who want to try something new.
Sometimes adopting The New Hotness is because some people are bored and uninspired in their jobs and just want to believe in something good again.
The success of things like MongoDB and blockchain are not because they are better or even very good at most things people use them for. They’ll never live up to the hype. But people need things to get excited about. It’s psychological. Attacking them purely on technical points can lead one to miss the actual point.
Managing this stuff is a job, and it requires discipline and organisation. It's one of the few functions of an IT department that is left once you go to cloud (as dealing with Oracle's bullshit is then gone). The younglin's doing development in the business functions have to play by the rules - and the business functions need to agree that there are rules or they will :
a) spend much monies on the issue you describe
b) get cracked open like fresh eggs by any passing hacker who is keen on sports cars
The problem we have in software is that we have engineers build bridges and then somehow expect them to stay around to paint the thing, change light bulbs, etc and stay fulfilled. A person who is an expert at making new things won’t like being in maintenance. It’s that simple.
It's much more fun to improve the status quo by inventing something nice as to repeat the work of the past generation (only painting things in a different color).
I wrote a blog post on Monday about a relatively old and not-hot language. The post nevertheless trended on Medium and received 50k views. A lot of the feedback was from people who were just happy to see someone senior say something positive about the not-hot language.
Anyway, working with The New Hotness also gives people something to put on their resumé: “I built system X on top of The New Hotness. Nobody had built such a system before for The New Hotness”.
So people keep moving on to the next project, leaving the old ones running live but to rot, and at best every once in a while someone will be assigned the dreadful task of having to figure out why the "legacy" (even though it's still fully in operation and servicing customers) system had an outage and can't recover.
Our field suffers from a serious lack of focus. People don't learn to work under constraints. Instead, they prefer to build a whole new thing, full of complexity and hidden issues, just to get around a few nuisances that, with the right mindset, actually force one to be creative (which should make work fun).
I've recently commented here on HN about how I feel like I'm outside the norm on these things [0]. I was happy to learn I'm not alone. I do think those of us who don't mind working on "legacy" systems and perhaps with limited tools are the minority though.
Sometimes it's not even about working on old systems. You work with current tech, build something, and as soon as it's barely deployable, you're expected to move on to the next thing. "It's a great opportunity for your career," they say, "you're gonna get to build something new and make key decisions."
Why does it have to be decided for me? It's my current predicament. I'm working on a project that took me two months to get in the right mindset for and get all the context I needed. I had to try things out, sort out ideas, until I got a clear picture of what to build. Now I've built the first iteration of it. It works, and I'm super excited about it. But it's not done by any means. It needs more features. It needs to be polished so it doesn't make operations miserable running the thing. And yet... I have to move on, and hand it over to someone who doesn't have the same excitement for it that I have, and has maybe 10% of the context.
All because building new things (and adopting The New Hotness, as you say) is what "everyone" wants.
[0] https://news.ycombinator.com/item?id=25404039
I've chosen to take on more than one heavy maintenance job, no client facing visibility in terms of fancy new features. Feedback in performance reviews would always be "it's hard for people to see what you're doing." In the end you have to find your own way to game the system to get appropriately recognised, but of course that kind of office politics isn't really what you want to spend time either.
I'm doing more 'new feature' style work now but I'm happy to not be tumbling through the waves of hype. We get a little bit of new hotness, because of course you need to embrace some aspect of that to avoid stagnating and getting too stuck in your ways, but not so much of it that the engineering team is incentivised to churn through half-assed features.
I cured this one when my native server running on not so expensive multicore CPU with enough RAM left their cloudy setup in a dust performing more than a 100 times faster and still having huge potential reserve in vertical scalability. They were just astonished at what a single piece of modern hardware in combination with the native code could do.
When the server gets even remotely suspected of having been hacked into, you must trash it overnight and never use any of it again.
Sure, if a drive breaks you'd probably want to replace it instead of just replacing the entire machine; it's a thing you don't need to do with VMs, but that doesn't mean it's difficult. It definitely doesn't take "hours".
Managing physical setups is different from managing fleets of cloud instances, but it's not necessarily automatically inferior.
It is pure scaremongering and making not very nice assumptions about people's abilities without any real substance. If this is the idea of attracting customers or advocating the approach it seems like a very poor job.
Anyways nice job advertising cloud and trying to scare people into using it. You happy with it - good for you. No need to convince me as I have my own reasons and do not have to buy into yours.
I'm about halfway through my career and I have seen something like that happen exactly once; the erratic behavior of the machine had my team scratching our heads for a fair bit until we ran memtest86 on it after running out of ideas. So I would say that it's not impossible, just extraordinarily unlikely.
The rest of your post I entirely agree with.
This is every company I’ve ever worked for or ever heard any colleagues, friends or associates work for, that tries to build their own SRE practice on top of data centers.
Large cloud platforms alleviate so so much of this pain. The “back to the 70s” workflow sarcastically described in the quoted passage you shared sounds like an absolute dream compared to the nightmare of shipping code on internally created developer tool & deployment platforms interfacing to bare metal or space leased from data centers.
I can’t give the cloud providers my money fast enough and I have only ever felt happy about that for years and years now.
Are all the commenters in this thread just cynical engineers who really can't see past their own narrow world view? I get snake oil is snake oil, but there is such a strong brand of "anything new must be snake oil" that runs through the HN community it boggles my mind.
The world is actually full of complex systems. And only once these systems are in place does the software engineer get to do his work! People on this site act as if they have all the understanding and knowledgeable necessary to decry something like cloud vendors because clearly these same people know better than everyone else and these are actually simple problems if only we had their perspective and knowledge.
The ego and arrogance of people in tech really shines through here.
A 1000 times this. These are solved problems. Anything new has to be better to be taken seriously.
I would say there's at least as strong a "brand" of "if it is a new way it is probably a better way" on HN as well.
I think making blanket assumptions in either direction is a terrible idea. Best to be aware of your own biases, and try to be as objective as you can in your decisions.
Experience matters, and there are new solutions that are inferior to older ones in many cases (yes, there is plenty of "snake oil" for sale). However, sometimes new people look at an old problem with fresh eyes and come up with better ideas on how to solve it.
This really isn't anything new. When I was fresh out of college I remember older programmers talking about "such-and-such" is just the old way of doing it repackaged. There is lots of oscillation in the computer industry.
You mean, "the ego and arrogance required to bend code and machines to their will and not the other way around". Maybe try to pass that and learn a thing or two.
> I can’t give the cloud providers my money fast enough
People run VMs and k8s on the cloud as well and run into every one of the problems you just described (aside from literal hardware failures).
Your rant seems to mainly be about having a good CICD pipeline and that’s not really related to cloud or serverless.
You toss out “build a good CI/CD pipeline” like that’s just some localized, solvable problem, and it’s exactly that lack of imagination and lack of understanding about the true multitude of use cases that causes this problem in the first place.
I agree “serverless” isn’t a panopoly solution to everything, but externalizing and democratizing all infra tools to cloud vendors is a major transformative solution that is worth a lot of money and solves a lot of problems.
For example, if someone like you had any leadership influence in my company’s infra org, I’d be trying to spend money on cloud tools just to get away from your authority as fast as I could, and I would be 1000% correct to feel that way and would end up with much healthier tool solutions and less blockages on use-case-specific delivery requirements by doing a giant spin move around your myopic perspective.
At my current client, my team is dependent on the "Platform Team" for _everything_ that isn't directly related to the code we're writing. Do we need a new Azure queue? That's a Platform ticket in JIRA. Do we need a new tool in our CI/CD pipeline? That's a Platform ticket in JIRA. Do we have a build failing for something other than tests? That's a Platform ticket in JIRA. Do we have an issue in production? We need to engage with Cloud Ops because we don't have access to anything in prod.
Plus, it takes six weeks to get a feature from development to production because horrible workflow and manual processes.
Plus, development teams are prevented from implementing anything without the approval of Solution Architects who are too far removed from the business to understand what's needed and too far removed from the technology to be able to design a working solution.
It's all fucking horrible, and yet we're using all the modern cloud tooling: Spring Boot, Azure, containers, K8S, Gitlab pipelines.
In contrast, a couple clients ago my team had full autonomy to do what we believed was the right solution. Stories would often be groomed, implemented, and delivered into production all within a 2-week sprint. No muss no fuss. When they weren't, it was normally due to us having to wait on another team to deliver their work.
Anyway, I think your comment proves that once we decide to invest in something, our brain does everything it can to justify that choice, Human nature really :)
Yes of course my team won’t solve bugs instantly just because of cloud tools and of course we will make mistakes implementing bad infra designs, especially early on as we are still gaining experience with efficient and cheap cloud patterns. Nobody said otherwise and my comment before has no connection to anything like that.
Rather, if we have control over our own infra, we can adaptively fix those issues for ourselves without getting delays and philosophical or gatekeeper arguments from central data center admins. We would have autonomy to change our own deployment system, provision more compute resources, generalize something to work with GPUs, test out different delivery and runtime frameworks, create our own ingress failure robustness patterns, all without blockages and mandated central infra bottle necks.
To create such a configurable and adaptable API-driven portal into bare metal or a data center yourself is way too slow and costly, only the largest companies can do it, and then they turned it around and made it into cloud vendor options.
When medium sized companies try to do it, the most common outcome is a ton of exceedingly arrogant SRE types will create piss poor tooling that barely works for maybe the top 2-3 use cases and then will whine and complain that they can’t support generalizing it for the long tail of other use cases, and because they have political power as the gorilla in the room sitting on top of the only interface to the data center, nobody can argue with them, and you get a burnout mill with constant turnover on all the other teams who have to do SRE’s job for them while listening to the arrogance about what a rip off cloud tools are.
I feel sorry that it worked for you this way (well rather it does not work). I have totally opposite experience. It takes single click of a button for me to build, test, deploy a code change using on premises and rented hardware.
I did cloud as well when it was mandated by client and it did not feel any simpler, faster etc. More complex and way more expensive it sure was but since it was not me paying the money I could safely ignore that part (not for my own projects and servers though as I do count my money).
Even without cloud APIs and containers and whatever, I find there's a creeping thing that happens here which has been helpful to become consciously aware of and intentional about fighting back on.
Basically, the thing is that wrappers become interfaces. You write tool A, and then tool B, and then some kind of script C which calls the two tools and produces a result. Then someone says "dang, so convenient", and sets up a Jenkins/Rundeck/Airflow/whatever job that runs script C against some source of inputs automatically— maybe there's even a parameterized build capability where arbitrary inputs may be submitted to it on demand.
Before you know it, the fact that the original tools and script exist has been forgotten, because everyone's interface to this functionality is a handy webpage. And when someone wants to add more features, they don't even go back and change the script, they end up just adding it directly into the job definition. Before you know it, the full scope of what is being done on "the system" isn't even really available any more to be installed, invoked, and tinkered with locally, not in a meaningful way, anyway.
It's not even necessarily all bad, but when I see this happening in a domain where I have the awareness of the underlying tools (or I may even have written or contributed to them), I do try to help demystify them by writing up docs on how to do the local setup, helping others get going with it, etc.
"Serverless" always reminds me of CGI/fastcgi, which was still a thing the last time I wrote web apps for money. It had the advantage of "scaling to zero" and could run a huge number of low-usage apps on a single 2000-era server.
One word: Worldcom
I still remember coming up to 2000 sitting in a meeting room of an equipment provider I recently joined looking at projections of market growth, competitors and our own sales and profitability. Shortly afterwards that part of the business was sold, I remained with the other non internet part and we sailed through the .com crash unscathed.
A lot of what AWS offers makes no sense. A lot what some AWS customer's teams are building is overly complex and a horror show to maintain. But there is also value: Maintained images, backups, on demand resources and specialized resources. We hit one of the inflection points in Moore's Law and that drives us into horizontal scaling and specialized hardware.
Like with Worldcom and the fantastic internet growth story back then there is a kernel of truth and value. Find people that understand it and be glad others are playing around and subsidizing the investments in the core.
I really don't know why that is, my guess would be that few people have really built something beginning to end with the most boring tech possible (start out with PHP+MySQL like everyone did 15 years ago). Or they always operate at levels of abstraction where everything is slow, so they have simply gotten used to it, like their text-editor not running smooth unless it's on an i9, because the text-editor now is a pile of abstractions running on top of Electron when vim was able to run smooth decades ago. It's sad and both an opportunity at the same time, because you can be the one with the correct gut feeling of "This should really be doable with a single machine, if not we're doing something fundamentally wrong".
We have the computing power to run complex physical simulations or AI training sequences on a normal home computer, but for some reason use programs that take 100 times longer to start than old software despite not having more features, and websites that sometimes even come with loading screens. Electron isn't even as bad or slow as many people think, but somehow developers still manage to throttle it so hard that I might as well just use a website instead.
As someone who is just starting with nodejs and web development I find a lot of the tech feels nice but sometimes also unnecessarily abstract. Sure, it tends to make a lot of the code very elegant and simple, but every additional framework adds another guide and documentation to look at, another config file that has to be correctly referenced by an already existing one so npm knows what to do, another couple of seconds of build time and another source of bugs and build problems. Then of course you need that trendy package for easier imports maintenance - which IDEs automatically handled in the past, but now we gotta use an editor running in a hidden web browser which started from scratch in terms of features but is just as slow in return.
Going through my current project has really worn me out with regard to tolerating UX slowdowns in developer tools. I am getting so incredibly tired of how shitty visual studio performs on a threadripper with effectively infinity ram. I've had to turn off all of the nice features to get our codebase to be even remotely tolerable. And, dont get me wrong. This is not a plea to Microsoft to fix VS. I think everyone involved can agree that fundamentally VS will be unfixable WRT delivering what I would consider an "ultra-low" latency user experience (mouse/keyboard inputs show on display in <5ms). Any "fixing" is basically a complete rewrite with this new objective held in the highest regard.
Current developer experiences are like bad ergonomics for your brain. This madness needs to end. Our computers are more than capable. If you don't believe me, go play Overwatch or CS:GO. At least 2 independently-wealthy development organizations have figured it out. Why stop there?
That's not possible, at least if you measure the full time from actuating the key to the screen showing the updated content. The record holder in that regard is still the Apple II with 30ms[0].
But I agree, modern software should react much faster. It's kind of weird how on one hand we have hardware that can run VR with 120Hz and extremely low input latency, and at the same time some programs that only need to display and change a couple bytes in a text file need 4 seconds to even start.
[0]: https://danluu.com/input-lag/
That's a lot of it, I believe. One startup I dealt with was honestly proud of how they were scaling up to hundreds of requests per second with only a few dozen AWS VMs to handle it.
I'm old enough to know how ludicrous those numbers are, having built systems handling more load on a single desktop machine with 90s hardware.
But I've come to realize some of the younger developers today have actually never built any server implementation outside of AWS! So the perspective isn't there.
When you have an organization with hundreds or thousands of developers delivering code that works together to support a business there is some types of complexity that are inescapable. So the question is “where do you want the interfaces between people and teams?”
SOA and microservices push that interface to the API layer which drastically increases the complexity of the code, but is a logical place to put the complexity if you consider the organization, people, and code a single system.
At scale there are levels of inescapable complexity. There are a lot of tools that help manage this complexity. However too many in smaller teams emulate bigger companies and inject wasted complexity by using tools that solve problems they don't have.
In fact this is quite seldom. So all the tech that solves problems that occur in such constellation should be quite exotic.
But it isn't. The reason is that computing "on a mainframe" (the cloud) can be sold orders of magnitude more expensive. The rest is marketing of the mainframe owners.
But yes, I agree that most companies jump to this level of complexity too quickly.
Also I think airlines do just fine on core system form the 80'ies until now. They also don't tend to have software development teams with "hundreds or thousands" of people to my knowledge. So maybe not a good example.
There is only a hand full of companies globally really in need of all that "serverless" stuff as I see it.
Some buy off the shelf, others buy components off the shelf and focus on custom where they feel it gives them an advantage, and others have built the whole thing themselves.
And they're doing "serverless" these days as their IT operations are a good fit for this paradigm?
Or could it be that they're small enough to get things managed with off the shelf software and/or something built and run in-house?
You mention of the shelf software, what do you think the organizations that build the COTS stuff (like Sabre) in this industry look like? Where do you think it is run? They’re all moving to microservices and cloud hosting too, including serverless where they think it makes sense (and a lot of times when it doesn’t, just like everyone else).
The smart ones (who understand Conway’s Law) are making these moves because they are enablers of smaller, more independent teams which we know are better at delivering software. The really smart ones know they’re paying the price in increased complexity the comes with a distributed system, but are willing to pay it because that cost is lower than the cost of the collaboration necessary to build a monolith of that size.
Logistics companies are one of the most interesting intersections of software and the real world. Most organizations are making a mistake by focusing on scaling their software too early or unnecessarily. Airlines and logistics are not, they have business problems that their existing software cannot solve in areas the COTS options do not fit in their operations.
This is the message we've needed.
Cloud is an expensive, overly complicated, lock-in boondoggle.
Open source needs to rally hard to prevent cloud providers from running their tech.
I don't love that I pay aws to run my open source domain(s), but the extent of their job is: keep my VM up, provide me with outbound email that works, offer backups. For now, I can live with that.
Only if you go with companies that make it such.
Renting web hosting on someone's collection of rack-mounted systems running some well-known OS is Cloud, and there's nothing inherently bad about that, unless you've defined Cloud to mean "Only The Bad Stuff About Using Computers Other People Own". You can run things like they were run in the 1990s, complete with Apache and mod_perl CGI and MariaDB swapped in for MySQL and everything held together with shell scripts and run via ssh and ftp. No magic, no lock-in, no complexity beyond what every other serious general-purpose computer system is these days.
I agree that Big Cloud Providers layer on pricing models and whatever middle-management-ware they can come up with to obfuscate the underlying mechanics, but if you're against that, say so. Don't catch everything in your blast radius because it reduces the overall impact.
However discounting it altogether because you don't need or understand why that complexity exists is naive.
I hear this a lot in interviews and it's a clear signal that you haven't worked on anything big enough yet.
Quite a bold wager, and totally incorrect. I hope you're not presumptive and rude to all of your candidates.
FWIW, I'm a senior engineer at a large tech company where I started pre-IPO and have had pretty great career growth. I've been on a variety of teams dealing with most of the levels of our stack: eventually consistent data models that are active-active, k8s, frontend, tearing down monoliths, etc.
I'm fine with microservices for medium to large orgs, and I love Kubernetes.
Lambdas and cloud are expensive lock-in, and holding this view doesn't mean you hate distributed systems or complexity. You can staff your own platform teams and manage your own capabilities. Imaging machines, upgrading, failing over - it's all automatable.
The excuse I hear is that companies don't want to build things that aren't related to their core competency. But I'd argue your software stack fundamentally relies on the environment it lives within and that ceding control poses a non-zero risk.
I'm incredibly fearful of the future where all software runs within AWS or Azure. We'll all be paying taxes to these companies and we won't be able to run or replicate similar environments on our own. It'll be difficult for new engineers to learn as our field will effectively become a "trade".
Can you imagine the world where it's expected that everything is composed of SQS and Lambda and other bullshit?
I'm going to focus on one part of it for now.
"I'd argue your software stack fundamentally relies on the environment it lives within and that ceding control poses a non-zero risk"
This is factually correct, but more nuanced than "just build it in house". Many large orgs externalise this risk by relying on AWS / Azure / Google. Nobody ever got fired for picking IBM is more relevant now than ever, and it is because the largest companjes on the planet have invested huge amounts of time and money solving these problems.
I think you misunderstood me at some point, because nowhere did I advocate that everything should be a lambda. Or that everything should be in the cloud.
The point I am making is that a lot of people I speak to are afraid of complexity and lock in, when they should be afraid of never being able to grow past what they are comfortable with.
To your point about this becoming a trade... Brace yourself, I would wager most development positions will be obsolete in 10-20 years as AI assisted infrastructure and development hits full stride. Especially dev ops and front end.
I’m sorry but this absurd and blatantly ignores the reasons why people choose serverless. No mention of scale to zero, virtually infinite auto scaling, automated deploys, arbitrary runtimes, ability to spin up code in multiple regions around the world with zero operational overhead, the list goes on...
These are non-trivial things. You’d have to pay engineers to build the same systems provided by serverless if you want smooth deployments and high availability. Also, imagine if you wanted to start operations outside of the continental United States. You’d need to find a data center provider, buy machines, pay someone to install them (and hope they do it correctly), and then, when ever you have a technical problem, you need to email the datacenter team to have someone plug in a KVM with 5 seconds of lag so your engineers can debug, and then you have to pay admins to constantly maintain the servers, and consistently pay for their rack space even if they aren’t being fully utilized. None of this is a problem on a fully managed serverless platform.
This article is just click bait capitalizing on FUD in the software community. Serverless will be a part of the future of software infra whether luddites are mad about it or not
Disclaimer: I work on serverless products so I am biased
The alternative is renting VMs from those same clouds. It works fine and is cheaper when you work on software that actually has users. It also doesn't result in the cloud provider cramming a crappy under-baked programming model down your throat.
Per request pricing and "scaling to zero" is just old fashioned unbundling. You are selling a smaller quantity of a product at a higher unit price and putting a bit of lipstick on that pig to convince people they are not getting hosed.
Because that’s what the author was comparing it to. Yes you can rent vms, and that works for many workloads. However serverless platforms provide other benefits like deployments, load balancing, and auto scaling that you’d need to setup yourself on raw vms. Plus, with services like AppEngine Flex you can run serverless on vms to cut costs if you have consistent traffic.
> Per request pricing and "scaling to zero" is just old fashioned unbundling. You are selling a smaller quantity of a product at a higher unit price and putting a bit of lipstick on that pig to convince people they are not getting hosed.
It really depends on your problem. Sometimes you might have a very low QPS service (like maybe a few QPH) and running a whole vm would be overkill. Some business have really bursty traffic and maybe some regions won’t get any traffic at all during the night but they still want to deliver low latency service to the people living on that side of the globe. Additionally, these are just reasons to serverless for scale to zero and ignores other benefits, like reduced operational costs, that business can use to justify paying more for compute time.
There are lots of reasons to use and not use serverless for compute. I don’t think it is the solution for everyone and everything, but I do think the hate and FUD directed towards it is unwarranted and uniformed.
Disclaimer: I work for Google, opinions are my own
Edit: typos and disclaimer
When you're working on your MVP, you can skip dev ops completely and get your infrastructure cost very close to zero. Then if you actually do end up building a product which is fortunate enough to have to worry about scale, you can work on cost optimization. I think this is a very relevant and valid use case for server-less.
And if I don't have users? What's the number of 'users' where it becomes cheaper?
Most of the web is idle 95% of the time. Theres no reason people should pay for so much VM time that they're not using.
Not going with Lambda in AWS doesn't mean you can't use EC2, EKS, ECS, Elastic Beanstalk, and so on and so forth. You're the one spreading FUD.
I suppose this wasn’t clear because another commenter had a similar question/response
What are you even talking about dude? You don't have to build a rack in a colo to run a non-Serverless app. There's this thing called The Cloud now, and Virtual Machines, and other fancy new technologies from 15 years ago.
"Serverless" as in Lambda is a joke. "Serverless" as in Fargate is legitimately great at lowering ops overhead. But "zero operational overhead" is absolute bullshit. Even Lambda has operational overhead.
Serverless is just the abstraction of a function into a compute unit. There's like 20 other things you need for a functioning product, many of which literally cannot be replaced by Serverless-anything. And the severe limitations of many Serverless technologies make it very unwieldy to use.
Give it 10 years and it may morph into something more like a monolith whose functions become "Serverless", like a Beowulf cluster where forks are executed on remote nodes and there's some ugly glue to tie it all together. That seems a lot more seamless and useful than trying to architect 100 tiny pieces together with crappy custom integrations everywhere.
I think you are miss understanding my comment. I was comparing serverless to colo because that’s essentially what the author was doing. I agree that there are plenty of other cloud compute solutions that work for many workloads.
> There's like 20 other things you need for a functioning product, many of which literally cannot be replaced by Serverless-anything
I totally agree. Serverless is usually just a piece of infrastructure that is combined with other resources like databases, batch clusters, blob storage, vms, etc. I don’t think you’d be able to create a very interesting app using stateless functions alone, though I could be proven wrong ;)
Not many people have the problems solved by a serverless architecture (or they need more freedom), which is why it's not as common as just running your own long running instances.
I would say that "literally any single person" benefits from serverless if they don't have the skills, knowledge, or time to handle the server/infrastructure responsibilities that come from running software.
If I'm a one man development shop with a sufficient budget, I'll probably find it far more productive when using serverless architecture to host my business logic than not.
And cloud debugging has its issues too: I remember endless fudging with print statements, deploying once more and waiting for the changes to propagate in a AWS lambda handler, because the bloody thing simply wouldn't run locally.
That’s a good question. I think serverless has a lot to offer, global compute, and infinite scaling are a subset of features. I think it really comes down to your specific workload—it’s definitely not a be all end all silver bullet
One example I can think of I mentioned in another comment in this article which I have pasted below:
the first company I worked at was very small, under 30 people, sales included, and I was hired on as an SA. If they were using serverless, they would not have needed to hire me. Deployments and system upgrades were very toilsome. The dev lead would spend a few hours deploying new code and we’d only do it around midnight in case there was a problem (our customers were almost entirely US based). I spent a lot of my time automating OS upgrades on the web servers behind our proxy. All of this work would be unnecessary for that small company if they were using serverless. Additionally, as mentioned before, the majority of their traffic was during the day so they could have scaled down automatically at night, cutting costs. Would this be cheaper than running machines 24/7? I’m not sure, but I could see why you’d be interested in at least looking into it
Ah, the bullshit. When we showed GCS engineers our hadoop workload, their eyes went really big and told us that they don't have that kind of capacity in a single AZ.
Maybe.
Right now I don't think that there is a single telco that has the scale to break GCP, or a single manufacturing company (exempt Tesla) or a single energy company. Or bank.
I agree that pharma, social media and streaming may do this. But since Netflix is on AWS I am doubtful of the last.
There's a big world out there, there are probably tons of big computer users that we are largely unaware of.
The LHC produces somewhere between 20-30 petabytes/year[0][1] (at a rate of about 200 Megabyte/second[2] after ignoring many "uninteresting" events)
The LHC is a big name, but there are many other applications in disciplines like physics, genomics, meteorology etc for whom "just use the cloud" is a terrible suggestion.
[0] https://www.popularmechanics.com/science/a16716/lhc-data-vis...
[1] https://home.cern/resources/faqs/facts-and-figures-about-lhc
[2] https://www.lhc-closer.es/taking_a_closer_look_at_lhc/0.lhc_...
also, some manufacturing giants are truly enormous..
samsung for instance makes everything from smartphones to artillery installations..
same goes for hyundai, mitsubishi etc..
Tesla is TINY.
Try things like Siemens.
EDIT Besides, the point is not the whole of AWS of GCS; it's the about availability zones vs singular datacentres. Distributed computing doesn't really like to throw data even within the same AZ - in-DC will always be faster. AZ and the modern idea of distribution suit certain things - streaming, for example, benefits from it, but not everything.
That being said, I think what gets me most about this is it seems to overlook the fact that serverless exists to remove the effort and time and cost spent on maintaining servers and aims to allow developers to focus purely on providing value. There’s no doubt you pay more to have it managed by someone else, but at the same time costs are optimized as best as possible by only charging on a per use basis rather than having servers running all or most of the time to host services.
Massive monoliths are very difficult to maintain and often lack resiliency based on the often coupled nature of them. Microservices aim to solve this and also provide more ownership for teams. They also allow for flexibility in hosting as well.
Nit: those were always integration tests.
Networks are unreliable even if it is your local area network. In my career I saw some unrelated network equipment was sending a RST packet to load balancer with application servers IP address and resulting in service unavaiable errors. It took us weeks to find that out since it was happening very rarely and there were many layers between load balancer and application servers.
Network means slowness, connection errors, unknown results, etc all of which should be handled carefully.
On the other hand, I agree with your point of multiple teams. I tried to mean same thing with "used by other applications"
Also, with the current network speeds and memory availability, it's hard to imagine the war size to be a bottleneck in web development.
Depending on other services via a well defined network interface is good for decoupling teams. So it depends on the number of employees you have coding for the same platform, rather than on clients or anything like that.
> Also, with the current network speeds and memory availability, it's hard to imagine the war size to be a bottleneck in web development.
Network speed was not the problem, decompression / expansion of the application was. There were war files about 100-150 MB size. Also I saw one with size of 1.5GB however it was an exception. These may or may not be a concern depending on your CPU utilization, disk speed, memory, etc.
Most decisions depends on environment. For some environments, as you said, deployments are not a concern for microservices, for some they are.
Anyway, can you share which kind of environment even a 5 minute deployment would be a problem? Most places I worked at it took a lot more than 5 minutes to actually implement the change that's going to production, so deployment times were pretty much irrelevant overall.
There are a few fundamental differences between these languages, but it's just plain wrong to say that any of them are not "structured." Did OP mean "schema-less" instead? Because the only fundamental difference at that level is that XML has a built-in mechanism for metadata referring to a file's own schema, while in JSON (and I believe the others, even though I've used them less) the schema is not linked from within the file itself. You can still verify the schema of these languages, though, just as you could with XML. And once you've verified the schema, local code can do stricter validation to make sure things hang together properly, that values are within limits, and so on. After that, I expect modern tools like Terraform and CDK to check with the target platform that the scheduled change can be applied before trying to change anything.
That said, OP has several valid points:
- Many projects try to make a microservices architecture long before it's needed, even in projects where it might never be needed. I suspect some people think microservices are actually easier to maintain, since they are smaller pieces, but only extremely trivial or shoddy microservices will be easier to maintain than a corresponding monolith, because many things which are well-understood in a local setting are much harder to do in a highly distributed setting, simply because the infrastructure is now a bazillion separately managed pieces.
- The feedback loop can be abysmal. Without any easy alternatives to do "local cloud" deployments, having to wait for several minutes for feedback about a tiny change is frustrating in the extreme.
I haven't worked with an API that uses XML in a long time, but does it express the difference between "0123" and 123 well? I think that it either doesn't, or you add some really awkward schema language with XML namespaces ...
I just translated my .travis.yml to JSON, and I learned that
gets translated to {"deploy": {"true": {"branch": "master"}}} because "on" is a synonym for true. I'd call that unstructured. It also has problems with keys that start with -, etc.Also, all of the languages are often used with say Go templates, which makes them more unstructured. They're no longer statically parsable, which is "table stakes" for doing anything else (validation or static analysis).
I took some pains to make shell statically parsable [2], and I think it's now a better foundation than these config languages, per my other comment here: https://news.ycombinator.com/item?id=25483270
I think HCL is probably a lot better, but it also seems to be the least used! It's structured, and has its own mechanism for expressing duplication that's not the equivalent of a 70's-style #include.
[1] https://github.com/oilshell/oil/blob/master/.travis.yml
[2] http://www.oilshell.org/blog/2019/02/07.html
And "no" is a synonym for false, hence "the Norway problem."
https://hitchdev.com/strictyaml/why/implicit-typing-removed/
Is as easy as declaring
vs with DTDs.I read him as referencing what people would have called "structured programming" in the 1980s, and pointing out that we have these settings/metadata/control files in languages that have no capacity for "structure" in the sense that, say, Pascal or C would have over BASIC or assembly, that is, thin-to-no factoring or abstraction capacity. Hence the contrast with "Configuration as Code."
Now, depending on your personality and experience, you might find the idea of using a complete language as config an opportunity for someone to do something non-obvious or horrifying, and the closer to the "settings" end of the setting/metadata/control spectrum you're on the more plausible that position is. When you get to a level of complexity that's the other end I'm not so sure and horizontally scaling systems might well be arriving there.
As long as you're using plain DTDs for your schema, yes. Like early XML-based config languages such as idk JSPs/web-app and Jakarta (née J2EE) deployment descriptors used and still use. But later XML configs use XML Schema and namespaces, which are similarly detached from an instance document as eg JSON Schema and the others you mention [1].
[1]: http://cmsmcq.com/2005/xml2005-166.html "How schema-validity is different from being married"
(1) Languages have gotten worse as distributed back ends have gotten more powerful. The IBM JCL and XML references in this post were good!
(2) Workloads that currently run on such back ends could run on a single computer, or at least with many fewer resources. This is one reason I got into shell in the first place! I wrote some shell scripts that saturated 32 cores instead of using distributed systems. In other words I try to avoid the "COST" or "parallelizing your overhead" problem.
We're paying a huge productivity tax and in many cases not reaping the rewards. I think a better better UNIX SHELL can help in the following ways:
(1) We need to bring the interactivity of Unix back to distributed systems. We're still in the days of "IBM Job Control Language" with Kubernetes config and similar kinds of "declarative cloud configuration". We need a flexible and efficient analogue of Bourne shell.
(2) Unix shell is already how you set up local development environments: Docker embeds shell; virtualenv changes your shell state, Ruby's bundler, OCaml's opam switch, etc. We need to evolve this into first class and polyglot environments specified in shell.
Debugging distributed systems locally could be the norm, but it's not.
The local topology should simply be a configured variant of the distributed topology, but it's not. I used to do this at Google with a trick of generating a shell script with BCL (Borg Config Language).
(3) A better shell should be able express configurations that evaluate to JSON, and also statically validate them before pushing, to solve this problem.
XML, however, was universally rejected in favour of things like JSON, Yaml, HCL, Toml - all free of structure, with zero indication whether a computer would find your prose gibberish or the next Shakespeare play until you actually pushed your code to some test cluster.
I want https://www.oilshell.org/ to go in this direction, and there is already some progress. (Feel free to contact me if this resonates with you.) And I have a draft of a blog post about shell, distributed systems, and languages that I need to publish, based on these comments from a couple weeks ago:
https://news.ycombinator.com/item?id=25343716
Those comments talk about the other problem with the cloud: that it locks you in to APIs! We need the POSIX of distributed systems. Kubernetes was trying to do that, but it's not good enough.
Shell can solve this problem more economically: it expresses UNPORTABLE glue code to leave your application PORTABLE. I did that with Oil's continuous builds, and the Unix-y gg FaaS framework also appears to do that in a pretty nice way.
All the best for your new Oil shell, and I agree that we need to have similar to POSIX standard for distributed systems but it must put local-first software as the first class citizen [1].
Personally I think the local-first software cannot happen soon enough given that what we have endured for the past 30 years with regard to inefficiency and drudgery of the web based applications.
Just wondering about your thoughts on SDLang [2] as the type alternative to JSON, YAML etc, and lightweight alternative to XML?
[1]https://martin.kleppmann.com/papers/local-first.pdf
[2]https://sdlang.org/
https://news.ycombinator.com/item?id=24945449
So I think Oil will be very familiar to SDLang users, BUT it has programmability, which you need in cluster configuration and build configuration, e.g. to express duplication.
So I would divide the space into a few categories:
(1) Stringly typed: .ini, YAML
(2) Typed Data: JSON, XML to some extent, SDLang
(3) Programmable configuration: HCL, Cue, Jsonnet, Nix expressions, BCL, Starlark, Oil
SDLang looks really similar to HCL too, except HCL is programmable:
https://www.terraform.io/docs/configuration/expressions/dyna...
Very important: With categories 1 and 2, people add template languages like Go templates on top to solve the programmability problem, which we don't want!!! We don't want to move typed data back to the stringly typed realm!
Also, I don't think the "expression style" of HCL, Nix, and BCL is great. It's good for small things, but it starts to get hard to read for large configs. I'd prefer simple for loops and if statements, basically like the Starlark language (a subset of Python used in Bazel).
---
Thanks for the paper reference, I bookmarked it!
This guy is trying too hard to sound smart.
Serverless functions are usually free. This is a huge boon for learning: I'm a college student with limited resources, and even spinning up a $5/mo VPS takes some thought and budgeting.
I learned web dev with free web hosts & free VPS services back when I was 13-15. Further democratization will only help newer generations explore & tinker with confidence (scale-wise and tech wise).
Still, what you said is right. I mean, heroku still exists, but the ever present complaints are that it's too expensive at scale and that it is too constraining. So, maybe the vertical is the best approach.
This hit home... the amount of money I've seen businesses pour down the drain in my past few years of experience is insane. Offices with less than 10 people being talked into server setups costing tens of thousands of dollars for nothing more than a NAS, recoridng camera footage, Door access, etc.
It’s no wonder any startup with half the brain would still pick rails/ruby for actually creating something of value, while enterprise middle managers and bean counters argue about how to not write code.
This is the best description of modern software development I have ever seen.
> It seems that, contrary to what everybody is saying, software developers are in plentiful and cheap supply so wasting a lot of their talent makes economic sense over the perceived cost of adapting more powerful programming systems.
> but we just throw more bodies at the problem because Gartner tells us this is the future.
This last part is the real reason for it: executives are blind morons who have no idea how to run technology-dependent companies, and their poor leadership bleeds down because hierarchical organization.
I used to blame executives because it seemed like they were making terrible decisions that everybody knew were terrible. But now I realize that the reason they're making the terrible decisions is they just have absolutely no concept of how the organization should run. They don't understand the customer, the employee, or the product, but they have generic ideas of how to make generic things, so they just do that and hope the people under them will fix everything.
So of course we're managing everything as tiny little components that get glued together. They're "Fordisming". And "Globalizing" is already here too. Many companies already have one microservice made in India, another in Romania, a third in Spain. Who cares if they barely talk? Who cares if they reinvent the wheel and use conflicting technology? It's all just a game of "reduce cost and increase profit".
The next big hotness is going to be the API-gateway-service-mesh-distributed-tracing monolith that every service will be plugged into. Developers will be given more power to turn on and off things that are connected to this monolithic service frontend. They will thus be able to have faster feedback, but nobody will be coordinating the changes they are making, so actually things will break more often due to conflicting changes.
Finally, some time after that, the developers will figure out that it would be better if, instead of depending on all the external Infrastructure/Configuration-as-Code, they should be using frameworks that have everything built into them, and have their application tell the framework what to do, so a dedicated person won't need to write it all using external tools and weird DSLs. This will persist as a sort of "Java framework for Kubernetes-native Applications", although it'll extend into all the thousands of shitty extension apps that plug all the holes in K8s. And then we'll truly have a new monolith, but a distributed one.
As an indie, lone dev, I can run my site, email and a basic web app for about 10 USD a month and, if my app kicks off, I can scale it to a million users without breaking a sweat.
To me, that is the true power of the modern cloud computing paradigm. An influx of great apps and services enabled by cheap scalable and sweet competitive cloud services.
Was this even possible in 2000?
Isn't it usually a mistake to design your app to be scalable to 10,000 users when all you may ever have is 10?
I was running multiple sites by myself. There was less encapsulation (unless you were working with someone on bsd), for sure, but I'd say the advantages are more visible in terms of security (hacking was much easier back then) than ease to deploy.
Doing your own ops has never been that hard (YMMV) and I don't think it improved much nowadays. Scaling to n machines didn't change as well. Deploy n instances, load balancers, scaling vertically then horizontally, db sharding. You still have the same fundamentals.
One pro of today is having a single artifact to deploy (tons of cowboys deployments and FTP uploads in the past).
One big cons you have nowadays is learning all the latest crap $currentCompany is using and having to peel the layers of abstractions to see why things work so badly.
I kind of disagree. It's doable, but still requires maintenance. For instance, even running a simple blog off of a VPS requires you to worry about setting up an SSL certificate and keeping it renewed, staying on top of security updates, and putting together some kind of scaling solution assuming you ever want to write a post which gets to the front page of hacker news.
I think there's definitely an argument at a certain scale to just outsourcing that complexity so you can actually work on what contributes to your unique value proposition.
Most users of the top public clouds are paying 2-10x more than they should just because using the public cloud "is what you do".
Also, which bare metal server providers would you recommend?
Complexity can be managed if you start out thinking about. The biggest problem with the cloud is ad-hoc. If you go in just building shit without a strategy and guardrails, it's going to cost a lot of money and end up in pretty bad shape.
Too many companies have to hope of hiring the level of competence required to not suck. It's painful but true. AWS is cheap and serverless is popular when setting up a linux image security and correctly can take weeks or months.
Unlike Lambda, CGI works on any web server and any OS. It's also extremely easy to mock because the input and output is stdin/stdout, not AWS API invocations that cost money every time you run.
What you miss out on is "scale to infinity" (read: your bill goes to infinity too).
That model does scale well for the web layer at least. Just lots of cheap VPS nodes. Of course, the bottleneck then is the database or whatever layer provides state and persistence.
It's weird to me that PHP uses FastCGI but Python support for it is really limited. I'm using FastCGI on Dreamhost and it was a pain to get it to work.
https://lobste.rs/s/9o4zsx/back_70s_with_serverless#c_9hxzjs
https://news.ycombinator.com/item?id=24683304
CGI and FastCGI can also be used with containers, although it appears that almost nobody does it.
I think if "bubblewrap" (user space container tool) starts to be installed on shared hosts via Red Hat, we could hack it.
Don't think the general notion that serverless is either a return to the past or a negative development is true.
Long feedback loops seem to be more of an issue with a lack of engineering discipline where I've worked. Implementing a comprehensive, fast test suite is up to the team to build, not AWS. Serverless is a tool like any other that needs to be used in a responsible manner.
If you're using cloud features that geniunely cannot be tested locally or quickly I would reconsider using more stable technologies that are perhaps enhanced by your cloud provider instead of totally replaced by them. _You_ are responsible for creating a short feedback loop regardless of your deployment environment.
I've been using serverless in both GCP and AWS for many years now and have helped migrate mid sized startups onto those platforms for event processing and API build. In that time I've seen bills go from thousands of dollars to a few dollars a month.
Mid sized startups are now able to compete in data heavy fields precisely because of technologies like serverless that have enabled small teams to build scalable systems. When you work in transaction processing or analytics I think you'll understand that scalability directly limits how many customers your platform can support.
Not everyone just makes simple json http endpoints.
I definitely agree that feedback loops, local dev, and testing are the things I hear about people struggling with.
For the most part that is a one time cost that your team can breeze through if you invest some time in initial setup.
If anyone out there is looking for a painless transition take a look at netlify and vercel. If you outgrow them later there is minimal lock in.
I also use pulumi for infrastructure mgmt, and it works extremely well as a lambda dev pipeline that manages complex stacks with ease.