How are we still talking about this? The original “architecture” was just stupid. Splitting thousands of hours of video into 1-2s chunks, uploading them to s3, then processing them in isolation?
Does anyone even consider the monster layers of abstraction that these services require? It’s crazy to me to think someone actually thought using step functions and lambdas in this way was a good idea.
I think there's a lot of projects within Amazon that get extra marks for dogfooding AWS and for providing materials / whitepapers for use cases of various AWS services.
I've done some AWS training, especially the introductory ones are intended to get you to use as many different services as possible. I'm sure Amazon's internal training is just that.
> I've done some AWS training, especially the introductory ones are intended to get you to use as many different services as possible
That sounds like an obvious good thing right? The training isn't supposed to teach you how to solve problems, its supposed to teach you how to use a particular set of tools.
Ish? If we are talking about "hammer training," then I can see the argument. Teach me about the different hammers and why I would pay attention to the weight and such.
But all too often, the training picks something like "build a clubhouse." And in that, I would expect legit solutions based on specifications of a clubhouse that you are wanting to build.
Language advocates fall hard on this. They will "build a TODO application" to show off part of the language. But end up with an application that can't do what folks would want a TODO application to actually do. It is very frustrating.
I once worked on a payment processing system. Files came in with about a thousand payments, in a textual format, 10-20 short lines of text each.
The geniuses who led the project before me decided that the first step was to split the file into single payment chunks, then put each one into RabbitMQ, from where a different service would pick them up and parse them - then encode the parsed payments in JSON, and put them back in RabbitMQ, for a third process to pick up and handle. This was all for performance, of course.
I tried to explain that just parsing the whole file in one go would be much faster than splitting it up, writing it to a queue, and reading it back in. That it probably wouldn't be much slower than the splitting. But nobody was interested. Obviously distributing the work would be much faster.
It was a weird experience. My colleagues just had no ability to reason about the "physics" of what computers did. I think they mostly had Rails backgrounds, and I wonder if you just don't exercise that muscle much doing Rails.
That intuitive feel of "what's faster" has been lost ever since computing power exploded. Back when the resources were at a premium, we were just better engineers. Not all of us had the command of Leetcode algorithm problems, but we rarely missed a database index and knew to stay close to the metal as much as possible, avoiding network trips, avoiding database trips.
It still holds true today, and software has become bloated and slower because these simple rules are being ignored. It's all in the cloud, it's all distributed, it's all "free".
As for Rails - it's the same problem but it's different. If you stay too high up in the abstraction orbit, it's hard to think of the nitty gritty of performance.
What is the cloud doing? What is the ORM doing? Once it's all "magic", you just keep piling on non-performant crap until it becomes an emergency. Then you just add more cache :)
I don't think it's necessarily abstraction, it's just complexity. We've been given a thousand footguns and we're expected to know how to use each one. You didn't consider microservices and lambda twenty years ago because they didn't exist. But you also didn't have to build an app that worked everywhere and looked good on every screen.
There’s also the fact that we shouldn’t be thinking about problems the way they used to be solved - not that we shouldn’t be considering database calls and network, but because compute/memory/storage are cheaper than they used to be, we’re able to make more tradeoffs to save developer time or to improve experience. Back when the option was to further optimize how many registers your assembly is using or to leave an application in a virtually unusable state, you didn’t really have a choice. You optimized the small details
Oh, compute and storage are cheap alright, until they are not.
At many jobs I had to deal with the all-hands-on-deck, code red emergency "our AWS bill got too damn high, what can we cut" situation. I can guarantee you a ton of companies are going through this exercise right now.
Some software engineers purposefully develop using fewer resources to force themselves to reason about efficiency.
Abundance leads to complacency and laziness, and those resources are no longer cheap if you are abusing them.
I saw this exact same pattern before. I think it's actually common, because I've also heard people in favor of micro services point to this design as a point in their favor.
One reason always given is the ability to replay chunks if the system goes down, at least for Kafka.
> It was a weird experience. My colleagues just had no ability to reason about the "physics" of what computers did. I think they mostly had Rails backgrounds, and I wonder if you just don't exercise that muscle much doing Rails.
Same experience here. I feel like I'm always repeating myself when explaining to other developers why we should avoid moving data around unnecessary and prefer to operate directly in the database when necessary, specially when it comes to large data migrations. Also important is why it is bad to do a bunch of small operations instead of batching them in chunks closer to the page size (8 kB).
I don't think that's on Rails, more that people with less experience tend to use more indirection in their solutions because each step of the problem is seen as independent and breaking everything down into its own class or method enhances 'readability'.
- Step one: put file in queue
- Step two: break down file and insert rows
- Step three: convert to json
- Step four: compute results
- etc...
Once your experience improves you start to understand the cost of such indirection and realise that the problem could have been solved in a much more straightforward manner.
This explains why Prime Video is the only streaming service that can never stay stable on my gigabit connection, ugh. Sometimes it drops resolution until it's 360p.
> Splitting thousands of hours of video into 1-2s chunks, uploading them to s3
Eh, that's a separate thing - that's how modern streaming works, i.e. it's what allowed streaming to become really, really good, and it's how pretty much every major streaming platform works today.
I don't know about 1-2s chunks, but have you ever had to process a UHD HDR ProRes master with 24 channels of audio? Processing that on a single machine (depending on the level of processing) can easily take up to 10 hours. I definitely agree the 1-2s chunks makes no sense, but cutting it up into segments is the only way to get realistically acceptable processing times.
> It’s crazy to me to think someone actually thought using step functions and lambdas in this way was a good idea.
It’s only crazy to you because you don’t know what you’re doing. But it seems the average HNer isn’t smart enough to digest anything beyond “AWS” and “Cost” before raging about stuff they know nothing about.
> Splitting thousands of hours of video into 1-2s chunks, uploading them to s3, then processing them in isolation?
Now that you place it like that, it does sound very dubious. The overhead required by uploading MBs go S3 go afterwards have to download it back to the lamdda's execution context should have the same order of magnitude as processing the payload. I'm sure they were aware of this but we're hoping that things would scale well due to the embarrassingly parallel nature of the problem.
> But if your microservice is ThatOneCatOnMyBlock(), it might be a sign that you have broken things down too far.
This is what I commonly see Lambda turn into and why I think it's real messy. People think that adding another Lambda is as cheap as adding a function in regular code and put them everywhere. I haven't seen it as much with microservices, might just be where I work though.
Cloudflare workers had a interesting solve for this where they don't let you make a http request to another worker, they mount the code for that worker in the same process. Kind of wish AWS could find a way to do that, although since they support multiple runtimes I don't know if it is possible.
I think they used the word "monolith" to denote that 4 different things[1] (orchestration, detection, media converter, aggregation) were running together within a ECS task/container. Rather than each in their own, separate, horizontally scalable thing. So there was horizontal scaling, but not independent horizontal scaling for each of the 4 things.
The original article[2] about moving to a monolith didn't say much about how those 4 things were built..one binary/script, more?
So I think just a bunch of confusion on what was meant by "monolith".
Some quality publications cannot seem to come to the point for some reason. This was full of so much fluff, I gave it twice as much time as it deserved but still don't know what its point is.
I prefer working on monoliths as opposed to microservices. Microservices (2-pizza team model microservices, or microservices uses for this one little thing) are a reaction to be development practices. Many companies have a monolith which requires a dedicated team doing builds and deploys because its built on decades of bad practice and bolted on scripts deploying to bespoke servers; the company then build a new cicd pipeline and makes microservices that use the pipeline to deploy into a cloud with a vm, then they are shocked at how much better the experience is. But the improvement is from better infra and cicd, not moving monolith to microservices.
The sweet spot is IMO: Monolith until the monolith becomes truly unwieldy, then break off core pieces of functionality into other services. So 1 monolith -> 4 "microservices". The 100s of microservices is a mistake.
What do you consider tryly unwielding, and what happens when the 4 microservices become that? Do they all break up into 16 microservices? In a few more iterations you'll have over 100 microservices running, so it seems inevitable for a large organization.
I think many teams should manage a single service, which really drastically lowers on call support burden. Unwieldy is a single team (10 or fewer people) supporting a service by themselves, or more than one service by themselves. If at the point you have 16 microservices, that means you should have 16 very distinct areas of technology and business. The monolith to 4 services was an example, and I think you should really be careful each time you carve out any area of business, and the better focus is on focusing on how to scale the monolith further.
I guess "monolithic" is relative -- a microservice is a monolithic alternative to a bunch of step functions and lambda functions.
I bet this particular story would be fascinating if we had more details, but mostly it's funny (and annoying) that so many people are invested in the idea that there is a "monolith/microservices" debate that makes sense outside of a particular engineering problem and context.
It's funny how in engineering religions arise from anti-religiosity. We have a religion of people yelling "monoliths!" thinking they are not religious but are only fighting against the microservices religion, and vice-versa, a religion of people yelling "microservices!" that think they are not religious but are only fighting against the monolith religion.
Also I think the sizing of the boundary context is wrong. If a service is too fine, it will be come chatty like this one. They colated the features into 1 hence making the granular level more coarse.
44 comments
[ 3.5 ms ] story [ 119 ms ] threadDoes anyone even consider the monster layers of abstraction that these services require? It’s crazy to me to think someone actually thought using step functions and lambdas in this way was a good idea.
Maybe Amazon used this to test step functions and lambdas.
I've done some AWS training, especially the introductory ones are intended to get you to use as many different services as possible. I'm sure Amazon's internal training is just that.
That sounds like an obvious good thing right? The training isn't supposed to teach you how to solve problems, its supposed to teach you how to use a particular set of tools.
But all too often, the training picks something like "build a clubhouse." And in that, I would expect legit solutions based on specifications of a clubhouse that you are wanting to build.
Language advocates fall hard on this. They will "build a TODO application" to show off part of the language. But end up with an application that can't do what folks would want a TODO application to actually do. It is very frustrating.
The geniuses who led the project before me decided that the first step was to split the file into single payment chunks, then put each one into RabbitMQ, from where a different service would pick them up and parse them - then encode the parsed payments in JSON, and put them back in RabbitMQ, for a third process to pick up and handle. This was all for performance, of course.
I tried to explain that just parsing the whole file in one go would be much faster than splitting it up, writing it to a queue, and reading it back in. That it probably wouldn't be much slower than the splitting. But nobody was interested. Obviously distributing the work would be much faster.
It was a weird experience. My colleagues just had no ability to reason about the "physics" of what computers did. I think they mostly had Rails backgrounds, and I wonder if you just don't exercise that muscle much doing Rails.
It still holds true today, and software has become bloated and slower because these simple rules are being ignored. It's all in the cloud, it's all distributed, it's all "free".
As for Rails - it's the same problem but it's different. If you stay too high up in the abstraction orbit, it's hard to think of the nitty gritty of performance.
What is the cloud doing? What is the ORM doing? Once it's all "magic", you just keep piling on non-performant crap until it becomes an emergency. Then you just add more cache :)
At many jobs I had to deal with the all-hands-on-deck, code red emergency "our AWS bill got too damn high, what can we cut" situation. I can guarantee you a ton of companies are going through this exercise right now.
Some software engineers purposefully develop using fewer resources to force themselves to reason about efficiency.
Abundance leads to complacency and laziness, and those resources are no longer cheap if you are abusing them.
One reason always given is the ability to replay chunks if the system goes down, at least for Kafka.
Same experience here. I feel like I'm always repeating myself when explaining to other developers why we should avoid moving data around unnecessary and prefer to operate directly in the database when necessary, specially when it comes to large data migrations. Also important is why it is bad to do a bunch of small operations instead of batching them in chunks closer to the page size (8 kB).
Not surprised. Rails seems to encourage the worst behavior in developers.
- Step one: put file in queue
- Step two: break down file and insert rows
- Step three: convert to json
- Step four: compute results
- etc...
Once your experience improves you start to understand the cost of such indirection and realise that the problem could have been solved in a much more straightforward manner.
Eh, that's a separate thing - that's how modern streaming works, i.e. it's what allowed streaming to become really, really good, and it's how pretty much every major streaming platform works today.
It’s only crazy to you because you don’t know what you’re doing. But it seems the average HNer isn’t smart enough to digest anything beyond “AWS” and “Cost” before raging about stuff they know nothing about.
Now that you place it like that, it does sound very dubious. The overhead required by uploading MBs go S3 go afterwards have to download it back to the lamdda's execution context should have the same order of magnitude as processing the payload. I'm sure they were aware of this but we're hoping that things would scale well due to the embarrassingly parallel nature of the problem.
This is what I commonly see Lambda turn into and why I think it's real messy. People think that adding another Lambda is as cheap as adding a function in regular code and put them everywhere. I haven't seen it as much with microservices, might just be where I work though.
Cloudflare workers had a interesting solve for this where they don't let you make a http request to another worker, they mount the code for that worker in the same process. Kind of wish AWS could find a way to do that, although since they support multiple runtimes I don't know if it is possible.
The original article[2] about moving to a monolith didn't say much about how those 4 things were built..one binary/script, more?
So I think just a bunch of confusion on what was meant by "monolith".
[1] https://imgur.com/a/6nUFga0
[2] https://www.primevideotech.com/video-streaming/scaling-up-th...
Shocking.
Huh?
The sweet spot is IMO: Monolith until the monolith becomes truly unwieldy, then break off core pieces of functionality into other services. So 1 monolith -> 4 "microservices". The 100s of microservices is a mistake.
I bet this particular story would be fascinating if we had more details, but mostly it's funny (and annoying) that so many people are invested in the idea that there is a "monolith/microservices" debate that makes sense outside of a particular engineering problem and context.
It's funny how in engineering religions arise from anti-religiosity. We have a religion of people yelling "monoliths!" thinking they are not religious but are only fighting against the microservices religion, and vice-versa, a religion of people yelling "microservices!" that think they are not religious but are only fighting against the monolith religion.