"The main goal of the team was to implement a job processing pipeline that was vendor agnostic and cloud-based"
I wonder if they feel they've achieved it?
How much work would it be to switch away from their choice of JS libraries and AWS?
And if it's not trivial, what's the point?
Do you need 20 microservices for that? If you choose to swap a video vendor, I guess you need to write wrapper API calls for the new vendor?
Even if the new vendor provides that out of the box, NYT amortizes those costs one way or the other...
It may be more or less, depending on how you count the, but what I'm really trying to figure out is whether the over engineering and the extra abstraction layers are justified in this case and in general.
My experience so far is the opposite - Microservices, orchestration etc hide the complexity but don't remove it, so if the shit hits the fan, you still have to look under the hood and with Microservices, probably under 10 different hoods or a hood under the hood of a hood :)
Multiple encoding vendors won't give you much. They're just different people writing different shell scripts that run ffmpeg (and not donating anything to the original project).
I guessit lets you avoid a little price lock-in, but you could also just write your own shell script.
I worked on something like this for a small video rendering startup a few years ago. Our backend needed to ingest short (<= 12s) videos and strip the parts we needed in our rendering stack. We polled S3 (SNS wasn't consistently fast enough at the time) for new files in our Upload directory.
The uploaded video was sent to multiple microservices in a group of tens or hundreds of instances. One would "normalize" the video so we had the same frame-rate, size, and bitrates to work with. Another would rip the audio into a couple formats. Another would generate HLS streams with a few resolutions. Another would generate the HLS playlist. Another would split the video up into images. Each service uploaded its results to S3, named as a SHA1 hash of itself, and reported to a single API which kept track of the "tree" of files generated from each upload.
The system was based upon an early version of NodeJS, which worked fantastically well as a generic daemonized wrapper around the command line tools we were using for all the ripping and modifying.
About a year before I left the team we changed to GCS for the upload server because they had reliable immediate notifications to webhooks for uploads, but kept the rest of the system on S3, since our processing instances were all EC2.
Would have loved to have Lambda back then!
We had a similar set of microservices for rendering the eventual videos in near-real-time - but that's a whole different conversation.
I made one of these 6 years ago for one of my clients using aws, on my todd and it's really not that complicated. Sure ours is a bit duct-tape and string, but seriously, it's not complicated programming. Probably took me a whole day.
At one point we switched from one encoding service to another and it took me less than half a day of chargeable time to change the code, so not sure why they need it to be so "flexible".
They've probably wasted more dev time on over-engineering it than they'd ever have saved manually adding a new encoding provider.
How it takes 4 of them to write it too I'm a bit mind-boggled. Reliable processing queues are a solved problem, encoding services are a solved problem, this whole thing was a solved problem 5 years ago, which is aeons in programming times.
10 comments
[ 3.3 ms ] story [ 41.5 ms ] threadI guessit lets you avoid a little price lock-in, but you could also just write your own shell script.
The uploaded video was sent to multiple microservices in a group of tens or hundreds of instances. One would "normalize" the video so we had the same frame-rate, size, and bitrates to work with. Another would rip the audio into a couple formats. Another would generate HLS streams with a few resolutions. Another would generate the HLS playlist. Another would split the video up into images. Each service uploaded its results to S3, named as a SHA1 hash of itself, and reported to a single API which kept track of the "tree" of files generated from each upload.
The system was based upon an early version of NodeJS, which worked fantastically well as a generic daemonized wrapper around the command line tools we were using for all the ripping and modifying.
About a year before I left the team we changed to GCS for the upload server because they had reliable immediate notifications to webhooks for uploads, but kept the rest of the system on S3, since our processing instances were all EC2.
Would have loved to have Lambda back then!
We had a similar set of microservices for rendering the eventual videos in near-real-time - but that's a whole different conversation.
I made one of these 6 years ago for one of my clients using aws, on my todd and it's really not that complicated. Sure ours is a bit duct-tape and string, but seriously, it's not complicated programming. Probably took me a whole day.
At one point we switched from one encoding service to another and it took me less than half a day of chargeable time to change the code, so not sure why they need it to be so "flexible".
They've probably wasted more dev time on over-engineering it than they'd ever have saved manually adding a new encoding provider.
How it takes 4 of them to write it too I'm a bit mind-boggled. Reliable processing queues are a solved problem, encoding services are a solved problem, this whole thing was a solved problem 5 years ago, which is aeons in programming times.
Here are the slides: https://speakerdeck.com/flavioribeiro/building-a-video-encod...