Love the transparency here. Would also love if the same transparency was applied to pricing for their core product. Doesn't appear anywhere on the site.
I had the same question, but I imagine that the "media pipeline" box with a line that goes directly from "compositor" to "encoder" is probably hiding quite a lot of complexity
Recall's offering allows you to get "audio, video, transcripts, and metadata" from video calls -- again, total conjecture, but I imagine they do need to decode into raw format in order to split out all these end-products (and then re-encode for a video recording specifically.)
Reading their product page, it seems like Recall captures meetings on whatever platform their customers are using: Zoom, Teams, Google Meet, etc.
Since they don't have API access to all these platforms, the best they can do to capture the A/V streams is simply to join the meeting in a headless browser on a server, then capture the browser's output and re-encode it.
If you want to support every meeting platform, you can’t really make any assumptions about the data format.
To my knowledge, Zoom’s web client uses a custom codec delivered inside a WASM blob. How would you capture that video data to forward it to your recording system? How do you decode it later?
Even if the incoming streams are in a standard format, compositing the meeting as a post-processing operation from raw recorded tracks isn’t simple. Video call participants have gaps and network issues and layer changes, you can’t assume much anything about the samples as you would with typical video files. (Coincidentally this is exactly what I’m working on right now at my job.)
At some point, I'd hope the result of zooms code quickly becomes something that can be hardware decoded. Otherwise the CPU, battery consumption, and energy usage are going to be through the roof.
The most common video conferencing codec on WebRTC is VP8, which is not hardware decoded either almost anywhere. Zoom’s own codec must be an efficiency improvement over VP8, which is best described as patent-free leftovers from the back of the fridge.
Hardware decoding works best when you have a single stable high bitrate stream with predictable keyframes — something like a 4K video player.
Video meetings are not like that. You may have a dozen participant streams, and most of them are suffering from packet loss. Lots of decoder context switching and messy samples is not ideal for typical hardware decoders.
This makes sense. I find it curious that a WASM codec could be competitive with something that is presumably decoded natively. I know Teams is a CPU hog, but I don't remember Zoom being one.
my guess is either that video they get use some proprietary encoding format (js might do some magic on the feed) or it's because it's latency optimized stream that consumes a lot of bandwidth
I don't mean to be dismissive, but this would have been caught very early on (in the planning stages) by anyone that had/has experience in system-level development rather than full-stack web js/python development. Quite an expensive lesson for them to learn, even though I'm assuming they do have the talent somewhere on the team if they're able to maintain a fork of Chromium.
(I also wouldn't be surprised if they had even more memory copies than they let on, marshalling between the GC-backed JS runtime to the GC-backed Python runtime.)
I was coming back to HN to include in my comment a link to various high-performance IPC libraries, but another commenter already beat me linking to iceoryx2 (though of course they'd need to use a python extension).
SHM for IPC has been well-understood as the better option for high-bandwidth payloads from the 1990s and is a staple of Win32 application development for communication between services (daemons) and clients (guis).
> I don't mean to be dismissive, but this would have been caught very early on (in the planning stages) by anyone that had/has experience in system-level development rather than full-stack web js/python development
Based on their job listing[0], Recall is using Rust on the backend.
Sometimes it is more important to work on proving you have a viable product and market to sell it in before you optimise.
On the outside we can’t be sure. But it’s possible that they took the right decision to go with a naïve implementation first. Then profile, measure and improve later.
But yes the hole idea of running a headless web browser to get run JavaScript to get access to a video stream is a bit crazy. But I guess that’s just the world we are in.
> rather than full-stack web js/python development.
The product is not a full-stack web application. What makes you think that they brought in people with that kind of experience just for this particular feature?
Especially when they claim that they chose that route because it was what was most convenient. While you might argue that wasn't the right tradeoff, it is a common tradeoff developers of all kinds make. “Make It Work, Make It Right, Make It Fast” has become pervasive in this industry, for better or worse.
I don't think that number is as easy to figure out as most people think.
Netflix has hardware ISPs can get so they can serve their content without saturating the ISPs lines.
There is a statistic floating around that Netflix was responsible for 15% of the global traffic 2022/2023, and YouTube 12%. If that number is real... That'd be a lot more
That's a good write-up with a standard solution in some other spaces. Shared memory buffers are very fast too. It's interesting to see them being used here. Nice write up. It wasn't what I expected: that they were doing something dumb with API Gateway Websockets. This is actual stuff. Nice.
I read the entire article and that wasn't my takeaway. After reading, I assumed that AWS was (somehow) billing for loopback bandwidth, it wasn't apparent (to me) from the article that CPU costs were the sticking point
The article describes why this isn't the problem. You might enjoy reading it.
The basic point is that WebSockets requires that data move across channels that are too general and cause multiple unaligned memory copies. The CPU cost to do the copies was what cost the megabuck, not network transfer costs.
> A single 1080p raw video frame would be 1080 * 1920 * 1.5 = 3110.4 KB in size
They seem to not understand the fundamentals of what they're working on.
> Chromium's WebSocket implementation, and the WebSocket spec in general, create some especially bad performance pitfalls.
You're doing bulk data transfers into a multiplexed short messaging socket. What exactly did you expect?
> However there's no standard interface for transporting data over shared memory.
Yes there is. It's called /dev/shm. You can use shared memory like a filesystem, and no, you should not be worried about user/kernel space overhead at this point. It's the obvious solution to your problem.
> Instead of the typical two-pointers, we have three pointers in our ring buffer:
You can use two back to back mmap(2) calls to create a ringbuffer which avoids this.
It's pretty funny that they assumed that memory copying was the limiting factor when they're pushing a mere 150MB/s around instead of the various websocket overheads, then jumped right into over-engineering a zero copy ring buffer. I get it, but come on!
>50 GB/s of memory bandwidth is common nowadays[1], and will basically never be the bottleneck for 1080P encoding. Zero copy matters when you're doing something exotic, like Netflix pushing dozens of GB/s from a CDN node.
well someone will feel like an idiot after reading your facts. This is why education and experience is important. Not just React/rust course and then you are full stack senior :D
The title makes it sound like there was some kind of blowout, but really it was a tool that wasn't the best fit for this job, and they were using twice as much CPU as necessary, nothing crazy.
What does this mean? The article says 'TB' which would be terabytes. Terabytes are made out of gigabytes. There is nothing faster than straight memory bandwidth. DDR5 has 64 GB/s max. 12 channels of that is 768 GB/s.
Terabytes per second is going to take multiple computers, but it will be a lot less computers if you're using shared memory bandwidth and not some sort of networking loopback.
Is this really an AWS issue? Sounds like you were just burning CPU cycles, which is not AWS related. WebSockets makes it sound like it was a data transfer or API gateway cost.
I doubt they would have even noticed this outrageous cost if they were running on bare-metal Xeons or Ryzen colo'd servers. You can rent real 44-core Xeon servers for like, $250/month.
AWS was working “fine” for about 10 years without live migration, and I’ve had several individual machines running without a reboot or outage for quite literally half a decade. Enough to hit bugs like this: https://support.hpe.com/hpesc/public/docDisplay?docId=a00092...
Anyway, depending on individual nodes to always be up for reliability is incredibly foolhardy. Things can happen, cloud isn't magic, I’ve had instances become unrecoverable. Though it is rare.
So, I still don’t understand the point, that was not exactly relevant to what I said.
I think they meant that Hetzner is offering specific machines they know to be faulty and should have EOLd to customers, not that they use deprecated CPUs.
It's not scary, it's part of the value proposition.
I used to work for a company that rented lots of hetzner boxes. Consumer grade hardware with frequent disk failures was just what we excepted for saving a buck.
I know serious businesses using Hetzner for their critical workloads. I wouldn’t unless money is tight, but it is possible. I use them for my non critical stuff, it costs so much less.
I don't know anything about Colo Crossing (are they a reseller?) but I would bet their $60 per month 4-core Intel Xeons would outperform a $1,000 per month "compute optimized" EC2 server.
For $1000 per month you can get a c8g.12xlarge (assuming you use some kind of savings plan).[0] That's 48 cores, 96 GB of RAM and 22.5+ Gbps networking. Of course you still need to pay for storage, egress etc., but you seem to be exaggerating a bit....they do offer a 44 core Broadwell/128 GB RAM option for $229 per month, so AWS is more like a 4x markup[1]....the C8g would likely be much faster at single threaded tasks though[2][3]
That's not dedicated 48 cores, it's 48 "vCPUs". There are probably 1,000 other EC2 instances running on those cores stealing all the CPU cycles. You might get 4 cores of actual compute throughput. Which is what I was saying
That's not how it works, sorry. (Unless you use burstable instances, like T4g) You can run them at 100% as long as you like, and it has the same performance (minus a small virtualization overhead).
Are you telling me that my virtualized EC2 server is the only thing running on the physical hardware/CPU? There are no other virtualized EC2 servers sharing time on that hardware/CPU?
If you are talking about regular EC2 (not T series, or Lambda, or Fargate etc.) you get the same performance (within say 5%) of the underlying hardware. If you're using a core, it's not shared with another user. The pricing validates this...the "metal" version of a server on AWS is the same price as the full regular EC2 version.
In fact, you can even get a small discount with the -flex series, if you're willing to compromise slightly. (Small discount for 100% of performance 95% of the time).
This seems pretty wild to me. Are you saying that I can submit instructions to the CPU and they will not be interleaved and the registers will not be swapped-out with instructions from other EC2 virtual server applications running on the same physical machine?
Wouldn't c8g.12xlarge with 500g storage (only EBS is possible), plus 1gbps from/to the internet is 5,700 USD per month, that's some discount you have.
If I try to match the actual machine. 16G ram. A rough estimate is that their Xeon E3-1240 would be ~2 AWS vCPU. So an r6g.large is the instance that would roughly match this one. Add 500G disk + 1 Gbps to/from the internet and ... monthly cost 3,700 USD.
Without any disk and without any data transfer (which would be unusable) it's still ~80USD. Maybe you could create a bootable image that calculates primes.
These are still not the same thing, I get it, but ... it's safe to say you cannot get anything remotely comparable on AWS. You can only get a different thing for way more money.
Neither the title nor the article are painting it as an AWS issue, but as a websocket issue, because the protocol implicitly requires all transferred data to be copied multiple times.
Only for people that just read headlines and make technical decisions based on them. Are we catering to them now? The title is factual and straightforward.
I disagree. Like @turtlebits, I was waiting for the part of the story where websocket connections between their AWS resources somehow got billed at Amazon's internet data egress rates.
They are presumably using the GPU for video encoding....
And the GPU for rendering...
So they should instead just be hooking into Chromium's GPU process and grabbing the pre-composited tiles from the LayerTreeHostImpl[1] and dealing with those.
> We do our video processing on the CPU instead of on GPU, as GPU availability on the cloud providers has been patchy in the last few years.
I dunno, when we're playing with millions of dollars in costs I hope they're at least regularly evaluating whether they could at least run some of the workload on GPUs for better perf/$.
And their workload is rendering and video encoding. Using GPU's should have been where they started, even if it limits their choice of cloud providers a little.
Here they have a nicely compressed stream of video data, so they take that stream and... decode it. But they aren't processing the decoded data at the source of the decode, so instead they forward that decoded data, uncompressed(!!), to a different location for processing. Surprisingly, they find out that moving uncompressed video data from one location to another is expensive. So, they compress it later (Don't worry, using a GPU!)
At so many levels this is just WTF. Why not forward the compressed video stream? Why not decompress it where you are processing it instead of in the browser? Why are you writing it without any attempt at compression? Even if you want lossless compression there are well known and fast algorithms like flv1 for that purpose.
> Usually you'd leave it compressed until the last possible moment.
Context matters? As someone working in production/post, we want to keep it uncompressed until the last possible moment. At least as far as no more compression than how it was acquired.
It does, but you just removed all context from their comment and introduced a completely different context (video production/post) for seemingly no reason.
Going back to the original context, which is grabbing a compressed video stream from a headless browser, the correct approach to handle that compressed stream is to leave it compressed until the last possible moment.
Since they aim to support every meeting platform, they don’t necessarily even have the codecs. Platforms like Zoom can and do use custom video formats within their web clients.
With that constraint, letting a full browser engine decode and composite the participant streams is the only option. And it definitely is an expensive way to do it.
Possibly because they capture the video from xvfb or similar (they run a headless browser to capture the video) so at that point the decoding already happened (webrtc?)
I think the issue with compression is that they're scraping the online meeting services rather than actually reverse engineering them, so the compressed video stream is hidden inside some kind of black box.
I'm pretty sure that feeding the browser an emulated hardware decoder (ie - write a VAAPI module that just copies compressed frame data for you) would be a good semi-universal solution to this, since I don't think most video chat solutions use DRM like Widevine, but it's not as universal as dumping the framebuffer output off of a browser session.
They could also of course one-off reverse each meeting service to get at the backing stream.
What's odd to me is that even with this frame buffer approach, why would you not just recompress the video at the edge? You could even do it in Javascript with WebCodecs if that was the layer you were living at. Even semi-expensive compression on a modern CPU is going to be way cheaper than copying raw video frames, even just in terms of CPU instruction throughput vs memory bandwidth with shared memory.
It's easy to cast stones, but this is a weird architecture and making this blog post about the "solution" is even stranger to me.
> I think the issue with compression is that they're scraping the online meeting services rather than actually reverse engineering them, so the compressed video stream is hidden inside some kind of black box.
I mean, I would presume that the entire reason they forked chrome was to crowbar open the black box to get at the goodies. Maybe they only did it to get a framebuffer output stream that they could redirect? Seems a bit much.
Their current approach is what I'd think would be a temporary solution while they reverse engineer the streams (or even get partnerships with the likes of MS and others. MS in particular would likely jump at an opportunity to AI something).
> What's odd to me is that even with this frame buffer approach, why would you not just recompress the video at the edge? You could even do it in Javascript with WebCodecs if that was the layer you were living at. Even semi-expensive compression on a modern CPU is going to be way cheaper than copying raw video frames, even just in terms of CPU instruction throughput vs memory bandwidth with shared memory.
Yeah, that was my final comment. Even if I grant that this really is the best way to do things, I can't for the life of me understand why they'd not immediately recompress. Video takes such a huge amount of bandwidth that it's just silly to send around bitmaps.
> It's easy to cast stones, but this is a weird architecture and making this blog post about the "solution" is even stranger to me.
Agreed. Sounds like a company that likely has multiple million dollar savings just lying around.
> Their current approach is what I'd think would be a temporary solution while they reverse engineer the streams (or even get partnerships with the likes of MS and others. MS in particular would likely jump at an opportunity to AI something).
They support 7 meeting platforms. Even if 1 or 2 are open to providing APIs, they're not all going to do that.
Reverse-engineering the protocol would be far more efficient, yes - but it'd also be more brittle. The protocol could change at any time and reverse-engineering it again could days between days and weeks. Would you want a product with that sort of downtime?
Also, does it scale? Reverse-engineering 7+ protocols is a lot of engineering work, and it's very specialized work that not any software engineer could just dive into quickly.
In comparison, writing web scrapers to find the video element for 7 different meeting products is super easy to write, and super easy to fix.
> Here they have a nicely compressed stream of video data
But they don't.
They support 7 different meeting providers (Zoom, Meet, WebEx, ...), none of which have an API that give you access to the compressed video stream.
In theory, you could try to reverse-engineer each protocol...but then your product could break for potentially days or weeks anytime one of those companies decides to change their protocol - vs web scraping, where if it breaks they can probably fix it in 15 minutes.
Their solution is inefficient, but robust. And that's ultimately a more monetizable product.
Masking in the WebSocket protocol is kind of a funny and sad fix to the problem of intermediaries trying to be smart and helpful, but failing miserably.
I for one would like to praise the company for sharing their failure, hopefully next time someone Googles "transport video over websocket" theyll find this thread.
236 comments
[ 4.1 ms ] story [ 238 ms ] threadJokes aside though, some good performance sleuthing there.
Recall's offering allows you to get "audio, video, transcripts, and metadata" from video calls -- again, total conjecture, but I imagine they do need to decode into raw format in order to split out all these end-products (and then re-encode for a video recording specifically.)
Since they don't have API access to all these platforms, the best they can do to capture the A/V streams is simply to join the meeting in a headless browser on a server, then capture the browser's output and re-encode it.
To my knowledge, Zoom’s web client uses a custom codec delivered inside a WASM blob. How would you capture that video data to forward it to your recording system? How do you decode it later?
Even if the incoming streams are in a standard format, compositing the meeting as a post-processing operation from raw recorded tracks isn’t simple. Video call participants have gaps and network issues and layer changes, you can’t assume much anything about the samples as you would with typical video files. (Coincidentally this is exactly what I’m working on right now at my job.)
Hardware decoding works best when you have a single stable high bitrate stream with predictable keyframes — something like a 4K video player.
Video meetings are not like that. You may have a dozen participant streams, and most of them are suffering from packet loss. Lots of decoder context switching and messy samples is not ideal for typical hardware decoders.
(I also wouldn't be surprised if they had even more memory copies than they let on, marshalling between the GC-backed JS runtime to the GC-backed Python runtime.)
I was coming back to HN to include in my comment a link to various high-performance IPC libraries, but another commenter already beat me linking to iceoryx2 (though of course they'd need to use a python extension).
SHM for IPC has been well-understood as the better option for high-bandwidth payloads from the 1990s and is a staple of Win32 application development for communication between services (daemons) and clients (guis).
They have APIs to schedule meetings.
They don't have APIs that give you access to the compressed video stream.
[0] https://www.workatastartup.com/companies/recall-ai
On the outside we can’t be sure. But it’s possible that they took the right decision to go with a naïve implementation first. Then profile, measure and improve later.
But yes the hole idea of running a headless web browser to get run JavaScript to get access to a video stream is a bit crazy. But I guess that’s just the world we are in.
The product is not a full-stack web application. What makes you think that they brought in people with that kind of experience just for this particular feature?
Especially when they claim that they chose that route because it was what was most convenient. While you might argue that wasn't the right tradeoff, it is a common tradeoff developers of all kinds make. “Make It Work, Make It Right, Make It Fast” has become pervasive in this industry, for better or worse.
As a point of comparison, how many TB per second of video does Netflix stream?
Netflix has hardware ISPs can get so they can serve their content without saturating the ISPs lines.
There is a statistic floating around that Netflix was responsible for 15% of the global traffic 2022/2023, and YouTube 12%. If that number is real... That'd be a lot more
"using WebSockets over loopback was ultimately costing us $1M/year in AWS spend"
then
"and the quest for an efficient high-bandwidth, low-latency IPC"
Shared memory. It has been there for 50 years.
From the article intro before they dive into what exactly is using the CPU.
The memcopys are the cost that they were paying, even if it was local.
The basic point is that WebSockets requires that data move across channels that are too general and cause multiple unaligned memory copies. The CPU cost to do the copies was what cost the megabuck, not network transfer costs.
They seem to not understand the fundamentals of what they're working on.
> Chromium's WebSocket implementation, and the WebSocket spec in general, create some especially bad performance pitfalls.
You're doing bulk data transfers into a multiplexed short messaging socket. What exactly did you expect?
> However there's no standard interface for transporting data over shared memory.
Yes there is. It's called /dev/shm. You can use shared memory like a filesystem, and no, you should not be worried about user/kernel space overhead at this point. It's the obvious solution to your problem.
> Instead of the typical two-pointers, we have three pointers in our ring buffer:
You can use two back to back mmap(2) calls to create a ringbuffer which avoids this.
>50 GB/s of memory bandwidth is common nowadays[1], and will basically never be the bottleneck for 1080P encoding. Zero copy matters when you're doing something exotic, like Netflix pushing dozens of GB/s from a CDN node.
[1]: https://lemire.me/blog/2024/01/18/how-much-memory-bandwidth-...
And since it behaves like filesystem, you can swap it with real filesystem during testing. Very convenient.
I am curious if they tried this already or not and if they did, what problems did they encounter?
that’s surprising to.. almost no one? 1TBPS is nothing to scoff at
assuming you're only shuffling bytes around, on bare metal this would be ~20 DDR5 channels worth
or 2 servers (12 channels/server for EPYC)
you can get an awful lot of compute these days for not very much money
(shipping your code to the compressed video instead of the exact opposite would probably make more sense though)
pro-tip: it's quite a bit bigger than a terabit
What does this mean? The article says 'TB' which would be terabytes. Terabytes are made out of gigabytes. There is nothing faster than straight memory bandwidth. DDR5 has 64 GB/s max. 12 channels of that is 768 GB/s.
Terabytes per second is going to take multiple computers, but it will be a lot less computers if you're using shared memory bandwidth and not some sort of networking loopback.
I doubt they would have even noticed this outrageous cost if they were running on bare-metal Xeons or Ryzen colo'd servers. You can rent real 44-core Xeon servers for like, $250/month.
So yes, it's an AWS issue.
GCP exposes their cpu models, and they have some Haswell and Broadwell lithographies in service.
Thats a 10+ year old part, for those paying attention.
Anyway, depending on individual nodes to always be up for reliability is incredibly foolhardy. Things can happen, cloud isn't magic, I’ve had instances become unrecoverable. Though it is rare.
So, I still don’t understand the point, that was not exactly relevant to what I said.
I used to work for a company that rented lots of hetzner boxes. Consumer grade hardware with frequent disk failures was just what we excepted for saving a buck.
AWS: E5-2680 v4 (2016)
Hetzner: Ryzen 5 (2019)
the AWS one is some emulated block device, no idea what it is, other than it's 20x slower
Hetzner isn't the best provider in the world, but it's also not as bad as you say they are. They're not just renting old servers.
I see "AMD EPYC 7502P 32-Core" for 236 EUR per month. Can you tell me where you see 48c/96t?
EDIT
I found it! Unbelievable that it is so cheap.
https://www.hetzner.com/dedicated-rootserver/#cores_threads_...
https://www.colocrossing.com/server/dedicated-servers/
[0]https://instances.vantage.sh/aws/ec2/c8g.12xlarge?region=us-... [1]https://portal.colocrossing.com/register/order/service/480 [2]https://browser.geekbench.com/v6/cpu/8305329 [3]https://browser.geekbench.com/processors/intel-xeon-e5-2699-...
That's not dedicated 48 cores, it's 48 "vCPUs". There are probably 1,000 other EC2 instances running on those cores stealing all the CPU cycles. You might get 4 cores of actual compute throughput. Which is what I was saying
In fact, you can even get a small discount with the -flex series, if you're willing to compromise slightly. (Small discount for 100% of performance 95% of the time).
If I try to match the actual machine. 16G ram. A rough estimate is that their Xeon E3-1240 would be ~2 AWS vCPU. So an r6g.large is the instance that would roughly match this one. Add 500G disk + 1 Gbps to/from the internet and ... monthly cost 3,700 USD.
Without any disk and without any data transfer (which would be unusable) it's still ~80USD. Maybe you could create a bootable image that calculates primes.
These are still not the same thing, I get it, but ... it's safe to say you cannot get anything remotely comparable on AWS. You can only get a different thing for way more money.
(made estimates on https://calculator.aws/ )
125 MB per second × 60 seconds per minute × 60 minutes per hour × 24 hours per day x 30 days = 324 TB?
If you want 1 Gbps unmetered colo pricing, AWS is not competitive. So set up your video streaming service elsewhere :-)
https://portal.colocrossing.com/register/order/service/480 offers unmetered for $2,500 additional per month, for the record.
If you have high bandwidth needs on AWS you can use AWS Lightsail, which has some discounted transfer rates.
If I said that "childbirth cost us 5000 on our <hospital name> bill", you assume the issue is with the hospital.
The idea that clearer titles are just babying some class of people is perverse.
Titles are the foremost means of deciding what to read, for anyone of any sophistication. Clearer titles benefit everyone.
The subject matter is meaningful to more than AWS users, but non-AWS users are going to be less likely to read it based on the title.
And the GPU for rendering...
So they should instead just be hooking into Chromium's GPU process and grabbing the pre-composited tiles from the LayerTreeHostImpl[1] and dealing with those.
[1]: https://source.chromium.org/chromium/chromium/src/+/main:cc/...
I dunno, when we're playing with millions of dollars in costs I hope they're at least regularly evaluating whether they could at least run some of the workload on GPUs for better perf/$.
Here they have a nicely compressed stream of video data, so they take that stream and... decode it. But they aren't processing the decoded data at the source of the decode, so instead they forward that decoded data, uncompressed(!!), to a different location for processing. Surprisingly, they find out that moving uncompressed video data from one location to another is expensive. So, they compress it later (Don't worry, using a GPU!)
At so many levels this is just WTF. Why not forward the compressed video stream? Why not decompress it where you are processing it instead of in the browser? Why are you writing it without any attempt at compression? Even if you want lossless compression there are well known and fast algorithms like flv1 for that purpose.
Just weird.
As it turns out, doing something in Rust does not absolve you of the obligation to actually think about what you are doing.
Context matters? As someone working in production/post, we want to keep it uncompressed until the last possible moment. At least as far as no more compression than how it was acquired.
It does, but you just removed all context from their comment and introduced a completely different context (video production/post) for seemingly no reason.
Going back to the original context, which is grabbing a compressed video stream from a headless browser, the correct approach to handle that compressed stream is to leave it compressed until the last possible moment.
With that constraint, letting a full browser engine decode and composite the participant streams is the only option. And it definitely is an expensive way to do it.
I'm pretty sure that feeding the browser an emulated hardware decoder (ie - write a VAAPI module that just copies compressed frame data for you) would be a good semi-universal solution to this, since I don't think most video chat solutions use DRM like Widevine, but it's not as universal as dumping the framebuffer output off of a browser session.
They could also of course one-off reverse each meeting service to get at the backing stream.
What's odd to me is that even with this frame buffer approach, why would you not just recompress the video at the edge? You could even do it in Javascript with WebCodecs if that was the layer you were living at. Even semi-expensive compression on a modern CPU is going to be way cheaper than copying raw video frames, even just in terms of CPU instruction throughput vs memory bandwidth with shared memory.
It's easy to cast stones, but this is a weird architecture and making this blog post about the "solution" is even stranger to me.
I mean, I would presume that the entire reason they forked chrome was to crowbar open the black box to get at the goodies. Maybe they only did it to get a framebuffer output stream that they could redirect? Seems a bit much.
Their current approach is what I'd think would be a temporary solution while they reverse engineer the streams (or even get partnerships with the likes of MS and others. MS in particular would likely jump at an opportunity to AI something).
> What's odd to me is that even with this frame buffer approach, why would you not just recompress the video at the edge? You could even do it in Javascript with WebCodecs if that was the layer you were living at. Even semi-expensive compression on a modern CPU is going to be way cheaper than copying raw video frames, even just in terms of CPU instruction throughput vs memory bandwidth with shared memory.
Yeah, that was my final comment. Even if I grant that this really is the best way to do things, I can't for the life of me understand why they'd not immediately recompress. Video takes such a huge amount of bandwidth that it's just silly to send around bitmaps.
> It's easy to cast stones, but this is a weird architecture and making this blog post about the "solution" is even stranger to me.
Agreed. Sounds like a company that likely has multiple million dollar savings just lying around.
They support 7 meeting platforms. Even if 1 or 2 are open to providing APIs, they're not all going to do that.
Reverse-engineering the protocol would be far more efficient, yes - but it'd also be more brittle. The protocol could change at any time and reverse-engineering it again could days between days and weeks. Would you want a product with that sort of downtime?
Also, does it scale? Reverse-engineering 7+ protocols is a lot of engineering work, and it's very specialized work that not any software engineer could just dive into quickly.
In comparison, writing web scrapers to find the video element for 7 different meeting products is super easy to write, and super easy to fix.
But they don't.
They support 7 different meeting providers (Zoom, Meet, WebEx, ...), none of which have an API that give you access to the compressed video stream.
In theory, you could try to reverse-engineer each protocol...but then your product could break for potentially days or weeks anytime one of those companies decides to change their protocol - vs web scraping, where if it breaks they can probably fix it in 15 minutes.
Their solution is inefficient, but robust. And that's ultimately a more monetizable product.
> Arrow libraries provide convenient methods for reading and writing columnar file formats
The linked section of the RFC is worth the read: https://www.rfc-editor.org/rfc/rfc6455#section-10.3
The RFC has a link to a document describing the attack, but the link is broken.