Ask HN: Video streaming is expensive yet YouTube "seems" to do it for free. How?
Can anyone help me understand the economics of video streaming platforms?
Streaming, encoding, and storage demands enormous costs -- especially at scale (e.g., on average each 4k video with close to 1 million views). Yet YouTube seems to charge no money for it.
I know advertisements are a thing for YT, but is it enough?
If tomorrow I want to start a platform that is supported with Advert revenues, I know I will likely fail. However, maybe at YT scale (or more specifically Google Advert scale) the economics works?
ps: I would like this discussion to focus on the absolute necessary elements (e.g., storing, encoding, streaming) and not on other factors contributing to latency/cost like running view count algorithms.
379 comments
[ 2.7 ms ] story [ 342 ms ] threadhttps://www.businessofapps.com/data/youtube-statistics/#:~:t....
That's... a lot! Plenty of historical precedent for fully advertising-supported media with high expenses, from OTA network television and radio to free weekly newspapers... or inexpensive subscriptions to daily newspapers subsidized by advertising. Advertising has been paying the bills for electronic media for a century now.
They have a direct financial incentive to have more people watch the same thing.
From 2015 "Some unnamed person at Google reportedly said that the site is "roughly break-even." https://www.cbsnews.com/news/4-reasons-youtube-still-doesnt-... which in turn quotes https://www.wsj.com/articles/viewers-dont-add-up-to-profit-f...
From March this year (2024): "Analyst Michael Nathanson of MoffettNathanson estimates that YouTube TV will become profitable this year" https://www.newscaststudio.com/2024/03/29/youtube-tv-profit-...
Part of this could be because they pay out roughly 40-60% of the revenue to the content creators, that leaves Google / Youtube with half the revenue that they use to pay salaries, maintain infrastructure, including storage, hosting and serving content.
Sandvine is the usual source in the industry and they say: "Sandvine's 2023 Global Internet Phenomena Report Shows 24% Jump in Video Traffic, with Netflix Volume Overtaking YouTube"
https://www.sandvine.com/press-releases/sandvines-2023-globa...
Netflix needs (1) a copy of say Top Gun Maverick (And maybe variations for different resolutions), YouTube needs a bazillion Copies of it with small variations, clips with different music, people "commenting" on it, people actually making commentary, Highlights, YTP's, etc all of it with a wider range of resolutions and also to way more people, plus the overhead of checking against their master copy of copyright infringement
YTTV is their "cable" product
The marginal costs go down a bit if your scale is truly immense. Google can afford to design/manufacture/deploy hyper-efficient custom silicon ASICs for encoding. Also because their critical mass of users provide valuable network effects, they can get away with particularly poor quality encoding (IMHO) and the vast majority of users still won't switch to other platforms with higher visual quality - but other (non-pornographic) video platforms generally don't have that luxury.
So in short, the only "on-demand" component is encoding, and if you don't have an 'available in an instant' promise, you can do it on spot instances on the cheapest cloud you can find; The rest is just storage and distribution - if you own a world-wide network of datacenters for your successful advertising service, that's kinda an already solved problem for you - just allocate a few racks to a new service.
I of course downplay everything and simplify massively - but at a high level, it's just a lot of ffmpeg -> S3 -> html5 player. The harder problems are in the long tail - high latency, content licensing & geo fencing, etc.
Source: used to SRE for a video streaming provider (not YT), also former GG
As Curly would say, "Eh, a big cheapskate, nyuk nyuk !" ;)
It's true they do have a "cheapening" effect, especially over time, but Curly's a knucklehead, I wouldn't want to compete with them on their own terms. That's a big gorilla.
>If tomorrow I want to start a platform
Seems like one approach would be to start out with what you can easily afford to begin with.
Which brings me exactly to the bare bones of storing, encoding, streaming and nothing else.
If nothing else to minimize complexity and cost of getting started.
And to possibly obviate the need for monetization up to a point.
To launch, just pick one fairly popular & accessible format/bit-rate and encode all your raw content (or a test portion of content) the exact same way in advance. Afterward, you're done with that phase and free of any need for real-time encoding. You still need to store and subsequently "outstream" your ready-to-deliver content.
It may actually cost you nothing to store a working copy of your encoded content "library" on your own private server on your own designated premises, especially if you already own the storage devices and there is plenty of unused storage space. There are also alternatives that are not without cost, only you could decide if it was worth money or not.
Naturally you will be limited by the bandwidth and infrastructure at each storage location, as to how many viewers at what resolution you can directly serve at one time, and whether or not the ISP/router can be configured to allow outside access to your server.
If you're going to use 100Mbps of surplus upload bandwidth from a business internet account for instance, and your content was encoded at 1.5Mbps (don't even think about 4K), it may be no additional cost to start serving viewers directly from that server, but you would not be able to serve more than about 50 viewers at one time.
That might get you started (at an appropriate scale) with no cash outlay whatsoever, and if the demand was there beyond a few dozen viewers then you could decide to pass your stream along to a more capable content delivery network of some kind, at various incremental cost.
Alternatively, the whole thing could be outsourced and hosted for world-wide access in a turnkey operation where all you do is supply the content. Cost may be a prohibiting factor, it does seem like there are hosting plans with a free tier but not with enough bandwidth to serve a meaningful number of viewers compared to YT.
Fortunately for YT, when they got started they didn't have competition already showing 4K stuff to compare to.
But if the action you take, has cost within the range of what you can easily bear, you could then afford to deliver a completely superior, ad-free experience for your fortunate few viewers. If you wanted to. Something a multi-billion-dollar company seems to be less and less able to afford. What a position to be in. If the whole thing actually was costing you no cash at all you'd be free to make it seem as free and frictionless as YT, probably more so because it was free from the ground up.
>a platform that is supported with Advert revenues
If you did decide to go this route and were sustainable without ads to begin with, you could very judiciously choose your sponsors to be ones that did not conflict with any feature that is more meaningful to the visitors. You would also be financially ahead beginning with the first ad you decided to run. And you could decide to stop at any time.
If you have the ipvfoo extension, you can see it in action. (its easier to see with IPv6)
https://github.com/pmarks-net/ipvfoo
The motto of so many social networks. Even if they don't know it.
Wow, I’m going to put this on a wall.
https://peering.google.com/#/infrastructure#edge-nodes
It also says "Static content that's very popular with the local host's user base, including YouTube and Google Play, is temporarily cached on edge nodes. Google's traffic management systems direct user requests to an edge node that provides the best experience."
it's arguable cheaper than buying but then again it's also the core business and outsourcing put's the whole operation in danger
A CDN is when you set up servers in datacenters around the country and distribute content to them so that your content is closer to your users.
What Netflix did is create guidelines for ISPs that want to cache Netflix content at the edge. So Netflix isn't running a CDN themselves, the ISP acts as a CDN.
> A CDN is when you set up servers in datacenters around the country and distribute content to them so that your content is closer to your users.
Yes, exacly - the server in the Netflix case is called an OCA, and is set up in datacenters all around the country - in this case they are owned by the ISP rather than being a 3rd party data center where the ISP has a fiber and the CDN rents space. Here's the thing though.... Akamai, Cloudflare, and Fastly all do the same as Netflix too. The ISP wins because it saves transit, the CDN wins because it doesn't need to rent dc space, and customers of both win because the content is delivered faster to the people using that ISP.
Or maybe you're talking about the peering guidelines? Hate to break it to you, every CDN is happy to peer with just about anyone with a mutual POP - keeping the transit bill lower is always a win.
Also: bandwidth gets a lot cheaper when you own the pipes.
And one that is less obvious: despite having hundreds of millions of videos available, a large contingent of people are watching the same ultra-popular ones. There are some economies of scale to be had there.
While I highly doubt Youtube is just plopping the videos into a queue and firing up `ffmpeg -i myvid.mp4 -c:v libx264 -y outvid.mp4` and presumably they some dedicated silicon (or at least FPGAs) handling it, but even still it's incredible how they can pull that off. Video conversion is fairly expensive!
Especially when you're already running ginormous data centers and can transcode on processors that are otherwise idle.
Also, video quality on YT is not great. When you're using ffmpeg to transcode a movie at home, you're probably doing it with fairly high quality settings because it's a movie.
But you can also configure ffmpeg to transcode very quickly if low quality output is fine.
Google has a lot of custom encoding silicon, too, AFAIK.
Storage is the biggest question of the three. Linus Sebastian specifically called this out when YouTube started really pushing to make the non-Premium experience dreadful. There isn't really some secret special sauce you can buy or make for storage. Literally everything is being stored with the same hard drives, SSDs, discs, or tapes you can just go out and buy. The only specialization you can do is build or buy equipment to handle extreme numbers of them. Google does buy these in bulk, so they probably get a discount on storage, but it's not something that would make storage costs just go away.
They'll have to do that eventually, right?
Especially if the amount of content uploaded keeps going up, so the relative benefit of deleting old stuff is small.
I agree. That's why I've recently made a practice of backing up things to which I'd regret permanently losing access.
https://ourworldindata.org/grapher/historical-cost-of-comput...
I have two accounts from my youth with mostly fansubs and "funny" vids. (A "Everybody Draw Mohammed Day"-Video that, 11 years after its upload, got band in Pakistan.) It includes two very successful (over 1M views combined) uploads of political TV shows. I will not log in to this this accounts. So this stuff will get deleted. So in the long run, stuff will disappear from YT.
[1] https://blog.google/technology/safety-security/updating-our-...
Would be nice if they would write that into the actual policy and don't just add a random sentence to a blog post.
The question is whether YT is serving up the one (redundantly-backed storage) copy they have of your almost-never-watched video, or whether it's serving it up from one of 1,000+ copies it's made across the globe for currently popular videos.
After that, you can plunge into colder storages and warm things up as you stream. Additionally, if you need longer to 'defrost' things, just cache a few more MB at the front. Cheat a bit by assuming 480p to start with if you need to; even less to store.
Maybe Google holds your content in 7 data centers round the world (~1 per continent for planned maintenance + latency + reduced oceanic fiber usage).
But with old rarely streamed content they might cut that down to just 3.
However, implementing this in practice is non-trivial. Knowing what is "everyday content" versus what is "once a month content".
To add more complexity -- you have these semi-predictable hype-waves especially two peaks in case of most YT videos where a "once-a-month" content becomes an "everyday" content before again becoming a "once-a-month" content. It feels you could specifically optimise for this -- reduce storage costs without sacrificing UX.
Caching is hard but this sounds like an ARC would likely catch this, if it occurs on a small number of videos concurrently.
My guess is that the first step will be to re-encode all the non-popular videos with severe lossy compression.
You know that because when they release a new format (eg. HDR or a different resolution), they re-encode from the original. Various people have tested that with moire patterns and various other ways to demonstrate if something was encoded more than once.
I suspect that’s Google beginning to trim unprofitable channels using a lot of storage: delete them for bullshit reasons.
Which makes competing with them effectively impossible except for a very-few other megacorps.
Most bandwidth is via settlement-free peering with thousands of ISPs around the world. At least that's how we did it at Twitch, and how we did it when I worked at a large CDN before that. There are still costs for backhaul, interconnect, colocation space, dark fiber, network hardware, and transit to fill the gaps. But this talk about how "Google can magically do it 5x cheaper" is nonsense.
I've worked at 2 ISPs and we obviously didn't have this peering policy, because it's dumb and it breaks the Internet. We also didn't throttle video, because it's dumb and breaks the Internet.
Genuine question - aren't those gaps essentially what make a video streaming service operate at scale though? It'd be like saying "ya this bus can get everyone from NYC to Philly at $10 but doesn't stop anywhere in between", or am I missing something about all of those gap filling components?
This is the model for all networks, indeed, most businesses - they pay a big upfront and moderate recurring cost to make a fast network (or restaurant or widget factory) and then sell it in slices with a large freedom to choose a pricing model. Pay per terabyte is a pretty reasonable way to pass on the network's fixed cost to consumers, just like part of the cost of the restaurant meal covers the interior decorations, even though the decorations don't actually cost more the more people eat, until the restaurant gets so busy it needs to expand.
So the 'terabit of network' the content provider needs to build need only span a few hundred feet within a single building.
Not all networks are the same. Some have terabit backbones and gigabit edges, some have the reverse. We'd still call both of them, roughly, terabits of network, and you still have to build them. The one with the terabit core might actually be easier because you have less of the expensive really fast equipment.
The only point I wanted to make is that the 'terabit of network' doesn't have to be end to end, so it's not as scary as it may sound.
If you're wondering, if you can't get peering, you wind up like Twitch. In South Korea. South Korean telecom law explicitly shifts the capital expenditure costs of ISPs over to other online services, which is sort of like the fucked-up opposite of Net Neutrality regulation. So Twitch was being bled dry to pay for the chaebols' network expansion. Hell, even after Twitch left, South Korea fined them for leaving!
...is there some reason for Twitch to pay such a fine? Were any grounds stated for it?
> According to Yonhap, the KCC made the decision that Twitch terminating the ability for users in South Korea to access VODs wasn’t necessary to keep the service alive. When asked to justify their claims, Twitch declined due to contractual obligations related to keeping user and site data private.
> Additionally, Twitch would have to present evidence that their decision to gradually take features away from South Korean users & leave the country was necessary. This means that Twitch isn’t likely to return service to South Korea any time soon.
> There’s also a good chance Twitch will be forced to provide refunds for those who have been affected by the service being discontinued, with the KCC warning Twitch that they need to prepare “user protection measures” as they cease service in the country.
https://www.dexerto.com/twitch/south-korea-fines-twitch-over...
95th billing, adaptive, progressive playing and just cap buffer to the minimum to keep playing. Equals ~$1M/month for +10 tbit/s egress.
Source: Worked at one of the largest bandwidth consumers in the world.
You would think the VMs are the expensive part, but no, egress is easily multiple times the cost of the compute.
Jump into a 'bare metal' datacenter and things can get much different.
You can get 100x cheaper and unmetered at a low cost provider like OVH or hetzner or similar bare metal data centers .
It doesn’t even need significant monthly commits to get that pricing if you are running video streaming at scale you are not running on AWS or even tier 2 like OVH for sure
* up to 2TB per month
* if our upstream is saturated we're going to look at our biggest users and if the number is really big we'll send them a polite email to please reduce it or pay more.
There are reports of people getting emails from Hetzner after sending multiple Gbps continuously for several months. That's the level you have to reach before the * kicks in. Only 1Gbps servers are unmetered, so you'd have to have several.
If you want to know a better approximation of their true cost just look at their non-unlimited plans: 20TB/month included for free; 1€/TB (excl VAT) after that.
I have one more interesting data point to add: I was quoted 950€/month for a dedicated 10Gbps between Berlin and Amsterdam (about 600km) plus peering at AMS-IX, or 300€ for 1Gbps. (They're not secretive and you can just ask for a quote using their sales contact form). Extrapolating, it seems that 1€ is worth about 2.5 petabyte-kilometers, at least within the dense interconnections of continental Europe. About twice the price of shipping a petabyte of hard drives the same distance.
I will point out that this is still about 50x-80x cheaper than Amazon. Not far off the claim of 100x.
Cloudflare and most other Object Storage Providers either fully free egress for all users or at-least for inter-cloud transit so you can then put a free/cheap CDN like Cloudflare in front and not pay all that much for b/w.
AWS refuses to participate. Costs of retrieval of all data plus associated bandwidth is a so high for many people that they stick to S3 including me.
The cloud bandwidth you get on your VPS has nothing to do with costs at the scale where you’ve racks in data centers.
At even higher scale (YouTube, Facebook or even Netflix ) you are going to be putting content caching servers at the local ISP PoPs : it is mutually beneficial to do so .
It is hard to directly compete on long-form video because of audience and content depth advantage Google has with YouTube.
There are successful niche players who are fairly large (like Vimeo or Twitch or even OnlyFans) who focus on specific markets that don't require social network advantage like corporate or smaller segments etc.
For general purpose media, creators are going to focus on the platform with most audience and vice-versa, very hard to break that.
The number above was a thousand dollars per month per 10gbps, and AWS would charge more than a hundred thousand dollars at the listed $0.05/GB price.
Nonetheless it does give a ballpark for the cost of bandwidth being a lot lower than people think. A 10G internet connection would be cheaper to provide in some parts of that equation and more expensive in others - should end up in the same ballpark.
And honestly 10 cents per Mbps sounds kind of high for raw transit, I interpreted it as a price for actual utilization.
Add 10-25% profit for that company to get closer to true "raw transit" pricing from the carriers directly.
But sure, if you want citations for $0.10 per Mbps in bulk for transit, that's easy to find/beat.
https://he.net/ "Get BGP+IPv6+IPv4 for $0.06/Mbps!"
https://www.fdcservers.net/ip-transit Europe, North America: 10Gbps $499/month
And telegeography just sells information, but they had a blog post that's now three years out of date reporting that "In Q2 2021, the lowest 10 GigE prices on offer were at the brink of $0.09 per Mbps per month. The lowest for 100 GigE were $0.06 per Mbps per month."
You need to factor in that your utilization won't be 100%, but if you're comparing 6 cents for a Mbps and and 5 cents for a gigabyte, then the exact point where AWS is 100x more expensive is when your line is 36% utilized.
It's a bit surprising that you were not getting significantly better prices than individuals.
[edit] and that have good deliverability worldwide, no weird paths to other consumer IPs that intermittently fail to route or inexplicably have dial-up transfer speeds. And have anything like a real SLA.
Mindgeek? :-)
Well, there's this:
https://www.microsoft.com/en-us/research/project/project-sil...
This sounds like a good answer, but falls away drastically when you realise the vast majority of consumer of YouTube are outside of the USA, which in turn means so are those bandwidth costs.
Are you guessing or have I missed something here? I can't see how this could be a significant enough factor to make the global model work.
How Zoom manages to do this in a client-server fashion and is still financially solvent is also a question I've had for a while, but like others say, discounts on the transport and peering arrangements will be a key part in making those economics work, as compression and storage are relatively solved problems here.
I doubt they send N-1 full resolution streams to each participant. They probably send only the currently focused stream in full resolution, the unfocused streams in low res, and don't stream any of the non-visible participants.
As you change focus between the streams you can sometimes see as it renders the low res stream briefly until the high res stream is received.
Besides the general advantage of having control over such a massive platform, which definitely plays an important part in the lives of hundreds of millions of people, Google likely views YouTube as important to control. If YouTube were a separate entity, it could e.g. freely choose their ad providers or even provides ads themselves, essentially creating competition for Google. Google also has trivial access to the data there and therefore the easiest access if they want to train AI on that data. Last but not least I think Google sees YouTube as vital for their corporate image and their social mission presented in Google Jigsaw.
The biggest cost is as you imagine the streaming - getting the video to the viewer. It was a large part of our variable cost and we had a (literal) mad genius dev ops person holed up in his own office cave that managed the whole operation.
Ive long forgotten the special optimizations he did but he would keep finding ways to improve margin / efficiency.
Encoding is a cost but I don’t recall it being significant
Storage isnt generally expensive. Think about how cheap you as a consumer can go get 2 TB of storage, and extrapolate.
The other big expense - people! All those engineers to build back and front end systems. That’s what ruined us - too many people were needed and not enough money coming in so we were burning cash.
Or both, similar to Skype's supernode model?
And generally torrent-based streamers don't hire financial analysts :)
The current definition requires 20Mbps of upload, and uploading a youtube-quality video to two other people would not take a big fraction of that. Though it would help if ISPs stop trying to set bandwidth caps at <5% utilization levels.
It's that you're also probably going to get CGNAT - and maybe even a firewall blocking unusual ports.
And you're going to be running the power-hungry data connection at least twice as much, bad for battery life.
And mobile connections are less reliable - transitions between towers, going through tunnels, switching between 4G and WiFi.
And mobile OSes are eager to suspend things - especially things that are using a lot of data and battery.
I'm thinking of the situation where most of the users are using home connections and have power cables always in or in reach.
Based on some power laws etc., I would guess most videos have only a handful of views, so storing them forever and the cost to encode them initially is probably significant.
The primary difference between live and static video is the bursts -- get to a certain scale as a static video provider, and you can roughly estimate your bandwidth 95th percentiles. But one big live event can blow you out of the water, and push you over into very expensive tiers that will kill your economics.
nvenc > See also: https://en.wikipedia.org/wiki/Nvidia_NVENC#See_also
NVIDIA Video Codec SDK v12.1 > NVENC Application Note: https://docs.nvidia.com/video-technologies/video-codec-sdk/1... :
> NVENC Capabilities: encoding for H.264, HEVC 8-bit, HEVC 10-bit, AV1 8-bit and AV1 10-bit. This includes motion estimation and mode decision, motion compensation and residual coding, and entropy coding. It can also be used to generate motion vectors between two frames, which are useful for applications such as depth estimation, frame interpolation, encoding using other codecs not supported by NVENC, or hybrid encoding wherein motion estimation is performed by NVENC and the rest of the encoding is handled elsewhere in the system. These operations are hardware accelerated by a dedicated block on GPU silicon die. NVENCODE APIs provide the necessary knobs to utilize the hardware encoding capabilities.
FFMPEG > Platform [hw video encoder] API Availability table: https://trac.ffmpeg.org/wiki/HWAccelIntro#PlatformAPIAvailab... :
> AMF, NVENC/NVDEC/CUVID (CUDA, cuda-nvcc and libnpp) (NVIDIA), VCE (AMD), libmfx (Intel), MediaCodec, Media Foundation, MMAL, OpenMAX, RockChip MPP, V4L2 M2M, VA-API (Intel), Video Toolbox, Vulkan
If you make 1 million, 10k isn't going to go very far towards paying devs to save you 1%
But if you're broadcasting something live and what's killing you is that everyone wants to watch it at the same time... wouldn't you serve it P2P so that everyone is downloading it from each other rather than you?
And home connections —while still largely asymmetric— are much faster than they used to be. Having 10mbps up means one client can serve two more. And there's a lot more FTTP with 100-1000mbps up too. These really make a difference when you have a large swarm.
A place this could work is streaming a conference, live-ish is the goal and the producers aren't rich. Sports would be the worst case.
Isn't the point of the P2P approach that it gets better the more this is true?
It doesn't really work for something you want to watch simultaneously and reliably. I have to wait for my neighbour to get the chunk I want, then I get it. If they got it from someone else, we form a bigger chain, and then you have all the broadcasting etc to figure out who to actually get a chunk of video from.
Hearing the street cheer while I watch my national team captain take a runup for a penalty is really quite bad.
>Due to the proliferation of P2P CDN (or PCDN for short), which includes a large amount of home broadband uplink bandwidth at the central office, increasing operational pressure, and cannibalizing the traditional CDN business revenue share of telecom operators […] access is technically detected If the user's traffic volume exceeds a certain threshold, the speed will be limited or even the user's Internet access service will be interrupted. If the user finds a complaint, the user will be required to ensure that he or she has not used or removed the PCDN corresponding access device in exchange for restoring normal access. access services; thereby preventing access users from overusing home broadband and infringing on the interests of telecom operators.
https://zh.wikipedia.org/wiki/內容傳遞網路#P2P_CDN
In a prerecorded video CDN managing that catalog is a PITA and does drive meaningful infrastructure cost. You need the "right" content to be in the correct location for low cost peering/transit/distribution, on the correct media for the total throughput:size, in the optimal number of encodings for efficient/quality playback, etc. This job is a lot easier when the provider controls the catalog, and has a limited catalog size. See some of the OpenConnect talks where they're "preloading" content offpeak to optimize IO allocation on the appliances. It was an absolute nightmare to try and manage with a many PB catalog with 3P content that service didnt control the release/popularity of.
Edit: source, principal at AWS and was responsible for a lot of the prime video delivery once upon a time.
With Netflix's Live events, you can seek anywhere up to time zero, not just the last few minutes.
Interestingly enough the Apple and I assume Netflix live streams come from the colo equipment in your ISP. So each box has their own recording as it happens.
Used to work at a live-streaming company on our stream infra.
I mostly disagree, unless it's pure live no replay at all and no closely timed events required. Usually live platforms will offer some sort of a VOD (VODs, Replays, Rebroadcasts), all of which will require a storage solution. Couple that in with the fact that anything requiring more complex timing than "show video live~ish" can get messy fast with sync and latency issues.
Edit: the above notwithstanding live sports etc is _still_ better on the storage side as viewers are so heavily synchronized. Lots of nice cache efficiencies when everyone is watching the same content at the same time.
The minimum possible expenditure on encoding is "we require videos to be encoded like so; here's our help page on how you can do that".
It's not even slightly expensive.
Encoding, scaling and transcoding are relatively cheap for stored content, and relatively expensive if you want real or near-real time.
If you want DRM (digital rights management = ~ineffective copy protection) then you need to add a bit more overhead for that, both in terms of processing and latency. If you need multi-DRM (different DRM systems for different devices the consumer owns) and a good cross-device experience (like pause and resume across devices), it gets real hard real fast.
It helps to be targeting a standard platform, for example a modern widescreen TV with H.265 support and solid 4K decoding. Otherwise you need a different version for every resolution, a different version for every CODEC, a different version for every bitrate, etc. We had great experience adjusting bitrates and encoding parameters for different device categories, for example if you had a certain phone and you ran it at max spec it might look great but if you were looking to preserve battery and were running on battery save mode the decode would fail and you'd get choppy performance and stuttering audio. This sort of thing was rife then.
As a series of specialist video providers emerged, ~all the cloud providers went and added these services, basically 95% of which are frontends to ffmpeg and some internal cloud storage scheme or similar.
Finally, billing is hard. Users have an expectation of free content now.
No experience with real time stream economics, but saw the inside of LA's stadium video control center one day. Didn't look inexpensive, I'll tell you that much. Probably for events with multiple cameras you're mostly paying site fees, ie. reliable bandwidth, humans, mixing desk if required. For studio broadcast these costs will be reduced. Both will have a slight real time encoding tax vs. stored content. If you want to figure out how to do it cheaply, look at the porn industry.
I wonder what the approximate net global economic benefit of ffmpeg is to this point?
(https://xkcd.com/2347/)
They can still afford to serve the occasional obscure video from the origin servers.
[1] https://www.qqtube.com/blog/how-much-storage-does-youtube-ha...
[2] https://www.qnap.com/solution/petabyte-storage/en/
YT financials and P&L were not broken out in audited financial statements back in the day.
Allocating costs for things from Google that they use (e.g. the Ads system) is difficult. The problem with any subsidiary.
I have premium, my wife accepts all of those ads
I'm sure that's what Google's accountants would love us (and the IRS) to continue to believe.
and also that streaming and storing video at that scale is almost a natural monopoly, with how much it must cost and how hard it would be to compete without existing resources
Routers have ASIC switching, why can't we have dedicated cache appliances with a bunch of RAM and some kind of modified GPU with network access and crypto acceleration in each core?
https://openconnect.netflix.com/en/appliances/
https://news.ycombinator.com/item?id=32519881
Yt for example deletes your 720p after a while and replaces it with a potato.
And if you watch a old not relevant yt and it starts after 10 seconds instead of now, no one really cares.
You can put that old highly encoded potato at your huge and cheap storage system de located somewhere around the globe were it's just cheap (energy).
You can also calculate in the time for a band robot and only store half or the first minute of that potato on your cheap storage and let the robot grab the rest of it.
After all if video is your main thing plenty of weird optimizations start to make sense.
Tape is for long term sunk storage, not cold infrequent access like a youtube video.
I know aws glacier has an "expedited retrieval time" of 1-5 minutes, but that is not how typical tape setups work. Frankly I would be very interested in what actually hides behind that product.
You can tell this is the case for at least the flexible retrieval tier, because small objects can be returned in a few minutes, whereas larger requests take hours - if the files were actually on a tape drive somewhere, small requests couldn't be fulfilled dramatically faster than large ones, given that tape has shitty random-access performance.
(I used to work on a downstream team at AWS)
It's also weird that the retrieval gives you a regular fast S3 object you can then access. Given that it's already on that hardware, is a copy even happening?
always bugs me to hell when i encounter a "high definition" video that has worse quality than pal/ntsc
But no, always goes to spinny wheel buffering after ad ends. Oh, and thats after having some spinny wheel to load the ad in first place ffs.
They are definitely not fetching video in the background.
I seem to remember Google own some network infrastructure? That saves some money. On top of that at their size you are going to get things cheaper.
> The other big expense - people! All those engineers to build back and front end systems. That’s what ruined us - too many people were needed and not enough money coming in so we were burning cash.
There should be economies of scale on that. Its harder to build and maintain bigger systems, but the work required does not scale linearly with size.
BandAid was a success. YouTube's history might look very different if it wasn't. It consisted of a massive crash buildout of a global CDN, something that Google historically hadn't had (CDNs aren't useful if everything you serve is dynamically generated).
One reason BandAid could happen was that Google had a large and sophisticated NetOps operation already in place, which was already acquiring long haul unused fibre wavelengths at scale for the purposes of moving the web search index about. So CDN nodes didn't need to be far from a Google backbone POP, and at that point moving bits around on that backbone was to some extent "free" because the bulk of the cost was in the fibre rental and the rackspace+equipment on either end. Also it was being paid for already by the needs of search+ads.
Over time Google steadily moved more stuff over to their infrastructure and off YouTube's own, but it was all driven by whatever would break next.
Then you have all the costs that YouTube has that basic video sites don't. ContentID alone had costs that would break the back of nearly any other company but was made effectively mandatory by the copyright lawsuits against YouTube early on, which Google won but (according to internal legal analysis at least) that was largely due to the enormous good-faith and successful effort demonstrated by ContentID. And then of course you need a global ad sales team, a ton of work on ad targeting, etc.
This story explains why Google can afford to do YouTube and you can't. The reality is that whilst they certainly have some sort of internal number for what YouTube costs, all such figures are inevitably kind of fantastical because so much infrastructure is shared and cross-subsidised by other businesses. You can't just magic up a global team of network engineers and fibre contracts in a few months, which is what YouTube needed in order to survive (and one of the main reasons they were forced to sell). No matter what price you come up with that in internal booking it will always be kinda dubious because such things aren't sold on the market.
Though the idea of such networks not being sold on the market makes be ponder if starlink will come to provide such a service. They’d need to scale out their laser links and ground stations.
(same for Meta wrt Instagram)
These products that have been scaled by multiple orders of magnitude since original acquisition are like ships of Theseus; almost everything about how they work, how they scale and how they make money, have completely changed.
On day one of the acquisition, Youtube's egress network was at least 4x the size of Google's, built and run by two guys. This shouldn't be a shock, you need a lot more bits to serve video than search results. For the hottest bits of content, third-party CDNs were serving videos and thumbnails.
There was no collapse imminent, but there were concerns about getting YouTube and Google infrastructure on a common path. BandAid was named as such because the goal was "not to break the internet." It was a small project with maybe a dozen members early on, all solid people.
YouTube had its own contemporaneous project, née VCR - "video cache rack". We did not necessarily believe that BandAid would arrive in a reasonable amount of time. Generally Google has two versions of every system - the deprecated one and the one that doesn't work yet.
VCR was a true YouTube project, 3 or 4 people working with one purpose. It was conceived, written, physically built and deployed in about 3 weeks with its own hardware, network setup and custom software. I believe it was lighttpd with a custom 404 handler that would fetch a video when missing. That first rack maxed out at 80Gbps a few hours into its test.
After several months, Bandaid debuted. It launched at ~800Mbps and grew steadily from then on into what is certainly one of the largest networks in the world.
YouTube mostly moved things to Google based on a what made good engineering sense. Yes, a few of them were based on what would break next - thumbnails leaps to mind. Search, which we thought was a no-brainer and would be "easy" took more than a year to migrate - mostly due to quality issues. Many migrations were purely whimsical or some kind of nebulous "promo projects." Many more stayed separate for more than a decade. When a company gets large enough, the foolish consistency tends to trump a lot of other engineering arguments.
To the ancestral poster, do not despair. You can transcode, store and serve video, as you've likely surmised it's not all that difficult technically. In fact, it's so much easier now than in 2005.
What makes a great product is hard to describe and not always obvious. The economics will depend on your premise and success. "the cloud" gets pricey as you scale. There will be a long path of cost optimization if you get big enough, but that's the price of doing business at scale.
YouTube continued building their own POPs AND network for ~18 months AFTER the google acquisition. Google did not have the network capacity to carry it. (Fun fact: YT had 25 datacenter contracts, and opened them at the rate of 1 a month) starting from March 2006 - 25 contracts were set up in 2 years. At the time of the google acquisition, there were, ~8. (So yeah, 17 additions over the next ~16 months)
Also YT had a far more streamlined (but less optimized) network architecture. Traffic was originally generated in the PoP and egressed out of the PoP. The was not a lot of traffic going across backbones (Unless if it was going to a settlement free peer). Initially, it was egressed as fast as possible. This was good for cost, not great for performance, but it did encourage peering, which also helped cost. Popular videos did go via CDN initially.
YouTube had a very scalable POP architecture. I agree with area_man that the collapse was not imminent. (See 17 additional pops) There were growing pains, sure, but there was a fairly good system.
Also, as it relates to bandaid from a datacenter and procurement perspective, the original bandaid racks were in YT cages. YT had space in datacenters, and Google didnt. (SV1, DC3). Also, the HWOps tech who went on-site to DC3, ended up tripping a breaker. (They were almost escorted out).
Side-note: the evolution/offshoot of bandaid into the offnet caching system - now called Google Global Cache, is what really helped scale into provider (end-user) networks, and remove a lot of load from their backbone, similar to an Akamai, or a Netflix open connect box. Last I heard GGC pushed significantly more traffic than the main google network.
The google netops teams that were of help in the first year of acquisition was the peering team, and some of the procurement team. The peering team helped us leverage existing network relationships, to pick up peers (eg: SBC)
The procurement team gave us circuits from providers that had a long negotiation time (eg: sprint)
Google also helped YouTube procure various Juniper equipment, which was then installed by the YT Team.
Sort of make's Cloudflare's R2 look more impressive since they do not charge for egress.
Build vs buy pushes you to "build" early on when your margins are slim and your volume is huge.
At the scale of the largest streaming apps (Disney, Netflix, YouTube, etc) you are moving petabytes of data PER DAY. At that size, you have access to significant savings on CDNs, backbone providers, etc. in many cases the discounts will be 90% - I have seen as high as 99% - or higher off the “list” price (which are usually never paid by anyone anyway).
You also tend to own your own backbone and can link in whichever ISP wherever you want for the “final mile.”
Final note, when you have been doing this long enough, you can start shaping the traffic based off previous patterns. I remember an eBay listing years ago for a Netflix local storage device that was meant to store shows at an ISP’s data center.
egress costs were enormous and YT was not profitable. I don't know if it is now, but I wouldn't be surprised to find it is. They sure have enough ads.
As several people say below, caching content around the world is key, so that not all requests are serviced in NoCal.
See how easy it is to make random statements on the internet.
Ruth Porat has been on record many times indicating that YouTube wasn't profitable in the 2010's. I think her public statements have only indicated that YouTube was free cash flow positive as of the 2020's, but I haven't found exactly where that happened - Google has experimented with a lot of different kinds of breakdowns of its finances. I assume that hiding the economics of YouTube is part of this (as well as protection against a zealous DOJ saying that Google's businesses are separable).
Because I'm not here to read a wall of text generated by ChatGPT.
Different user, but I'm here to read what people are writing. IMO, pasting GPT content is about on par with replying to something with a LMGTFY link.
that's the key issue with this kind of ChatGPT writing. Code you can relatively easily check for correctness - just run it. For analysis on this level, it really had to be based on facts and reality, not generated by a bullshit generator to be of actual use.
In this case, digging through all the material to find the factual basis is the hard part, and corroborating it is not (for those who care).
Here's an answer to you & all the downvoters: just download an annual report or a 10-K for Google, for the years in question.
Realize, I’m not anyone up the chain claiming any knowledge about YouTube’s specific profitability in any given year. I’m just commenting about how frustrating the internet is becoming. HN can be cool because you’ll be in threads like this and someone who worked on some original piece of YouTube compression algo might pop in to offer insight.
On the other hand, multiple paragraphs of ChatGPT regurgitation are next to useless. If you really want to share that kind of thing, maybe link to the publicly available chat instead of quoting, so people can read it if they want to or ignore it at their leisure.
So rather than assserting it's all wrong, why don't you find one thing in it that is?
We've seen plenty about media literacy / verifying news over the years of course, but the time and surface area considerations will change exponentially if people can start generating fake news with an AI (especially audio and video).
And I think it's too naive to go straight from there to the Dead Internet Theory, because that's like switching from "believing everything you read on the Internet" to "disbelieving everything you read on the Internet" (which is necessarily wrong with the opposite percentage margin).
For this thread, I'm inclined to believe the original comment(er). I'm not going to look it up, look up the username, ask an AI, or really care all that much about the debate, because I'm not directly interested in the outcome.
I _am_ interested that, in my own mind, the original comment lost an enormous amount of credibility when the author then reached for ChatGPT to defend the claim. So the only thing I can do without sinking a large amount of time into looking for sources is to leave this thread thinking "I don't know, maybe I'll actually look it up one day, and evaluate it properly when it becomes relevant to me (probably never)". And maybe that's a better outcome than my initial "yeah seems like a legit HN comment" anyway.
EDIT: There's a second level issue too where the people using AI to generate "facts" won't themselves know whether they're generating true or false information - presumably(?) not the case in this thread.
http://abc.xyz/investor
Content ads: not nearly as much.
https://www.macrotrends.net/stocks/charts/V/visa/profit-marg...
https://www.macrotrends.net/stocks/charts/VRSN/verisign/prof...
Epic electronic health records software is not a publicly listed company, but I would guess they have massive profit margins too.
> You never hear about someone’s salesforce being ransomwared or their servicenow being held hostage
You may hear about business organizations being unable to do work due to ransomware. Likely nobody mentions their inability to access salesforce specifically, because 1. the data in there usually isn't controlled and 2. likely nobody is going to die if they can't log into salesforce.
The cost of YouTube Music is $11 and YouTube Premium, which include Music is $14. To me that indicates that you can run YouTube for a given user for around $3 - $5 per month. Trying to watch YouTube with ads, the shear amount of ads and the length, could be a sign that ads on YouTube is almost worthless, at least they seem to struggle to get $5 per user per month.
YouTube isn't going to die at the hands of competitors anytime soon though, because the cost will deter anyone interested.
(It’s a joke, I know YouTube started going downhill the moment they’ve decided to squeeze every penny out of it)
E: do you happen to know by any chance why algorithm for recommendation become so shit?
i suspect privacy legislation and this cohort thing to be at the bottom of this
Every month I notice the temperature of the pot is up a few degrees. This month it’s unskippable 15 second ads before most videos. Last month it was the first search result now being an ad. Before that it was how 5 second ads are now 7 seconds.
If I thought to write them all down I’d have a dozen more steps to share.
My kids now call it “the Bad YouTube” vs. YouTube Kids because the former is flooded with ads.
> I know advertisements are a thing for YT, but is it enough?
Yes, it is -- virtually certainly. We can assume YouTube is profitable. It's not broken out directly in quarterly reports, but it doesn't make any sense that Google would still be running it after all these years (almost 20) if it weren't.
But obviously YouTube didn't start out as profitable. You need scale, which provides two things:
1) Marginal storage and streaming costs go down (Google is big enough to save huge amounts of money by running its own data centers, peering agreements, caching near customers, etc.)
2) More advertisers running more ads that can be targeted to more users whose preferences you know more about
So no, you can't run it profitably.
This is a classic example of a business that is only profitable at scale, that needs to lose a lot of money at first as it grows until it achieves scale. And it's not just scale on the traditional tech/users side, it's scale on the advertising side as well -- advertisers aren't going to bother running ads on your platform until you have enough users for them to care.
It's also pretty strongly a "winner-takes-all" network effects situation, where video publishers want to put up their videos where the viewers are, and viewers want to visit the site where all the content is. So if you wanted to create a YT competitor, I don't know how you'd convince content creators to post their videos to your site in addition to YT, or how to convince consumers to watch said videos on your site instead of YT.
This applies to the content which would exist anyway, and then doubly to content created for Youtube. Grand Pooh Bear would be on Twitch anyway, but it doesn't really make sense for a Tom Scott, let alone "Corrections" which is a Youtube-only addition to Seth Myers "Late Night" show.
Likewise until it gets fairly "big" it doesn't make sense to officially put your music videos on Youtube. Today that's basically the main way they're getting seen.
When you look at costs per unit, then it gets cheaper at scale, not more expensive.
For streaming, at scale you can afford to do peering yourself, instead of buying bandwidth.
For encoding, at scale you can afford special purpose encoding hardware, instead of using general purpose hardware.
For storage, at scale you can get cheap bulk deals with drive manufacturers.
They're not paying a margin to advertising companies because they are the advertising company, they're not paying a margin to datacenters because they are the datacenter.
The data gleaned from YT views helps them to run search and vice versa.
Additionally, they don't pay the same electricity and water bill that others pay for their data center. They get a discount because they are creating jobs.
Getting streaming to be cost effective starts from decades of R&D investment + getting low cost electricity and water + owning supply chain.
On the "Revenue" side you will quite probably need to have enough eyeballs that advertisers come to you directly to display ads and do so in volume.
On the "Costs" side you'd want to be big enough that you can just store your content in ~3 of your own datacentres, cache the "hot content" in a site or two per country then give away caches to ISPs (who will gladly host them in their own network for free).
Biggest cost will be bandwidth/streaming servers. Encoding/storage is comparatively cheap. If you were small you would likely start to do this from a few 100Gbps dedicated servers per continent. https://www.fdcservers.net/configurator?fixedFilter=15&fixed... If we set an average of 3-4Mbps per stream you're looking at each server handling 20,000 videos served and the hourly cost of the server would be around say $4/hour so you're looking at around $0.20 per 1000 video hours in theory, in practice it will be higher. Worst case closer to $0.50 per 1000 video hours due to utilisation rates.
Even if you just put it all on S3 infrequent, which would be one of the most expensive ways of doing it, it's still not really expensive compared to serving the content.