Will the server-side code for this be available as open source, so that it can be used by someone who wants to run their own CDN but doesn’t want to be part of the network?
>How well does this scale? Are peers all necessarily running in major cloud environments?
The network scales very well. The network is divided into multiple GeoZones. We can add remove these zones easily. GeoZones are not connected to each other, so there is no single point of failure. Actually the major cloud envs are the most expensive possible variants for Peers. Bare metal servers are much cheaper in our case.
>Do you imagine seeing peers in the form of smaller players with beefy home setups?
Home internet has a couple of disadvantages:
- traffic is not "unlimited". Xfinity doesn't allow you more than 1 Tb in and out traffic per month
- home internet is asymmetrical: you could have 1 Gbps in and only 5 Mbps out.
Something doesn't add up here. They're actively telling third parties to lease cloud servers with large amounts of free bandwidth, and allocate them to this purpose. I see four possibilities:
- They are paying the third party less than what it costs to rent the server, in which case the third party is losing money and shouldn't be doing this.
- They are paying the third party at least what it costs to rent the server, in which case why aren't they just renting the server themselves? They could probably negotiate a bulk discount.
- They are trying to hide this activity from the cloud server provider by involving third parties. While this may not be a ToS violation as such, I'd guess that if they're hiding it, it is something the cloud providers will want to crack down on if it becomes significant.
- This entire scheme is actually a plan to get access to countries with weird regulatory requirements, e.g. foreign companies are not usually allowed to own servers in China. This one's pretty legit actually, but it's a really complex version of the scheme.
I think at least one of those must be true, and none of them would be a good sign for the future.
Edit: I guess "paying the third party less than their costs" and/or "not making a profit themselves" might be what they mean by "Uber-like", so it would just be an elaborate joke?
>Something doesn't add up here. They're actively telling third parties to lease cloud servers with large amounts of free bandwidth:
Bare-metal servers are much better is our case. Please note, bandwidth is not free. You can follow the links and see the actual price for it. If LeaseWeb tells you: 1 Gbps dedicated/unlimited per $143/month, it means you can use it 24x7.
>- They are paying the third party less than what it costs to rent the server
we don't know prices in all datacenters all over the world. That's why the _Peer_ sets the price for his service.
> If LeaseWeb tells you: 1 Gbps dedicated/unlimited per $143/month, it means you can use it 24x7.
While that is true in theory, if you in practice saturate your bandwidth 24x7 by reselling it your contract will probably get cancelled (or at least that is the case for most "unlimited" contracts).
I don't think you actually answered the questions of the parent.
There is a difference in this case between 'unlimited' and 'unmetered'. The limit is on the throughput (1Gbps) rather than the total bandwidth which is itself naturally limited by the throughput x time.
That being said I could still see this being against TOS if you end up chewing up SSDs, hosting dodgy content or running at full power constantly.
> That's why the _Peer_ sets the price for his service.
But that breaks the economics of this, no? If you're in the US and charge $6/TB and I'm in Germany and charge $100/TB, why would traffic ever go to my server? If your US server gets overloaded, am I now making $100/TB? How does the network ensure that this is both fair and competitive? Moreover, how can a customer know that they're not going to get a surprise bill for an order of magnitude more than they planned for?
I think it's indirectly exploiting providers that give you a fixed bandwidth rate for a low cost. Those providers do this knowing some large percentage of customers don't use it all.
So, if this becomes popular, flat rate servers with unmetered bandwidth at a fixed speed would become more expensive, or come with a more specific ToS to prohibit that.
Similar to how data hoarders ended the "really cheap personal cloud storage" bubble.
I think it's quite simple. The math adds up, they just want third parties to network with them. The more servers join this system, the more their product is worth. At the same time, the cost of managing hundreds or thousands of servers is moved to others.
The more this trick will be used, the higher the cost of the dedicated fast internet connections with cloud hosts will become. Most people buying dedicated, unlimited uplinks don't upload the amounts mentioned in this article, which is why bandwidth is relatively affordable. If everyone buys into this network to make a profit, the prices would probably rise to the point that profit drops. This isn't against any kind of ToS I know of.
The "make 2.4x your investment" promise also assumes your server will actually serve 100TB of content. The more people join, the fewer files your server will serve and the less profit you'll make.
In the end, the system is profitable until the network has reached the size where everyone breaks even. If people leave the network, profits go up again, if people join the network, profits drop and people back out. People renting servers game the system trying to one-up each other and in return Farba gets a worldwide CDN for cheap.
The best trick may perhaps be that server providers may directly buy into the system, as they can get bandwidth and servers for lower margins than their customers. This gets Farba high-quality service for cheap while the service providers add a relatively easy income stream on the side.
All of this comes at an additional cost to the customers, because now your battery-starved mobile phone is validating cryptographic signatures of images and other content.
This also leaves legal questions: who's responsible for illegal content hosted on servers hooked into this network? If your hosting provider of choice gets a takedown notice for child porn or malware, will you still be able to use their services?
>now your battery-starved mobile phone is validating cryptographic signatures
with a native app, it costs nothing. Actually, the main power consumer is the phone screen.
>who's responsible for illegal content hosted on servers
this is a very important question. We are going to run AI models against all content we host on Peers. Right now, we don't accept third-parties and "fake" all Peers ourself. Want to ask lawyers what is the best way to handle this case.
For the first bullet point, it could be worth it for the third party if they are using the server for something else, but have surplus resources that could be used for this CDN. And they might be paying higher rates initially to get enough peers to make the CDN worth using.
There's a bunch of really obvious objections to this idea around performance, privacy, economics, UX, and reliability. Impressively, the post did not manage to address any of them up front.
1. Performance. In terms of latency, this is adding an extra round-trip to each page request (to get the metadata). It's also likely going to load different resources from different servers (hostname, IP) preventing connection reuse. I guess they can't even start fetching data until the full page has loaded? And it feels like this scheme would make it much harder to reliably cache resources, since the browser will cache by URL while they'd struggle to make sure the URLs are stable across calls.
2. Privacy. Their threat model only addresses malicious peers changing content, not malicious peers trying to track users.
3. Economics. If running peer servers really was profitable, the company would do it themselves rather than outsource it. Given the "where can you get a server" section, it's not even that they're expecting this to just be running on spare capacity.
The problem with the setup is that incentives of the peers are badly misaligned with the network, the clients, and the end-user. A peer wants to maximize traffic; the clients and end-users want to minimize it. So a peer would be incentivized to set cache-control headers to prevent caching, to increase traffic. Likewise a peer would be incentivized to only keep copies of the most accessed resources, to be able to serve as high a proportion of the traffic as possible with a given disk budget, while all the other stakeholders would like each file to be available just N+1 times in each region.
4. UX. URLs will be really crappy compared to a proper CDN. Right-click copy a link to an image and send it to somebody? It'll initially be to some super-dodgy URL, and stop working in a day or two as the CDN node starts caching different content. And AMP showed that people do actually care about URLs.
>1. Performance. In terms of latency, this is adding an extra round-trip to each page request (to get the metadata). It's also likely going to load different resources from different servers (hostname, IP) preventing connection reuse. I guess they can't even start fetching data until the full page has loaded?
fair point. We use same technique as lazy-load (you have to wait the full page has loaded).
>3. Economics. If running peer servers really was profitable
Uber doesn't drive taxis himself, airBnB doesn't own property. This is how Uber-economy works. We believe it works in the CDN industry as well. We just don't want to be another CDN provider.
1. It started out burning huge amounts of investor capital, basically transferring all of that money to drivers and in the form of subsidized rides, to riders.
2. It works today by a combination of the above and underpaying drivers, who generally make less than minimum wage when they manage to cover their completely costs at all. Despite this, prices have gone up dramatically, so that taxis are once again competitive on a price basis.
3. Uber was profitable in 2018, and not before nor since[0]. Deeply, profoundly unprofitable. Why would anyone want to be the Uber of anything?
1. Is basically price dumping using unlimited investor money and is blatantly illegal. But since it "this great big disruptive innovation unicorn", no one batted an eye. And still no one bats an eye in 3. :)
It always underpaid drivers. It exploited drivers not calculating car amortization costs and sometimes not even gas correctly. Also, people do not calculate the value of their time spent waiting (unpaid) and the cost to drive to the pick up (also unpaid). I guess Haraszti laughed his ass off when he learned of this bullshit. https://www.amazon.com/worker-workers-state-Piece-rates-Hung... He basically have proven the socialist state was paying factory workers only when their machine was actually working on a piece and then per the minute. And of course wrote a book criticizing the entire system on the side.
Don't do something differently for the sake of being different. Do something differently because it unlocks some new capability.
Uber was special for a few reasons. Primarily, it was a better UX to use an app for a taxi, and apps couldn't exist when taxis were first popularized. Secondarily, Uber not driving taxis themselves was "different" in labor law handling. Uber didn't own cars/employ drivers not because "they didn't want to be another taxi company" but because they wanted to unlock the capability to pay people differently. It's the same story for AirBnB - what's legal for rental units is not the same as what's legal for hotel rooms.
Any company can rent or own thousands of servers across the planet, we know this because CDNs do it today, and because if an individual can do it, so can a company.
>Hopefully you at least make sure to try and return a single host for all the objects in a request.
our goal is to saturate the _User's_ network. Now it's hard to say how many Peers we need to meet this requirement. It looks like 2-3 Peers are enough.
Also, when you have 50 Peers, you have to do 50 DNS/SSL requests in _parallel_. With current hardware, it's not a big deal.
1 is something every CDN in the world already deals with. The metadata system ultimately needs to be its own little CDN, and can achieve the same level of performance (e.g. by deploying at the edge) just without the bandwidth costs.
2 is definitely an issue.
3 is a maybe. There's definitely a model where people with access to cheap servers and bandwidth can get on board and not have to worry about the application-level details, finding customers, billing etc.
4 isn't an issue as long as you are selling the CDN specifically for website assets that don't need to have pretty URLs.
> 1 is something every CDN in the world already deals with.
Not exactly. For a normal CDN, your webpage HTML directs the client directly to the CDN server, which in theory already has the object. Only if it doesn't have the object does it need to go get it, but there isn't an extra lookup. The CDN does it's direction magic via DNS or IP routing, not with double lookups.
1. None of the issues I listed affect traditional CDNs though.
3. Why would the people running a peer have cheaper access to servers and bandwidth than a company specializing in that business? Everything about this business should have huge economies of scale (buying hardware, system administration, transit/peering).
The only situation where I can see a distributed model winning out is if the entity running the peer does not pay for resources but gets them with fraud or theft, e.g. paying for cloud servers with stolen credit cards.
It's a matter of locale. Having an entrepreneur in South Africa or Tibet hear about the money making opportunity, setting up a peer, and then paying them for it is better than flying there for a month to set up a business presence and figuring out all the local everythings. Akamai et al has already done this, and charge money commensurate with having done that. This group seems to be banking on someone in Kathmandu hearing about this opportunity and setting up a peer there.
> 2. Privacy. Their threat model only addresses malicious peers changing content, not malicious peers trying to track users.
I'm narrowing the scope around 'user tracking' specifically to 3rd parties, because 1st party user tracking is a requirement in order to perform some necessary functions (Providing accounts & user sessions, for example).
In the current centralized CDN model (Cloudflare, AWS, Akamai, etc.), the 3rd party is made clear (those aforementioned companies), and their need to track users is documented on the provider's ToS. Likewise, reputational damage is accrued to the providers in question.
In their proposed CDN model, there is no clear 3rd party, and so it must be defensively assumed that all of the data sent for requests is recorded at all times (The FBI Tor exit node problem). As a consequence, the minimal amount of request data can only be sent to these providers (IP address, content hash). Tor-like routing could be implemented to further disperse the requests from one IP address to multiple addresses, but now the monitoring problem's increased to the routers as well, on top of the additional latency introduced for said privacy. Mandating said routing into the protocol would make such tracks useless, as the IP address collected will roughly appear as random noise in the optimistic case.
Personal opinion below
However, it is likely that no technical solution provided would resolutely satisfy the question being asked.
Travis built RedSwoosh (https://en.m.wikipedia.org/wiki/Red_Swoosh) and then went on to build Uber. Now someone saw Uber and claims it inspired them to rebuild RedSwoosh. ¯\_(ツ)_/¯
There are several red flags here, not the least of which is that a CDN isn't just about bandwidth, it is about disk. If a cache node doesn't have the object, it has to go to origin to get it. That has a cost at origin, which most CDN customers want to minimize.
Cache management (eviction, invalidation, etc) is done where? By the look of that diagram, you are actually doubling your origin traffic just to push to the cache after serving the original client request. Perhaps subsequent requests could get served from their 'peer' but the hit rate will be abysmal.
Maybe there is more to it than they describe here, but I don't see how this can work. Are they going to rewrite the object in a way that doesn't require the host certificate?
>a CDN isn't just about bandwidth, it is about _disk_
The disk performance - is the weakest part of the Peer. It is a part you can't guarantee any QoS on a cheap server. The idea is to keep popular cache on Peers in memory. Everything else we will host from a limited number of regular servers (belong to Farba), with high-performance disk subsystem.
>Cache management (eviction, invalidation, etc) is done where?
yes. You can invalidate either a single URL or a folder with a wildcard
>you are actually doubling your origin traffic just to push to the cache after serving the original client request
this is something we can proxy and do only one request per file. Current implementation was done for simplicity and reliability.
>but I don't see how this can work. Are they going to rewrite the object in a way that doesn't require the host certificate?
you can easily check this up. Please open the network console on our demo-page and see how it works:
- we have two different certificates (one for the main site, for for the Peer)
Your example servers have 16-32 GB of memory, which is basically nothing. If you ever ramp to real world traffic the peer will be constantly evicting objects (if you use a normal LRU). Even if you backstop that with a mid-tier belonging to Farba to protect the origin, what kind of cache hit rate are you expecting at the peer?
> we just need to add more Peers and evenly distribute the workload.
That doesn't make sense! If you add more peers just to keep more files in memory, each peer is serving less and less data. There's a fixed, finite amount of usage that the system will receive, and the more peers you add, the more that usage gets subdivided among the peers. Which means they make less money, which makes the system unprofitable. Moreover, you're spending more money to distribute the file to more peers (well, you're not, you're offloading that to the customer right now), which means the more peers, the more expensive for you or the customer.
The problem you have is that handling more files means cutting peer profits without adding any benefit for yourself or the peers. Storage isn't factored into pricing, which means that any accommodation you make to increase availability (an important part of being a CDN!), the worse the economics end up being.
> The disk performance - is the weakest part of the Peer. It is a part you can't guarantee any QoS on a cheap server. The idea is to keep popular cache on Peers in memory. Everything else we will host from a limited number of regular servers (belong to Farba), with high-performance disk subsystem.
I use a CDN to serve about 50GB of files stored on my origin, which turn over every week or so (a new 50GB every week). What you're saying means that _at best_ I need at least 3-4 peers in order to keep one copy of all of my files available. That's your using your numbers for a LeaseWeb or OneProvider peer. And if I'm competing with other customers, my content is getting evicted, resulting in more hits to my origin. If I'm actually using your system as a CDN where peers should be physically close to my users, that means that I'm filling up RAM of lots and lots of peers just so that enough peers that are close to my users can serve the files.
for unpopular content, we will deploy a limited number of servers (belong to Farba) with a performant disk subsystem. A web cache storage is a separate option in every CDN: if you want your files always in "hot" cache, you have to pay extra.
How does this compare to IPFS? I was playing around with IPFS and it seems like a lower risk higher node version of this. Performance wasn’t half bad either. I guess it depends on which node you end up downloading from and whether there was a nearby cache hit. P
IPFS is very slow. It's like a cold storage. Not good for web-cache at all. Though Cloudlare provides gateways for IPFS, and again, we have a CDN layer here. It would be nice to see IPFS performance numbers to compare.
The value of this largely depends on what you're optimizing for. If your constraint is you need a lot of cheap bandwidth but don't care that much about latency, there are many places to find it (find a cheap host, set up Varnish, you're good to go).
However, a lot of people use CDNs because they are fast. They want the latency to be ultra low and they want it to be reliable. This is where the author's solution falls short: CDNs can get premium quality bandwidth because they have more scale than their customers. Sure you can find servers from various hosts in lots of different data centers, but you might not get the latency or quality of bandwidth you were looking for. This can be critical for some applications like streaming or serving images.
If your constraint is speed, you almost have to go with a CDN (even if you are huge). If it's volume, you are in a better position. If your volume is small, you might as well go with something off the shelf like Cloudfront or Cloudflare. If you are ultra huge like YouTube or Netflix, then you have to peer directly with ISPs.
This sounds a lot like CoralCDN. You should look them up, they had some good papers. We actually used CoralCDN on reddit for a while, where we'd send users to the CoralCDN link of the webpage they were trying to get to instead of the actual page. It sadly didn't work very well.
One of the problems I have with the VP of the product discussed is that while I trust CloudFlare, Akami or any of the big players, to run a tight network -- with high and STABLE performance -- I can't say the same about a random vendor (uber driver?) in this network.
Was the issues you were seeing in this general area?
This says it's a system for delivering content "efficiently", but it doesn't look like this manages that. Normally loading a resource is a round trip to your server, which might be a bit inefficient because your server could be on the other side of the world. But what they replace it with is:
1. Round trip to Farba to get metadata.
2. Round trip to your "driver" to get the resource.
3. Computing the hash of the resource to ensure the "driver" didn't cheat.
It could still be worth it as a way of saving money on bandwidth, but I'd be quite surprised if you got the speed benefits of a CDN.
What stops a peer from, say, injecting malicious JavaScript into pages? Or going rogue and serving porn for every request for a JPEG? What if they only do it for every 1000th request?
You could send a hash of the file to the client from the coordination server and check it. If the hash isn't valid, you can re-request the file from another peer. But I don't think you can trust the client to report abuse: a malicious client could report innocent peers (e.g., a vindictive Chrome extension).
Or, the site owner could manipulate the coordination JavaScript to report the downloaded file as invalid/corrupt/tampered with, and then just not re-request the file (and there's nothing you can do to stop them). So in that case, the site owner is managing to not pay for the service (assuming you don't bill for downloads that were reported as invalid).
But on top of all of that, this is _slow_. It's slow on my (quite fast) internet. If you're using this, it's because you're trying to get cheap bandwidth, not because you value performance. The whole point of a proper CDN is to improve performance by spreading out load and moving the edge server physically closer to the user. This might be putting the edge server physically closer to the user, but the mechanism to do that seems to take almost as long as the request/download of the asset itself.
> Since the requested file comes from an untrusted source, the next step is to verify the integrity of the file. The script calculates the hash sum and compares it to the digest from the JSON reply. If the sums match, the script displays the downloaded file to the User.
>Or, the site owner could manipulate the coordination JavaScript to report the downloaded file as invalid/corrupt/tampered with, and then just not re-request the file (and there's nothing you can do to stop them). So in that case, the site owner is managing to not pay for the service (assuming you don't bill for downloads that were reported as invalid).
This fraud is possible. We collect logs from three sources:
- the Balancer (belong to Farba)
- the Peer
- the User
We can analyze every single case of suspicious behavior.
The math doesn’t work. If I sign up for a server and host a node and charge $1 more than I’m paying, it would just make more sense to do that yourself and save yourself the dollar.
To everyone saying, this business does not make sense because they could just buy the servers and serve it themselves and make all the money, please consider that the same argument applies to any intermediated business whatever: to Uber; to wholesalers; to franchise businesses like McDonald's.
The reason businesses like this work in general is because they break a big, complicated business into smaller businesses where there is specialization. There may not be be meaningful potential for specialization here; but that is the argument to have. For example, it may be that Farba can be a leaner business if they don't have to think about how to get the best pricing in dozens of different hosting markets.
It seems a little odd to use GitHub as the CDN for your JavaScript library. Like what if this got really popular, what’s to stop GutHub for shutting it down for abusing their hosting? If I were a customer for a CDN, I would feel more comfortable if they owned their own domain and could control how (and if!) their JavaScript is available to customers.
Anyways, other than that it’s a fun concept. I like the P2P nature of it.
This isn’t good for SEO, which is highly dependent on speed. The CDN data is loaded via JavaScript, meaning crawlers might not load it. I don’t know what this is trying to solve, CloudFlare pretty much already fixes the egress charges problem.
81 comments
[ 3.0 ms ] story [ 158 ms ] threadDo you imagine seeing peers in the form of smaller players with beefy home setups?
The network scales very well. The network is divided into multiple GeoZones. We can add remove these zones easily. GeoZones are not connected to each other, so there is no single point of failure. Actually the major cloud envs are the most expensive possible variants for Peers. Bare metal servers are much cheaper in our case.
>Do you imagine seeing peers in the form of smaller players with beefy home setups?
Home internet has a couple of disadvantages: - traffic is not "unlimited". Xfinity doesn't allow you more than 1 Tb in and out traffic per month - home internet is asymmetrical: you could have 1 Gbps in and only 5 Mbps out.
- They are paying the third party less than what it costs to rent the server, in which case the third party is losing money and shouldn't be doing this.
- They are paying the third party at least what it costs to rent the server, in which case why aren't they just renting the server themselves? They could probably negotiate a bulk discount.
- They are trying to hide this activity from the cloud server provider by involving third parties. While this may not be a ToS violation as such, I'd guess that if they're hiding it, it is something the cloud providers will want to crack down on if it becomes significant.
- This entire scheme is actually a plan to get access to countries with weird regulatory requirements, e.g. foreign companies are not usually allowed to own servers in China. This one's pretty legit actually, but it's a really complex version of the scheme.
I think at least one of those must be true, and none of them would be a good sign for the future.
Edit: I guess "paying the third party less than their costs" and/or "not making a profit themselves" might be what they mean by "Uber-like", so it would just be an elaborate joke?
Agreed, I'm mildly annoyed that farba.js repo has not been updated since 2021 (no further info) + grammar issues [1]
[1] https://github.com/FarbaCDN/farba-public/blob/master/src/far...
Bare-metal servers are much better is our case. Please note, bandwidth is not free. You can follow the links and see the actual price for it. If LeaseWeb tells you: 1 Gbps dedicated/unlimited per $143/month, it means you can use it 24x7.
>- They are paying the third party less than what it costs to rent the server
we don't know prices in all datacenters all over the world. That's why the _Peer_ sets the price for his service.
While that is true in theory, if you in practice saturate your bandwidth 24x7 by reselling it your contract will probably get cancelled (or at least that is the case for most "unlimited" contracts).
I don't think you actually answered the questions of the parent.
Yes 0.0005$ per GB egress and a provider in Europe still makes money.
That being said I could still see this being against TOS if you end up chewing up SSDs, hosting dodgy content or running at full power constantly.
But that breaks the economics of this, no? If you're in the US and charge $6/TB and I'm in Germany and charge $100/TB, why would traffic ever go to my server? If your US server gets overloaded, am I now making $100/TB? How does the network ensure that this is both fair and competitive? Moreover, how can a customer know that they're not going to get a surprise bill for an order of magnitude more than they planned for?
So, if this becomes popular, flat rate servers with unmetered bandwidth at a fixed speed would become more expensive, or come with a more specific ToS to prohibit that.
Similar to how data hoarders ended the "really cheap personal cloud storage" bubble.
The more this trick will be used, the higher the cost of the dedicated fast internet connections with cloud hosts will become. Most people buying dedicated, unlimited uplinks don't upload the amounts mentioned in this article, which is why bandwidth is relatively affordable. If everyone buys into this network to make a profit, the prices would probably rise to the point that profit drops. This isn't against any kind of ToS I know of.
The "make 2.4x your investment" promise also assumes your server will actually serve 100TB of content. The more people join, the fewer files your server will serve and the less profit you'll make.
In the end, the system is profitable until the network has reached the size where everyone breaks even. If people leave the network, profits go up again, if people join the network, profits drop and people back out. People renting servers game the system trying to one-up each other and in return Farba gets a worldwide CDN for cheap.
The best trick may perhaps be that server providers may directly buy into the system, as they can get bandwidth and servers for lower margins than their customers. This gets Farba high-quality service for cheap while the service providers add a relatively easy income stream on the side.
All of this comes at an additional cost to the customers, because now your battery-starved mobile phone is validating cryptographic signatures of images and other content.
This also leaves legal questions: who's responsible for illegal content hosted on servers hooked into this network? If your hosting provider of choice gets a takedown notice for child porn or malware, will you still be able to use their services?
>now your battery-starved mobile phone is validating cryptographic signatures
with a native app, it costs nothing. Actually, the main power consumer is the phone screen.
>who's responsible for illegal content hosted on servers
this is a very important question. We are going to run AI models against all content we host on Peers. Right now, we don't accept third-parties and "fake" all Peers ourself. Want to ask lawyers what is the best way to handle this case.
https://news.ycombinator.com/item?id=34639212
1. Performance. In terms of latency, this is adding an extra round-trip to each page request (to get the metadata). It's also likely going to load different resources from different servers (hostname, IP) preventing connection reuse. I guess they can't even start fetching data until the full page has loaded? And it feels like this scheme would make it much harder to reliably cache resources, since the browser will cache by URL while they'd struggle to make sure the URLs are stable across calls.
2. Privacy. Their threat model only addresses malicious peers changing content, not malicious peers trying to track users.
3. Economics. If running peer servers really was profitable, the company would do it themselves rather than outsource it. Given the "where can you get a server" section, it's not even that they're expecting this to just be running on spare capacity.
The problem with the setup is that incentives of the peers are badly misaligned with the network, the clients, and the end-user. A peer wants to maximize traffic; the clients and end-users want to minimize it. So a peer would be incentivized to set cache-control headers to prevent caching, to increase traffic. Likewise a peer would be incentivized to only keep copies of the most accessed resources, to be able to serve as high a proportion of the traffic as possible with a given disk budget, while all the other stakeholders would like each file to be available just N+1 times in each region.
4. UX. URLs will be really crappy compared to a proper CDN. Right-click copy a link to an image and send it to somebody? It'll initially be to some super-dodgy URL, and stop working in a day or two as the CDN node starts caching different content. And AMP showed that people do actually care about URLs.
It's pretty hard to take this seriously.
fair point. We use same technique as lazy-load (you have to wait the full page has loaded).
>3. Economics. If running peer servers really was profitable
Uber doesn't drive taxis himself, airBnB doesn't own property. This is how Uber-economy works. We believe it works in the CDN industry as well. We just don't want to be another CDN provider.
1. It started out burning huge amounts of investor capital, basically transferring all of that money to drivers and in the form of subsidized rides, to riders.
2. It works today by a combination of the above and underpaying drivers, who generally make less than minimum wage when they manage to cover their completely costs at all. Despite this, prices have gone up dramatically, so that taxis are once again competitive on a price basis.
3. Uber was profitable in 2018, and not before nor since[0]. Deeply, profoundly unprofitable. Why would anyone want to be the Uber of anything?
0. https://www.macrotrends.net/stocks/charts/UBER/uber-technolo...
> We just don't want to be another CDN provider.
Don't do something differently for the sake of being different. Do something differently because it unlocks some new capability.
Uber was special for a few reasons. Primarily, it was a better UX to use an app for a taxi, and apps couldn't exist when taxis were first popularized. Secondarily, Uber not driving taxis themselves was "different" in labor law handling. Uber didn't own cars/employ drivers not because "they didn't want to be another taxi company" but because they wanted to unlock the capability to pay people differently. It's the same story for AirBnB - what's legal for rental units is not the same as what's legal for hotel rooms.
Any company can rent or own thousands of servers across the planet, we know this because CDNs do it today, and because if an individual can do it, so can a company.
also please note:
if you have 1 image on the page - we have to do 2 requests (1 metadata, 1 download)
if you have 50 images on the page - we have to do 51 requests only (1 metadata for all images, 50 downloads)
But unless you return the same host for 50 objects, I'm going to have to do a bunch of DNS lookups too, right? And a bunch of SSL setups.
Hopefully you at least make sure to try and return a single host for all the objects in a request.
our goal is to saturate the _User's_ network. Now it's hard to say how many Peers we need to meet this requirement. It looks like 2-3 Peers are enough.
Also, when you have 50 Peers, you have to do 50 DNS/SSL requests in _parallel_. With current hardware, it's not a big deal.
You should make this your marketing slogan.
"Try our CDN, with current hardware the gross inefficiencies other CDNs spend a lot of time and effort eliminating are not a big deal."
How do you know the nearest peer has all the images at hand?
- we divide web cache in buckets
- these buckets have different sizes
- we know a list of files in every bucket
- we know what buckets every Peer has
2 is definitely an issue.
3 is a maybe. There's definitely a model where people with access to cheap servers and bandwidth can get on board and not have to worry about the application-level details, finding customers, billing etc.
4 isn't an issue as long as you are selling the CDN specifically for website assets that don't need to have pretty URLs.
Not exactly. For a normal CDN, your webpage HTML directs the client directly to the CDN server, which in theory already has the object. Only if it doesn't have the object does it need to go get it, but there isn't an extra lookup. The CDN does it's direction magic via DNS or IP routing, not with double lookups.
3. Why would the people running a peer have cheaper access to servers and bandwidth than a company specializing in that business? Everything about this business should have huge economies of scale (buying hardware, system administration, transit/peering).
The only situation where I can see a distributed model winning out is if the entity running the peer does not pay for resources but gets them with fraud or theft, e.g. paying for cloud servers with stolen credit cards.
I'm narrowing the scope around 'user tracking' specifically to 3rd parties, because 1st party user tracking is a requirement in order to perform some necessary functions (Providing accounts & user sessions, for example).
In the current centralized CDN model (Cloudflare, AWS, Akamai, etc.), the 3rd party is made clear (those aforementioned companies), and their need to track users is documented on the provider's ToS. Likewise, reputational damage is accrued to the providers in question.
In their proposed CDN model, there is no clear 3rd party, and so it must be defensively assumed that all of the data sent for requests is recorded at all times (The FBI Tor exit node problem). As a consequence, the minimal amount of request data can only be sent to these providers (IP address, content hash). Tor-like routing could be implemented to further disperse the requests from one IP address to multiple addresses, but now the monitoring problem's increased to the routers as well, on top of the additional latency introduced for said privacy. Mandating said routing into the protocol would make such tracks useless, as the IP address collected will roughly appear as random noise in the optimistic case.
Personal opinion below
However, it is likely that no technical solution provided would resolutely satisfy the question being asked.
Akamai acquired RedSwoosh precisely on the same economics promised and if this were economical/viable they would already have deployed it, no?
I did hack with Travis on RedSwoosh code base optimizing his TCP Zipper algorithm which was pretty cool BTW
I don't know how it works in details, but it looks like you need to install a browser plugin to enable the p2p network here.
Our solution works transparently for the end user.
Cache management (eviction, invalidation, etc) is done where? By the look of that diagram, you are actually doubling your origin traffic just to push to the cache after serving the original client request. Perhaps subsequent requests could get served from their 'peer' but the hit rate will be abysmal.
Maybe there is more to it than they describe here, but I don't see how this can work. Are they going to rewrite the object in a way that doesn't require the host certificate?
The disk performance - is the weakest part of the Peer. It is a part you can't guarantee any QoS on a cheap server. The idea is to keep popular cache on Peers in memory. Everything else we will host from a limited number of regular servers (belong to Farba), with high-performance disk subsystem.
>Cache management (eviction, invalidation, etc) is done where?
yes. You can invalidate either a single URL or a folder with a wildcard
>you are actually doubling your origin traffic just to push to the cache after serving the original client request
this is something we can proxy and do only one request per file. Current implementation was done for simplicity and reliability.
>but I don't see how this can work. Are they going to rewrite the object in a way that doesn't require the host certificate?
you can easily check this up. Please open the network console on our demo-page and see how it works:
- we have two different certificates (one for the main site, for for the Peer)
- everything works flawlessly
we just need to add more Peers and evenly distribute the workload.
> what kind of cache hit rate are you expecting at the peer?
I don't know. We have too many moving parts right now and can't predict anything.
That doesn't make sense! If you add more peers just to keep more files in memory, each peer is serving less and less data. There's a fixed, finite amount of usage that the system will receive, and the more peers you add, the more that usage gets subdivided among the peers. Which means they make less money, which makes the system unprofitable. Moreover, you're spending more money to distribute the file to more peers (well, you're not, you're offloading that to the customer right now), which means the more peers, the more expensive for you or the customer.
The problem you have is that handling more files means cutting peer profits without adding any benefit for yourself or the peers. Storage isn't factored into pricing, which means that any accommodation you make to increase availability (an important part of being a CDN!), the worse the economics end up being.
I use a CDN to serve about 50GB of files stored on my origin, which turn over every week or so (a new 50GB every week). What you're saying means that _at best_ I need at least 3-4 peers in order to keep one copy of all of my files available. That's your using your numbers for a LeaseWeb or OneProvider peer. And if I'm competing with other customers, my content is getting evicted, resulting in more hits to my origin. If I'm actually using your system as a CDN where peers should be physically close to my users, that means that I'm filling up RAM of lots and lots of peers just so that enough peers that are close to my users can serve the files.
P2P CDN: https://hn.algolia.com/?query=p2p%20cdn, https://hn.algolia.com/?query=peer%20to%20peer%20cdn
However, a lot of people use CDNs because they are fast. They want the latency to be ultra low and they want it to be reliable. This is where the author's solution falls short: CDNs can get premium quality bandwidth because they have more scale than their customers. Sure you can find servers from various hosts in lots of different data centers, but you might not get the latency or quality of bandwidth you were looking for. This can be critical for some applications like streaming or serving images.
If your constraint is speed, you almost have to go with a CDN (even if you are huge). If it's volume, you are in a better position. If your volume is small, you might as well go with something off the shelf like Cloudfront or Cloudflare. If you are ultra huge like YouTube or Netflix, then you have to peer directly with ISPs.
Zero-cost CDN for 2 USD/TB! Sounds like a great deal to me.
On a more serious note, is price the problem with other CDNs?
Here is one of their papers: https://www.usenix.org/legacy/event/nsdi10/tech/full_papers/...
Was the issues you were seeing in this general area?
1. Round trip to Farba to get metadata.
2. Round trip to your "driver" to get the resource.
3. Computing the hash of the resource to ensure the "driver" didn't cheat.
It could still be worth it as a way of saving money on bandwidth, but I'd be quite surprised if you got the speed benefits of a CDN.
You could send a hash of the file to the client from the coordination server and check it. If the hash isn't valid, you can re-request the file from another peer. But I don't think you can trust the client to report abuse: a malicious client could report innocent peers (e.g., a vindictive Chrome extension).
Or, the site owner could manipulate the coordination JavaScript to report the downloaded file as invalid/corrupt/tampered with, and then just not re-request the file (and there's nothing you can do to stop them). So in that case, the site owner is managing to not pay for the service (assuming you don't bill for downloads that were reported as invalid).
But on top of all of that, this is _slow_. It's slow on my (quite fast) internet. If you're using this, it's because you're trying to get cheap bandwidth, not because you value performance. The whole point of a proper CDN is to improve performance by spreading out load and moving the edge server physically closer to the user. This might be putting the edge server physically closer to the user, but the mechanism to do that seems to take almost as long as the request/download of the asset itself.
This fraud is possible. We collect logs from three sources:
- the Balancer (belong to Farba)
- the Peer
- the User
We can analyze every single case of suspicious behavior.
The reason businesses like this work in general is because they break a big, complicated business into smaller businesses where there is specialization. There may not be be meaningful potential for specialization here; but that is the argument to have. For example, it may be that Farba can be a leaner business if they don't have to think about how to get the best pricing in dozens of different hosting markets.
Anyways, other than that it’s a fun concept. I like the P2P nature of it.
as a web developer, no thanks, I do not want to trust my app’s speed to your profit incentive or margin.