Ask HN: How to distribute files in CDN?

1 points by hknmtt 3y ago ↗ HN

I am building a private CDN and am wondering what is a good approach to distributing the files?

So far I have decided that I want the files to be downloaded into CDN node only when requested by client and files that won't be read for a week will be deleted.

Except that I am not sure how to approach the distribution because I could overwhelm the source of files with too many request for files from the nodes. Also I am not sure if using websocket is good enough to transfer files(so I can re-use the source's api http server connection) or whether I should send data directly via length-value "protocol" or some another one. Compression is obvious. But other than that I am not sure what modern CDNs use today.

What are some standard CDN architectures for distributing the files these days?

3 comments

[ 2.4 ms ] story [ 13.6 ms ] thread

phillipseamore 3y ago ↗

CDNs are just reverse proxies. Cloudflare was built around nginx (though they have now developed their own reverse proxy) and so do most CDNs. Fastly uses Varnish. A few use Apache Traffic Server. A correctly configured nginx can easily manage caches by various rules like minimum hits, maximum age etc.

hknmtt 3y ago ↗

i was thinking about doing just that - when request comes, check local file, if not present, fetch from remote url. but the problem is with large files(GBs) that you cannot serve as fast and the client has to wait instead of being redirected to the source which bypasses the cdn node and then what? you have to somehow get the large files too in a way that makes sense.

after a bit more thought, you are right. this is the right approach. i just need to make sure the file is being fetched only once and if client closes the connection the file needs to keep being downloaded.