Sounds like someone people are downloading it in their CI pipelines. Probably unknowingly. This is why most services stopped allowing automated downloads for unauthenticated users.
Make people sign up if they want a url they can `curl` and then either block or charge users who download too much.
Do they email heavy users? We used Nominatim free api for geocoding addresses in 2012 and our email was required parameter. They mailed us and asked us to cache results to reduce request rates.
I continue to be baffled the geofabrik folks remain the primary way to get a clean-ish OSM shapefile. Big XKCD "that one bloke holding up the internet" energy.
Whenever I read about such issues I always wonder why we all don’t make more use of BitTorrent. Why is it not the underlying protocol for much more stuff? Like container registries? Package repos, etc.
It became disliked because of various problems and complaints, but mainly disappeared because Blozzard got the bright idea of preloading the patchset, especially to new expansions, in the weeks before. You can send down a ten gig patch a month before release, and then patch that patch a week before release, and a final small patch on the day before release, and everything is preloaded.
The great Carboniferous explosion of CDNs inspired by Netflix and friends has also greatly simplified the market, too.
I used to work at a company that had to deliver huge files to every developer every week. At some point they switch from a thundering herd of rsyncs to using BitTorrent. The speed gains were massive.
To use bittorrent, your machine has to listen, and otherwise be somehow reachable. In many cases, it's not a given, and sometimes not even desirable. It sticks out.
I think a variant of bittorrent which may be successful in corporate and generally non-geek environments should have the following qualities:
- Work via WebSockets.
- Run in browser, no installation.
- Have no rebel reputation.
It's so obvious that it must have been implemented, likely multiple times. It would not be well-known because the very purpose of such an implementation would be to not draw attention.
It's not all about how you distribute content. We must also decide which content do distribute, and that is a hard problem.
The most successful strategy so far has been moderation. Moderation requires hierarchical authority: a moderator who arbitrarily determines which data is and is not allowed to flow. Even bittorrent traffic is moderated in most cases.
For data to flow over bittorrent, two things must happen:
1. There must be one or more seeders ready to connect when the leecher starts their download.
2. There must be a way for a prospective leecher to find the torrent.
The best way to meet both of these needs is with a popular tracker. So here are the pertinent questions:
1. Is your content fit for a popular tracker? Will it get buried behind all the Disney movies and porn? Does it even belong to an explicit category?
If not, then you are probably going to end up running your own tracker. Does that just mean hosting a CDN with extra steps? Cloud storage is quite cheap, and the corporate consolidation of the internet by Cloudflare, Amazon, etc. has resulted in a network infrastructure that is optimized for that kind of traffic, not for bittorrent.
2. Is a popular tracker a good fit for your content? Will your prospective downloaders even think to look there? Will they be offended by the other content on that tracker, and leave?
Again, a no will lead to you making your own tracker. Even in the simplest case, will users even bother to click your magnet link, or will they just use the regular CDN download that they are used to?
So what about package repos? Personally, I think this would be a great fit, particularly for Nix, but it's important to be explicit about participation. Seeding is a bad default for many reasons, which means you still need a relatively reliable CDN/seed anyway.
---
The internet has grown into an incredibly hierarchical network, with incredibly powerful and authoritative participants. I would love to see a revolution in decentralized computing. All of the technical needs are met, but the sociopolitical needs need serious attention. Every attempt at decentralized content distribution I have seen has met the same fate: drowned in offensive and shallow content by those who are most immediately excited to be liberated from authority. Even if it technically works, it just smells too foul to use.
I propose a new strategy to replace moderation: curation. Instead of relying on authority to block out undesirable content, we should use attested curation to filter in desirable content.
Want to give people the option to browse an internet without porn? Clearly and publicly attest which content is porn. Don't light the shit on fire, just open the windows and let it air out.
People like Geofabrik are why we can (sometimes) have nice things, and I'm very thankful for them.
Level of irresponsibility/cluelessness you can see from developers if you're hosting any kind of an API is astonishing, so downloads are not surprising at all...If someone, a couple of years back, told me things that I've now seen, I'd absolutely dismiss them as making stuff up and grossly exaggerating...
However, on the same token, it's sometimes really surprising how API developers rarely ever think in terms of multiples of things - it's very often just endpoints to do actions on single entities, even if nature of use-case is almost never on that level - so you have no other way than to send 700 requests to do "one action".
I don't understand why features like S3's "downloader pays" isn't more widely used (and available outside AWS). Let the inefficient consumer bear their own cost.
Major downside is that this would exclude people without access to payment networks, but maybe you could still have a rate-limited free option.
Some years ago I thought, no one would be stupid enough to download 100+ megabytes in their build script (which runs on CI whenever you push a commit).
I mean, at this point I wouldn't mind if they rate-limit downloads. A _single_ customer downloading the same file 10.000 times? Sorry, we need to provide for everyone, try again at some other point.
It is free, yes, but there is no need to either abuse it or give as much resource for free as they can.
>Just the other day, one user has managed to download almost 10,000 copies of the italy-latest.osm.pbf file in 24 hours!
Whenever I have done something like that, it's usually because I'm writing a script that goes something like:
1. Download file
2. Unzip file
3. Process file
I'm working on step 3, but I keep running the whole script because I haven't yet built a way to just do step 3.
I've never done anything quite that egregious though. And these days I tend to be better at avoiding this situation, though I still commit smaller versions of this crime.
Ah, responsibility... The one thing we hate teaching and hate learning even more. Someone is probably downloading files in some automated pipeline. Nobody taught them that with great power (being able to write programs and run them on the internet) comes great responsibility. It's similar to how people drive while intoxicated or on the phone etc. It's all fun until you realise you have a responsibility.
But that raises the complexity of hosting this data immensely. From a file + nginx you now need active authentication, issuing keys, monitoring, rate limiting...
Yes, this the the "right" solution but it is a huge pain and it would be nice if we could have nice things without needing to do all of this work.
Speaking as the person running it - introducing API keys would not be a big deal, we do this for a couple paid services already. But speaking as a person frequently wanting to download free stuff from somewhere, I absolutely hate having to "set up an account" just to download something once. I started that server well over a decade ago (long before I started the business that now houses it); the goal has always been first and foremost to make access to OSM data as straightforward as possible. I fear that having to register would deter many a legitimate user.
Meta: the title could be clearer, e.g. "Download OSM Data Responsibly" would have helped me figure out the context faster as someone not familiar with the domain name shown.
Why put a load of money into infra and none into simple mitigations like rate limiting to prevent the kind of issues they are complaining they need the new infra for?
Sounds like someone has an internal deployment script with the origin mirror in the update URI. This kind of hammering is also common for folks prototyping container build recipes, so don't assume the person is necessarily intending anyone harm.
Unfortunately, the long sessions needed for file servers do make them an easy target for DoS, and especially if supporting quality of life features like media fast-forward/seek features. Thus, a large file server normally does not share a nimble website or API server (CDNs still exist for a reason.)
Free download API keys with a set data/time Quota and IP rate limit are almost always necessary (i.e. the connection slows down to 1kB/s after a set daily limit.) Ask anyone that runs a Tile server or media platform for the connection-count costs of free resources.
It is a trade-off, but better than the email-me-a-link solution some firms deploy. Have a wonderful day =3
Chances of those few people doing very large amounts of downloads reading this are quite small. Basic rate limits on IP level or some other simple fingerprinting would do a lot of good for those edge cases, as those folks are most likely not aware of this happening.
Yes, IP rate limiting is not perfect, but if they have a way of identifying which user is downloading the whole planet every single day, that same user can be throttled as well.
Seems to me like the better action would be to implement rate-limiting, rather than complain when people use your resource in ways you don't expect. This is a solved problem.
Oh hey, it's me, the dude downloading italy-latest every 8 seconds!
Maybe not, but I can't help but wonder if anybody on my team (I work for an Italian startup that leverages GeoFabrik quite a bit) might have been a bit too trigger happy with some containerisation experiments. I think we got banned from geofabrik a while ago, and to this day I have no clue what caused the ban; I'd love to be able to understand what it was in order to avoid it in the future.
I've tried calling and e-mailing the contacts listed on geofabrik.de, to no avail. If anybody knows of another way to talk to them and get the ban sorted out, plus ideally discover what it was from us that triggered it, please let me know.
Hey there dude downloading italy-latest every 8 seconds, nice to hear from you. I don't think I saw an email from you at info@geofabrik, could you re-try?
55 comments
[ 0.28 ms ] story [ 73.6 ms ] threadMake people sign up if they want a url they can `curl` and then either block or charge users who download too much.
Also, everyone go contribute/done to OSM.
https://rwmj.wordpress.com/2013/09/09/half-baked-idea-conten...
But yes I do think Bittorrent would be ideal for OSM here.
https://en.wikipedia.org/wiki/InterPlanetary_File_System
It became disliked because of various problems and complaints, but mainly disappeared because Blozzard got the bright idea of preloading the patchset, especially to new expansions, in the weeks before. You can send down a ten gig patch a month before release, and then patch that patch a week before release, and a final small patch on the day before release, and everything is preloaded.
The great Carboniferous explosion of CDNs inspired by Netflix and friends has also greatly simplified the market, too.
If A is a copyrighted work, and B is pure noise, then C=A^B is also pure noise.
Distribute B and C. Both of these files have nothing to do with A, because they are both pure noise.
However, B^C gives you back A.
I think a variant of bittorrent which may be successful in corporate and generally non-geek environments should have the following qualities:
It's so obvious that it must have been implemented, likely multiple times. It would not be well-known because the very purpose of such an implementation would be to not draw attention.It's not all about how you distribute content. We must also decide which content do distribute, and that is a hard problem.
The most successful strategy so far has been moderation. Moderation requires hierarchical authority: a moderator who arbitrarily determines which data is and is not allowed to flow. Even bittorrent traffic is moderated in most cases.
For data to flow over bittorrent, two things must happen:
1. There must be one or more seeders ready to connect when the leecher starts their download.
2. There must be a way for a prospective leecher to find the torrent.
The best way to meet both of these needs is with a popular tracker. So here are the pertinent questions:
1. Is your content fit for a popular tracker? Will it get buried behind all the Disney movies and porn? Does it even belong to an explicit category?
If not, then you are probably going to end up running your own tracker. Does that just mean hosting a CDN with extra steps? Cloud storage is quite cheap, and the corporate consolidation of the internet by Cloudflare, Amazon, etc. has resulted in a network infrastructure that is optimized for that kind of traffic, not for bittorrent.
2. Is a popular tracker a good fit for your content? Will your prospective downloaders even think to look there? Will they be offended by the other content on that tracker, and leave?
Again, a no will lead to you making your own tracker. Even in the simplest case, will users even bother to click your magnet link, or will they just use the regular CDN download that they are used to?
So what about package repos? Personally, I think this would be a great fit, particularly for Nix, but it's important to be explicit about participation. Seeding is a bad default for many reasons, which means you still need a relatively reliable CDN/seed anyway.
---
The internet has grown into an incredibly hierarchical network, with incredibly powerful and authoritative participants. I would love to see a revolution in decentralized computing. All of the technical needs are met, but the sociopolitical needs need serious attention. Every attempt at decentralized content distribution I have seen has met the same fate: drowned in offensive and shallow content by those who are most immediately excited to be liberated from authority. Even if it technically works, it just smells too foul to use.
I propose a new strategy to replace moderation: curation. Instead of relying on authority to block out undesirable content, we should use attested curation to filter in desirable content.
Want to give people the option to browse an internet without porn? Clearly and publicly attest which content is porn. Don't light the shit on fire, just open the windows and let it air out.
Level of irresponsibility/cluelessness you can see from developers if you're hosting any kind of an API is astonishing, so downloads are not surprising at all...If someone, a couple of years back, told me things that I've now seen, I'd absolutely dismiss them as making stuff up and grossly exaggerating...
However, on the same token, it's sometimes really surprising how API developers rarely ever think in terms of multiples of things - it's very often just endpoints to do actions on single entities, even if nature of use-case is almost never on that level - so you have no other way than to send 700 requests to do "one action".
Major downside is that this would exclude people without access to payment networks, but maybe you could still have a rate-limited free option.
Then I learned about Docker.
It is free, yes, but there is no need to either abuse it or give as much resource for free as they can.
Whenever I have done something like that, it's usually because I'm writing a script that goes something like:
1. Download file 2. Unzip file 3. Process file
I'm working on step 3, but I keep running the whole script because I haven't yet built a way to just do step 3.
I've never done anything quite that egregious though. And these days I tend to be better at avoiding this situation, though I still commit smaller versions of this crime.
getfile.sh processdata.sh postresults.sh doall.sh
And doall.sh consists of: ./getfile.sh ./processdata.sh ./postresults.sh
Yes, this the the "right" solution but it is a huge pain and it would be nice if we could have nice things without needing to do all of this work.
This is tragedy of the commons in action.
Unfortunately, the long sessions needed for file servers do make them an easy target for DoS, and especially if supporting quality of life features like media fast-forward/seek features. Thus, a large file server normally does not share a nimble website or API server (CDNs still exist for a reason.)
Free download API keys with a set data/time Quota and IP rate limit are almost always necessary (i.e. the connection slows down to 1kB/s after a set daily limit.) Ask anyone that runs a Tile server or media platform for the connection-count costs of free resources.
It is a trade-off, but better than the email-me-a-link solution some firms deploy. Have a wonderful day =3
Appeals to responsibility aren't going to sink in to people that are clearly careless
Yes, IP rate limiting is not perfect, but if they have a way of identifying which user is downloading the whole planet every single day, that same user can be throttled as well.
Maybe not, but I can't help but wonder if anybody on my team (I work for an Italian startup that leverages GeoFabrik quite a bit) might have been a bit too trigger happy with some containerisation experiments. I think we got banned from geofabrik a while ago, and to this day I have no clue what caused the ban; I'd love to be able to understand what it was in order to avoid it in the future.
I've tried calling and e-mailing the contacts listed on geofabrik.de, to no avail. If anybody knows of another way to talk to them and get the ban sorted out, plus ideally discover what it was from us that triggered it, please let me know.
Edit: I sent you the email, it got bounced again with a 550 Administrative Prohibition. Will try my university's account as well.
Edit2: this one seems to have gone through, please let me know if you can't see it.