Show HN: OpenFreeMap – Open-Source Map Hosting (openfreemap.org)
Hi HN,
After 9 years of running my own OpenStreetMap tile server infra for MapHub (https://news.ycombinator.com/item?id=11389989), I've open-sourced it and launched OpenFreeMap.
You can either self-host or use our public instance. Everything is open-source, including the full production setup — there’s no 'open-core' model here. Check out the repo (https://github.com/hyperknot/openfreemap). The map data comes from OpenStreetMap.
I also provide weekly full planet downloads both in Btrfs and MBTiles formats.
I aim to cover the running costs of the public instance through donations.
Looking forwards for your feedback.
242 comments
[ 4.4 ms ] story [ 341 ms ] threadI know nothing about mapping, but I’ve always had a dream of making a private neighborhood map which labels each house with people, sort of like a visual Rolodex. It would be great to have property boundaries, too. Do you have any suggestions for places to start? I don’t even know what file formats would hold this sort of data.
public domain map data in GeoJSON format.
also, for google maps I know you can respond to interactions with the supplied points of interest like buildings, including homes, and from that point you can obtain an address via reverse geocoding or some other technique. I expect other mapping tools may have similar abilities but I don't have any experience with anything other than Google.
I work for a company (onXmaps) which produces user-friendly maps of such information. You could use our app to create such a "rolodex" of your neighborhood, by annotating individual parcels with your neighbors' information.
https://onxmaps.com
In my mind I’d really like to host my own map which neighbors could share, just starting from a baseline of parcel boundaries and a way to drop info on each plot (onX does this wonderfully already).
Back in the day, my mom would make a little neighborhood diagram using MS Publisher and drawing boxes at the approximate locations of houses, and include names & phone numbers of each house. She would then distribute copies to the neighbors to hang on fridges. I loved that idea and basically want an online version of that, in a perfect world!
https://baremaps.apache.org/
Question, why use btrfs?
- MBTiles - an SQLite file, each file is a row in a table, you need a server to serve it.
- PMTiles - a single file, optimised for serverless usage.
- Extract them into a directory, which in practice should be on a partition image. This is the approach I choose.
I tested ext4 and Btrfs and I choose Btrfs. The reason is how ext4 and Btrfs handles inodes. Btrfs handles inodes so much better compared to ext4, it doesn't allocate them at start and also allows putting tiny files right with the metadata.
Because here the average tile size is only 405 bytes, most of the tiles can actually stored with the Btrfs metadata block. From the latest run:
Btrfs data is 51.57GiB
Btrfs metadata is 84.37GiB
I've been making some custom map stuff for mountain bike trails and I'd really like to move to self-hosted vector tiles for all layers, but too much of what I find says to start with Planet.osm when all I really need is a State (in the US) or even a few-miles-wide area.
(My goal is to basically snapshot OSM data, generate tiles, and use that until I decide to do another snapshot down the line, so the underlying data doesn't change. And limit it to a small area because that's all I need.)
Examples of maps I've done this way, and want to improve, are: https://trailmaps.app/ramba/ and https://trailmaps.app/dte/
Another option is to use the extract functionality in pmtiles [4] to extract your area of interest from their daily full-planet pmtiles builds. You can then statically host that file and use that in your client with one of their client libraries.
[1] https://github.com/onthegomap/planetiler
[2] https://download.geofabrik.de/
[3] https://gdal.org/en/latest/programs/gdalwarp.html
[4] https://docs.protomaps.com/pmtiles/cli#extract
I wanted to give repackaging the btrfs image a shot but the download was pretty slow - I assume your server is getting HN-hugged a bit so I didn't want to make it worse and stopped the download.
[1] https://erofs.docs.kernel.org/en/latest/index.html
btrfs.openfreemap.com just a public Cloudflare bucket, no idea why it might be slow.
btrfs has certainly been around for longer, but in my (embedded systems only) experience, EROFS has been pretty solid - it's slowly being picked up by Android, so it is definitely seeing a lot of use in the wild (probably surpassing btrfs by the number of installations already).
> btrfs.openfreemap.com just a public Cloudflare bucket, no idea why it might be slow.
I'm getting 30 MiB/s (on a gigabit uplink) - not great, not terrible. A .torrent would be nice but I guess outside of being on the HN front page full-planet downloads by different people won't synchronize enough for this to be useful (and using web seeds is problamtic in its own right with small-ish chunks).
Making this automatic might not be easy, but I might do it one day, it might help with the donations.
That total has gone up. :-)
Not currently having any plans of using openfreemap, but I like the cut of your jib. I'm currently in the process of generating aerial tiles for Colorado, and updating 2 of our map tile sets (bitmap tiles, Here and OpenStreetMap), but at some point we should switch to vector tiles. When we do I'll try to get a business sponsorship going, but it is nearly impossible for me to get the company to pay for sponsorship. :-(
Slightly off-topic, one thing I'm having a hard time finding is satellite imagery. I need to self-host an offline web application using CesiumJS, and while I can spin up a map server, I can't find satellite imagery for free or cheap. I've used MapTiler[0], their server works great and they offer low-res satellite imagery for free/testing, but their high-res images are out of my price range.
Anyone know of resources for downloading offline images of satellite imagery, compatible with OpenFreeMap or other server for use with CesiumJS? Doesn't have to be super recent images but would be nice.
[0]: https://www.maptiler.com/
Basically Protomaps / PMTiles allows you to do this serverless, but it has it's downsides.
There are two ways to use PMTiles with Cloudflare:
- Putting the file in a public bucket and use HTTP range requests.
- Deploy a worker to access it.
For the range request version, you can test PMTiles here: https://pmtiles.io/?url=https%3A%2F%2Fdata.source.coop%2Fpro....
It loads in 8-10 sec for me on a cold start. Of course it gets faster if many of us try to request it at the same time, but if you check it once per day for example, it'll be really slow, up to 10 sec.
Compare that with OpenFreeMap here: https://openfreemap.org/quick_start
It's near instant for me. (I know it's not full screen but you can make the app and then see how it's in full screen).
For the Cloudflare Workers version, I didn't find any publicly available full planet test, so I don't know the performance, but Workers are not free. It might not matter for a small project, but the $0.30/million requests per month can easily add up if your project gets popular.
In terms of OpenFreeMap, hosting it on a Cloudflare Worker would be prohibitively expensive.
The latency you see on https://pmtiles.io/?url=https%3A%2F%2Fdata.source.coop%2Fpro... is representative of how PMTiles works on AWS S3, coming from the us-west-2 region. It will be reasonable to load for those in the western US and likely quite slow from Europe or Oceania.
If you want to make a direct comparison of Protomaps to OpenFreeMap, you need to compare serving OpenFreemap with NGINX from btrfs on disk, to running `pmtiles serve` on a `.pmtiles` file on disk, as described here: https://docs.protomaps.com/deploy/server
The OpenFreeMap page for me (in Taiwan) takes 1-2 seconds per tile, which is more than double the load tile for the PMTiles in us-west-2 example linked above.
The best solution to get latency competitive with commercial providers, for a global audience, is to cache tiles at CDN edge locations. I describe automated ways to do that with Cloudflare Workers and AWS Lambda + Cloudfront here:
https://docs.protomaps.com/deploy/cloudflare https://docs.protomaps.com/deploy/aws
I'm also experimenting with http://tigrisdata.com which lets you geo-distribute a static storage bucket like in this example: https://pmtiles.io/?url=https%3A%2F%2Fdemo-bucket.protomaps....
I can imagine workers to be fast, it's the range requests which are super slow. It's also outside of your control, it depends on how Cloudflare and S3 handles range requests to 90 GB files.
I think if you could make PMTiles split into files <10 MB, it'd be perfect with range requests.
I would be interested to see a comparison of Btrfs + nginx serving latency, vs `pmtiles serve` from https://github.com/protomaps/go-pmtiles on a PMTiles archive on disk. That would be a more direct comparison.
I think there's potentially some interesting use case for tiles in Btrfs volumes and incremental updates, which I haven't tackled in PMTiles yet!
The fact that there is no service running was the more important for me. Mostly for security and bugs. I had so many problems with various tile servers in production, they needed daily restarting, they had memory leaks, etc.
Basically I wanted to go nginx-only for security and to avoid tile server bugs.
For PMTiles the module is loadable directly as a Caddy plugin (https://docs.protomaps.com/deploy/server#caddyfile) which I prefer to nginx for security and bugs (and automatic SSL), and also enables serving PMTiles from disk or a remote storage bucket without a separate service running.
At that point, the difference between the two projects is mostly which schema is being used.
The nearest thing (and it's not really really similar, just related) that ZFS has is special VDEVs.
When you have an array of disks (usually "slow", like regular HDDs) you can attach to it another array of disks (usually very fast, like NVMe) where you can store metadata (file information) and optionally small files up to a size that you can define.
So for example you have a 50TB (or 500, who knows) array of SATA disks, and a small but superfast array of NVMe drives. Lets say 128GB. Or 512, or 1TB depending on you want to do.
File metadata is saved there, so doing a find, ls, tree... operation is now very fast. And if you save, for example, all files smaller than 32KB there (it will depend on your needs, also) all the small file operations will be way faster.
https://github.com/bdon/TileSiege
On a related subject, I remember seeing MapHub sometime ago and I have it in my bookmarks for one of my (forgotten?) projects. Whenever I find some free time I need to sit down and try the free tier to see if it will do what I have in mind.
I'm not sure if their hosting works for you because you might be serving byte ranges. I think we're paying 2k/month usd for like 300TB/month. Backblaze-to-fastly is free, so no egress.
Love your project, more power to you. I'll be using this for a few side projects for sure.
I don't think this project will ever make $2k per month. TBH, I'd be happy it it ever reaches $200 per month. I have to be very careful about what hosting providers to use and CDNs are out of the question.
https://docs.hetzner.com/robot/dedicated-server/network/10g-...
As for other cdns - I remember https://neocities.org/ built their own cdn - but I couldn't immediately find a blog post about the cdn, or something related in their GitHub (unless every node runs part of it, which would make sense, I guess).
They could use you as a PoC/demo for their serverless stuff. Same with cloudflare.
It's worth it to ask. I can send you the contact for our fastly rep if you'd like.
Maybe it's worth front-ending it with cloudflare free, just to see/measure cache hits/requests/bandwidth?
In addition to Cloudflare, bunny was on the radar too.
I believe Hetzner should not have any problem with using 100 mbps out of the 1 Gbps on their servers. That's 30 TB/month for free, per server. It'd be $300-$1800 per month on Bunny.
Just like the cloud becoming easier to run on your own environments, tons of bandwidth is accessible too.
Also thanks for the donation!
> Unfortunately, making range requests in 80 GB files just doesn't work in production. It is fine for files smaller than 500 MB, but it has terrible latency and caching issues for full planet datasets.
Wondering which part incurs the latency here.
In summary, on a cold start (so now when it's on HN it might be totally meaningless), the following page loads in like 8-10 sec for me: https://pmtiles.io/?url=https%3A%2F%2Fdata.source.coop%2Fpro...
Technically why it's slow is entirely on CloudFlare, so I can only guess. But probably they are not mirroring files of this size across their datacenters / servers. I don't know how their public buckets hosting is done, but it's definitely very slow.
OpenFreeMap on the other hand seems almost instant for me.
As a user, free or paid, it'd give me a lot more faith in this living past tomorrow.
And the risk of this project is largely mitigated by offering the full repo as open-source, so you can always self host this if needed.
The latter should be easier to manage with more predictable traffic, while giving you more leeway to deal with abuse on the free servers if it ever becomes an issue.
You may think it noble to "only take donations" just to pay the hosting cost, etc. but providing people support is a completely different thing. People are stupid, needy and inconsiderate of your time, feelings and experience.
If you don't want to run ads, great! But your time should be compensated fairly as well. You should be charging businesses for things, with a substantial cost for substantial work. Charging a business is not the same as charging an end user and you should not be averse to doing so.
(a) They're making millions, you deserve a cut of that.
(b) If making money makes you more likely to be around in 10 years, that's attractive to businesses. Nobody knows when your donations might suddenly dry up.
I honestly believe this project can cover $175 per month eventually.
Of course there is the countless hours which went and which can possibly go into this, but to cover the hosting is still meaningful.
Let's say 1 year from now your day job gets toxic and you have to work 18 hours a day just to pay your rent and expenses. Are you still going to maintain this?
What if the economy goes to shit and you have trouble scraping that $175/month? How long are you going to pay $175/month for something that isn't making money?
What if, instead, this became a part-time job of yours?
If I was a commercial user of your product, some assurance that your random personal situations aren't going to affect the product stability for a long time would be a good thing.
Why OMT instead of protomaps? The latter is clearly where the community is moving towards (albeit very slowly).
I'm somewhat sceptical about the "free with no API keys" idea. I guess your service is not guaranteed to be up so no one too big will rely on it. But what if you start getting abuse or someone using them on some humongous site (e.g. one of those cheap restaurant email builders that always embed a map), and you start getting way too much traffic from random sources and websites. What would you do?
> What about PMTiles?
> I would have loved to use PMTiles; they are a brilliant idea!
> Unfortunately, making range requests in 80 GB files just doesn't work in production. It is fine for files smaller than 500 MB, but it has terrible latency and caching issues for full planet datasets.
> If PMTiles implements splitting to <10 MB files, it can be a valid alternative to running servers.
[1] https://github.com/hyperknot/openfreemap
The main reason PMTiles is one file and not two or more files is that it enables atomic updates in-place (which every mature storage platform supports) as well as ETag content change detection in downstream caches. All of the server and serverless implementations at http://github.com/protomaps support this now for AWS, S3-compatible storage, Google Cloud, and Azure.
https://github.com/bdon/cloudflare-r2-latency
Also, I'm pretty sure range requests are going to be difficult to cache. That implies going to origin every request which is bad.
First 100KB of a 100GB+ file:
curl -H "Range: bytes=0-100000" https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2... --output tmp -w "%{time_total}"
First 100KB at the 100GB mark:
curl -H "Range: bytes=100000000000-100000100000" https://overturemaps-tiles-us-west-2-beta.s3.amazonaws.com/2... --output tmp -w "%{time_total}"
BTW, we are discussing latency with bdon in this issue, it seems to be specific to Cloudflare: https://github.com/hyperknot/openfreemap/issues/16
- There is the Protomaps schema. It's competing with OpenMapTiles and Shortbread (https://shortbread-tiles.org/)
- There is the Protomaps file format. It's competing with MBTiles and say Btrfs images in OpenFreeMap.
- There is the Protomaps styles. It's competing with OSM Liberty and OSM Bright styles (https://github.com/maputnik/osm-liberty)
- There is the way to host the tiles, on CloudFlare workers, range requests, etc. It's competing with MBTiles servers or OpenFreeMap's nginx servers.
About getting abuse, how do you imagine it gets abuse? It's a static file showing OpenStreetMap data.
I mean traffic is the only thing what can be a problem, but I think Hetzner should be good till 30-100 TB per server.
2) The file format (PMTiles) addresses a different audience than either MBTiles or Btrfs images. Both of those require administering a server for tiles, while PMTiles requires static blob storage and nothing else. You do have the option of using a server like MBTiles/btrfs which ought to be comparable in latency, and that's documented here: https://docs.protomaps.com/deploy/ as well as Lambda, Cloudflare Workers, Google Cloud Run and Azure Serverless functions.
3) There are no existing styles for MapLibre GL that work off the Tilezen layer, generalization and tagging scheme, so we need to develop one style, with multiple themes.
Can you elaborate on this ? I'm derivating OMT and am quite worried now ^^
This means that software that implements OMT, even if written from scratch, cannot be re-used by other FOSS projects (Apache, BSD, GPL, AGPL, other software in the OpenStreetMap ecosystem, etc) without affecting the license.
Ideally for Protomaps it should be possible to re-use just one portion - like only the label layer with your own layers from other sources, or even bundle it as a JS dependency in another open source project - without affecting the license of downstream projects.
Even so, CC-BY is permissive and you could include CC-BY content in a, say, GPL project. You just need to include both licences.
That is exactly the opposite of OMT's copyright interpretation: https://github.com/openmaptiles/openmaptiles?tab=readme-ov-f...
> You just need to include both licences.
That is the definition of license incompatibility as described in the Creative Commons documentation above. The license is open source and a good fit for if you are running a paid map SaaS or free service as an end product, but is not compatible with the open source ecosystem as a building block.
They are entitled to their opinion, but I don't consider a database to be a derivative work of a schema (this should be clear, since databases of facts aren't copyrightable anyway), same for software (it's at least fair use in the US, see Google v. Oracle). This goes back to the debate on copyrightability of APIs, where a decision for copyrightability would ruin swathes of the software industry and much of free software.
Though I understand if you'd rather avoid using projects by people with weird legal opinions.
> That is the definition of license incompatibility as described in the Creative Commons documentation above.
Where do you see that? All I found was a statement that the CC share-alike licences can be converted to GPL, but no word on non-SA licences:
> Version 4.0 of CC’s Attribution-ShareAlike (BY-SA) license is one-way compatible with the GNU General Public License version 3.0 (GPLv3).
The FSF says that CC-BY is compatible with the GPL, and I believe this extends to all reasonable FLOSS licences: https://www.gnu.org/licenses/license-list.html#ccby
It's not uncommon for a piece of code to have multiple licences in a big project. One for licencing to that project, and another for sublicencing to the end user. Proprietary apps that include FLOSS code do this all the time.
"Additionally, our licenses are currently not compatible with the major software licenses, so it would be difficult to integrate CC-licensed work with other free software. Existing software licenses were designed specifically for use with software and offer a similar set of rights to the Creative Commons licenses." https://creativecommons.org/faq/
> Though I understand if you'd rather avoid using projects by people with weird legal opinions.
The scale of the Protomaps project is small (I'm the only full-time developer) and I don't have the resources to interpret novel copyright situations. I think it's best for the ecosystem to abide by the license terms stated by the open source developer, instead of challenging their validity.
If you want free servers/bandwidth around the world, hit me up. My email is in my profile.
The pitch for the project is here if anyone in Taiwan is interested in joining: https://hackmd.io/7K4kLkseTYCPI7DWoNv5mA?view
Hackathon info: https://g0v-jothon.kktix.cc/events/g0v-hackath63n
1. https://www.weather.gov/documentation/services-web-api
2. https://news.ycombinator.com/item?id=34329988
3. https://pirateweather.net/
I think it would be quite a straightforward task to transform it into a Docker file. I'd be happy to link to it on GitHub.
In total, there are 271 million hard links. So out of 300 million files 271 are hard links!
The file system limit is 64k hard-links for the same file, so I have to handle the case when it's reached and then start a new file for the next 64k.
I had never heard that, so I went sniffing around and it seems to be ext4 specific[1] but I wasn't able to easily get the limits for ZFS (or xfs, etc), so depending on how much glucose one wished to spend it may be better to use a different FS than all that renaming work around
1: https://en.wikipedia.org/wiki/Hard_link#Limitations
If not, declaring one with IMGui+layers can be done in a week from any experienced C++ programmer. The included demos for OSGEarth already depict a minimal client with impressive 3D maps for a nearly non-supported demo.
I think KDE/Plasma lost a great oportunity there by not adopting OSGEarth.
Do you know about https://panoramax.fr ?
Last year I made a website for my gf where I had to build a custom map of Paris, and I struggled a lot trying to figure out how to actually make a map from scratch while avoiding paid services like mapbox.
I finally managed to hack something up using openstreetmap data, then some manual work in QGIS to customize the look, and voila - I had a bunch of folders filled with raster tiles.
This site is deployed for free on Netlify and is basically just a React SPA, a public folder with tiles, and I give the tile URL template to the OpenLayers lib to display it all nicely on the screen. Simple and it works!
I always wanted to improve the map a bit by using vector tiles as I think it looks nicer, but I thought you need a dedicated server for that? (unless I'm mistaken, correct me if I'm wrong)