The code is available for free and you can host it on your own Cloudflare account. Cloudflare provides a free plan (up to 100k daily requests across all your Workers), and paid plan ($0.50 per 1m requests)
Love the ingenuity and thank you for the performance comparison on Cloudflare Workers vs AWS Lambda. I personally wouldn’t consider what you built a Geolocation service, but glad it solved your use case!
I always find it funny when people brag about cost savings on cloud services while overlooking the insane premium they're paying by using the "cloud" to begin with.
It's kinda like a construction firm using a supercar to haul materials and then bragging about their efforts to make cosmetic repairs cost-effective... maybe you could just not use an overpriced car to begin with and then damage won't be an issue?
Am I missing something? Isn't a $3 VM referring to a VM service in a cloud somewhere? If you try to do the same thing without cloud you'll have to end up renting space in a datacenter, install your own servers, connect to an ISP, power bills, etc etc. Cloud definitely seems cheaper in comparison
GP and you aren't using the same definition of cloud. By cloud, they mean infrastuctures like AWS/GCP/Azure, not simple VM hosting (called VPS outside "clouds", and compute nodes in "clouds")
What's the definition of "cloud" then which excludes VPS from the umbrella of services offered in cloud? Not really sure how a VPS is any different from a Google Compute Engine
I think the main difference is the pricing structure. Usually "cloud" refers to the auto-scaling servers with the pay-as-you go pricing, while VPSs are usually manually instantiated and have a fixed monthly cost. Also, sometimes with VPSs you get more control over the server's settings than with "cloud" services.
So "cloud" is like renting a dynamically changing number of VPSs each month and which are usually more costly if your consumption is not spiky (eg. when a cheap VPS rented 24/7 is enough).
The main advantage of the cloud for me that it provides PaaS services. You don't have to administer a VM, etc, you just use the services and it's someone else's job to install security updates and stuff.
And sometimes automatic updates break and then the site may be down until you find the problem which may require much reading and therefore a longer time if you don't deal with sysadmin stuff often.
With PaaS it's someone else's job who does this every day.
...so someone else can install the updates that break the server? Check your SLA, no cloud provider is going to pay you much compensation when the service goes down
I had a geo-location service running in google app engine. for free. After pimping caching the daily traffic of 400.000 requests/day was in the free-tier.
This one I made couple years ago (and haven’t checked in years) is still running for free thanks to Heroku and Cloudflare: https://github.com/jlxw/geoip
So this finds the country based on some cloudflare-populated info of an incoming request (which sounds like it solved OP’s problem which is good), but if you want to use MaxMind’s database to find the country of any public IP I built the following a few months back:
How does MaxMind prevent someone from releasing an open source version of their database? If you are about to answer "Copyright", remember - you can't Copyright facts. This has been upheld in the Courts system many times.
MaxMind have two versions of their database, the free to use version called GeoLite and the paid version (whos name I'm not aware of).
I realise this doesn't directly answer your question but I guess one reason might be that they already provide a free version so releasing the paid version would just get you in strife?
Facts cannot be copyrighted, but a unique compilation of facts can be. This became settled law many decades ago with sports almanacs.
Furthermore, Maxmind adds non-factual entries to the database so they can identify it uniquely as their work. Map makers do something similar: https://en.wikipedia.org/wiki/Trap_street
I (like many others it seems!) have also built a geolocation service however mine is built on top of MaxMind's DBs that are mentioned in the post. Its on a few boxes running OpenResty and now handles 130m+ requests/day. Was really fun to build!
At the bottom of the page it credits the MaxMind GeoLite database. The GeoLite2 database is licensed as CC BY-SA 4.0. It's the GeoIP2 database that has stricter licensing.
As another reply states I publish that the source of data is indeed MaxMind. I agree ethically it doesn't quite sit right so thats why I don't charge for the service.
Google App Engine had this quite a while ago (almost a decade, I think)and a very generous free tier. Also has approx lat-long, city, state and country.
Can’t you just ask the client which timezone it’s in? Every browser knows, and it will be a more accurate answer: the user might be on a foreign VPN or temporarily in another TZ, but not working on that timezone.
I’d like to set a system clock timezone via geoip only, without location lookup via wifi. On my more secure systems I have location services disabled and it’d be nice to have an accurate local time automatically.
Ah, I haven’t seen a dataset I could use for that. Everything on the links I’m putting up is based on the country and state information.
I’ve seen timezone maps drawn, so this information does exist somewhere, but one would have to draw polygons for each timezone and do coverage checks, which seems complicated. The time zone lines zig and zag a lot.
This is surprisingly difficult task if you need it to work really globally and accurately. You need specific, detailed and uptodate map data for it, and timezones change somewhere in the world all the time. Especially daylight savigs. In 99% of cases you need to find some workaround or limit your task to specific regions, and get something to work in 95% cases
Try https://blip.runway7.net/v2.beta – I've added unix timezone based on the country and state code. The ISO databases have timezones associated with the level 2 code, which roughly corresponds to a state inside a country.
I had to tackle this problem as well for a small project and found that I could just send a GET request to https://www.cloudflare.com/cdn-cgi/trace and parse the response with a simple regex: /loc=(\w*)$/gm
I mean, you can do that but it's a hideous hack and we might change that output at any time and you could easily hit DDoS mitigation or other security features if you do that. That endpoint is used for debugging and is in no way an API.
I'd advise you to just create a Cloudflare Worker if you want do this. You'll get more reliability etc. and Workers have a very generous free tier.
It is worth mentioning that /cdn-cgi/trace doesn't only work under www.cloudflare.com. It works for any https?://DOMAIN_WITH_CLOUDFLARE/cdn-cgi/trace . It is an easy to determine whether a website is using CloudFlare or not too.
since this article mentions maxmind, this reminded me of when geolocation by IP address goes horribly wrong, and non-technical persons interpret the results as something to be relied upon as factual:
ISP perspective here: Geolocation by granular /24 to /20 sized block of ipv4 space is often wildly inaccurate on a regional basis. It's entirely possible for the ARIN registration (used by maxmind) to be a street address in Seattle, but serving end user ISP customers a 4.5 hour drive away in a far eastern corner of WA state.
Why would you need to depend on a third-party service and network being available for such a basic task? MaxMind provides their GeoLite database for free and it's extremely easy to embed it in your app.
I've used the free MaxMinds database for a while but since the last year I've been using iplist.cc. It supports IPv4, IPv6, shows if an IP is tor, spam, which ASN and a lot more.
It's also free and fast but with services like these I always wonder how long they manage to stay free.
Right. The first worry is "will this be here tomorrow?" The second is "how much more will they end up charging?"
I had good success with maxmind's free database. I can't find tooth results for accuracy but my sampling was pretty good. They also have an incentive to keep it.
That would be a lot more appealing if it explained where the data is sourced from, included the update scripts, and how it is licensed - the only issue asking that question was closed without a word.
AFAIK, an unresolved problem is a proper geolocation service - at least for city level resolution - for mobile IP addresses. There are some services in this field (digital element), but they are very unreliable.
Kosovo is not universally recognised though, to be fair. But yes, I'd include it. Even if you are against in principle, it's de facto a state and treated as such even by many countries not recognising it in name.
Which is precisely what for example Mapbox does - they allow consumers to specify a worldview, which adjusts borders and other geographic features based on the specified culture: https://docs.mapbox.com/help/glossary/worldview/
A somewhat related problem is that overseas regions (at least for France) are handled as different countries. It makes sense for a CDN, but for most other purposes when I use Internet when I'm on Réunion island my country is "FR", not "RE", since RE is not actually a country. It's also quite inconsistent, I've tried with several IPs from Réunion and some of them are FR while the others are RE.
It doesn't seem to have the same problem with Hawaii for some reason, they're all just US from what I can see.
Since most CDNs are already doing GeoIP lookups (for request headers or log entries), you can leverage that to provide the data back in the response body via origin, worker or even CDN edge config.
Programmatically populating the response body, as in the Cloudflare worker example from the post, is better than going to the origin just to echo some headers back in the response. To me, something like Fastly's VCL config language is even simpler. It directly executes on every CDN edge node worldwide upon request.
For example, I just whipped this up on Fastly using VCL. It returns GeoIP as json data for your IP at the root path:
You could do the same via query params, headers, etc. Have URL endpoints that only return some of the data, and so forth.
The VCL syntax gets a little gross when you handle quoting strings and assembling json and testing if the string is empty, but it gets the job done.
Of course what you might want from GeoIP data may not be what you get. It's really kind of a useful kludge that gets treated sometimes as a panacea.
This dataset right now thinks that I'm about 5 miles east of my location, but when subnets are repurposed it could be much more significant. And the data sources are always changing, so who knows what it will think tomorrow.
I haven’t worked in the space in a while but I’d doubt it’s via anything like HTTP headers. From a total guess I’d look at packet inter frame gaps & jitter to imply client csmacd or l2 behavior. Maaaaybe MTU and TTLs to infer intermediate routed networks or devices like the tunnel. And of course various TCP options and behavior, like say timestamps and dsack, to fingerprint the client or intervening ip proxies.
it used to be, for years, thats the older stuffin the "geoip.<key>" namespace.
the stuff in the "client.geo.<key>" space is from a newer/better/higher-tier service (they say, i forget the name). also I think some of it is mixed in with other sources and some of the info is self-sourced.
Can you make a geolocation service from static files? 2^32 IP4 addresses with 2 floats per address would take only 34 gigabytes of storage. Put 256 addresses in a given file, and turn the other three octets into folders. E.g. https://example.com/192/168/0.txt would contain the location for addresses 192.168.0.0-192.168.0.255.
Would this be cheaper than running these services?
You could also generate all those files, put that into a GCS Bucket and put an API Gateway in front.
easier would be and thats just a thought: You could build a go/c++/java app which contains that database pre compressed in some arbitrary better format than just 2 floats, get it down from your original 34gig and just keep it in memory.
Like if you know that 99% of all ips in a certain range are from US, then only store the 1% as a list.
GCS for example and its the same with S3, costs space, egress and requests themself.
Data Storage 50 GB Standard Storage * $0.026 per GB $1.30
Network 10 GB egress * $0.12 per GB $1.20
Operations 10,000 Class A operations * $0.05 per 10,000 operations $0.05
Operations 50,000 Class B operations * $0.004 per 10,000 operations $0.02
186 comments
[ 3.1 ms ] story [ 89.3 ms ] threadIt's kinda like a construction firm using a supercar to haul materials and then bragging about their efforts to make cosmetic repairs cost-effective... maybe you could just not use an overpriced car to begin with and then damage won't be an issue?
So "cloud" is like renting a dynamically changing number of VPSs each month and which are usually more costly if your consumption is not spiky (eg. when a cheap VPS rented 24/7 is enough).
With PaaS it's someone else's job who does this every day.
https://www.united-coders.com/christian-harms/detailed-perfo...
https://github.com/Risk3sixty-Labs/geoapi
I realise this doesn't directly answer your question but I guess one reason might be that they already provide a free version so releasing the paid version would just get you in strife?
Furthermore, Maxmind adds non-factual entries to the database so they can identify it uniquely as their work. Map makers do something similar: https://en.wikipedia.org/wiki/Trap_street
Nope, US law was changed over a decade ago.
https://github.com/jloh/geojs https://www.geojs.io/open/
It's very borderline from them (at least ethically)
https://dev.maxmind.com/geoip/geoip2/geolite2/
https://blip.runway7.net https://github.com/runway7/blip
I lost track of the consumer apps I've used this on and still haven't received a $1+ bill.
Moment.js is probably the most prevalent time and date library.
I’d like to set a system clock timezone via geoip only, without location lookup via wifi. On my more secure systems I have location services disabled and it’d be nice to have an accurate local time automatically.
I need to be able to send a lat/long/timestamp to get tinezone.
I’ve seen timezone maps drawn, so this information does exist somewhere, but one would have to draw polygons for each timezone and do coverage checks, which seems complicated. The time zone lines zig and zag a lot.
The response time averages 10ms and is free, granted I'm not sure if there is a ToS or anything attached to this endpoint, I've only found non-conclusive discussion here https://community.cloudflare.com/t/what-are-the-terms-of-use...
I'd advise you to just create a Cloudflare Worker if you want do this. You'll get more reliability etc. and Workers have a very generous free tier.
https://arstechnica.com/tech-policy/2016/08/kansas-couple-su...
https://www.theguardian.com/technology/2016/aug/09/maxmind-m...
https://mashable.com/2016/08/11/ip-addresses-kansas/
ISP perspective here: Geolocation by granular /24 to /20 sized block of ipv4 space is often wildly inaccurate on a regional basis. It's entirely possible for the ARIN registration (used by maxmind) to be a street address in Seattle, but serving end user ISP customers a 4.5 hour drive away in a far eastern corner of WA state.
https://developers.cloudflare.com/workers/templates/pages/co...
It's also free and fast but with services like these I always wonder how long they manage to stay free.
I had good success with maxmind's free database. I can't find tooth results for accuracy but my sampling was pretty good. They also have an incentive to keep it.
I've been meaning to make automatic Github releases for it.
AFAIK, an unresolved problem is a proper geolocation service - at least for city level resolution - for mobile IP addresses. There are some services in this field (digital element), but they are very unreliable.
It doesn't seem to have the same problem with Hawaii for some reason, they're all just US from what I can see.
6 million requests per month is only 2.2 requests per second, a raspberry PI could do this (technically).
Programmatically populating the response body, as in the Cloudflare worker example from the post, is better than going to the origin just to echo some headers back in the response. To me, something like Fastly's VCL config language is even simpler. It directly executes on every CDN edge node worldwide upon request.
For example, I just whipped this up on Fastly using VCL. It returns GeoIP as json data for your IP at the root path:
http://geo.zombe.es
Or if you want a particular IP, just append it to the path:
http://geo.zombe.es/2a04:4e42:600::313
You could do the same via query params, headers, etc. Have URL endpoints that only return some of the data, and so forth.
The VCL syntax gets a little gross when you handle quoting strings and assembling json and testing if the string is empty, but it gets the job done.
Of course what you might want from GeoIP data may not be what you get. It's really kind of a useful kludge that gets treated sometimes as a panacea.
This dataset right now thinks that I'm about 5 miles east of my location, but when subnets are repurposed it could be much more significant. And the data sources are always changing, so who knows what it will think tomorrow.
https://gist.github.com/simonkuhn/a380a6fa205db87a3625f26ad0...
vcl_recv and vcl_error are the important bits, the rest is VCL boilerplate from https://developer.fastly.com/learning/vcl/using/ for unused subroutines.
conn_speed: https://developer.fastly.com/reference/vcl/variables/geoloca...
conn_type: https://developer.fastly.com/reference/vcl/variables/geoloca...
proxy_desc: https://developer.fastly.com/reference/vcl/variables/geoloca...
proxy_type: https://developer.fastly.com/reference/vcl/variables/geoloca...
conn_type is interesting to me, I'm not sure how you would distinguish wifi vs. wired based on HTTP header data.
the stuff in the "client.geo.<key>" space is from a newer/better/higher-tier service (they say, i forget the name). also I think some of it is mixed in with other sources and some of the info is self-sourced.
Would this be cheaper than running these services?
easier would be and thats just a thought: You could build a go/c++/java app which contains that database pre compressed in some arbitrary better format than just 2 floats, get it down from your original 34gig and just keep it in memory.
Like if you know that 99% of all ips in a certain range are from US, then only store the 1% as a list.
GCS for example and its the same with S3, costs space, egress and requests themself.
Data Storage 50 GB Standard Storage * $0.026 per GB $1.30 Network 10 GB egress * $0.12 per GB $1.20 Operations 10,000 Class A operations * $0.05 per 10,000 operations $0.05 Operations 50,000 Class B operations * $0.004 per 10,000 operations $0.02