Ask HN: Who here would be interested in a free daily updated GeoIP database?
I recently was playing around with Maxmind's free GeoIP database but was disappointed at how inaccurate the free version was.
I started hacking away yesterday at making an alternative database that seems to be more accurate, but it's still in the works. Who here would be interested in a free alternative, including and excluding cities, updated on a daily basis, and if so, which format's would you like to see?
127 comments
[ 4.5 ms ] story [ 193 ms ] threadBut in specific answer to this question, if it's really public and free, why not see about adding it to Amazon's public datasets [1]? S3 is fast, durable, available, and I've enjoyed these other data sets.
[1] http://aws.amazon.com/datasets/
The biggest one is just routing information and rDNS hosts which give away a lot of the information required anyway. If you were really thrifty you would make agreements with services (or use services you run yourself) that have users supply their location information and use that to construct a more granular database. I suspect this already happens to a degree, some of the fake information I give out has ended up in at least one IP address database.
I strongly considered doing this at one point but lacked the connections to pull it off properly (not enough sources of data I could get without approaching random people, basically). A secondary concept would just be to crowd source it and have the information freely available, but I was never sure how to deal with people poisoning the database or just honestly supplying conflicting information.
Seems like anyone who can look at BGP tables and whois information can get more current and accurate data than MaxMind.
I would be interested. Although a service like ip-api.com is not expensive ($8 per month or so if commercial use).
The structure of the data seems very simple: ip address range plus a location, I guess. So maybe just dump as a simple csv text file ? Always easy to process.
You have to make sure the legal side is covered: how do you get your changes ?
A free service like locationkit (seen here a few days ago IIRC) looks OK, and I'm reasonably sure it's supported by revenues for other services from the parent company. However, even that feels like it might get shut down if it becomes too big a burden.
Something with a good understandable license that I can pay a reasonable amount ($5-$20/year, for most low-mid volume use cases) would be great, as long as I can really use it for commercial purposes. I had some high hopes for Yahoo services a few years back, but they seemed determined to plague everything with confusing or commercial-hostile licensing.
I would personally not want to pay up front without knowing anything about this database from a company (or private person) that only asks 5 bucks a year for it.
That seems to be way too low.
$20/month+ could make sense, but less than that should be simply free (limited edition).
Dealing with payments is just too much overhead.
With maxmind, $10/year might be all that I use (considering 1000 calls might only be $1 charge).
Should work too.
To echo others, aim for low-priced and not free. That'll give you some operating revenue to avoid it going the way of the dodo.
I myself, could also be interested, but I think, I will leave that to my payment-provider ... (ugly, EU!)
Edit: Ok, I should not, but why the downvote, please? When I wrote something wrong, than please explain!
But, you (because you are so sure about the best method) by sure know, that the laws are explicitly for non-physical goods, where no delivery-address must be provided, right?
Here is a link, how one company is doing it (forget about government information, they seldom go so much in details!): https://rachelandrew.co.uk/archives/2014/12/08/consensus-and...
They use the IP address to validate the user given address and for later proof.
BTW: Downvoting all the time does not help the discussion. I try to discuss this topic seriously and I was bringing some very relevant topic here, and am constantly downvoted. (It is not my fault, that the EU came up with such laws, and provide no useful guide how to implement it!)
That "all" are doing it, does not guarantee, that one time a judge or tax controller will come and argue with you that you have no proofs at all. In this case, you are really bust, because the new law also says, that you are accountable for the tax to all the countries tax authorities. Much fun, when you get citations from different EU countries.
(Don't say it is not realistic: There are cases already public, where British shops got claims from remote EU countries, because of tax differences of <10 Euro).
And as long the EU does provide laws that are hard to implement in real life (only in the minds of the politicians they are simple) and nobody says how to implement them, it is better to collect all proof you can get.
https://www.gov.uk/government/publications/vat-supplying-dig...
There is clearly stated, that IP information is one of the proofs accepted for the country determination.
So, maybe next time, please first ask before judge.
As far as formats go, if anyone is thinking of putting together a custom database with information keyed on IP address or IP ranges, I'd encourage you to look into our MaxMind DB format:
http://maxmind.github.io/MaxMind-DB/
There are open source file readers available in many languages: https://github.com/maxmind/?utf8=%E2%9C%93&query=MaxMind-DB
There's also an open source writer https://metacpan.org/pod/MaxMind::DB::Writer
Our officially supported libraries are actively maintained and pull requests are always welcome.
I'm actually about to begin writing a blog post on how you can create your own custom databases. I'll share it on HN once it's ready.
https://www.maxmind.com/en/geoip2-city-database-accuracy
To get the actual product comparisons, you'll need to enter a country in the input box. For instance:
https://www.maxmind.com/en/geoip2-city-database-accuracy?cou...
You can try refining the distances to see how the different products compare. If you fiddle with it a bit, you'll see that the results do vary from country to country. So, "good enough" will depend on some combination of your tolerance for error + which countries you care about.
A couple years ago I worked on a project that involved processing the "Internet Census of 2012" data.
The MaxMind DB format was a fast, efficient way to store simple data keyed on IPs and subnets.
http://dev.maxmind.com/geoip/geoip2/downloadable/
https://cloud.google.com/appengine/docs/python/#Request_Head...
and it would likely be free if the number of requests was below a certain threshold per month.
Please don't try to use the dat file format. The binary files ending in ".dat" are the Legacy format, which has never been documented. Use the MMDB format linked by oalders in the top comment.
But I understand that's not really an argument: you might say you don't want nukes to exist because they hurt people. I just wanted to mention it.
That said, maybe. Usually an rdns lookup does the trick already though (a4-8e-ef-78.dynamic.ziggo.nl or cpe-12-34-56-78.rr.com or 80-100-131-150.adsl.xs4all.nl is clear enough).
https://blog.mozilla.org/services/2015/03/06/combain-deal-he...
(I work at Mozilla and I used to work on the Location project. :)
Think it would be an interesting way to deliver services and data, just not sure how it turns into a business (since the API/service portion is designed to be zero profit).
- Free version: only updated once [every 6 months / a year] and non-commercial use only
- Paid version: $9.99 for a year of free updates
- Updates come both in diff and in full version (for the commercial version)
- Use a common format, or use multiple formats. People love csv, but more serious applications probably use an existing format (I see maxmind a lot in this thread).
- Perhaps an API with heavy ratelimiting for people who need only a very occasional lookup (for very occasional use, I use ip2location.com/demo currently; for more serious stuff I download a database).
Having a free version with a nice license might also get you included in a lot of open source software, which increases name visibility and makes people write parsers for your format of choice.
A minor note on payment: accept Paypal and/or Bitcoin, or I (and many other Europeans) can never pay you. These credit card-only services are impossible to order from.
Idea for the diff: if you have a binary format, perhaps use this: https://www.chromium.org/developers/design-documents/softwar...
Re: CC's, I'm also based in Europe (The Netherlands) and also hate services that only allow credit cards.
Thanks for the link regarding diffs, I'll definitely give it a read.
Getting the person who has the card to enter the necessary card details to a web site is not always convenient. On the other hand passing the card number, cvc etc to many people in the company has certain risks associated with it. Using personal credit card is painful, especially with monthly fees, since you need to do all the paperwork to get money from company.
Usually the credit card invoices are not enough for the bookkeeping. You need to have more detailed record about what was purchased and about the VAT included in the price. With shared company credit cards somebody then ends up chasing the actual receipts, which happen to be in somebody's inbox.
In most cases it is just so much easier to handle things with invoicing. The invoices are handled electronically and usually companies have systems in place which allow the accounting department to just pass the incoming invoice to whoever for approval.
A quite good solution this would be a virtual credit card service where somebody could assign "virtual cards" to whoever on the company and put specific monthly/yearly limits on those. It would be pretty nice to add the collection of receipts to this one as well. So the person responsible for specific virtual card, would also upload the invoices that match to the payments made.
I refuse to have a paypal account, and recent changes to how they process payments mean I can't just pay with a cc without having an account.
If you are accepting payments, it might be a good idea to have more than one payment processor.
Especially if paypals notoriously badly timed account suspensions might force a existential crisis for your business.