Show HN: DNS-based alternative to Semantic Web – 23M data points now in DNS
We’ve just launched NUM [1] in the UK, it’s an alternative to the semantic web [2] but based in DNS. We’ve published 23 million data points about 4.8m UK businesses.
We had some really interesting HN feedback when we previously posted about one of our example apps [3] so keen to hear what the community has to say this time round.
1. https://www.num.uk
2. https://num.uk/blog/we-crawled-5m-uk-websites-and-published-23m-data-points-to-dns
3. https://news.ycombinator.com/item?id=27598164
19 comments
[ 3.1 ms ] story [ 57.1 ms ] threadI’m working on making the web semantic (on your search query) by using large language models which is more equipped for reality since businesses can keep doing their websites as they want
With NUM, we’re not asking people to store this data in DNS because it’s simpler (although it does separate it out from your website which has benefits in my opinion), the point of storing it in DNS is that we can pre-populate it at scale and offer basic tools to manage this data safely in DNS.
Your project sounds cool. Is it searchable through a centralised database or some other way? Google have done the centralised model and that’s what we’re trying to offer an alternative to.
The project is focused on repetitive tasks on the internet. It’s not intended to replace one off searches but rather to build datasets/tables/graphs on your query.
One off search: best pizza in Berlin
Our usecase: show me the best rated salami pizzas sorted by price in Berlin
Whereas you’d visit each website with a one off search and go through their menu (and TripAdvisor ratings) we’d serve you a table/graph/written summary…
EDIT:
it’s due to the workings of the agent that we do not need to index the internet to use it
On NUM: I like the idea of building open graphs, but to move the needle it gotta be easier for business owners to provide that information. Lots of orgs use let’s encrypt some of them update their dns records every 90 days with a dns challenge. If there was to be some way to use a process like this (with a companion tool) to get business info into NUM records it would help adoption. I find the idea of calling a domain, sending money to a domain compelling I have to say.
[1] https://get-sentinel.io
It works for 90% of UK businesses already, because we’ve pre-populated the DNS with their phone numbers, addresses, social media.
So they don’t need to adopt it, for it to work. They can if they want to override the pre-populated data but they can also do that with our simple tools.
See https://companydirectory.uk for an example app powered by NUM.
I was initially curious, then cautious, then a bit disappointed as I dug into how you're doing what you're doing.
What would make this data truly decentralized and "DNS-based" is if the data was published to the companies' respective domains as extra standardized records (and even thinking about it for 30 seconds I can see the problems with that). As it stands, everything is hosted on num.net.
It's literally just a JSON API with extra steps.
Now don't get me wrong; you've opened my mind to the fact that yes, hosting certain things over DNS has use cases. I think you built something neat, but I too am concerned that you're directly contributing to a tragedy of the commons effect. You're piggy-backing off decentralized infrastructure that gets to be free because its costs are globally minimal.
In other words, you're paying for your infrastructure by fishing for pennies in the Trevi fountain.
I find the disappointment fascinating and I guess it's maybe a network traffic purist view point?
> What would make this data truly decentralized and "DNS-based" is if the data was published to the companies' respective domains as extra standardized records (and even thinking about it for 30 seconds I can see the problems with that). As it stands, everything is hosted on num.net.
Surely you can see that there's zero chance of getting to this truly decentralised way of doing things without some kind of fallback (See Semantic Web vs Google). NUM enables organisations to do exactly what you want – publish this data to their own DNS and that overrides data in num.net
We package it up as a JSON API-like library because that's what developers are used to, but under the hood it's nothing like an API – requests aren't tracked, rate limited or restricted.
> In other words, you're paying for your infrastructure by fishing for pennies in the Trevi fountain.
This is a colourful description!
Conceptually the idea had legs, but I can't see wholesale adoption of it. Whilst all (most?) of the records are hosted at Num.net it doesn't meet it's promise of decentralised.
It feels like you need a few big-name companies to stand up and endorse you, and set their own NUM entries on their DNS.
We don't promote NUM as decentralised, but it has that potential with enough independent adoption.
We talk about NUM as an alternative to centralised APIs for two reasons: (a) it can be adopted independently; (b) there are no rate limits or restrictions, which we see as the big problems with APIs.
I agree, some big endorsements would be good but I don't think that'll drive adoption on the domain side. The point of num.net is that we don't need wholesale adoption on the domain side, we need adoption on the developer side (and by extension users) to drive adoption on the domain side – either by people claiming their records on num.net or publishing independent records.
Why didn't you go for RDAP, the new protocol that AfriNIC, APNIC, RIPE and even LACNIC are adopting?
[1] https://www.icann.org/rdap
As I’m sure you know, WHOIS and RDAP are about domain registrant data supplied by registrars and registries.
We didn’t do this in RDAP because: a) we’re not a registrar or registry; b) RDAP has strict limits designed specifically to prevent exactly what we’re trying to enable - unlimited and unrestricted access to data for developers; c) NUM is much broader than just contact data
I'm not sure what you're trying to argument with here. Why is RDAP limited if anybody can just use a client and any server on the internet that fulfills the same spec? [1]
The argument works both ways, because as of now, your solution is literally "n (servers) = 1", therefore it's the maximum restricted access possible. Arguing that an IETF- and ICANN-backed RFC is limited and restricted; whereas you are using a proprietary NUM protocol on top of DNS, an RFC by the IETF, is ridiculous.
Don't you realize that you're trying to push a proprietary protocol here?
RDAP is an IETF-RFC, so I'd argue that adoption is much more likely; as it is also backed by ICANN, RIPE and all internet registries.
You're just using DNS as a generic database here, and I'd argue that the DNS protocol wasn't designed for the purpose that the NUM protocol seems to have as its goals; given the overhead of schema validation that each client has to do for themselves. [2]
> NUM is much broader than just contact data
RDAP supports different schemas for entries, objects and models; just as any other database. Add a new rtype="whatever" and define its properties, and you're set to go.
Honestly in times of GraphQL making predictable queries as easy and as built-in as possible; the NUM protocol feels like a huge setback in terms of transport encryption, integrity, predictability of state complexity and confidentiality.
[1] https://datatracker.ietf.org/doc/html/rfc7482
[2] https://www.numprotocol.com/specification
> Why is RDAP limited if anybody can just use a client and any server on the internet that fulfils the same spec?
RDAP as a technology and RFC isn't limited or restricted of course, just like restful APIs as a technology are not limited. However, the implementation of the technology is limited. The servers offered by registrars and registries will be limited and the access and use of data restricted.
We promote NUM as unlimited and unrestricted because NUM records (whether in authoritative nameservers or under num.net) are hosted in DNS and requests can not easily be tracked or rate limited.
> I'd argue that the DNS protocol wasn't designed for the purpose that the NUM protocol seems to have as its goals
I disagree, we're using vanilla DNS TXT records which have existed in DNS since '94 [1]: "This paper proposes the use of the DNS TXT resource record [...] to contain new types of information."
Most NUM Records can be transported over UDP, even under the original 512b limit. A NUM record containing let's say 20 typical key value pairs is smaller than a DNSSECd A record and smaller than millions of DNS TXT records at a domain apex (dig target.com TXT).
> given the overhead of schema validation that each client has to do for themselves
Validation is minimal and is dealt with at a library level, as it will be with RDAP or any other API. The fact that mainstream data serialisation formats (e.g. JSON, YAML, etc) are not designed for DNS, does not mean that DNS is not designed to store serialised data. We created MODL [2] to solve this problem.
To demonstrate low overhead, we have a heavy example implementation that fetches 62 NUM records, validates them and displays them in 1.6 seconds [2].
> Honestly in times of GraphQL making predictable queries as easy and as built-in as possible...
This data is already centralised in graphs by Facebook, Google and more. There was an effort to decentralise it with the Semantic Web. We're offering an alternative to both, where users can retain their privacy, developers can actually build scalable, profitable products (as opposed to using Google / Facebook APIs) and the data is actually available (as opposed to SemWeb).
> ... the NUM protocol feels like a huge setback in terms of transport encryption, integrity, predictability of state complexity and confidentiality.
These seem like complaints about DNS which are addressed by DNSSEC, DNS over TLS (DOT), DNS over HTTPS (DOH), DPRIVE and Oblivious DNS-over-HTTPS (ODOH).
Thanks again for your thoughts, happy to dive deeper into any of this.
1. https://datatracker.ietf.org/doc/html/rfc1464
2. https://companydirectory.uk/barclays.co.uk/contact-informati...
3. https://www.modl.uk
You may have been suggesting that we serve out our data from num.net using RDAP instead of DNS. In answer to that: NUM is a DNS-based protocol because every online organisation has access to DNS - it’s a standardised way for them to store and serve out data, they’ve already got it and already pay for it.