This is really cool, thanks for the write up. I did the same thing in my homelab but much simpler, my shell script only fetched from a few sources and converted it into unbound format so it was much shorter. And I didn't care about statistics, but the idea was the same.
> Grab and parse the Top 10 milion domains (based on Open PageRank data) so that they can be used by dnsperf(1).
Be vary of using such sources.
Go to the source [0], make "Display: 100" and scroll to "1,351 to 1,450 of 10,000,000 websites".
You will find a bunch of bullshit domains clearly used for something nefarious (and some no longer exists), but to believe what those sites are as popular as spreadsheets.google.com? NO WAI
Tiny sample for the lazy:
1,361 ecoquipo.ga 5.92/10 ga
1,362 h4uvpev4tx.ga 5.92/10 ga
1,363 lfsni36qvn.ga 5.92/10 ga
1,364 dashseo.ga 5.92/10 ga
1,365 velocityconf.com 5.92/10 com
1,366 filezilla-project.org 5.92/10 org
1,367 bx2vnequhc.ga 5.92/10 ga
1,368 incblizzard.ga 5.92/10 ga
1,369 anisima.ru 5.92/10 ru
1,370 ou14ib22tf.ga 5.92/10 ga
1,371 i12tfukfft.ga 5.92/10 ga
1,372 blankmedia.ga 5.92/10 ga
1,373 nicelia.ga 5.92/10 ga
1,374 finlandtoday.fi 5.92/10 fi
1,375 spreadsheets.google.com 5.92/10 com
So I've always been curious, does anybody have a writeup of how DNS servers like unbound etc are able to handle evaluating these massive blocklists? Like the algorithms and data structures they use to evaluate queries against these lists efficiently. My pfsense+pfblocker unbound instance has something like 400K hosts blocked and I've always been amazed that it doesn't even break a sweat scanning that many hosts with every query.
> My pfsense+pfblocker unbound instance has something like 400K hosts blocked
But it is not 400k hosts to search:
com
facebook
www
ad
google
ad
ads
notadswepromise
googletagmanager
tags
ads
to block ad.facebook.com you only need to scan 10-100k of .com domains in your block list and only a couple of records in facebook.com. Don't forget, DNS is a hierarchical system. And we, humans, actually record, use and remember all the DNS records in the wrong, reverse direction ;-)
> 400K hosts
I bet 30% of those hosts are don't resolve anymore and another 30% resolve to domain parking/404 hosts.
I bet 30% of those hosts are don't resolve anymore
I've seen some evidence of this. I periodically dig all the blocked domains and noticed that some of them come and go so I assume people cycle them in and out to evade blocking. A handful of the domains just change the A record so I just block the apex domain which makes it a little harder to keep the block lists optimized but does reduce the size a little.
I should add that blocking the apex of some domains includes risk of dropping legit traffic but if I see more than {n} badware domains on the apex, I write it off as compromised or at very least poorly implemented and poorly secured.
I suspect parent meant the way the list is processed and how the structure resides in memory.
We'd optimally want, e.g., aaa.ai to resolve as fast as zzz.zx. If we just loaded the file for every query and ran down the list linearly we'd probably see a large disparity with the former being much quicker than the latter.
So, the question would be: how is it done in an efficient way? Is it loaded into a btree, is each domain broken into various parts, are certain domains prioritized, threading, compression?
Plus, there's the queries for dns updates, and how the results get integrated.
As a fun experiment one could set this up on a handful of VM's and leave the Unbound TLS port open or DoH if compiled with libnghttp2 to the world in addition to the graphs. I think it would be interesting to see how many clients each VM could handle. Maybe enable the rate limits in Unbound first. I'm half tempted to do this with a few of mine but I do not have the fancy graphs, just shell scripts to parse the logs which I guess could be written out as text files for nginx to serve.
Unfortunately for me the Unbound package on Alpine is not compiled for DoH, only DoT and I am too lazy to recompile things any more.
In spite of my entire satisfaction with it, I had to ditch my OPNsense router... Its unbound ad filtering DNS is among the features I miss most from it.
Bus factor. I had a very nice home setup (three 24 ports switches and a 6-port router) with multiple VLAN, GPON terminated on my own device (emulating the ISP's hardware), bonded Ethernet links, fully routed IPv6, sane firewalling, ad-filtered DNS, reverse proxy etc. - but I was the only person in my home (six kids + adults) to know how to maintain it. Anything happens while I'm away and Internet access stops. I borked an update and was offline for a couple of days - that was once too many for the users.
So we are back to the ISP's router, no VLAN and soon some Wireguard overlay for my hosting.
I'm sad about this: replacing infrastructure I control with the ISP invading my home, replacing tailor-made with generic consumer products, replacing native Internet with yet another overlay network... But MTTR rules my world.
As somebody who spends some time on this, identifying common infrastructure and targeting that is the best bang for the buck: all kinds of stuff lives in the sewer that you never thought of. OTOH it does require some whitelisting, because all kinds of stuff lives in the sewer.
This is what I do. I was trying to figure out what pi-hole did one easy thing was it black-holed domains, so I got the same list put it on my dns relay box and there you go same thing on openbsd.
30 comments
[ 2.5 ms ] story [ 44.1 ms ] threadAlso, why use ftp instead of curl?[1]
1. https://curl.se/docs/manual.html#time-conditions
Be vary of using such sources.
Go to the source [0], make "Display: 100" and scroll to "1,351 to 1,450 of 10,000,000 websites".
You will find a bunch of bullshit domains clearly used for something nefarious (and some no longer exists), but to believe what those sites are as popular as spreadsheets.google.com? NO WAI
Tiny sample for the lazy:
[0] https://www.domcop.com/top-10-million-domainshttps://dotat.at/prog/qp/README.html
But it is not 400k hosts to search:
to block ad.facebook.com you only need to scan 10-100k of .com domains in your block list and only a couple of records in facebook.com. Don't forget, DNS is a hierarchical system. And we, humans, actually record, use and remember all the DNS records in the wrong, reverse direction ;-)> 400K hosts
I bet 30% of those hosts are don't resolve anymore and another 30% resolve to domain parking/404 hosts.
Eg: see my comment about "TOP 10M SITES"
https://news.ycombinator.com/item?id=33150554
I've seen some evidence of this. I periodically dig all the blocked domains and noticed that some of them come and go so I assume people cycle them in and out to evade blocking. A handful of the domains just change the A record so I just block the apex domain which makes it a little harder to keep the block lists optimized but does reduce the size a little.
I should add that blocking the apex of some domains includes risk of dropping legit traffic but if I see more than {n} badware domains on the apex, I write it off as compromised or at very least poorly implemented and poorly secured.
We'd optimally want, e.g., aaa.ai to resolve as fast as zzz.zx. If we just loaded the file for every query and ran down the list linearly we'd probably see a large disparity with the former being much quicker than the latter.
So, the question would be: how is it done in an efficient way? Is it loaded into a btree, is each domain broken into various parts, are certain domains prioritized, threading, compression?
Plus, there's the queries for dns updates, and how the results get integrated.
Not sure how out of date that is
Unfortunately for me the Unbound package on Alpine is not compiled for DoH, only DoT and I am too lazy to recompile things any more.
So we are back to the ISP's router, no VLAN and soon some Wireguard overlay for my hosting.
I'm sad about this: replacing infrastructure I control with the ISP invading my home, replacing tailor-made with generic consumer products, replacing native Internet with yet another overlay network... But MTTR rules my world.
https://nlnetlabs.nl/projects/unbound/about/
but yes it was made to avoid a bind monoculture. and comes in two parts nsd is the authoritive name server. and unbound is a recursive name server.
https://pgl.yoyo.org/adservers/
basicly all it is is
where yoyo.conf is a dns blacklist- https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound
Remove the header with &showintro=0 and for plain text, use &mimetype=plaintext:
- https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound&mi...