Shell scripts that download more shell scripts. Kind of terrifying, no? I feel like I would personally rather take my chances getting scammed out of a CS skin than running these installers as root at will.
To resolve in Windows 10[1]: Disable DNS Client by changing the value data for HKLM\SYSTEM\ControlSet001\Services\Dnscache\Start from 2 (automatic) to 4 (disabled) (recent versions of Windows 10 no longer allow disabling DNS Client via services.msc).
There's a concern that a malicious list could point domain names to a malicious IP address. I don't think its a big concern with https:// since the cert will be invalid, but it's still a concern.
I suspect it’s very likely that somewhere in the world is a domain-validation server, used by a trusted CA, which has this very anti-advertising hosts file installed onto it.
It could be a lesser-known CA, perhaps the national CA of a small country (under 2m people) that normally only issues less than tens-of-thousands of certs and only exists for regulatory reasons (e.g. the country requires all of its own gov services to use its internal CA, while all commercial/popular services use CA based in another country, usually LetsEncrypt)
I love awk as much as the next person and I've written some pretty large scripts with it, but the original grep/sort/uniq is a lot easier to understand in two seconds. I don't think it's "useless" at all.
Also your awk isn't exactly identical as it's not sorted, but in this case that probably doesn't matter.
One of the advantage of the array[item]++ / for (item in array) construct is that it avoids possibly expensive sorts (at the cost of generating potentially large arrays), and tends to be quite fast.
That's even if you end up sorting the output by frequency, as the summary listing is typically shorter than the overall input data:
{a[$1]++}
END {for (i in array) printf("%6i %s\n", a[i], i) | "sort -k1nr | cat -n";}
The accumulator/loop idiom is ... fairly readily recognisable to someone familiar with awk.
This is the default list used by the pihole project as well; it's a very good list.
FWIW, something like ublock origin is going to be better for in-browser blocking (for example, DNS poisoning won't work with youtube ads, which are served from youtube.com, but addins like UBO can remove them because they can hook into the actual traffic).
The ultimate adblocking solution is to use UBO in your browsers AND spin up a pihole to block ads on your entire network: https://pi-hole.net/
I'm not worried about the non-techies being tricked into thinking .com is an executable, I doubt they even know its possible. They almost certainly know about .zips
If I hover over the link like my phishing training says to do, Chrome correctly shows it as "https://v1271.zip"
If I don't bother hovering over the link then the domain never mattered at all. Could have hosted the file anywhere since the link text in HTML (including HTML email) need not have any relation to the destination. You can simply write a legitimate github.com link but the href goes somewhere else.
Any situation where you're relying on users to visually inspect every link and decide if the domain looks plausible is already a security failure.
If I'm downloading and executing files that are randomly in a Medium article or in my inbox from an untrusted source then we're in trouble whether they are hosted on github or not.
I’m confused what you’re advocating. If your family isn’t checking the domain before downloading and executing files then the .zip tld doesn’t matter. They’re just as vulnerable to phishing without it.
I'm advocating using a pi-hole to block problematic domains on your network using list like the one from StevenBlack and entire TLDs that could be confusing. If you don't think .zip & .mov could be confusing then feel free to not block them.
The explanation I've seen is that certain programs (mail, chat, forums, etc) automatically recognise urls that are not hyperlinked and convert them into a hyperlink.
As I understand, the attack is preformed by when someone mentions attaching a file to an email (i.e. "I've attached invoice.zip"), then that filename becomes a clickable link, and the user clicks it expecting to download the file, but is actually taken to the attacker's website which looks like WinRAR, and download malware from there.
To me this attack sounds too convoluted to actually get people, since you'll need to buy a lot of domains to match someone's email's contents
Which email client automatically creates links? Virtually everyone uses HTML email, which can link any text to any destination. If your users are clicking links to download files that they got via email from untrusted sources...then the domain doesn't really matter.
To be fair all extensions are potentially executable. Extensions are just a mapping to an application maintained by the OS. .com is just mapped to a program that executes by default. File type is a whole different ball of yarn to untangle.
Extra fun: measure bandwidth use before and after implementing, the savings are fucking nuts. All that adtech webshit loading in hundreds of mb of obfuscated JS and assets to sell you widgets…
Reminds me of SlashDot where there was a relatively well known submitter that was a huge proponent of host-based ad blocking on Windows and was also a proponent of getting very angry at almost anyone else.
Now it sort of works in practice as most of the advert networks do not host on the same domain.
It is good at catching out whole bad domains but partial ones it is not very good at. Which some domains will do using a reverse proxy.
Also at one point a hosts file was linear scan. No sort of hashing or binary search lookup. Not sure if that was ever fixed in windows or linux. So a small number of hosts it was ok. But as the file grows it starts to add up.
The only problem with this is that on interface up/down it can take a few minutes for the network to be available. This may not be an issue with a server or idle desktop but may be confusing for non tech people or laptop users.
If you can manage it, a pihole/pivpn instance may be better. Both of those can be installed on any debian distro easily.
Sure, you could manage /etc/hosts manually, but Little Snitch Mini has a nice interface for managing blocklists, auto-updates them, and has nice visualization, and you can also manage blocks per app.
Does anyone have any updates on Mike Burgess (from winhelp2002.mvps.org)? He used to keep my go-to hosts file, but his health seems to have declined in the last few years. I would like to know if he is still around and if there is currently any method to donate or support him. Any news would be greatly appreciated.
It's almost the 40th anniversary of RFC 882. Back when the Domain Name System was being invented as the replacement for everyone sharing hosts files, I wonder whether anyone expected that 40 years later there'd be a shared hosts file with its own copyright licence, Docker file, and code of conduct.
Interestingly, from what I remember of the April Fools columns of electronics magazines of the time, one thing that people would not have been surprised by is that the purpose was zapping advertisements.
Yes. pihole coupled with a router that can intercept outbound traffic on port 53 and force it to the pihole is how my home network is configured. That way it catches all those sneaky devices that have hard coded dns servers.
I've also added (pfSense with pfBlocker) a list-rule to block all Public DNS servers (https://public-dns.info/). If you want to resolve a name, you gotta go through my DNS server.
The UDP:53 block gets almost everything, but I'm preparing for DoH.
I use Brave (not so much for the built-in ad block, but mostly because it's not actually Chrome) and I've noticed that more and more sites have ad-block detection that appears to work by sending a unique token to their ad server and then not loading content until after the token has been verified to have been received. I imagine there are ways around this on a per-site basis, it's generally easier to just give in and disable the ad blocker.
There's no host file syntax for NXDOMAIN, it has to be an IP. 127.0.0.1 with Port 80 assigned to a dummy page can be an alternative solution if you don't like 0.0.0.0.
I’m a fan of dbab. It’s DNS based as well but pretty light weight using dnsmasq in the backend. No fancy GUI and such like pi-hole, just simple domain blocking.
hosts file will grow with records + you need to run specific scripts to update it. One of the similar options will be to use DNS resolver with built in filters (like pi-hole, but "cloud"), for example: https://nextdns.io, even free version is OK.
intiially i wanted to point out a potential issue with using large /etc/hosts file in windows, but ended up finding a (potential) fix in this thread.
regardless, after using this list for a long time, i do recommend setting up your own dns server or use a service that does (such as pi-hole and nextdns already suggested by others) with this list enabled.
on mobile, i found that ad-blocking dns services that offer DNS-over-HTTPS (such as adguard dns) offer a much more elegant option that does not require an app nor configuring every network you connect to.
I must say: the lists we offer — 31 variants in all — result from fine work by all our curators. Some are extremely diligent and maintain their lists every day. It's just remarkable what dedicated people can do together over time.
78 comments
[ 3.4 ms ] story [ 120 ms ] threadDon't forget about hosts.deny (but you probably need it at your router if you are behind one).
[1] https://tinyapps.org/blog/201809300700_large_hosts_file.html
* https://porkmail.org/era/unix/award#grep
Also your awk isn't exactly identical as it's not sorted, but in this case that probably doesn't matter.
That's even if you end up sorting the output by frequency, as the summary listing is typically shorter than the overall input data:
The accumulator/loop idiom is ... fairly readily recognisable to someone familiar with awk.FWIW, something like ublock origin is going to be better for in-browser blocking (for example, DNS poisoning won't work with youtube ads, which are served from youtube.com, but addins like UBO can remove them because they can hook into the actual traffic).
The ultimate adblocking solution is to use UBO in your browsers AND spin up a pihole to block ads on your entire network: https://pi-hole.net/
edit: HN is actually erroring out if I post an example .zip domain of the problem
[1] https://medium.com/@bobbyrsec/the-dangers-of-googles-zip-tld...
If I don't bother hovering over the link then the domain never mattered at all. Could have hosted the file anywhere since the link text in HTML (including HTML email) need not have any relation to the destination. You can simply write a legitimate github.com link but the href goes somewhere else.
Any situation where you're relying on users to visually inspect every link and decide if the domain looks plausible is already a security failure.
If I'm downloading and executing files that are randomly in a Medium article or in my inbox from an untrusted source then we're in trouble whether they are hosted on github or not.
How do you hover on mobile? Instead of training my family members to hover on their phone's I will continue to just block these TLDs.
I'm advocating using a pi-hole to block problematic domains on your network using list like the one from StevenBlack and entire TLDs that could be confusing. If you don't think .zip & .mov could be confusing then feel free to not block them.
E.g. "example.com" becomes "<a href="http://example.com/">example.com</a>".
Such features would now (potentially) also work on mentions of zip files, i.e. on messages that state "download install.zip from github".
Edit: ghi, HN actually does this. Not yet with zip TLDs though.
To me this attack sounds too convoluted to actually get people, since you'll need to buy a lot of domains to match someone's email's contents
testlink.com link.gov abc.zip test.xyz
I agree about being able to hyperlink pretty much anything having the same effect, that's why I don't think the .zip TLD is a big issue
When my Pi stopped working and I couldn’t find a new one, I gave up and moved to NextDNS [1] and have been very happy.
[0] https://www.pivpn.io/ [1] https://nextdns.io/
(Edit: typo)
Slashdot has a, shall we say _engaged_, community.
They could not wrap their head around the idea that the hosts method is limited in some use cases.
This could not be blocked without blocking the main host too. https://example.com/adverts-folder/advert.jpg
Now it sort of works in practice as most of the advert networks do not host on the same domain.
It is good at catching out whole bad domains but partial ones it is not very good at. Which some domains will do using a reverse proxy.
Also at one point a hosts file was linear scan. No sort of hashing or binary search lookup. Not sure if that was ever fixed in windows or linux. So a small number of hosts it was ok. But as the file grows it starts to add up.
What a strange place that was.
If you can manage it, a pihole/pivpn instance may be better. Both of those can be installed on any debian distro easily.
https://www.obdev.at/products/littlesnitch-mini/index.html
Sure, you could manage /etc/hosts manually, but Little Snitch Mini has a nice interface for managing blocklists, auto-updates them, and has nice visualization, and you can also manage blocks per app.
Forgot about it for a few years, but this post jogged my memory.
We currently use a PiHole for house-wide, network-wide ad and telemetry blocking, though, but perhaps that hosts list is useful to someone else.
Interestingly, from what I remember of the April Fools columns of electronics magazines of the time, one thing that people would not have been surprised by is that the purpose was zapping advertisements.
* https://worldradiohistory.com/UK/Electronics-Today-UK/80s/El...
The UDP:53 block gets almost everything, but I'm preparing for DoH.
https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
99% of people don't understand (because they don't even try) what you can run a full resolver even on Windows machines. butmuhpihole
regardless, after using this list for a long time, i do recommend setting up your own dns server or use a service that does (such as pi-hole and nextdns already suggested by others) with this list enabled.
on mobile, i found that ad-blocking dns services that offer DNS-over-HTTPS (such as adguard dns) offer a much more elegant option that does not require an app nor configuring every network you connect to.
I'm happy to answer any questions arising.
I must say: the lists we offer — 31 variants in all — result from fine work by all our curators. Some are extremely diligent and maintain their lists every day. It's just remarkable what dedicated people can do together over time.
And https://mayakron.altervista.org/support/acrylic/Home.htm works as well, and it also accepts wildcards.