141 comments

[ 2.9 ms ] story [ 205 ms ] thread
Is this reasonably small enough not to cause performance issues? I notice it mentioned trying to keep the size more reasonable.

Ad-blocking via hosts files can often lead to a noticeable performance hit.

I remember reading a bit on how the kernel stores the hosts list, but I stopped before finding how it scales. All I remember is that it seemed like a linear list. If it was a trie it would be alright.
Compared to the overhead of a network request, the overhead of going to though even 1000 items doesn't seem that much to me.
My Linux system runs a 60k+ /etc/hosts files, plus dnsmasq for local resolution. Nice thing about dnsmasq is that it treats domain-level entries as domain-level assignments.

No measurable lag.

I'd used the various blockfiles used by uBlock Origin, as well as some additional entries of my own, de-duplicated. There are some overaggressive entries, I've commented those.

A nice plus: I found the dozen or so hosts/domains associated with autoplay video crap, added them, and have no more bother from that.

You can pay for a basic license of OpenDNS and "null route" (to a block page) ads at the device / network level. Those preferences get synced to anycasted DNS resolvers and performance hits are negligible.

Some ideas here, but it's pretty straightforward and functions just like their adult content blockers: http://iradar.blogspot.com/2011/07/useful-free-tool-use-open...

I've been using this method for months if not years. I haven't noticed anything unusual about browsing speed, except that it's increased due to less ads.
I highly doubt it. I use AdAway for Android, and my current (/system)/etc/hosts file is 58626 lines with no noticeable lag on my Nexus 7 2012 and my Nexus 9.

I would assume local IO would usually beat network IO.

you're getting downvoted, but I remember this issue cropping up for windows XP boxes. Other Operating systems and more modern versions of windows cope with large hosts lists more competently.
The problem isn't completely solved, though.

https://www.reddit.com/r/Windows10/comments/401h2o/hosts_fil...

In windows 10, the DNS Client does something that is O(n^2)

Isn't the O(n^2) operation described in that thread happening only when the hosts file gets updated, and not with every DNS lookup? I would imagine even sequentially searching a local list of 350,000 entries, like the post describes, would still be extremely fast in the context of loading a web page.

But, that said, there can be a huge difference between how one might think things should work and what actually happens.

Yeah maybe it's mostly specific to Windows. Wouldn't surprise me if that's the case. I was speaking from a Windows perspective (I should have specified).
(comment deleted)
See, the thing is what if a website is broken due to host files, you can't easily re-enable ads for just this one website you need.

A situation we can all imagine ourselves in: You need to check the google analytics for your website/company site. You can't because it's blocked at Host level.

What solution would there be for this use case?

$man mv
You're suggesting renaming the hosts file to something else temporarily? I guess that would work. How quick does the operating system pick up on this change, do I need to reboot my machine for it to reload the Hosts file?
No reboot needed. I have updated my hosts file regularly in linux, windows and macosx without needing a reboot.

Generally your browser will pick it up quickly as well. It doesn't cache host file entries in the same way as dns lookups so the effect of adding or removing is pretty much instant. I use Chrome mostly, so I am not as sure about other browsers.

No need to reboot, as soon as you change the hosts file the OS should immediately start to use the new version, the hosts file is the first thing the OS checks when resolving names, this is what enables this approach to work.
It takes immediate effect. I often have a terminal open and toggle commenting out an entry when testing something. As soon as you save it, it's good to go.
Keep in mind that Firefox has its own DNS cache and that has confused me more than once.
Soe does Chrome. You can clear the internal cache by visiting chrome://net-internals/#dns
A situation I've found myself in before: asking our devs why the site is broken, only to find it's my overzealous script blocking.
I bet your devs love you for this
That's not as bad as complaining about the speed of your site, only to realize you're running a torrent on your laptop.
https://github.com/jakeogh/dnsgate has a whitelist command for this situation. It's really necessary if you use the block-at-psl[1] config option.

[1] strips domains to the top level that the public can register using https://publicsuffix.org/

It also can quickly "dnsgate disable/enable". (dnsmasq only, quick enable/disable for /etc/hosts is not supported yet, patches appreciated)

uMatrix chrome plugin is working well for me
I've been using the hosts file method for years and have never had an issue with checking Google analytics. I use the "Someone who cares" link.

Aside Google sponsored links and the odd ad sponsored link on pseudo-news sites not working (due to them being tracking URLs), I can't see it ever gets in my way.

However to answer your question, these days I run dnsmasq on my home server and have my DHCP server assign that as my primary DNS. So every device (phone, laptop, smart TV, etc) gets their ads blocked as well - which is particularly good for my TV as it's bad enough having regular adverts on TV without LG pushing out sponsored content as well. So if there was a rare occasion that I needed to turn off my ad blocking, I'd just change the DNS to 8.8.8.8 (Google DNS) then switch back to my dnsmasq server once i was done (the only complication being that I sometimes need to close and reopen the browser due to that particular application caching the DNS lookup)

The nice thing about using dnsmasq is that you can import those hosts files verbatim. Which means your update script can be simple.

I used to block various domains that served TV ads for UK Channel 4, but they hosted the client and that not detect ad loading. Seems you could probably truncate the video stream, maybe o using iptables, but their ad-load has reduced so I'm not motivated to try right now.
The weirdest side effect I've had was with the Sky HD box. If it was connected via ethernet then it wouldn't power up while my ad blocking was enabled. I was able to replicate this behavior by enabling and disabling my ad blocking, so my Sky box was definitely phoning home and failing to start if a specific domain was disabled. The weird thing is it would start up fine on WiFi or if the internet was disconnected completely. So I ended up just connecting it to WiFi as my wife was growing impatient by that point!

I did intend to reinvestigate the issue; throw wireshark on a promiscuous NIC and look for what domain Sky was trying to connect to and what data it was sending. I was thinking it might make an interesting article - depending on what I find. But in all honesty I had then forgotten about it until now.

If you're on a Mac I'd suggest Gasmask. It's a lightweight freeware that lets you switch between multiple hosts files right from the menubar. I have a productivity hosts file I switch on everytime I want to block social media, another one for development purposes and a standard one to revert too in case I don't want any hosts overhead and a clean default file. It's extremely useful.

https://github.com/2ndalpha/gasmask

Some of these map the domains to 127.0.0.1 which is wrong. It should be 0.0.0.0.

On second thought, you shouldn't be using the hosts file for this at all.

Can you explain the reasoning behind using 0.0.0.0 instead of 127.0.0.1? Genuinely interested.
Basically it's slower if you use 127 because it actually checks if something is running (example a server or whatever). 0.0.0.0 means explicitly "there is zilch, nip, nada". It's a little bit faster.
NXDOMAIN is also an option if you use dnsmasq. I havent tested if 0.0.0.0 actually returns NXDOMAIN, but from you comment, it sounds like yes. Also see --dest-ip in "dnsgate config --help".
telnet 0.0.0.0 $port works fine on my machine, it connects to localhost, so besides the speed, I don't think there's any other difference at least on Linux.
You can have processes listen on 0 - which means all available IPv4 IPs (available to that machine).

And since you can have processes listen on 0, it means you can equally curl 0; just like you could with 127.0.0.1. Here's an example from my IRC server (the only process I run on 0):

    $ curl https://0:9997 -kis | head
    HTTP/1.1 200 OK
    Date: Fri, 12 Feb 2016 07:49:24 GMT
    Server: ZNC - http://znc.in
    Content-Length: 1878
    Content-Type: text/html
    Set-Cookie: 9997-SessionId=54245f15ba592bc691e09ac75e6778e6d4c33841fad71a8d6c56addc998e043f; path=/;
    Connection: Close

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
(just in case someone queries, 0 is just another notation of 0.0.0.0)
I do some blocking with my hosts file, and I used to put 127.0.0.1. That hits my default nginx vhost, so i usually get 404 or 403 pages which is not so bad since my default vhost is an empty docroot. But it is not very intuitive and sometimes I think a server is down because of the nginx error page before I realize its my hosts file. I considered making a custom error message for my default host to remind me what I'm hitting, but I'm lazy.
Security.

I suppose it would be possible to craft a url that attacks local web services sometimes found on developer machines. If someone can confirm this is indeed the case, I'll submit a pull request to their README.

Which ones? Looking at [1], the only entries I can find for 127.0.0.1 are for localhost or are commented out

[1] https://github.com/StevenBlack/hosts/blob/master/hosts

Either way I'm not personally comfortable with this method since I often run servers on my personal computers for development. So whether it's 127.0.0.1 or 0.0.0.0 the request will still reach my system and possibly be handled by any port depending on who makes the request.

I would much rather have a browser plugin for this.

Those servers could be anything from MySQL, redis to any web app.

I get that the hosts-method is meant to affect all apps but that's not a big problem for me running Mac OS and Fedora.

Last time I had to block ads this way was when Opera had them embedded and it was much less memory hungry than Phoenix on my 256M RAM laptop. Back then I blocked them in ipfw instead.

(comment deleted)
Just gonna plug hostsman, which has been doing this on windows since forever:

http://www.abelhadigital.com/hostsman

Lets you chose which lists to use, and automatically update those lists. Also makes it easy to temporarily disable your rules if you need something that's blocked. Has a button for flushing the DNS cache.

I loved using Gas Mask for multiple host file management on OSX. Not so much for Ads but a great app I had trouble discovering.
I wanted a service for my laptop with custom blacklisting/whitelisting, blocking stats and a webserver to serve a blank HTML page for any domains in DNS list so I made:

https://github.com/jdoss/dockerhole

It was inspired by https://pi-hole.net/ and I am glad to see there are others making similar things to block Ads.

Are you talking about the ability to NXDOMAIN *.example.com while also whitelisting this.example.com? AFAICT dnsmasq cant do this, I'm interested if bind can. I have a simple DNS request forwarder half-written to deal with rule trees.

Edit: https://github.com/paulchakravarti/dnslib looks interesting

> Are you talking about the ability to NXDOMAIN [star].example.com while also whitelisting this.example.com?

Yeah, you can totally do that! Details are here [0][1], but in your RPZ zone file, you use a CNAME with a value of . to return NXDOMAIN, and a CNAME with a value of rpz-passthru. to process the query normally:

  ;allow www.sinfest.net, but deny all others, including sinfest.net
  www.sinfest.net CNAME   rpz-passthru.
  sinfest.net CNAME       .
  *.sinfest.net CNAME     .
And to demonstrate:

  $ dig +short www.sinfest.net ; dig +short sinfest.net; \
  dig +short www.sinfest.net @8.8.8.8 ; dig +short sinfest.net @8.8.8.8
  64.29.145.9
  64.29.145.9
  64.29.145.9
  $
If you're interested in a complete, but simple RPZ zone file I can provide one. If you have more questions, feel free to ask, and I'll try to answer.

[0] http://www.zytrax.com/books/dns/ch7/rpz.html

[1] http://www.zytrax.com/books/dns/ch7/rpz.html#rpz-examples

Fantastic. I'll add bind output and optional integration asap.
This already works quite well on my Android phone using AdAway.

Edit: AdAway uses an /etc/hosts file.

> Using 0.0.0.0 [instead of 127.0.0.1] is faster because you don't have to wait for a timeout.

I'm not going to argue that localhost is better than 0, but that specific argument they've raised is incorrect. You don't have to wait for a timeout on localhost either. It will either fail instantly due to no listening processes on that IP and port, or it will connect to whatever process you have open on that address (eg a local instance of a http daemon).

Can you more precisely describe why it is incorrect? Both implementations requests data from the network stack, so both are very similar?
You should raise an issue on the issue tracker.
I'm using my own VPN server and I setup unbound DNS server there. It's the only way for my old iPhone and iPad to browse internet without ads. And it's really fast. I use https://pgl.yoyo.org/adservers/ for ad servers list and a little awk script to convert it to unbound format.
Yes, unbound is great—I run it on my local machine with a list of ad servers to block. I use this little script to download and convert the Someone Who Cares hosts list to unbound format[0] every few weeks. BTW, the pgl.yoyo.org list is available[1] in Unbound format since a few months ago[2].

[0] https://github.com/jodrell/unbound-block-hosts

[1] https://pgl.yoyo.org/adservers/serverlist.php?hostformat=unb...

[2] https://pgl.yoyo.org/adservers/news.php?#unbound

I'm running this for years.

using 127.0.0.1, I have a httpd responding to every request by a 200. this avoid some anti-ad-block check. (such as "watch this ad before your video")

you can also configure your server to reply with a cat gif. but who would like to see a such Internet?

Cats for ads. Has a nice ring to it. I'm surprised no one built it yet.
I guess I could plug a silly little script I wrote for GreaseMonkey which runs on facebook.com. I hated looking at the "Trending" and "Recommended Pages" sidebar so I axed them out in favor of random imgur images of your favorite subreddit. By default you'll get `/r/aww` but you can flip it with a fixed little box in the top right to whatever floats your boat.

Full disclaimer: I'm a horrid at javascript, and the result below is mainly due to a lot of copy paste from various internet sources.

https://gist.github.com/GrahamBlanshard/d7211436088e0159164a

Care to explain the software you use? I'm looking into it with nginx, can't find anything explaining "always return 204 whatever the URI" :(
Could someone please explain why advertisers do not re-use functional domain names to defeat domain-based filtering? I always find it fascinating that they still use such obvious ad-only (sub)domains to host assets.
Possibly because of the threats of getting hacked by XSRF, XSS etc.
That's really the next step in the evolution here. Content creators/producers and publishers need to make money. They're not going to make all that content for free if ads (their revenue) suddenly disappeared tomorrow. Sure, you'll still have a few donation driven sites, and a few subscription-based websites, but 80% of the internet (probably more) relies on ads. Without that revenue stream, the content goes poof. Not to mention, all that innovation and the innovation yet to come. The next youtube, reddit or facebook would not get invented.

Anyways, yeah, some ad companies are starting to do exactly this. They're serving ads up from the domain/website the ads are displayed on. Host files are completely ineffective here unless you've already previously spotted the ads and manually blocked them (and provided they haven't changed the file name of the ad/picture since your last visit).

But the ad companies doing this are tiny. It's quite likely you've never been on a website that has enacted this method. Most ads on the 'net are served up from google's adchoice/adsense program. Once Google themselves start doing it, it's game over for adblockers and for host blocking. I heard through the grape vine that they're actually working on this very issue (the issue of ad blockers).

Thanks for the tip! I would image that once HTTP/2 becomes more popular, serving ads from the same domain might even be preferred simply for the speed improvement.
I had been doing this until some time ago to block ads and to prevent Google from collecting my web browsing history via Google Analytics. During the time I witnessed a strange phenomenon. Every time I added "127.0.0.1 www.google-analytics.com" to C:\Windows\System32\Drivers\etc\hosts. I saw the line removed from the file some hours later. Although I had added tens of lines I only saw the Google Analytics line removed. IIRC finally I decided to figure out whet caused the removal. I used Filemon to watch file changes, though the line got removed again while watching the file and nothing appeared on the log. I suspected Ring-0 processes were secretly running and causing the removal, but I knew nothing about the Windows kernel so I gave up here. I wonder what was the cause to this day.
Wow, really? Can anyone back up a similar story? That's really interesting if so.
Mine is the reverse. I have this line added to /etc/hosts on my mac, a local A record I want to play with. Now, I am done with my project, and I wanted to remove the A record. I keep rebooting and that line persists. God knows why.
Did you have local Google services running, like the Google Updater (afaik also comes with Chrome). Google also adds some entries into the task planner, you can also check there what is getting called.

Though I believe you should have seen something in Filemon.

Probably anti-virus software preventing malware from hijacking google analytics.
Honest question. why use 127.0.0.1 instead of 0.0.0.0?

EDIT: Now I've read the discussion below regarding this matter. No need to answer, I guess. I asked before reading all the comments, sorry.

Assuming you meant why not: "Using 0.0.0.0 is faster because you don't have to wait for a timeout. It also does not interfere with a web server that may be running on the local PC."
He said he used 127.0.0.1 for google analytics. I was asking why he used 127.0.0.1 and not 0.0.0.0, exactly for the reasons you wrote. Sorry if that wasn't clear.
I thought I had similar, we use Junos Pulse and it rewrites your /etc/hosts file. I think it takes a backup at some point and rebuilds the file from that when it needs to. This means some local changes just disappear. Nothing sinister though afaik.
Hopefully all the child posts will see this too.

TO edit the hosts file, you need to have admin privileges. That means closing whatever editor you're using, reopening it with 'run as administrator', and then opening the hosts file. You need to do this even if you are an admin account.

Another way to do it is to open the hosts file under normal editing privileges, editing it, saving it somewhere else, and pasting it into the drivers folder. The system will ask you if you want to run as admin, and you need to say 'yes'.

Nothing could (or should, I guess) be changing the hosts file otherwise (AFAIK, my source being many, many SO posts and random forums) without it being given explicit admin privileges when it attempts to change the file.

Just put the entries in the "hosts" file at router-level (e.g. using OpenWRT).
And duct tape said router to your laptop so you can take it with you every where you go?
Running a local dnsmasq server is effectively the same thing.
Yup. You could also use something like PeerBlock which gives you a little easier control.
Instead of Filemon I'd suggest firing up Process Monitor [1] with a filter of "path contains system32\drivers\etc\hosts" and then Filter -> Drop Filtered Events.

Let this run while you go about your normal work, then check back after you notice the change. Look through the Operation column for WriteFile or something similar, then see what Process Name did it. This'll let you figure out what's actually making the change and you can appropriately assign blame.

[1] https://technet.microsoft.com/en-us/sysinternals/processmoni...

on the subject of sysinternals Process Monitor. Did you know procmon.exe REQUIRES Workstation service running in order to start. It uses it to enumerate something and will silently die without it. This is not documented anywhere and pretty bogus.

Older versions worked fine without this service. It was silently added somewhere between win7 and win8 releases.

Hmm, interesting. No I didn't... I'll poke with this and maybe open a Premier ticket on it.
In Win10 fontdrvhost.exe tries to modify windows Firewall rules every single day to whitelist itself :o

``` A change was made to the Windows Firewall exception list. A rule was added.

Profile Changed: All

Added Rule: Rule ID: {59F33BF3-EAFF-424C-BB26-C2DF4A709398} Rule Name: Usermode Font Driver Host ```

Why would a simple Usermode Font Driver Host need internet access??!?!

binisoft.org Windows Firewall Control has an option to safeguard firewall rules and automagically deletes all unauthorized (by the only person that matters - ME) rules.

I'm using http://pi-hole.net running on a Raspberry Pi. I use it as my home dns, it runs dnsmasq and points a list of a million ad hostnames to its own IP, answering every request with a blank HTML page.
How many websites does it break? I'm a bit hesitant to set up pihole since it's rather hard to enable ads for just one domain, as more and more websites just stop working with ads removed/blocked.
It's not breaking much websites I'm using regularly. The only really breaking effect I see is when clicking on forwarder links of affiliate networks, of course. Happens for bargain/deal websites mostly.

And of course you shouldn't have to work with GA or Flurry or other analytics services, because they are blocked by pi-hole.net of course. But you can easily whitelist via ssh.

I'm now using it since 2 months and am pretty satisfied. The traffic saving effect is also nice, which make websites load faster as well.

While this is ok as an idea, I prefer Privoxy [1] to get my ad blocking outside of the browser. It has the benefit that I can turn it on and off (I use a proxy switcher). It also means that I can have other devices use it either via LAN or SSH tunnel or whatever).

[1] http://www.privoxy.org/

Privoxy is being hosted on Sourceforge, and Sourceforge is still hiding malware in the files. I checked yesterday.
How ironic. I just get it via apt-get in debian.
Wouldn't it be more efficient to add blocking on router level? Making all of your devices (at home) ad-free.

Anyone exprerienced doing this?

Should be very easy to do in OpenWrt and similar firmwares.
Yes, if you had access to every router you use. I travel a lot and have a laptop so I use the blocking via hosts method.
There are a couple of products for this already: "AdTrap" and for the Raspberry Pi "pihole". (Content warning on https://pi-hole.net/ : contains piping curl into bash, slow-loading SPA site)
Pi-hole is great and I had no trouble setting it up, but I couldn't continue using it as it constantly triggered NoScript's ABE (Application Boundaries Enforcer) which blocks scripts served from a LAN address. This is for a good reason, and it may be possible to white-list the Pi in the ABE settings, but I didn't have a good enough understanding of the implications to feel comfortable just allowing it or disabling ABE.
I've installed some home routers with the OpenDNS ip's as DNS servers and a manual block list. Works quite well. It has its flaws of course.
been doing this for a long time with various facebook related domains - upsets people if they borrow my laptop though.
Whilst I'm sure this is no longer the case, I used to do this back in the day, but it was soooo slow!
It took me about a minute to find 12 false entries just by looking what lines end in .de
Relevant: adblock with DNS Server https://hub.docker.com/r/kolyunya/afdns/

This is for people who cannot edit /etc/hosts, but can change DNS server.

"... people who cannot edit /etc/hosts, but can change DNS server."

e.g., "mobile" or "tablet" users who choose Apple iPhone, iPad, etc.

The idea of an ARPA-networked devices that have no user-editable HOSTS file seems inferior to ones that do, i.e. the vast majority of ARPA-networked computers for three decades, but that's just my uninformed view.

The experts selling these things must know better.

DNS works very well for blocking ads. It allows for things that cannot be done with HOSTS alone.

But if you trust a third party for your DNS resolution needs (ad-supported search engine company "free" public DNS, ad-supported, corporate-sponsored browser, etc.), then all bets are off.

If and when advertisers complain and start to cut back on spending, then these third parties could remedy the situation, easily. In my opinion.

If the user is running her own DNS services, then it may be too much trouble for advertisers and the companies they prop up. It is a stretch to think that any ad-supported company could stop users from exchanging lists of names and numbers, whether through a HOSTS file, zone files, or some other mechanism.