Ask HN: How to locate ads on websites programmatically?

byoung2 7y ago ↗

What are you using to fetch the pages? Something like CasperJS/PhantomJS? If the banners are images or iframes from an external domain you should be able to traverse the DOM and locate them this way. You would have to have a list of ad providers domains to look for and you might miss some.

InGodsName 7y ago ↗

Using chrome headless

detaro 7y ago ↗

Modify an ad-blocker to instead of blocking highlight ads?

InGodsName 7y ago ↗

Nice idea! Is there any resource on how modern ad blockers detect and block ad banners?

greenyoda 7y ago ↗

Some ad blockers, like uBlock Origin, are open source, so you can examine the code and see what they're doing:

https://github.com/gorhill/uBlock

The basic algorithm is to match URLs of loadable resources (images, JavaScript files, iframes, etc.) against huge lists of ad/malware/tracking sites. If the URL matches, the resource isn't loaded. The lists that uBlock Origin uses are enumerated here:

https://github.com/gorhill/uBlock/blob/master/README.md

detaro 7y ago ↗

As greenyoda suggested, I'd start with source and documentation of uBlock. From what I know, the major blockers share the format for their blocklists. There's two types of blocks: visual blocks, which contain a CSS selector to hide, and network blocks. Since the latter commonly block scripts that load ad elements, you'd probably need to load the page twice, to observe what elements are not created when the ad-scripts are blocked.