HN Explorer

14 comments

[ 3.5 ms ] story [ 132 ms ] thread
thevania
link to bot detection page - https://bot.incolumitas.com/ - is down
Forbo
I just get a 403. I assumed it was because of my VPN but even without it I get the same result.
risingson
It's been down for months, sadly. I was working on some dummy Puppeteer human-behavior script back then and the site helped a lot. Hope the guy's doing alright :/
[deleted]
(comment deleted)
1vuio0pswjnm7
Not down for me.

   echo 167.99.241.135 bot.incolumitas.com|sed -i -e1r/dev/stdin -e1N /etc/hosts
   FTPUSERAGENT= tnftp -4vdo/dev/stdout https://bot.incolumitas.com
   links -no-connect https://bot.incolumitas.com
   sed -i 1d /etc/hosts

   curl --resolve *:443:167.99.241.135 https://bot.incolumitas.com
1vuio0pswjnm7
False. It's up.

Alas, does not accept TLS1.3.

     printf 'HEAD / HTTP/1.0\r\nhost: bot.incolumitas.com\r\nconnection: close\r\n\r\n'|openssl s_client -connect 167.99.241.135:443 -ign_eof
     printf 'GET / HTTP/1.0\r\nhost: bot.incolumitas.com\r\nconnection: close\r\n\r\n'|openssl s_client -connect 167.99.241.135:443 -ign_eof
     echo|openssl s_client -connect 167.99.241.135:443  -showcerts|openssl x509 -text
batch12
Ugh, just remembered hotornot circa 2004. Those neurons haven't been exercised in a while...
[deleted]
(comment deleted)
luth0r
Kind reminder Google broke almost all TCP fingerprints (i.e. JA3) when introducing grease values and other stuff such as random order in the ciphers in Chromium. I work in a bot detection company and we put a lot effort on it, currently have an advanced implementation that ignores some extensions, order the fields, remove the grease values etc.
acheong08
I have an open source implementation of normalized JA3 fingerprinting at https://github.com/fidraC/canary (Includes CreepJS and other stuff to be added / WIP) it’s not very complicated but I did waste some time before realizing that grease was everywhere, not just extensions.

Note: That is my secondary account for school. I’m not stealing someone else’s work. I use a different name as a sort of mental compartmentalization between different spheres of my life.

mike_d
Google's plan is to break every type of client fingerprinting and then funnel everyone into using their browser attestation.

iOS isn't any better and already sends a unique "Private Access Token" with requests to Cloudflare.

didntcheck
Well I get a 403 when trying to load the detection page (in Firefox) from behind a VPN. Guess that's a fail then?
mateuszbuda
Some parts are outdated now. Bot detection vs web scraping is an ever evolving arms race.

Anyway, incolumnitas blog is a great source of information about bot detection and techniques to avoid being detected. We have implemented many successful methods based on it in https://scrapingfish.com/.

dharma1
What are the current SOTA open source projects for bot detection?