Show HN: Hacker News bot that automated archive finding and commenting (github.com)

Love the idea. Too bad it’s not allowed, it would save me some annoyance.

Cool idea. It's kind of silly that "Hacker" News doesn't allow hacking the comments section with useful bots. Why not let the community decide through comment voting?

I wish there was a bot that automatically archived [flagged] posts. the fact users here can use botnets to flag/remove any submission or post they dislike makes Hacker News positively useless for whistleblowing. I learned from experience anything even tangentially controversial is almost guaranteed to be false-flagged and taken down.

@dang, would you consider whitelisting this bot on HN? I think the community would agree that this provides meaningful value to us.

I misread it as "Hacker News bot that automated archive finding and tormenting"...

It's a nice idea, but I like how HN is devoid of blatantly automated posts. Reddit is full of them and it's annoying.

Dang has actually written a response to OP a few days ago, saying essentially that he understands the intention is good, and that this sort of automation is at odds with the core philosophy of HN existing to promote curiosity and interesting conversation between human beings. When a post is popular enough, the community tends to organically post the un-paywall links, making the bot unnecessary.

Imagine having machine-automated archive link posts on every NYT and WaPo submission, it would be annoying and the early comment count would become meaningless noise, not even indicating that some person found it interesting enough to dig up the archive links for.

Thank you very much for sharing this. I was hoping to see the code as soon as you posted it for the first time.

"I build this bot to automate the process of going to wayback machine (or archive.ph) to find the archive link for posts with articles that have subscription blocks"

It is easy to automate this using a forward proxy. I use one on a loopback address. The way I have it set up, I add a given domain to a text file, reload,^1 and from then on, the proxy will rewrite the URLs using that domain to point to archive.org or whatever.

I have never encountered an archive.ph URL on HN where the site proxied, purportedly to avoid a "subscription block",^2 was not accessible without using the archive.ph URL. For example, sometimes the issue is simply that Javascript needs to be disabled, or one needs to use an AMP URL without Javascript. (The later looks great in a text-only browsersuch as links.) Whereas submissions for articles on sites that are password-protected, e.g., ft.com, rarely or never have archive.ph URLs.

This notion of "bots" is a never-ending source of amusement.

Everyone thinks they know what a "bot" is, but IMO that term alone tells us next to nothing. For example, one can write scripts, here using Python, that are disrespectful of server resources or whatever "rules" are in place and, as a result, get "banned". One can also write scripts that are respectful of server resources and any rules and never get banned. Automation is not per se "bots" and "bots" are not per se "bad". Computers and automation are everywhere. We do not label all automation using the internet as "bots". Whether something is "good" or "bad" depends on what a script is doing and, sometimes, who is using it.^3

I have been using automation for reading HN for over 10 years. Yes, it is possible to get banned but it depends on what a script does. The HN software does have limitations and one has to learn to work within those constraints.

1. Something like

   echo reload |socat stdio /path/to/control/socket

2. IMO a true "subscription" would require a password. That is, non-subscribers would not have access to the content. Why do sites make content avaiable to to non-subscribers. Is it only to try to attract new subscribers. The site is gathering data about computer users and serving advertising. The site operator profits from the traffic from non-subscribers. If a site operator wants to make content only accesible to sucbscribers it can make that content password-protected. If the operator chooses not to use password protection then she may have a difficult time convincing a court the content published was not public. Annoying people with Javascript is not access control.

3. Corporations performing telemetry and remotely installing and running "updates", automatically, without explicit consent on computers that do not belong to them generally do not get labeled as "bots". Whereas using automation to "subscribe" to "tech" company websites that aim to collect data about users in order to target advertising might be labeled as "bots" in a perjorative sense, if one is in the business of online advertising.

      http-request set-uri https://web.archive.org/web/20301234567890if_/http://%[req.hdr(host)]%[pathq] if { hdr(host) -m str -f hosts-archive.txt }

      http-request set-uri https://web.archive.org/web/1if_/http://%[req.hdr(host)]%[pathq] if { hdr(host) -m str -f hosts-archive.txt }

    echo example.org >> hosts-archive.txt
    echo reload|socat stdio /path/to/control/socket

   cat > 071.l 
     /* ft.com */
     int fileno(FILE *);
     #define jmp (yy_start) = 1 + 2 *
     #define echo do{if(fwrite(yytext,(size_t)yyleng,1,yyout)){}}while(0)
    %option noyywrap noinput nounput
    %s xb
    %%
    \"articleBody\":\" jmp xb;
    <xb>\",\"wordCount\" jmp 0;
    <xb>\134\156 putchar(10);
    <xb>\40\* putchar(32);
    <xb>. echo;
    \n|.
    %%
    int main(){ yylex();putchar(10);exit(0);}
   ^D 
 
   flex -8iCrf 071.l
   cc-O3 -std=c89 -W -Wall -pedantic -pipe lex.yy.c -o yy071

   #example usage
   curl -4s https://www.ft.com/content/fa956177-5daa-43fd-8a64-79f5eebcc309|yy071
   curl -4o 1.htm https://www.ft.com/content/fa956177-5daa-43fd-8a64-79f5eebcc309
   yy071 < 1.htm

How are you able to automate anything involving archive.ph?

Everything on that site that triggers a page-scrape is behind Google's ReCAPTCHA. And they've got the threshhold set at "hyper-aggressive" -- way above any other non-email-sending site I know of. It's the one site that I still get hit with a captcha from every time (excluding direct links to already-archived pages -- those don't seem to trigger the captcha).

Perhaps this would be better served as a plugin? For the wayback machine plugin I can right click and open that link in the archive, but this is of course for only one website.