It's not really a good idea to restrict crawlers at directory level since it doesn't prevent any of those pages being indexed. You can get weird behaviour such as googlebot trying to crawl a page but not being allowed to. If this happens a lot then they are going to penalise you for it because they don't want a bunch of pages in their index that they don't know the content of. If this number of pages is a significant percentage of your site then you are in real trouble.
Much better to use x-robots noindex, nofollow for pages you don't want to be in the public domain.
If you are a neutral site (in this case, caters to a lot of people - especially in Asia, where Google as a search engine is a hit-or-miss proposition), then this might backfire sine other crawlers won't understand X-Robots. You can mitigate this by knowing which spiders knows X-Robots (Google, Bing and Yandex AFAIK) and whitelist them while still using a disallow directive for the rest of them.
I'm not sure if I should focus on the detailed logo at the end or the allowed parts for Chinese spiders (especially that it's not blocked for other spiders - is this how you signal priorities for Chinese spiders?)
Edit: I've now noticed the disallow directive for other countries specifically targeted for Chinese spiders.
Indeed. I am also somewhat puzzled about what there is to see here.
I see the Chinese spiders are only allowed on the /cn/ directory, while disallowed from the root (/).
Another thing I find mildly strange is that the help for certain countries is restricted. This is not what robots.txt should be used for, and it creates incentive for bots to disregard robots.txt.
> Another thing I find mildly strange is that the help for certain countries is restricted. This is not what robots.txt should be used for, and it creates incentive for bots to disregard robots.txt.
It seems that they did exist, but moved to specific TLDs (like for Brazil, https://www.nike.com.br). Lazy that they didn't redirect all help files to the correct location though.
1) don't give a robot a gun
2) don't teach it how to put more bullets into gun after breaking #1
3) don't teach it how to make bullets after breaking #2
3a) don't give it your amazon API keys or other credit card info
The *just crawl it*.
Also technically it's quite a big sitemap.
It uses sub-sitemap files.
It also specify to ignore everything but the chinese translation of the website for 3 chinese search engines.
"It's morally abhorrent to try to sell me anything and yet please continue providing content and services at no cost to me. Information wants to be free!"
Well, why not? We also talk to each other for free, giving advice, information, lots of things of some kind of value, and mostly it's free. People organize on platforms to help each other with info, for free, for example you can ask people on Reddit to identify a bug or stone for you. Now I know hosting Reddit isn't free, but this is for another time I think. Now I just want to say that what you pointed out is not really something that's purely illogical or entitled.
Moderators' time is free, and maintaining reddit could be free. Hosting uses electricity, hardware and internet connection, and neither of those are given away currently.
Ah, agree to disagree on that one. That view seems exploitive to me. Moderation, OSS maintainers, etc are all doing those jobs for free and providing them to us for free but there is a large opportunity cost.
That's where the saying "it's not worth my time" comes from. If I asked you why you don't maintain and moderate reddit for free (I'm sure they would be happy to have you there and pay you nothing) I would assume your answer would be I have better things to do with my time or I don't have enough time. i.e. I need to be paid for that to be worth it.
Well, yeah, ideally I'd like an UBI-like system, so that one can moderate Reddit full time, and still pay rent and groceries. But letting that aside, do you think that all volunteers are in fact exploited? Or just these examples, because effectively they help someone else make money for free?
I mentioned exploitive in response to "Moderators' time is free". Which I feel like is an exploitive view. I don't think being a volunteer means you are exploited, but I think someone not valuing their time is almost the definition of it to me.
I think moderators time is extremely expensive. They could be doing one million other things with that time and getting paid for it. They could actually be doing the EXACT SAME thing and getting paid for it if they worked for a Reddit/Facebook/StackOverflow or were a community manager for a brand.
> Now I know hosting Reddit isn't free, but this is for another time I think.
You can't just pretend like the key fact isn't relevant.
Platforms don't grow on trees.
Somebody has bills to pay and its up to them how to decide how to pay them. If you don't like their decision, find another platform or moan incessantly about it.
I know the realities and I don't think this factor is irrelevant - hosting can cost a lot. I bemoan ads specifically, because I think as they are and were, they are a net negative for humanity. Having an addiction fueling platform in exchange for it doubles down on this negativity. I'm thinking where ads are the answer, or the main enabler of content, then the content itself is questionable, is what I'm trying to get at.
It seems that all of them were built for recursive archiving in the early '00s for personal storage (so they were programs literally installed on computers rather than a specific group). Probably a legacy thing (considering at that time Wikipedia literally runs on a single server). I found a robots.txt file that even list a greater variety of said software: https://www.iatse.com/robots.txt
The bad actors don't follow the robots.txt, or rather are bad actors because they don't follow robots.txt neither the informal courtesy practices. Then the main role of robots.txt is public shaming?
There are robots which follow robots.txt but which site operators prefer to block or content-restrict, for various reasons. See eg the annotated wikipedia robots.txt me tioned upthread.
There are really three types of crawlers: good actors, bad actors, and "grey actors". The grey actors can be just by making a mistake (or outright incompetence), or they're testing where the limits are, etc.
robots.txt does nothing for the bad actors, but for grey actors it's a bit more complicated as they can move from "grey" to "good". Adding something for that in robots.txt in addition to some other limits (if needed) can be useful.
As a search engine operator, I can confirm webmasters put a lot of weird crap in robots.txt. ASCII art, poems, emoji, love letters, sudoku puzzles, philosophical questions, whatever. It's cute, but odds are at least some parser will choke on them and depending on how the crawler is coded, either ignore your robots.txt; or skip crawling your site. It's not great either way as the format is more of a set of conventions than a proper well defined language all specified up in BNF.
Webmasters also put highly specific URLs they don't want the world to see in robots.txt. Also not a great idea.
Both old-school webhosting and modern cloud hosting commonly have data caps. Unlimited data is common only on mid-to-high-end VPS or dedicated servers and unlimited bandwidth doesn't really exist.
True, when there is no deliberate cap stated, it's unmetered, not unlimited, meaning you can use what however much fits through the pipes (usually best effort instead of guaranteed bandwidth), and that's the natural limit. A 1 Gbit/s link for example will get you close to 300 TiB in each direction in a 30 day month if you are able to saturate that link 24/7, while a 100MBit/s link will get you a tenth so "only" close to 30TiB.
Free unlimited bandwidth always has a catch somewhere, that this use could trip on, unless the service is fairly expensive.
There is often a “fair use policy”, and even without anything like that you need to consider the potential effect on the stuff you are actually running the service for (while a search engine is pulling random data as fast as they can, that transfer is competing against other network load). I have a couple of inexpensive hosted servers with genuinely unlimited bandwidth, but only at 100 and 250mbps respectively, better rates cost a fair chunk more without some sort of cap (usually of the form “after some-TB-or-some-tens-of-TB we'll throttle you to 100mpbs until your next billing month”.
Unlimited throughput on gbit/more lines is commonly available, but never both cheap and reliable.
I think the concern here is about congested peerings to certain providers, not uptime.
For example Deutsche Telekom: They don't accept free direct peerings. Sending data to them indirectly frequently runs into congestion issues. For a while Hetzner offered a paid option for better connectivity to DTAG. It looks like Hetzner added a direct peering in 2020.
> reliable unmetered gbit+ has been cheap for years
I suppose that depends on what you call cheap, and whether you include “guaranteed”/“dedicated” in the definition of reliable for bandwidth.
I'm looking at things mainly from a personal projects & packups PoV, but demanding at least two drives for RAID1+ rather than the cheapest units. If you have money-making projects then the lines between cheap/inexpensive/reasonable might move.
> OVH / Hetzner
I have Hetzner down as “inexpensive” rather than cheap, assuming you count the older hardware in their auction lines (with their new-kit offerings being “reasonable”).
OHV is cheap if you are considering their lower-spec offerings, but those are at limited rates (100mbit usually in the case of Kimsufi branded services, 250mbit for SYS) and IIRC that rate is not guaranteed (while not a massively oversold resource like you'll see in many VPS/shared/similar hosting arrangements, there are enough machines sharing a larger resource that if many try saturate their allocation at the same time they'll hit congestion even within the DC rather than just when your traffic touches external peering).
(Don't take the above as a criticism of Kimsufi/SYS - they are honest & open about the limits of the services they sell (older hardware, limited max network throughput) - I get what I pay for, which is no less than promised, and I've found them to be reliable over the years I've had services with them)
This isn't physically possible. I think you're confusing (a) an ISP saying "we'll give you all the bandwidth we get (from our tier 1 networks / backbone links)" with (b) genuinely unlimited bandwidth, which would suppose a infinitely large carbon fibre cable subsuming the universe, or something along those lines.
(a) may not be that much, in reality, depending on how their network is set up, and how much other traffic there is at any given moment. It also depends where you're routing to, whether e.g. (i) your neighbour or (ii) a DC across the world (neither is clearly better, it all depends). You have to think of it through the lens of physics and information theory, rather than law and business models.
I'd agree if the upper bound of unmetered and uncapped bandwidth were so far in excess of the possible upper bound for this application that it were silly to consider - but, like I addressed in my comment, it's very much not.
If he were only talking about meters or caps, then that would therefore make his point a bit of a non sequitur in the context of an argument about whether tarpitting/blackholing a connection is risky for the serving party.
(FWIW, at past companies I've not-infrequently encountered network saturation even without an artificial cap from whomever we were peering with.)
Genuine unmetered/unlimited bandwidth claims also include a port speed. And the really good claims will include a 'minimum sustained' rate (with well connected peers), which lets you complain when you can't get decent throughput because all the servers on the 10G switch share a t1 out to the local internet exchange.
Please tell me what provider has free unlimited bandwidth.
I’ve heard this false notion so many times on HN and every time I check it out, the provider says clearly that they don’t allow for unlimited bandwidth.
So please tell me the provider. I’ll pay for a month of service for 10 Gbps unlimited service and then I’ll saturate that line up and down stream for an entire month.
> Please tell me what provider has free unlimited bandwidth.
Unmetered rather than unlimited of course, but OVH, Online.net, Hetzner. These are rather well-known providers.
I haven't checked all of them for the exact language but Hetzner explicitly says "All dedicated root servers have unlimited traffic" for example.
Of course, they do have a limit for servers with 10 Gbps connections probably because of people like you who like to push the limits beyond what is reasonable.
Yep, those. Also FDCservers, Nforce, psychz, worldstream and countless of others.
I have extensive experience maxing out unmetered 10gbit lines with these providers 24/7. It’s been years and years since a host last tried to FUP me, bandwidth is just really cheap now.
Yep, the cheap dedicated servers I have are with Kimsufi and SoYouStart (both OVH brands), though the lines are limited to 100mbit and 250mbit respectively and IIRC neither is guaranteed (something like X servers sharing a gbit+ connection, where Xrate > TotalLineRate) so I can usually* but not always get 100mbit out of kimsufi. I've not really pushed the SYS machine's resources yet, it is a recent addition to my dominion.
A search engine pulling data from /dev/urandom as in the post that started this discussion is likely to impact other uses of the 100mbit link from Kimsufi. Worse if bad luck means multiple hits of that sort at the same time.
I hear it's decades of bad luck to do unless you check that the crawler's user-agent isn't "search.marginalia.nu"; but you could just tarpit the connection.
The certificate is only for "*.marginalia.nu", which simply doesn't cover "marginalia.nu". It should give the same error on any platform and browser, unless their SSL implementation is broken.
Some browsers try to be smart and insert www automatically though.
That's not the certificate I get served though (this one is from 5 february, the one served is from 25 january) do you actually get this certificate on https://marginalia.nu?
Ah, I see what's happening. search.marginalia.nu goes through cloudflare which does its creepy MITM thing and re-encrypts on its end; the others do not.
Yeah, It's 100% about bot mitigation. I got hit pretty bad by some botnet a while back, saw 10-15 searches/second. Can't have nice things on the Internet I guess.
Don't really understand their motive either. Maybe they thought I was cloud hosted or used some expensive API to do searches and were attempting to rack up big bills or something.
I wish I had a good solution to this. Cloudflare to mitigate DDOS attacks has somewhat of a “baby with the bathwater” vibe (considering how much of a pain it is if you can't pass the CAPTCHA, or if you're on Tor), but I can't think of an alternative.
I've considered having a naked endpoint with a rate limiter that, when it hits some ceiling (dunno, sustained load of 2 RPS or so), offers the alternatives of going through an unlimited cloudflared domain, or waiting until the bots give up. But that might be annoying too.
You could try to push forward the frontier of accessible CAPTCHAs. You just need some tasks that basically all humans can do, but barely and bots can, that don't rely on vision, hearing, emotional response or cultural knowledge. Should be easy, right?
It's simple, your cert points to *.domain.tld, domain.tld doesn't have a subdomain, <anything>.domain.tld does. You should expand it to include both domain.tld and *.domain.tld.
I serve my robots.txt files with Transfer-Encoding: gzip and Content-Encoding: gzip, chunked and then just serve a two layered 10G gzip bomb (240kB file size) instead.
There are fail2ban configs for most popular software ready to use. E.g. WP fail2ban¹ is a config that bans an IP after failed login attempts. The defaults it ships with are well-balanced between security and not-locking-yourself-out.
Drupal, PHPBB, etc all have such generic config defaults online. For frameworks such as Rails, Django etc its a bit harder because the logging and url-schemes are not standard, but defined by the person developing in that framework.
Fail2ban, by default, bans an IP for a limited time. For me that's 20mins.
This means that commonly only about 30 IPs are on the banlist. The chance that anyone using the same IP as a bot who is visiting any of the apps and sites on that server is not 0%, but tiny.
The only goal is to stop bots and worms from hammering my server. It hardly improves security; it shouldn't anyway.
I should try something like that for the endless bots that seem to try and access our site. Instead of outright banning them, put them in a special user group that gets a gzip bomb whenever they try to access the site again.
Is there a way to make it so it doesn't outright make processes crash or lock up, but instead really slow them down / hang them up for a long time?
At some point I just got annoyed by all the 4chan script kiddies and went in active defense mode :D
Also lookup "slowloris attack" -It's pretty efficient if the attacker has more network bandwidth than you have, and can even out the advantage.
Won't be as effective against them having more networked machines though, so you have to use some other tricks against shodan, websnort, burp suite etc.
There are some other tricks though, and you can also abuse HTTP smuggled headers (in the response) to specifically crash their proxies, for example :)
Most common vulnerability scanners use some standard http libraries, so they are vulnerable to slowloris + chunked encodings, or they can't recover from a broken http trailing header, or from an http2/3 upgrade/downgrade loop that breaks their UDP packet parser.
What also works is if you spoof their SOCKS5 proxy with UDP bind requests until they can't use it anymore. So for a lot of DDoS attack scenarios there are active defense options to mitigate the damage which the redteam can cause.
BTW, how's your stealth browser project coming along? (I'd go look but I'm in a weird situation right now where I can't view most of the web. It's browser related so maybe this would amuse you: I'm running Firefox 91.5.1esr on FreeBSD from the official pkg. The vast majority of HTTPS sites give a "Your connection is not secure" error (Error code: NS_ERROR_NET_INADEQUATE_SECURITY) with no recourse (no button to let you make an exception or anything like that. It literally says, "The website administrator will need to fix the server first before you can visit the site.") The fix is to disable SPDY for HTTP2 connections. That works, but then FF crashes (with a core dump, hello FreeBSD) within ~10 seconds no matter what I do. There are a few clues, but I'm just not interested in shaving this yak. Frankly, I'm shopping for a new browser as I find the situation ridiculous. Your project seems way more interesting than what FF has become.)
Did you try to breakout using pwnat to get a public port? In enterprise grade environments it usually works :D
I think I'd also try to run a UDP tunnel or try to use an amplification redirect on a UDP relaying proxy to get an open port that you can send requests through. Sometimes public DNS resolvers can help to punch a hole through the NAT because their IPs are allowlisted to unbreak secure browsing modes.
> re: stealth
I forked webkit into retrokit [1] these days and am gradually removing all APIs that could be abused for tracking purposes ... with the idea to create a minimalistic webview that is embeddable in a browser and still is able to parse/lex/render modern HTML5 and CSS4.
This way I can later bundle nodejs with the retrokit webview that just displays the Browser UI that's served locally.
I don't think it's a network issue (we're at home) because all the other browsers work. This version of FF is just weird about HTTP2/SPDY. I don't want to shave this yak.
> Webmasters also put highly specific URLs they don't want the world to see in robots.txt. Also not a great idea.
I can confirm; I tried to helpfully put a bunch of addresses which would be pointless for a web crawler to crawl in there (links to the login page, etc), but as soon as you say "don't" somebody's gotta go find out why.
I like the idea of putting gzip bombs at commonly-scanned URLs.
That’s fascinating. But assuming you aren’t one of the top two or three search engines, wouldn’t it hurt you as much if not more to stop crawling a major site like Nike which visitors surely expect to be crawled? If you are one of the top two or three, I would like to think their crawlers are amazingly complex and robust but maybe I’m just naive.
Can you point to your source? From a quick glance on their documentation, it seems like you got it backward.
> A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.
They won't crawl it, but it they already did crawl it or if there's a link to it, it will still be part of their index (thus search results).
They even go further on their noindex page, by saying that it need to be crawlable (thus not part of the robots.txt) so that they can see the no index directive on the page...
> Important: For the noindex directive to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex directive, and the page can still appear in search results, for example if other pages link to it.
My high school's was the strangest before they overhauled their site. One of the URLs was some sort of CGI spell check app: just a button and a field that said "I am spelled wrang"
I love the organization! Not sure if the comments are exactly to-spec, but sounds reasonable enough for parsers to ignore broken lines just like how css is parsed.
Also, it's a really common misunderstanding that robots.txt can be used to keep pages out of search results. You need to use noindex meta tags or noindex headers for that. Robots.txt tells crawlers which pages they can't fetch, but they're still allowed to include those pages in search results if they want.
And even those are advisory, if you really need to block them, you have to add rules to your webserver of known user agents and IPs (because plenty of indexers will spoof their user agent as well. Google probably does that so they can penalize those that offer different content to search engines than regular users. Doesn't really work though).
176 comments
[ 3.9 ms ] story [ 240 ms ] threadIf you also know why we're making some basic mistakes here, especially apply here.
Much better to use x-robots noindex, nofollow for pages you don't want to be in the public domain.
https://developers.google.com/search/docs/advanced/crawling/...
If you are a neutral site (in this case, caters to a lot of people - especially in Asia, where Google as a search engine is a hit-or-miss proposition), then this might backfire sine other crawlers won't understand X-Robots. You can mitigate this by knowing which spiders knows X-Robots (Google, Bing and Yandex AFAIK) and whitelist them while still using a disallow directive for the rest of them.
Why is only poland blocked for all robots? Why is there no /deutchland?
Seems that Bloomberg hosts something for the Polish, but I'm not sure what.
[1] https://en.wikipedia.org/wiki/I,_Robot
I’m using Safari on iOS 15.2.1, running on an iPhone X.
Edit: I've now noticed the disallow directive for other countries specifically targeted for Chinese spiders.
I see the Chinese spiders are only allowed on the /cn/ directory, while disallowed from the root (/).
Another thing I find mildly strange is that the help for certain countries is restricted. This is not what robots.txt should be used for, and it creates incentive for bots to disregard robots.txt.
It seems that they did exist, but moved to specific TLDs (like for Brazil, https://www.nike.com.br). Lazy that they didn't redirect all help files to the correct location though.
It makes sense considering they have this: https://purpose.nike.com/statement-on-xinjiang
https://www.last.fm/robots.txt
Honeypot or snarky?
> Second Law. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
> Third Law. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
https://en.wikipedia.org/wiki/Three_Laws_of_Robotics
If you ever get the chance to read a bit of Asimov, I highly recommend it. I'm particularly fond of his Foundation series.
https://xkcd.com/1613/
Wired even wrote an awful article about it at the time: https://www.wired.com/2010/08/robot-laws/
Absolutely delighted people remember it 12 years later.
I think it's play on the Nike's famous slogan "Just do it".
That's where the saying "it's not worth my time" comes from. If I asked you why you don't maintain and moderate reddit for free (I'm sure they would be happy to have you there and pay you nothing) I would assume your answer would be I have better things to do with my time or I don't have enough time. i.e. I need to be paid for that to be worth it.
I think moderators time is extremely expensive. They could be doing one million other things with that time and getting paid for it. They could actually be doing the EXACT SAME thing and getting paid for it if they worked for a Reddit/Facebook/StackOverflow or were a community manager for a brand.
You can't just pretend like the key fact isn't relevant.
Platforms don't grow on trees.
Somebody has bills to pay and its up to them how to decide how to pay them. If you don't like their decision, find another platform or moan incessantly about it.
[1] https://www.youtube.com/watch?v=mpe1R6veuBw
Everyone Loves Rodrigo
># unless they're feeding search engines.
>[disallows UbiCrawler, DOC and zao]
Anyone understand this bit? They are obedient spiders yet are disallowed because of purpose? Are these obedient but still overly intensive spiders?
robots.txt does nothing for the bad actors, but for grey actors it's a bit more complicated as they can move from "grey" to "good". Adding something for that in robots.txt in addition to some other limits (if needed) can be useful.
* https://www.tripadvisor.com/robots.txt
Webmasters also put highly specific URLs they don't want the world to see in robots.txt. Also not a great idea.
edit: s/unlimited/unmetered/
True, when there is no deliberate cap stated, it's unmetered, not unlimited, meaning you can use what however much fits through the pipes (usually best effort instead of guaranteed bandwidth), and that's the natural limit. A 1 Gbit/s link for example will get you close to 300 TiB in each direction in a 30 day month if you are able to saturate that link 24/7, while a 100MBit/s link will get you a tenth so "only" close to 30TiB.
There is often a “fair use policy”, and even without anything like that you need to consider the potential effect on the stuff you are actually running the service for (while a search engine is pulling random data as fast as they can, that transfer is competing against other network load). I have a couple of inexpensive hosted servers with genuinely unlimited bandwidth, but only at 100 and 250mbps respectively, better rates cost a fair chunk more without some sort of cap (usually of the form “after some-TB-or-some-tens-of-TB we'll throttle you to 100mpbs until your next billing month”.
Unlimited throughput on gbit/more lines is commonly available, but never both cheap and reliable.
It’s been a long time since you could call OVH “unreliable”, Hetzner chugs along just fine too.
I think you might just be stuck in 2004, reliable unmetered gbit+ has been cheap for years.
For example Deutsche Telekom: They don't accept free direct peerings. Sending data to them indirectly frequently runs into congestion issues. For a while Hetzner offered a paid option for better connectivity to DTAG. It looks like Hetzner added a direct peering in 2020.
https://web.archive.org/web/20200205014018/https://wiki.hetz...
https://www.hetzner.com/news/03-20-dtag/
I suppose that depends on what you call cheap, and whether you include “guaranteed”/“dedicated” in the definition of reliable for bandwidth.
I'm looking at things mainly from a personal projects & packups PoV, but demanding at least two drives for RAID1+ rather than the cheapest units. If you have money-making projects then the lines between cheap/inexpensive/reasonable might move.
> OVH / Hetzner
I have Hetzner down as “inexpensive” rather than cheap, assuming you count the older hardware in their auction lines (with their new-kit offerings being “reasonable”).
OHV is cheap if you are considering their lower-spec offerings, but those are at limited rates (100mbit usually in the case of Kimsufi branded services, 250mbit for SYS) and IIRC that rate is not guaranteed (while not a massively oversold resource like you'll see in many VPS/shared/similar hosting arrangements, there are enough machines sharing a larger resource that if many try saturate their allocation at the same time they'll hit congestion even within the DC rather than just when your traffic touches external peering).
(Don't take the above as a criticism of Kimsufi/SYS - they are honest & open about the limits of the services they sell (older hardware, limited max network throughput) - I get what I pay for, which is no less than promised, and I've found them to be reliable over the years I've had services with them)
Then there are no cheap options. Even frivolously expensive “cloud” providers don’t give you this.
The reality is that there are very few customers that actually have a real need for “guaranteed”/“dedicated” bandwidth.
(a) may not be that much, in reality, depending on how their network is set up, and how much other traffic there is at any given moment. It also depends where you're routing to, whether e.g. (i) your neighbour or (ii) a DC across the world (neither is clearly better, it all depends). You have to think of it through the lens of physics and information theory, rather than law and business models.
If he were only talking about meters or caps, then that would therefore make his point a bit of a non sequitur in the context of an argument about whether tarpitting/blackholing a connection is risky for the serving party.
(FWIW, at past companies I've not-infrequently encountered network saturation even without an artificial cap from whomever we were peering with.)
I’ve heard this false notion so many times on HN and every time I check it out, the provider says clearly that they don’t allow for unlimited bandwidth.
So please tell me the provider. I’ll pay for a month of service for 10 Gbps unlimited service and then I’ll saturate that line up and down stream for an entire month.
And we’ll just see what happens, ok?
Unmetered rather than unlimited of course, but OVH, Online.net, Hetzner. These are rather well-known providers.
I haven't checked all of them for the exact language but Hetzner explicitly says "All dedicated root servers have unlimited traffic" for example.
Of course, they do have a limit for servers with 10 Gbps connections probably because of people like you who like to push the limits beyond what is reasonable.
I have extensive experience maxing out unmetered 10gbit lines with these providers 24/7. It’s been years and years since a host last tried to FUP me, bandwidth is just really cheap now.
A search engine pulling data from /dev/urandom as in the post that started this discussion is likely to impact other uses of the 100mbit link from Kimsufi. Worse if bad luck means multiple hits of that sort at the same time.
Check out https://discord.com/invite/7Gv8tdM
There are really good deals available, all of these still come with a decent margin for the reseller.
I was hitting it plaintext first, so a simple redirect to some subdomain instead of a bare redirect to https would probably work fine.
I don't trust letsencrypt and I don't want to give them or anyone else a list of which subdomains I use.
The certificate is only for "*.marginalia.nu", which simply doesn't cover "marginalia.nu". It should give the same error on any platform and browser, unless their SSL implementation is broken.
Some browsers try to be smart and insert www automatically though.
[1] https://crt.sh/?id=6125359537
https://marginalia.nu = https://crt.sh/?id=6046506678 (includes wildcard but NOT apex)
https://search.marginalia.nu = https://crt.sh/?id=6125359537 (includes both wildcard and apex)
Don't really understand their motive either. Maybe they thought I was cloud hosted or used some expensive API to do searches and were attempting to rack up big bills or something.
I wish I had a good solution to this. Cloudflare to mitigate DDOS attacks has somewhat of a “baby with the bathwater” vibe (considering how much of a pain it is if you can't pass the CAPTCHA, or if you're on Tor), but I can't think of an alternative.
I've considered having a naked endpoint with a rate limiter that, when it hits some ceiling (dunno, sustained load of 2 RPS or so), offers the alternatives of going through an unlimited cloudflared domain, or waiting until the bots give up. But that might be annoying too.
See https://news.ycombinator.com/item?id=30301015
Oof, I'm not sure if I would do that. Since "" stands for all sub domains.
When I visit https://search.marginalia.nu I'm served with this different cert, which does include both wildcard and apex: https://crt.sh/?id=6125359537
I serve my robots.txt files with Transfer-Encoding: gzip and Content-Encoding: gzip, chunked and then just serve a two layered 10G gzip bomb (240kB file size) instead.
Can recommend. Works pretty often.
¹anything .php,.asp and some other tech we never use, also phpmyadmin, cgi-bin, and some more, actually.
Drupal, PHPBB, etc all have such generic config defaults online. For frameworks such as Rails, Django etc its a bit harder because the logging and url-schemes are not standard, but defined by the person developing in that framework.
¹https://wordpress.org/plugins/wp-fail2ban/
nobody own their IP, and at some point a legitimate person will be assigned that IP
and most people who do bad things on the internet use some proxies, usually residential ones
it always makes me sad when people put such rules
This means that commonly only about 30 IPs are on the banlist. The chance that anyone using the same IP as a bot who is visiting any of the apps and sites on that server is not 0%, but tiny.
The only goal is to stop bots and worms from hammering my server. It hardly improves security; it shouldn't anyway.
Is there a way to make it so it doesn't outright make processes crash or lock up, but instead really slow them down / hang them up for a long time?
Also lookup "slowloris attack" -It's pretty efficient if the attacker has more network bandwidth than you have, and can even out the advantage.
Won't be as effective against them having more networked machines though, so you have to use some other tricks against shodan, websnort, burp suite etc.
There are some other tricks though, and you can also abuse HTTP smuggled headers (in the response) to specifically crash their proxies, for example :)
Most common vulnerability scanners use some standard http libraries, so they are vulnerable to slowloris + chunked encodings, or they can't recover from a broken http trailing header, or from an http2/3 upgrade/downgrade loop that breaks their UDP packet parser.
What also works is if you spoof their SOCKS5 proxy with UDP bind requests until they can't use it anymore. So for a lot of DDoS attack scenarios there are active defense options to mitigate the damage which the redteam can cause.
BTW, how's your stealth browser project coming along? (I'd go look but I'm in a weird situation right now where I can't view most of the web. It's browser related so maybe this would amuse you: I'm running Firefox 91.5.1esr on FreeBSD from the official pkg. The vast majority of HTTPS sites give a "Your connection is not secure" error (Error code: NS_ERROR_NET_INADEQUATE_SECURITY) with no recourse (no button to let you make an exception or anything like that. It literally says, "The website administrator will need to fix the server first before you can visit the site.") The fix is to disable SPDY for HTTP2 connections. That works, but then FF crashes (with a core dump, hello FreeBSD) within ~10 seconds no matter what I do. There are a few clues, but I'm just not interested in shaving this yak. Frankly, I'm shopping for a new browser as I find the situation ridiculous. Your project seems way more interesting than what FF has become.)
I think I'd also try to run a UDP tunnel or try to use an amplification redirect on a UDP relaying proxy to get an open port that you can send requests through. Sometimes public DNS resolvers can help to punch a hole through the NAT because their IPs are allowlisted to unbreak secure browsing modes.
> re: stealth
I forked webkit into retrokit [1] these days and am gradually removing all APIs that could be abused for tracking purposes ... with the idea to create a minimalistic webview that is embeddable in a browser and still is able to parse/lex/render modern HTML5 and CSS4.
This way I can later bundle nodejs with the retrokit webview that just displays the Browser UI that's served locally.
[1] https://github.com/tholian-network/retrokit
Retrokit looks great. :)
I can confirm; I tried to helpfully put a bunch of addresses which would be pointless for a web crawler to crawl in there (links to the login page, etc), but as soon as you say "don't" somebody's gotta go find out why.
I like the idea of putting gzip bombs at commonly-scanned URLs.
# comments
// comments
<!-- comments -->
/* comments */
plain text lines as comments
TeX-markup
script tags
HTML
PHP code
Whatever this is: https://www.costablancaworld.com/robots.txt
search-engine specific directives (Yandex has a few strange ones)
That's an RTF file that was saved as a TXT (without removing formatting)...
https://en.wikipedia.org/wiki/Rich_Text_Format
Somebody screwed up and made their robots.txt in a rich-text editor like wordpad.
Even stranger things can be found in a `humans.txt`[0] file!
[0] https://humanstxt.org/
Ah the never ending joy I get by putting a 1GB file as the robots.txt or even having a 1GB `favicon.ico` file.
That’s not really how it works now. Google pretty much ignores robots.txt now and crawls your site anyway.
They can and will crawl even if you say no, they've officially said so as well in their docs.
> A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.
https://developers.google.com/search/docs/advanced/robots/in...
They won't crawl it, but it they already did crawl it or if there's a link to it, it will still be part of their index (thus search results).
They even go further on their noindex page, by saying that it need to be crawlable (thus not part of the robots.txt) so that they can see the no index directive on the page...
> Important: For the noindex directive to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex directive, and the page can still appear in search results, for example if other pages link to it.
https://developers.google.com/search/docs/advanced/crawling/...
Google, facebook and bbc's are surprisingly boring :)
I'd be interested in browsing these pages.
https://www.checkbot.io/robots.txt
Also, it's a really common misunderstanding that robots.txt can be used to keep pages out of search results. You need to use noindex meta tags or noindex headers for that. Robots.txt tells crawlers which pages they can't fetch, but they're still allowed to include those pages in search results if they want.