Given that current LLMs do not consistently output total garbage, and can be used as judges in a fairly efficient way, I highly doubt this could even in theory have any impact on the capabilities of future models. Once (a) models are capable enough to distinguish between semi-plausible garbage and possibly relevant text and (b) companies are aware of the problem, I do not think data poisoning will be an issue at all.
> So the compressed data in a JPEG will look random, right?
I don't think JPEG data is compressed enough to be indistinguishable from random.
SD VAE with some bits lopped off gets you better compression than JPEG and yet the latents don't "look" random at all.
So you might think Huffman encoded JPEG coefficients "look" random when visualized as an image but that's only because they're not intended to be visualized that way.
> It seems quite likely that this is being done via a botnet - illegally abusing thousands of people's devices. Sigh.
Just because traffic is coming from thousands of devices on residential IPs, doesn't mean it's a botnet in the classical sense. It could just as well be people signing up for a "free VPN service" — or a tool that "generates passive income" for them — where the actual cost of running the software, is that you become an exit node for both other "free VPN service" users' traffic, and the traffic of users of the VPN's sibling commercial brand. (E.g. scrapers like this one.)
That's just a variant of a botnet that the users are willingly joining. Someone well-intentioned should probably redirect those IP addresses to a "you are part of a botnet" page just in case they find the website on a site like HN and don't know what their family members are up to.
Easiest way to deal with them is just to block them regardless, because the probability that someone who knows what to do about this software and why it's bad will read any particularly botnetted website are close to zero.
I was very excited 20 years ago, every time I got emails from them that the scripts and donated MX records on my website had helped catching a harvester
> Regardless of how the rest of your day goes, here's something to be happy about -- today one of your donated MXs helped to identify a previously unknown email harvester (IP: 172.180.164.102). The harvester was caught a spam trap email address created with your donated MX:
That said, these seem to be heavily biased towards displaying green, so one “sanity” check would be if your bot is suddenly scraping thousands of green images, something might be up.
So how do I set up an instance of this beautiful flytrap? Do I need a valid personal blog, or can I plop something on cloudflare to spin on their edge?
There is a particular pattern (block/tag marker) that is illegal the compressed JPEG stream. If I recall correctly you should insert a 0x00 after a 0xFF byte in the output to avoid it. If there is interest I can followup later (not today).
Ok this is correct for traditional JPEG. Other flavors like Jpeg2000 use a similar (but lower overhead) version of this byte-stuffing to avoid JPEG markers from appearing in the compressed stream.
You're referring to JPEG's byte stuffing rule: any 0xFF byte in the entropy-coded data must be followed by a 0x00 byte to prevent it from being interpreted as a marker segment.
They do have a robots.txt [1] that disallows robot access to the spigot tree (as expected), but removing the /spigot/ part from the URL seems to still lead to Spigot. [2] The /~auj namespace is not disallowed in robots.txt, so even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice.
Faking a JPEG is not only less CPU intensive than making one properly, but by doing os you are fuzzing whatever malware is on the other end; if it is decoding the JPEG and isn't robust, it may well crash.
they have facebookexternalhit bot (they sometimes use default python request user agent) that (as they documented) explicitly ignores robots.txt
it's (as they say) used to validate links if they contain malware. But if someone would like to serve malware the first thing they would do would be to serve innocent page to facebook AS and their user agent.
they also re-check every URL every month to validate if this still does not contain malware.
the issue is as follows some bad actors spam Facebook with URLs to expensive endpoints (like some search with random filters) and Facebook provides then with free ddos service for your competition.
they flood you with > 10 r/s for days every month.
Is there reason you couldn’t generate your images by grabbing random rectangles of pixels from one source image and pasting it into a random location in another source image? Then you would have a fully valid jpg that no AI could easily successfully identify as generated junk. I guess that would require much more CPU than your current method huh?
This makes me wonder if there are more efficient image formats that one might want to feed botnets. JPEG is highly complex, but PNG uses a relatively simple DEFLATE stream as well as some basic filters. Perhaps one could make a zip-bomb like PNG that only consists of a few bytes?
This is pure internet mischief at its finest. Weaponizing fake JPEGs with valid structure and random payloads to burn botnet cycles? Brilliant. Love the tradeoff thinking: maximize crawler cost, minimize CPU. The Huffman bitmask tweak is chef’s kiss. Spigot feels like a spiritual successor to robots.txt flipping you off in binary.
33 comments
[ 1.9 ms ] story [ 54.6 ms ] threadhttps://xkcd.com/810/
A refreshing (and amusing) attitude versus getting angry and venting on forums about aggressive crawlers.
I don't think JPEG data is compressed enough to be indistinguishable from random.
SD VAE with some bits lopped off gets you better compression than JPEG and yet the latents don't "look" random at all.
So you might think Huffman encoded JPEG coefficients "look" random when visualized as an image but that's only because they're not intended to be visualized that way.
https://www.ty-penguin.org.uk/~auj/spigot/pics/2025/03/25/fa...
Some kind of statement piece
Just because traffic is coming from thousands of devices on residential IPs, doesn't mean it's a botnet in the classical sense. It could just as well be people signing up for a "free VPN service" — or a tool that "generates passive income" for them — where the actual cost of running the software, is that you become an exit node for both other "free VPN service" users' traffic, and the traffic of users of the VPN's sibling commercial brand. (E.g. scrapers like this one.)
This scheme is known as "proxyware" — see https://www.trendmicro.com/en_ca/research/23/b/hijacking-you...
Easiest way to deal with them is just to block them regardless, because the probability that someone who knows what to do about this software and why it's bad will read any particularly botnetted website are close to zero.
I was very excited 20 years ago, every time I got emails from them that the scripts and donated MX records on my website had helped catching a harvester
> Regardless of how the rest of your day goes, here's something to be happy about -- today one of your donated MXs helped to identify a previously unknown email harvester (IP: 172.180.164.102). The harvester was caught a spam trap email address created with your donated MX:
Does it? Encryption increases entropy, but not sure about compression.
That said, these seem to be heavily biased towards displaying green, so one “sanity” check would be if your bot is suddenly scraping thousands of green images, something might be up.
Related: https://en.wikipedia.org/wiki/JPEG#Syntax_and_structure
[1]: https://www.ty-penguin.org.uk/robots.txt
[2]: https://www.ty-penguin.org.uk concatenated with /~auj/cheese (don't want to create links there)
they have facebookexternalhit bot (they sometimes use default python request user agent) that (as they documented) explicitly ignores robots.txt
it's (as they say) used to validate links if they contain malware. But if someone would like to serve malware the first thing they would do would be to serve innocent page to facebook AS and their user agent.
they also re-check every URL every month to validate if this still does not contain malware.
the issue is as follows some bad actors spam Facebook with URLs to expensive endpoints (like some search with random filters) and Facebook provides then with free ddos service for your competition. they flood you with > 10 r/s for days every month.
https://www.youtube.com/watch?v=0me3guauqOU