This is a scraping tool that can download a list of URLs very fast. It's not an AI tool per se but the advertised use is to download corpora for AI training. Traditionally, polite Web spiders tried not to hit any one server too hard but when your goal is to download a bunch of data from a small number of sites fast you're inevitably going to be putting real load on each site. I'm not sure there's a solution that can satisfy everyone.
I guess the "solution" is that sites detect abuse and ban the ip for 24 hours. When enough users of this tool get banned, they will fill the github issues tracker with complain and the maintainers will have to learn how to write a polite spider.
We have the inverse problem in the university. We are all (many thousands) behind a proxy, so some popular sites detect the avalanche of request and add a captcha or other anti abuse tool to everyone.
The tool has instructions for how to opt out. They could also ban the user agent. This is just someone venting about something that's currently popular.
>> Websites can pass the http headers X-Robots-Tag: noai, X-Robots-Tag: noindex , X-Robots-Tag: noimageai and X-Robots-Tag: noimageindex By default img2dataset will ignore images with such headers.<<
>>To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'<<
Fair enough, though it's open source right, so if it didn't have the setting it could be trivially changed anyway. In either case, it's the user ignoring the expressed will of the site as opposed to the maintainers of the tool
1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure them to avoid the opt-out.
2) Add a general rate limit per ip, Bonus points for a te4mporal ban in case they don't understand the polite version.
it seems like what would be reqired is to whitelist, known acceptable user agents, then review requests from green agents and whiten them based on X criteria, otherwise leave green agents green and put them in a hole.
I once accidentally got my company's office blocked from w3.org automatically downloading the schemas for some XML documents.
What then happens is all requests to w3.org then redirect to a page explaining how anyone automatically downloading those schemas from those URLs became part of an accidental DDOS attack.
Ironically, the reason this tool is popular is because copyright prevents sharing of the datasets, and so all organizations like LAION can do is share URLs. The fuss people are kicking up about scrapers is nothing compared to what they'd do if their copyrighted images were in a public ML dataset.
If they (or legislators) just agreed that being part of training data was fair use, it would save a lot of servers a lot of bandwidth, and make it more convenient for people working with these datasets. It would be win/win, but there's a prisoners dilemma + misplaced greed that means it will never happen like that.
> Why should people contribute to making their own jobs obsolete and not even be compensated for it?
Haven’t technologist been saying for years about blue collar workers whose jobs have been made obsolete by technology that they should learn a new skill?
Now that it’s hitting white collar workers, we are singing a different tune.
And most of the time especially here on HN most people want an expansive definition of fair use as long as it is directed toward using work created by the big media companies.
It’s fair use because training sets are useless except for creating a new thing which transforms the inputs into something new. It’s not merely selling photos that already exist or something. Stable Diffusion has only about enough bits in the weights to represent a single pixel out of each image in the training set.
“training sets are useless except for creating a new thing”
So by useless you mean absolutely essential?
Hard to argue the fair use thing but it was designed around humans not AI so just because it is fair use now does not mean it should or will be in the future.
What is what? If you mean distinguishing between AI and humans, it’s exactly what it sounds like. Fair use is a legal construct not a universal truth and subject to change like anything else. Just like how the US Founders writing the 2nd amendment with muskets in mind didn’t foresee full auto assault rifles in the hands of citizens. In hindsight, overly permissive law, all things not equal, hello gun control.
Automatic weapons were in fact in the public consciousness at the time of the founding, and the Gatling Gun existed and was used in the Civil War with no restrictions on who may own one.
I dare say, this thought that people so well read they could architect a country were not aware of the State of the Art in weaponry is rather naive.
If it is naive to disbelieve the founders foresaw a future in which automatic weapons and bullets would be so lightweight, cheap and accessible to civilians that anyone could buy them and go on a rampage by themselves that the State could not suppress without dozens of police and SWAT teams then I am naive.
Gatling Guns came well after the Founding, were large, heavy, hand cranked, cost between $1000 to $1500 to purchase and required a four man crew to operate. I don’t feel they can be compared to modern weapons.
The Founders were intelligent and capable men but they were not infallible. They knew that too, that’s why Amendments exist.
I don't think any of that is relevant. It's obvious they didn't mean that they only wanted citizens to have the right to bear inferior arms and that only the government should get the more advanced or powerful weapons. The entire point is being able to oppose a tyrannical government.
I can see an argument over nukes, but this "they didn't imagine assault weapons" is just some kind of gaslighting. They imagined sufficient arms to defend oneself against authority.
While this may seem quaint, let's review the github package's README:
```
Opt-out directives
Websites can pass the http headers `X-Robots-Tag: noai`, `X-Robots-Tag: noindex` , `X-Robots-Tag: noimageai` and `X-Robots-Tag: noimageindex` By default img2dataset will ignore images with such headers.
To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'
This does not pass the minimum thresholds of ethics for me. It may be legal, but it's not ethical. It's good that the default behaviour respects the headers. It's fine that there is an opt-out (if you didn't offer one, someone would just clone your code and offer it).
But if the most important information you provide is "here's how to use my tool unethically" rather than "here's why the default ethical use is important, though if you really have a good use case, here you go", then you're complicit in unethical use of this tool.
The idea that you should be able to publish information and then control how people use it is indeed popular, but claiming that failing to embrace this idea is inherently unethical … well, I’d say “citation needed” at a minimum.
I'm happy to discuss further, but "citation needed" makes it sound like you think I must prove to you with some facts of the validity of the position. That's not the conversation we need to have.
I don't think you can take a general summary of this situation, round off edges and nuance, and then say that I made the argument that this DIFFERENT scenario is "inherently unethical".
Ethics are nuanced and highly specific to the context, and the framing. A single word, phrasing, emphasis (or lack thereof) can change the scenario from an ethical to an unethical one, and vice versa.
For example, here is how I would phrase the exact same content in a way that i believe WOULD be ethical:
```
//First sentence unchanged
Websites can pass the http headers `X-Robots-Tag: noai`, `X-Robots-Tag: noindex` , `X-Robots-Tag: noimageai` and `X-Robots-Tag: noimageindex` By default img2dataset will ignore images with such headers.
//Then pull up the policy link first
To understand why image creators and artists may choose to specify such headers for their images, and choose to actively not consent to their images being collected, see [AI use impact](https://github.com/rom1504/img2dataset#ai-use-impact).
AI training would not be possible without the contribution of artists, and it is the recommendation of this tool's authors that you should respect their communicated wishes. However, if you have a legitimate reason to bypass these headers, to disable this behavior and download all images, you may pass --disallowed_header_directives '[]'
I wonder what happens when sites start using techniques like Zip/Compression bombs to defend against impolite mass downloaders. Particularly, in this case of downloading images without honoring robots.txt, a PNG/JPEG/ZIP bomb [1] might be a countermeasure in this arms race.
This PNG image <link> is an example of a compressed image file that might expand to a huge size upon decompression. PLEASE DO NOT DOWNLOAD IT. We are not responsible for consequences if you do not follow this instruction.
and someone does that anyways, they are to blame, aren't they?
Especially downloaders who claim they are entitled to download it and hosters must use their proprietary opt-out scheme (like the ones on the original linked article) should also learn that they should follow the hoster's instruction and NOT download the zip bomb.
(And the hoster also lists that specific file in robots.txt)
> AI art is democratizing art and letting hundred of millions of people express themselves through art. Making art much more prevalent and unique
My contemporary art major friend blows a vein every time they read something like this. It's clear what it tries to say, but that's not art. It's fancy pictures. Which isn't to say it's somehow wrong, or doesn't allow people to express themselves — but if your creative input is limited to some AI prompt, then the art, if anything, is the prompt itself.
(And, just to be clear, the definition of "art" here isn't "you have to paint something yourself". It's "you have to perform a creative action yourself". It's fine if that action is just instructions — that's why the prompt could be art.)
In a way, yes. But both for a photograph as well as AI-generated art, you'd want (or even need) to show you can in fact execute it. That's where the actual result comes together. (There's also slightly distinct terms to throw around here, art, artwork, creative spark, …)
The problem with most "AI art" is that it has absolutely no visible creative spark. It's entirely possible to create art with AI, just — millions of random images or text aren't it.
(btw: I'm trying to explain the understanding of "art" as it is [fuzzily] agreed upon by people active in the field. I may be getting things wrong, or explaining them badly. Of note, however, is that this isn't "just my personal opinion", I'm trying to convey an existing consensus of that field.)
I'm not really sure what the hope here is for the people in the thread. Okay so the author makes it opt-in by default. The first user downloads the tool, tries to run it, goes "oh, well that's useless" forks the repo deletes the header check and runs it. Now to block them you have to employ "big league" blocking techniques instead of setting a header.
Having it be opt-out actually makes it so server operators have the ability to turn it off because a few people doing it will go unnoticed.
It's like how Microsoft killed DNT by making it the default. Sites can be like "sure, we can support DNT for the users that are really privacy conscious" but once it's every user the rubber meets the road of ad revenue.
47 comments
[ 4.8 ms ] story [ 104 ms ] threadhttps://github.com/rom1504/img2dataset
https://github.com/rom1504/img2dataset#opt-out-directives
edent: OK, so every time someone builds a tool, I have to specifically opt out of it. Is that what you're saying?
rom1504: Yes, for example if you want to ban the tool "Google Chrome" from accessing your website, you can do so.
We have the inverse problem in the university. We are all (many thousands) behind a proxy, so some popular sites detect the avalanche of request and add a captcha or other anti abuse tool to everyone.
>> Websites can pass the http headers X-Robots-Tag: noai, X-Robots-Tag: noindex , X-Robots-Tag: noimageai and X-Robots-Tag: noimageindex By default img2dataset will ignore images with such headers.<<
>>To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'<<
1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure them to avoid the opt-out.
2) Add a general rate limit per ip, Bonus points for a te4mporal ban in case they don't understand the polite version.
What then happens is all requests to w3.org then redirect to a page explaining how anyone automatically downloading those schemas from those URLs became part of an accidental DDOS attack.
Considering that wouldn't a simple optimization be to shard by domain instead of arbitrarily?
If they (or legislators) just agreed that being part of training data was fair use, it would save a lot of servers a lot of bandwidth, and make it more convenient for people working with these datasets. It would be win/win, but there's a prisoners dilemma + misplaced greed that means it will never happen like that.
It's not though, and never has been. Why should it be?
Why should people contribute to making their own jobs obsolete and not even be compensated for it?
> It would be win/win
In what way would this be a win for artists?
Haven’t technologist been saying for years about blue collar workers whose jobs have been made obsolete by technology that they should learn a new skill?
Now that it’s hitting white collar workers, we are singing a different tune.
And most of the time especially here on HN most people want an expansive definition of fair use as long as it is directed toward using work created by the big media companies.
So by useless you mean absolutely essential?
Hard to argue the fair use thing but it was designed around humans not AI so just because it is fair use now does not mean it should or will be in the future.
I dare say, this thought that people so well read they could architect a country were not aware of the State of the Art in weaponry is rather naive.
Gatling Guns came well after the Founding, were large, heavy, hand cranked, cost between $1000 to $1500 to purchase and required a four man crew to operate. I don’t feel they can be compared to modern weapons.
The Founders were intelligent and capable men but they were not infallible. They knew that too, that’s why Amendments exist.
I can see an argument over nukes, but this "they didn't imagine assault weapons" is just some kind of gaslighting. They imagined sufficient arms to defend oneself against authority.
``` Opt-out directives
Websites can pass the http headers `X-Robots-Tag: noai`, `X-Robots-Tag: noindex` , `X-Robots-Tag: noimageai` and `X-Robots-Tag: noimageindex` By default img2dataset will ignore images with such headers.
To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'
See [AI use impact](https://github.com/rom1504/img2dataset#ai-use-impact) to understand better why you may decide to enable or disable this feature. ```
This does not pass the minimum thresholds of ethics for me. It may be legal, but it's not ethical. It's good that the default behaviour respects the headers. It's fine that there is an opt-out (if you didn't offer one, someone would just clone your code and offer it).
But if the most important information you provide is "here's how to use my tool unethically" rather than "here's why the default ethical use is important, though if you really have a good use case, here you go", then you're complicit in unethical use of this tool.
I'm happy to discuss further, but "citation needed" makes it sound like you think I must prove to you with some facts of the validity of the position. That's not the conversation we need to have.
I don't think you can take a general summary of this situation, round off edges and nuance, and then say that I made the argument that this DIFFERENT scenario is "inherently unethical".
Ethics are nuanced and highly specific to the context, and the framing. A single word, phrasing, emphasis (or lack thereof) can change the scenario from an ethical to an unethical one, and vice versa.
For example, here is how I would phrase the exact same content in a way that i believe WOULD be ethical:
``` //First sentence unchanged
Websites can pass the http headers `X-Robots-Tag: noai`, `X-Robots-Tag: noindex` , `X-Robots-Tag: noimageai` and `X-Robots-Tag: noimageindex` By default img2dataset will ignore images with such headers.
//Then pull up the policy link first
To understand why image creators and artists may choose to specify such headers for their images, and choose to actively not consent to their images being collected, see [AI use impact](https://github.com/rom1504/img2dataset#ai-use-impact).
AI training would not be possible without the contribution of artists, and it is the recommendation of this tool's authors that you should respect their communicated wishes. However, if you have a legitimate reason to bypass these headers, to disable this behavior and download all images, you may pass --disallowed_header_directives '[]'
```
[1]: https://www.bamsoftware.com/hacks/deflate.html
WARNING
This PNG image <link> is an example of a compressed image file that might expand to a huge size upon decompression. PLEASE DO NOT DOWNLOAD IT. We are not responsible for consequences if you do not follow this instruction.
and someone does that anyways, they are to blame, aren't they?
Especially downloaders who claim they are entitled to download it and hosters must use their proprietary opt-out scheme (like the ones on the original linked article) should also learn that they should follow the hoster's instruction and NOT download the zip bomb.
(And the hoster also lists that specific file in robots.txt)
> AI art is democratizing art and letting hundred of millions of people express themselves through art. Making art much more prevalent and unique
My contemporary art major friend blows a vein every time they read something like this. It's clear what it tries to say, but that's not art. It's fancy pictures. Which isn't to say it's somehow wrong, or doesn't allow people to express themselves — but if your creative input is limited to some AI prompt, then the art, if anything, is the prompt itself.
(And, just to be clear, the definition of "art" here isn't "you have to paint something yourself". It's "you have to perform a creative action yourself". It's fine if that action is just instructions — that's why the prompt could be art.)
The problem with most "AI art" is that it has absolutely no visible creative spark. It's entirely possible to create art with AI, just — millions of random images or text aren't it.
(btw: I'm trying to explain the understanding of "art" as it is [fuzzily] agreed upon by people active in the field. I may be getting things wrong, or explaining them badly. Of note, however, is that this isn't "just my personal opinion", I'm trying to convey an existing consensus of that field.)
The artist's unmade bed says hello!
1) Selecting the prompt
2) Selecting the output
Having it be opt-out actually makes it so server operators have the ability to turn it off because a few people doing it will go unnoticed.
It's like how Microsoft killed DNT by making it the default. Sites can be like "sure, we can support DNT for the users that are really privacy conscious" but once it's every user the rubber meets the road of ad revenue.
Detect the scraper's IP/fingerprint and return deplorable porn instead of the original image.
Practices like this are a big part of the reason the EU and California adopted GDPR.
The creator is clearly unethical and too lazy to ask for permission.