I'm using replicate as the AI backend. They're the ones that are flagging the content. There's probably some tuning that I can do but I haven't dug into it.
You'll have to retry, rephrase, or edit the image to get past that.
The NSFW filter is in the stable-diffusion library you're calling. Disable the line that sets image, has_nsfw_concept in image2image.py, (or whatever your code calling the library looks like).
I don't understand how these various AI companies even try to filter NSFW content. The world contains NSFW, trying to avoid it by "filtering" is a effort not worth taking, the undertaking is huge and filled with pitfalls and false-positives.
I think they are aware they can't filter it, they just don't want to be the first company with a large scandal because of e.g. some celebrity/politician deepfakes going mainstream.
Doesn’t matter, they will still be blamed. Anything bad that big tech does becomes a story because it’s them doing it. A startup might not be noticed at first, but it can be blamed on “a Silicon Valley startup.”
The reason I saw mentioned is that NSFW content was so prevalent in their training set that, if they did not explicitly exclude it, it was actually difficult for it to NOT produce NSFW results even from normal inputs. For example, if you ask for anything with a woman in it, it would probably give you a naked woman in the results.
This exact example made me wonder how the 'bias' situation will ever be worked around (besides training the model to an individuals preexisting biases).
What should the picture of a 'woman' on wikipedia be? Should they be clothed? In western clothing? What skin color? Age? Surely everyone imagines something slightly different when they think 'woman'.
Which is close to what I would've guessed by picking a high population density area, and a likely location and candidate for a decent portait photo on a wildly popular wiki.
Still not sure why the top was edited from red to grey.
Perfect example: English Wikipedia has bikeshedded lead images of this kind ad nauseum over these two decades, with many times more text written in discussions of this than in the articles themselves. The current image for [[woman]] was chosen last year in an RfC that ended with a rare counted vote.¹ This prompted a re-debate of [[human]]² that trailed off in dissastisfaction toward the only established guideline in the area, which forbids montages. That, in turn was last affirmed by a series of debates in 2015/2016.³
No, if you ask SD without a filter for a woman, it won't probably give you a naked woman. It is actually rather rare, but I don't see what's so horrible when it does happen.
When you go to a gallery you will see a lot of naked people. It's not a big deal.
- naked and posed in a way that is artistically beautiful
- naked and posed in a way that is sexually provocative
SD is convinced past a certain threshold of flesh-colored space that it could fall into the sexually proactive group so they err on the side of caution. I'm not mad at them for defaulting to that. If you want to push past the default filter it's easy enough to remove. Seems sensible to me when the training set does include a lot of literal pornography.
I have yet to see an innocent prompt that results in a naked person posed in a sexually provocative way, but if someone discovered one, please post it along with a seed.
I don't fault SD for having a simply removable filter. I was ranting about the applications built on top of stable diffusion that force the filter to be engaged.
(But of course everyone is free to create the software they wish, I'm not entitled to complain it should work differently.)
I use SD with no filter and I wouldn't say it's rare. I do think some words trigger it more often than others. For example, I've seen a prompt with "beautiful" turn up half the results with nudity, whereas removing the word I got no nudity.
`Beautiful` is already pushing the model toward sexualizing her, I mean the subject is going to be beautiful by default since the model was trained on LAION Aesthetics.
But yes, `a beautiful woman` is a fairly innocent prompt. My experience (also using SD without the filter all the time) is that the model very rarely produces sexual results unless intentionally pushed that way. Perhaps our mileages vary.
Galleries aren't typically capable or in the business of producing millions of "naked pictures of celebrity x"? The body should be normalized but let's not think these tools used en masse will have that effect.
"Beautiful woman" gives you nudity roughly 90% of the time.
In a recent open office hours, they basically said that without the filters, everything quickly becomes overrun with either gore or things that verge on porn, because it only takes a small fraction users to flood every available space.
In the meantime, they've watched the gender demographics on periodic user surveys tilt dramatically toward men more recently, and they're concerned this might be one factor in that.
There are a billion different ways to generate porn images, and billions of porn images already existing. They'd rather SD not be a source of more.
>"Beautiful woman" gives you nudity roughly 90% of the time.
This is simply not factual. I've just tried `Beautiful woman` 64 times on Stable Diffusion v1.4:
- 0 results were fully nude
- 8 results were topless and sexually explicit (I counted pictures without nipples as well)
- 2 results were topless with breasts covered by hands in a facebook friendly way
12.5% is very far from 90%. Even then, what kind of a prompt is that anyway? The only attribute you give to the subject is sexualizing her. More descriptive prompts would contain even less nudity.
>They'd rather SD not be a source of more.
This isn't what their founder says[0]. Anyway it doesn't matter since SD already released with an easy to disable filter. We are not talking about whether SD should include an optional filter (I agree it should), but whether other apps built on top of it should have a non-optional filter.
My bad! I mixed up Stable Diffusion and Midjourney in my head. MJ has the open office hours and is worried about gore and porn and so on, not Stable Diffusion.
YMMV - I’ve been fuzzing permutations of various prompt phrases to come up with some art for a female D&D character. “Temptress”, “seductress”, “dominatrix”, and “femme fatale” are on my list of 22 phrases, and I’m only getting ~30% NSFW outputs - pretty much all of which are just weirdly suggestive curves in random places, vaguely phallic silhouettes, and artsy breast shapes.
Probably not stuff that should be shown to kids, but in the >1k samples I’ve run so far, I’ve yet to see anything jarring.
Are those surveys self-reported/verified or algorithmic assumptions? I feel like a lot of algorithms and criticisms assume default/normal/non-characteristic person as men, and are wrong because of that.
From the office hours, it's a self-reported quarterly survey, IIRC. So they're not sure how much of the growing skew is due to the demographics of Discord itself, how much is real, etc.
SD has rather specific ideas of what women look like. They don't necessarily resemble the average real woman. Instagram is already giving little girls eating disorders, maybe such idealized depictions of women's bodies isn't healthy for society
The LAION-5B dataset was curated by researchers at Heidelberg University, and while it was filtered for nsfw content, the human form is less of an issue in much of Europe compared to views and attitudes in the US.
> fake content could be something that minimizes their damage
I don't claim to have any knowledge in the subject area, but it seems plausible that feeding habits of those who enjoy child pornography could instead embolden them to commit actions they would not have otherwise. At the very least, this seems like something to approach with extreme caution.
> At the very least, this seems like something to approach with extreme caution.
Like you, I'm not overly informed on the factual question here, but I don't understand why you think banning the content is somehow the cautious course of action.
The heuristics for which action is the "cautious" one is usually some combination of:
a) which outcome is less likely to be harmful
b) which outcome is more passive (ie primum non nocere)
c) which outcome is more predictable (eg the status quo)
It seems to me that bans fail on the first two counts[1] and squeak by on the third. Even this status quo argument is weakened by how new these bans are, and by the many contemporary countries in which fictionalized CP is legal.
[1] In the former case, this is based on the practically universal consensus that sexual preferences, even dangerous and deviant ones, are burned deeply into one's psyche and can't simply be ignored through lack of exposure. This view was most famously one of the strongest pillars of the successful iteration of the gay rights movement, and is why we consider conversion therapy to be horrific. One can make the claim that this factual claim only applies to gender preferences, but that seems quite an extraordinary claim.
> I don't understand why you think banning the content is somehow the cautious course of action.
I didn't intend to say that I think banning is the cautious course of action. My point was that the decision to ban or not ban should be approached carefully.
Ah, sorry for incorrectly assuming. I fully agree with you.
It's a very common tendency for people to smuggle their moral intuitions into calls for the precautionary principle, and this runs directly counter to _actually_ approaching these discussions with caution.
They're responding to a hysterical, Luddite portion of the public that would make them pay a PR cost if they didn't at least make a sloppy attempt at "safety". It's idiotic and futile, but blessedly short-lived.
Thanks to the miracle of open-source, it took me about 20 minutes yesterday to get Stable Diffusion up and running on my laptop's CPU, with no content filter. SD inference has low enough compute needs that using cloud GPU compute is cheap as hell (for now, perhaps), and discrete GPUs aren't that expensive in the grand scheme of things.
Ultimately, reality often does win. It's just not a stable equilibrium for companies to continue these laughable attempts at "filtering", any more than physical media formats were able to withstand the arrival of digital formats in the presence of piracy. Combine this with the cultural headwinds against the influence of mentally-ill hysterics on social media, and I don't expect the current level of filtering to last very long.
As a central european I don't see what would be bad about showing my friends or co-workers a tool like this and accidentally producing NSFW results. If anything, we would find it funny.
This comment is not an argument, just a cultural exchange. I appreciate your feelings too and of course an option to toggle the filter is always the best.
Are any of your friends famous? Because if you use it to generate an image of said famous person naked, then you might see why some feel uncomfortable about the implications of this tech.
I highly doubt SD can generate a nude image of an actual famous person unless it is specifically asked to do that.
Even then, if you are showing DrawAnything to your famous friend and concerned about this, why would you choose to use your friend's name in the demo prompt?
Only the author knows their reasons for this specific case, but in general it seems internet services tend to follow american conservative views on nudity. I mean why is it strictly forbidden to post a topless photo on almost any mainstream social network again?
Another reason is liability, in case the model truly happens to generate something that might be illegal in author's jurisdiction and they are afraid of ever so shortly storing the image on their server.
This is just something you have to accept, because if these sites did not block not safe for work content everyone using them would almost immediately start using it to generate nude images of the people they know in real life and other very scummy things.
In the long run this is something we're just going to have to learn to deal with as a society, here is a new power we have and it's not going back in the bottle.
But in the short run having these filters is a good way to slow down the inevitable and give us a little bit more time to adapt to this new reality.
In the meantime, you can run it yourself and I think that's a perfectly fine compromise.
> everyone using them would almost immediately start using it to generate nude images of the people they know in real life and other very scummy things.
Even if this were true, I don't see what is scummy about it. Is it scummy for you to imagine people naked? If it's not scummy for you to do it, why is it scummy for you to use a computer to do it?
It's not like you're getting actual photographs. It's all made up.
The scummy part is that they are physical concrete images that you can share with other people.
Imagination cannot escape your head, but these images can.
The threat is somewhat unique from something like Photoshop as well, because the barrier to entry is much much lower quality can be much much higher.
Right now, at this transfer point, having nude images of yourself all over the internet when you did no such picture taking is going to be really tragic and infuriating and frustrating and harmful.
I kind of agree with you, it doesn't necessarily hurt anyone, but this is something we're going to have to adapt and learn how to deal with, not something we can just be reckless with.
>The threat is somewhat unique from something like Photoshop as well, because the barrier to entry is much much lower quality can be much much higher.
I disagree. The quality in an image editor would be much much higher and more consistent. The barrier to entry of using an image editor in such a way is very low. The barrier to entry to fine tune a model to someone you know is very high.
Imagination (in the sense of "mind images" not creativity) is not universal, and even those with the capacity, the image is usually less distinct than a real image would be.
Plus, people in general find images convincing: In some cases you can even get people to believe impossible things, like meeting Bugs Bunny at Disney World as a kid if you give them a photoshopped picture of it.
You can download the source and run it locally (or in Google Colab for a few bucks) if that's what you wanted to do. The nsfw filter is one function call you can short-circuit with a 'return False'.
I think the bigger issue is liability for these companies because they have to host the images in order to generate and display them. There's probably an argument that they can do this without liability, but that's an argument for an expensive lawyer to make, which they probably don't want to pay for.
SD had a single user show up and discover how good the model was at generating gore, and suddenly many, many users showed up to generate gory pictures, and every channel was filled with gory pictures until they put in overly-broad keyword filters to stem the flood.
They've been really open about why they're doing what they're doing on the regularly weekly "office hours" Discord audio chats.
I’m getting NSFW warning. On the sample prompt provided by the app itself:
> Photo realistic oil painting of a baby wearing a bib sitting at a wooden table in front of an egg. Painting by Leonardo da Vinci in the style tof mona lisa (1519). The baby has a Laughing expression. Medium shot with a tilted frame. The person is to the left of the table. Warm lighting. Photorealism.
The prompt "A ten year old boy daydreaming" is a bit haunted, as it's not specific enough to generate good results.
But the results from running that does not result in NSFW content, contrary to what you say. Most of these NSFW "blockers" simply act on keywords like "ten year old boy" rather than trying to understand the output.
Here is 9 images of the prompt "A ten year old boy daydreaming" with various settings. None of them are good (bad prompt + no tuning on my side), but none of them are NSFW either: https://imgur.com/a/FcBv05w
(slightly off-topic, seems there is a lot of misinformation in this thread, from misunderstanding the training data, to how the NSFW content block some platforms are implementing works)
I was just being snarky. I didn’t actually think the content filter would take into account what the model would actually output. I thought it probably was just a really bad keyword based filter.
Thanks for sharing these examples. Good to see what the model actually produces with that prompt.
More likely, the NSFW classifier has a lot of false positives because that’s the only way to effectively prevent those outputs, even though it causes it to detect nsfw when there is none.
Well, it is true that a 10 year old can be quite distracting at work. However, so is playing with stable diffusion models in general. (Which is probably why we are getting that message on basically every prompt.)
with literally hundreds of competitors popping up daily you cannot afford to have an overly sensitive nsfw filter; right now this tool is almost useless
But yeah, bounce. I think this will be a big enough factor in the “AI” thing that whoever comes up with a filter combo to just not generate things Americans fear, regardless of the input, will win big.
I recently made a game pretty much just like this. It's like a game of Drawful mixed with telephone so you can see how the image changes through cycles of AI drawing and human interpretation.
Photo realistic oil painting of a baby wearing a bib sitting at a wooden table in front of an egg. Painting by Leonardo da Vinci in the style tof mona lisa (1519). The baby has a Laughing expression. Medium shot with a tilted frame. The person is to the left of the table. Warm lighting. Photorealism.
113 comments
[ 3.0 ms ] story [ 178 ms ] threadYou'll have to retry, rephrase, or edit the image to get past that.
What should the picture of a 'woman' on wikipedia be? Should they be clothed? In western clothing? What skin color? Age? Surely everyone imagines something slightly different when they think 'woman'.
Which is close to what I would've guessed by picking a high population density area, and a likely location and candidate for a decent portait photo on a wildly popular wiki.
Still not sure why the top was edited from red to grey.
¹ https://en.wikipedia.org/wiki/Talk:Woman/Archive_19 ² https://en.wikipedia.org/wiki/Talk:Human/Archive_35#Argument... ³ https://en.wikipedia.org/wiki/MOS:NOETHNICGALLERIES
When you go to a gallery you will see a lot of naked people. It's not a big deal.
- naked because that's the default human
- naked and posed in a way that is artistically beautiful
- naked and posed in a way that is sexually provocative
SD is convinced past a certain threshold of flesh-colored space that it could fall into the sexually proactive group so they err on the side of caution. I'm not mad at them for defaulting to that. If you want to push past the default filter it's easy enough to remove. Seems sensible to me when the training set does include a lot of literal pornography.
I don't fault SD for having a simply removable filter. I was ranting about the applications built on top of stable diffusion that force the filter to be engaged.
(But of course everyone is free to create the software they wish, I'm not entitled to complain it should work differently.)
But yes, `a beautiful woman` is a fairly innocent prompt. My experience (also using SD without the filter all the time) is that the model very rarely produces sexual results unless intentionally pushed that way. Perhaps our mileages vary.
https://pixlr.com/ is far better suited for producing naked picture of celebrity x than https://www.drawanything.app/
In a recent open office hours, they basically said that without the filters, everything quickly becomes overrun with either gore or things that verge on porn, because it only takes a small fraction users to flood every available space.
In the meantime, they've watched the gender demographics on periodic user surveys tilt dramatically toward men more recently, and they're concerned this might be one factor in that.
There are a billion different ways to generate porn images, and billions of porn images already existing. They'd rather SD not be a source of more.
This is simply not factual. I've just tried `Beautiful woman` 64 times on Stable Diffusion v1.4:
- 0 results were fully nude
- 8 results were topless and sexually explicit (I counted pictures without nipples as well)
- 2 results were topless with breasts covered by hands in a facebook friendly way
12.5% is very far from 90%. Even then, what kind of a prompt is that anyway? The only attribute you give to the subject is sexualizing her. More descriptive prompts would contain even less nudity.
>They'd rather SD not be a source of more.
This isn't what their founder says[0]. Anyway it doesn't matter since SD already released with an easy to disable filter. We are not talking about whether SD should include an optional filter (I agree it should), but whether other apps built on top of it should have a non-optional filter.
[0] - https://twitter.com/EMostaque/status/1562200385401499650
Probably not stuff that should be shown to kids, but in the >1k samples I’ve run so far, I’ve yet to see anything jarring.
(it kind of sucks because assuming you can't eliminate pedophiles, fake content could be something that minimizes their damage)
I wonder how we will get around this one. Or do we get stuck and human artists will start only creating NSFW art? ;)
I don't claim to have any knowledge in the subject area, but it seems plausible that feeding habits of those who enjoy child pornography could instead embolden them to commit actions they would not have otherwise. At the very least, this seems like something to approach with extreme caution.
Like you, I'm not overly informed on the factual question here, but I don't understand why you think banning the content is somehow the cautious course of action.
The heuristics for which action is the "cautious" one is usually some combination of:
a) which outcome is less likely to be harmful b) which outcome is more passive (ie primum non nocere) c) which outcome is more predictable (eg the status quo)
It seems to me that bans fail on the first two counts[1] and squeak by on the third. Even this status quo argument is weakened by how new these bans are, and by the many contemporary countries in which fictionalized CP is legal.
[1] In the former case, this is based on the practically universal consensus that sexual preferences, even dangerous and deviant ones, are burned deeply into one's psyche and can't simply be ignored through lack of exposure. This view was most famously one of the strongest pillars of the successful iteration of the gay rights movement, and is why we consider conversion therapy to be horrific. One can make the claim that this factual claim only applies to gender preferences, but that seems quite an extraordinary claim.
I didn't intend to say that I think banning is the cautious course of action. My point was that the decision to ban or not ban should be approached carefully.
It's a very common tendency for people to smuggle their moral intuitions into calls for the precautionary principle, and this runs directly counter to _actually_ approaching these discussions with caution.
Thanks to the miracle of open-source, it took me about 20 minutes yesterday to get Stable Diffusion up and running on my laptop's CPU, with no content filter. SD inference has low enough compute needs that using cloud GPU compute is cheap as hell (for now, perhaps), and discrete GPUs aren't that expensive in the grand scheme of things.
Ultimately, reality often does win. It's just not a stable equilibrium for companies to continue these laughable attempts at "filtering", any more than physical media formats were able to withstand the arrival of digital formats in the presence of piracy. Combine this with the cultural headwinds against the influence of mentally-ill hysterics on social media, and I don't expect the current level of filtering to last very long.
smiling forest, photorealism: nsfw detected
Tech:
SvelteKit
Supabase
Replicate
Fabric JS/TLDraw
This was my Labour Day weekend hack. Source is gross. I'm going to keep it off my Github for now. But it's not hard to build from scratch.
Tech:
SvelteKit
Supabase
Replicate
Fabric JS/TLDraw
I did not intend for this to drain my wallet so I may start limiting requests to authenticated users.
I'm going to be posting updates on my Twitter[0].
[0]: https://twitter.com/KieranGilliam
Other than that, both the idea and the app is pretty neat, congratulations!
What I want more than anything is to see what kind of horrifying H.R. Geiger imagery these things can come up with.
> What I want more than anything is to see what kind of horrifying H.R. Geiger imagery these things can come up with.
Why does Google have a safe mode? Sorry, people should have the option for unwanted content.
I would like to show these projects off to friends and co-workers without the risk of generating explicit content.
These generators need settings to toggle such behavior.
This comment is not an argument, just a cultural exchange. I appreciate your feelings too and of course an option to toggle the filter is always the best.
Even then, if you are showing DrawAnything to your famous friend and concerned about this, why would you choose to use your friend's name in the demo prompt?
Only the author knows their reasons for this specific case, but in general it seems internet services tend to follow american conservative views on nudity. I mean why is it strictly forbidden to post a topless photo on almost any mainstream social network again?
Another reason is liability, in case the model truly happens to generate something that might be illegal in author's jurisdiction and they are afraid of ever so shortly storing the image on their server.
In the long run this is something we're just going to have to learn to deal with as a society, here is a new power we have and it's not going back in the bottle.
But in the short run having these filters is a good way to slow down the inevitable and give us a little bit more time to adapt to this new reality.
In the meantime, you can run it yourself and I think that's a perfectly fine compromise.
Even if this were true, I don't see what is scummy about it. Is it scummy for you to imagine people naked? If it's not scummy for you to do it, why is it scummy for you to use a computer to do it?
It's not like you're getting actual photographs. It's all made up.
The threat is somewhat unique from something like Photoshop as well, because the barrier to entry is much much lower quality can be much much higher.
Right now, at this transfer point, having nude images of yourself all over the internet when you did no such picture taking is going to be really tragic and infuriating and frustrating and harmful.
I kind of agree with you, it doesn't necessarily hurt anyone, but this is something we're going to have to adapt and learn how to deal with, not something we can just be reckless with.
I disagree. The quality in an image editor would be much much higher and more consistent. The barrier to entry of using an image editor in such a way is very low. The barrier to entry to fine tune a model to someone you know is very high.
I agree with the rest of what you say.
Plus, people in general find images convincing: In some cases you can even get people to believe impossible things, like meeting Bugs Bunny at Disney World as a kid if you give them a photoshopped picture of it.
I think the bigger issue is liability for these companies because they have to host the images in order to generate and display them. There's probably an argument that they can do this without liability, but that's an argument for an expensive lawyer to make, which they probably don't want to pay for.
They've been really open about why they're doing what they're doing on the regularly weekly "office hours" Discord audio chats.
so I changed it to man and it generated something totally random and unrelated.
I'm not sure that this stable diffusion tool can replace illustrators in this current state but more of a tool they can utilize or not.
> Photo realistic oil painting of a baby wearing a bib sitting at a wooden table in front of an egg. Painting by Leonardo da Vinci in the style tof mona lisa (1519). The baby has a Laughing expression. Medium shot with a tilted frame. The person is to the left of the table. Warm lighting. Photorealism.
But the results from running that does not result in NSFW content, contrary to what you say. Most of these NSFW "blockers" simply act on keywords like "ten year old boy" rather than trying to understand the output.
Here is 9 images of the prompt "A ten year old boy daydreaming" with various settings. None of them are good (bad prompt + no tuning on my side), but none of them are NSFW either: https://imgur.com/a/FcBv05w
(slightly off-topic, seems there is a lot of misinformation in this thread, from misunderstanding the training data, to how the NSFW content block some platforms are implementing works)
Thanks for sharing these examples. Good to see what the model actually produces with that prompt.
;)
"a river between mountains"
Could not generate: NSFW content detected, please try a different prompt
https://i.vgy.me/VEYd13.png
omg i laughed so loud :D
so realistic, Leonardo would be proud :D
https://i.vgy.me/OvbtNs.png
https://imgur.com/a/GXaf9kH
But yeah, bounce. I think this will be a big enough factor in the “AI” thing that whoever comes up with a filter combo to just not generate things Americans fear, regardless of the input, will win big.
Like Drawful [1] or Quickplash [2], but enhanced by AI. Everyone gets a task and has to come up with the best prompt or drawing.
I feel like this would be fun so play, even if you can't draw.
[1] https://www.jackboxgames.com/drawful-two/
[2] https://www.jackboxgames.com/quiplash/
http://www.dallestrations.com/