Heh, remember when you signed up and still bought into Google's mission is to organize the world's information and make it universally accessible and useful.
It's kinda sad that Google desperately wants the cool points for having its own DL models but you can only see it in the form of a store window display.
> Heh, remember when you signed up and still bought into Google's mission is to organize the world's information and make it universally accessible and useful.
They probably also remember signing an NDA, and maybe taking some training about how not all of the world's information should be made universally accessible and useful to everyone. For instance, the contents of a user's inbox.
I just tried the prompts with Imagen and Parti. They are similar to Dalle-2, with a bit more "variety" but none reproducing the author's specific prompt the way the author wants. For the prompt "a graph with 3 lines" both produce graphs with 3 lines at least 1/6 of the time.
Curious. I've played with quite a few of these models and one of the very consistent "tells" is that they're extremely bad at counting things. A friend of mine likes tarot and I tried a few prompts... great results for the major arcana, but good luck with "ten of cups"... without capability to edit & re-prompt, the only viable strategy appears to be "ask it to draw a bunch of cups repeatedly until you've collected all the numbers."
Getting 3 of something 1/6 of the time doesn't really sound like it groks the request.
Sure, but it doesn't have to count to be useful. When I run this locally on desktop GPU, I get 16 results in a few seconds. I can visually select the 2-3 that match what I want and pick one of them. I can try again ten times for 20-30 options, and it still takes less time than Fiverr.
I would not use these models for graphs yet, but for cool look tarot inspired "clipart" or background images, I think they are already usable.
The inability to count has some interesting knock-on effects. My favorite being eyes and hands. Especially the hands. The closer you look, the creepier it gets. That thumb has fingers of its own?! Thus far, the greatest utility I've seen has been on par with B-movies.
I am somewhat surprised at how bad these tools are at generating hands and feet. Is it just a matter of not having enough images to ingest?
The faces often look very good and they also have symmetric complexity and individual elements that come in a specific quantity (2 eyes, etc). Lower quality models do generate fly-like multi eye faces, but newer ones are so much more precise!
It's not at all surprising that an AI is bad at drawing graphs, and it is also not surprising that even a non-artist human can draw graphs pretty well.
Why is it not surprising? I don't see any fundamental reason for it. I think these models will be able to produce sensible graphs fairly soon.
You could equally say "it's not surprising that DALL-E can't draw words"... except that Imagen seems to be pretty good at it.
I think the real reason it's not surprising to you is that you've already seen enough DALL-E results to understand its limitations. It's not surprising that DALL-E can't draw graphs.
We don't know if it's surprising, the author never tells us their hypothesis. They don't state any particular reason for the prompt they used, they don't explore the contents or qualities of the prompt compared to other AI-generated art, and they don't run multiple trials. Because of that, we can't conclude anything useful from this article. There's no frame of reference or scientific inquiry involved. If you find it entertaining, that's fine. As a scientific comparison, this verges on parody.
From reading the comments here, they're overthinking it because they seem to be taking this as a pre-planned "attack" on AI art generation, rather than just an interesting anecdote on the limitations of these tools.
As someone who has not played with said tools, it was an outcome I found interesting to know: DALL-E et al. can't do specific graphs or even specific logical things very well a lot of the time. That's good to know, and I didn't previously!
I played with these tools a lot, so expected bad results as soon as I read the prompt.
Still found the post really interesting as it explores a very realistic use case. A client needs something simple designed for a blog post. Should they use AI or a human designer?
I read somewhere in the comments here that these tools are very bad at counting. Which is an interesting limitation with far reaching implications.
> I read somewhere in the comments here that these tools are very bad at counting. Which is an interesting limitation with far reaching implications.
It may not make much difference, but it's not so much that they're bad at counting, as that they don't even try. The way the prompt is parsed and diffused doesn't allow for that sort of logic.
All a "two people" prompt or some such provides, is a hint to push the AI towards that section of latent space where training-set images titled "two people" exists.
That's not "counting", and it would be truly amazing if any sense of math emerged de-novo from this training process. Doesn't mean it can't be done — it means we aren't trying.
I think it's just a bad experiment. The author must have been truly ignorant about the capabilities and utilities of DALL-E if he thought this experiment would yield interesting results.
I am personally annoyed because I expected something more sensible from an article called “DALL·E 2 vs. $10 Fiverr Commissions”.
The author could also compare how well DALL·E draws text, but what would be the point of that? Is not being a scientific article a good defense for posting nonsense?
I do see a fundamental reason: The current crop of AI tools are horrible at logic. It's a complete inversion of how we think of computers.
If I want to convey happy emotions in the style of Rembrandt, SD or DALL-E will do brilliantly. If I want an apple BELOW a table, or worse, a geometric shape like a triangle, they'll crash-and-burn.
GPT-3 is also really empathetic, but struggles simple logic (and especially mathematics).
Graphs are like the horror case for these.
I can think of ways to make them better at this, but it's not a weekend of hacking.
One missing dimension for this comparison is how long it takes to get results.
I've been using Imagen/Parti/Stable Diffusion already as a replacement for "clip art" because it takes ~15 seconds to get results and they are free. Fiverr takes at least 100 times that long and costs $10.
For tasks where the exact content isn't important and you can't invest more than a few seconds or wait a more than a few seconds for results, generative models are already a great solution.
Drawing graphs is probably one of the worst comparisons one can do in terms of evaluating these models. They seem to be trained to generate either photorealistic or stylized images.
I think the path forward for something like this is models that learn to execute python code and incorporate the results into their outputs. There are already projects that can generate correct matplotlib calls for prompts like yours, but I don't think we are to the point where those python outputs can be automatically combined with a diffusion model for style or whatever.
It's not about it being theoretical, it's moreso that the language model is still far more simplistic than our own, and struggles with anything but the most basic relations between nouns. The "horse riding an astronaut" post is a good example of this.[0]
Did you even read the linked article? They even cite the picture you link, it still proves their point though these models have no understanding of the language (like Google claimed).
One thing they're not very good at is deducing spatial relationships. Concepts like "above", "inside" and "behind", or "after". I'd say the prompts you gave make sense to a human who is thinking of a visual progression from left to right.
I bet you could write a few copilot prompts to generate code which would draw a graph like this, though.
I was surprised while reading your article how good the AIs did. I think it's fascinating that your intuition was that these tools would be able to do a good job of drawing a graph based on a description of it...
After reading your comment I asked Stable Diffusion to create photorealistic images of a graph with three lines (similar to the smallest prompt in the article).
Here's the results of three attempts with slightly different prompts:
Cool article! I think on a more general note it underlines what I have been thinking for some time now: most people, also on HN, get this generative art stuff totally wrong.
Yes, this will be the end for some artists but not for others. DALLE2 et al. are merely new tools for new generation of artists. And, we are still figuring out how to use these tools effectively.
In other words: The “AI” is a tool that we humans will use to get things done faster/better etc. Nothing less, but also not much more.
To your point, I read in a board game group the other day of an artist using AI generation for a first pass with new clients to save themselves time. "Is this close to what you want?" then does the actual art with alterations by hand. If the client says no or flakes out, a lot less effort was lost than before.
The more worrisome aspect for me isn’t that it’s already going to replace some artists.
I always thought in my head that this level of creativity would remain our domain for centuries. Even as of like two or three years ago I thought that.
It’s insane to me that today some artists feel they’re going to be replaced soon. The idea of centuries is completely shattered for me and now I don’t know if we’re a year or 50 years away from AI replacing humans entirely in the creative domain. I spent the other day completely in an existential crisis, tbh.
The only reason an artist would think they’re being replaced is someone told them they would. So the solution is to not tell them that, as it’s not true.
(The main instigator on Twitter is a guy who draws “realistic Pokemon” and hates that an AI may have stolen the art he already stole from The Pokemon Company.)
>I always thought in my head that this level of creativity would remain our domain for centuries.
From what I've seen these networks are rehashing learning set images into something matching some criteria to produce visually pleasing results. Not to belittle the results - it's impressive - but the stuff I'm not seeing here is understanding of generated material - nonsensical z-order, scale/proportions, configuration.
Fantasy images are an easy target because it's all about visually pleasing nonsense.
From what I've seen, these tools aren't making _new_ styles (yet? I guess they will eventually, now that would be existentially fascinating/horrifying) -- so my worry with them is that they'll basically lock us in to what we have today.
But then that's sort of a self-limiting factor: it means theres still space for human creativity in creating new things, new styles (as not every style exists yet!) -- at least until said new style gets loaded into the model, I suppose?
I think people overlook the fact that there's more to a photo than shutter speed, more to a comic than the drawing... There's message, composition, design, focused iteration, etc, etc.
I really enjoy using DALLE for simulating photographs as it forces me to think outside of just the viewfinder.
While this is true to a certain extent, you're probably underselling it a bit. A lot of "evocative" art (e.g. art on Magic The Gathering cards) can now be done by complete non-artists playing around with prompts for a little while, in less time than it would take a professional artist to make the art manually.
Now, if you're actually Wizards of the Coast, you probably wanna spend the money with real artists anyway, but for any smaller teams, I can see the appeal of just using AI for that kind of use case now.
Complicated generative models are the wrong tool for the job here.. And arguable Fiverr commissions are too, these graph prompts look like they would take about the same amount of time it took to write the prompt to do in a vector art program once you got some beginner skills in one. To me this is almost like asking it to graph functions and comparing it to excel's graphing tools.
This is a great idea executed very poorly. I would love to see a larger sample size of AI vs Fiverr with a wider range of prompts. Graphs are difficult for current models and that was already well understood.
These text-to-graph problems seem like a good candidate for someone to create a training-dataset/benchmark of.
Bear in mind that the training data for these models has been mostly images and their alt text, scraped off the web. There is a good chance that there's nothing remotely like the examples given here in the training data. (People don't caption their graphs like that.) These models are undoubtably good at doing what they have been trained to do - but I think no-one disagrees that there's plenty of room for improvement.
(And bear in mind that these text2image models only released this year, and that this tech in general has only been invented in the last couple of years, so it's very early days...)
Thanks for sharing. Interestingly, I wrote a blog post about a similar topic: What will happen if ML builders and domain experts had co-ownership of the data and the model.* I am planning to generate the first training seed images by using Fiverr and giving the logo designers ownership rights of the data/model/profits.
* vs the current trend of training diffusion models on 400M images from the Internet (many of them being garbage) with mixed licenses and letting the user take responsibility of the generated images licensing issues.
The prompt he did is almost a captcha lol in terms of difficulty for AI vs human. Try a painting... Do a video game concept art or magic the gathering card. See how long it takes a human vs AI. The results are way farther to the side of AI, such that he might find it cost prohibitive to try to commission people to do it on fiverr for a blog post.
How is this not the right tool? Where on DALLE-2 website does it say that it should not be used for artistic graphs?
Yes, we all have seen badly generated graphs from DALLE-2 before, so it feels like this is an obviously limitation of AI image generation tools. But why should this be such an obvious thing to absolutely everyone?
Doesn't change the fact that it's a bad experiment. Anyone familiar with DALL-E would predict this result. He should have taken some time to understand what DALL-E can do and proposed an actually interesting experiment.
funny timing, just yesterday I finished a little app I was hacking on and needed a somewhat decent looking logo that was blocking the release.
Instead of trying my luck in sketch and doodling around, I went to DALL E, and with my first prompt was able to generate better logos than I could have drawn. I was immediately unblocked and super happy with the results
It’s just amazing that non design people like me can just conjure up decent looking, and usable stuff with AI. I will definitely use DALL E much more going forward for creative work
The logos are a bit noisy and need redrawing in a proper vector tool but its a great starting point to try out different ideas immediately
Maybe it's better when hands are a central element, but I don't think I've ever seen it draw some that aren't weird when they are just a peripheral element of an image. But I haven't used it that much and those may be better too.
Text to graph is a great idea, but you don’t want an image generation AI like DALL-E. You want a natural language-to-code model like GPT-3/Codex that is able to accurately translate your requirements into code that programatically generates the image using a good graph library.
Wouldn’t be surprised if this is already possible with today’s tech, and just waiting to be built.
Good thinking. Yes, it's not quite accurate, but that's not because Codex failed, it's because the author's graph is made up nonsense that doesn't actually represent any concrete properties/data.
Now I am somewhat sleep deprived but I found the description of the graph incomprehensible. "Starts near the bottom and goes up" I interpreted as, it is a vertical line, and that its direction would be expressed in the graph as a vector or something. (The horizontal position of this vertical line appeared to be unspecified, which puzzled me.)
In fact, my first urge was to ask you to just draw the dang thing already, so I am very glad you included the sketch later!
This might say more about me than your prompt, though, but I thought I'd share the data point.
Perhaps I would have been more successful if I read the instructions with pencil in hand, sketching it out as I went along instead of trying to fit the whole instructions in my head first and then visualize it.
In my experiments with these tools I came to a conclusion that they are not very good at understanding very detailed clear directions. Which is fine!
I have a lot of fun treating AI as an absurdist philosophical visualizer. Feeding it very abstract prompts and getting back bizarre results that somehow make sense!
Agreed. What a contrived, unnecessary verbose way of describing a 3 line graph. You don't need any skill beyond basic motricity to draw a back-of-the-envelope illustration of what you need, especially in this situation that involves just a few abstract strokes, as opposed to, say, subjects/objects/animals.
I was hoping for a chance to actuslly compare things like artistic license, stylistic choices, etc. But instead the author chose an absolutely terrible prompt. AI image generation is not intended to generate graphs, and I'm surprised it was even able to do anything passable given how few it was probably trained on (if anything I'm more impressed with the AI than I was expecting to be).
Not GP but I did the same thing with trying to design a t-shirt… how is this not relevant? We’re trying to asses various tools to get our jobs done, not trying to create a peer reviewed scientific paper.
An experiment is only a single part of the scientific method, and one can easily neglect the rest of the steps. This article doesn't start with inquiry or a hypothesis from the author. We just get data and "I told you so" at the bottom, which doesn't illustrate anything.
It's funny that what we don't see is a shorter prompt. If you ran this experiment with just "A graph with 3 slightly wavy lines", maybe the difference between AI and human results would be closer. Maybe that's the basis for a legitimate research project, but it's frustrating that the author takes the ball to the 80-yard-line and just gives up.
And it's a legit criticism. There are three major issues I see here:
1) The prompt uses fairly complex grammar which is incompatible with a token-based parser. In particular, symbolic references like "The third […] starts below the second, and generally follows the second" are going to be lost on it.
2) The prompt includes details which a generative network is spectacularly unlikely to be able to handle, like asking for text labels with words like "prosecution" which are unlikely to be present in its training material. (Generally speaking, image generation models can only output short words which they've seen many times, like "STOP" or "PIZZA", and even those can be iffy.)
3) Speaking of training material, most of the training material given to image generation models consists of photographs and artwork. Technical diagrams are much less common, and when they do encounter those images, they're unlikely to be paired with the sorts of detailed descriptions that would be required to produce them on demand.
An experiment to prove that what? Humans are better at drawing graphs than computers? Well, Excel would like a word. That a neural net trained on photographs and artwork is bad at drawing graphs? Nobody expected otherwise. This "experiment" sets out to prove a hypothesis nobody had any doubts about.
A far more interesting blog post would have been looking at Fivr artists vs AI when it came to producing unique character artwork for games, or logos, or almost anything except what was done instead.
I'm pretty sure the author spent more time writing and tweaking their prompt than it would've taken them to simply draw the graph they wanted. This isn't merely a matter of illustrating prompt engineering in humans vs machines.
I get the point the author is trying to make, but I really wish the example felt less contrived.
The actual timeline is reversed; I started with the sketch, submitted it to fiverr, realized I wanted to make a comparison to dalle, and only then I tried to come up with a prompt that could encapsulate the whole image.
I can see how it felt contrived, but I hoped to make an apples-to-apples comparison on a real use case. Then to reduce the complexity I tried it on a much simpler prompt.
I have used fiverr for web assets before and I really had to make clear that what I wanted were SVG assets with transparent background. Nevertheless it was routinely not understood and I always had to ask for redo for the purpose of correcting this. Also in the same manner as this author, I found that word prompts for images were inferior to me simply doodling out what I wanted and having the more artistic Adobe inclined person create a polished version with small variations.
This is like asking a caricature artist to design a bridge. DALL-E is not a graphing tool, so it's weird to see it treated as one. A better version of this article might explore the differences between DALL-E and Fiverr-designed characters, to contrast how AI and humans approach visual storytelling.
Almost feel like this was intentionally framed this way to build more engagement (via comments where it's posted). It's pretty well known dalle and stable diffusion are bad at text and precise vector-style graphics. Do this on a professional art piece and let's see how much $10 gets you.
My man just look at the title, I clicked wondering if dall-e made better anime characters than $10 fiverr artists. But all I got was plots, who in their right mind asks for plots on fiverr.
Am I being engaged right now, was your comment also to generate engagement.hm.
I disagree and I downvoted you because I think you're being condescending and uncharitable.
I don't think OP chose graphs because they're "obviously" going to make AI look bad; I think he chose it because it's an incredibly simple image - extremely so. If the AI can't do this, how can you trust it to generate something complex? If it literally can't yet draw basic lines as described, how can it illustrate a story or any form of media where specifics matter?
And I don't think his post title implies that he was going to use some complex art prompt, either. Not in any way.
I agree that it's complex relative to what AI can currently handle (clearly) but I don't agree that it's complex in general. For a human, it's a simple description. You or I could draw it freehand correctly given 5 minutes, with no training or preparation.
I don't see how this isn't the task these AIs are supposed to solve. They are meant to take a text description and output a corresponding visual result. This just demonstrates the narrow limits on the complexity of the input they can take.
If you're saying they're not designed to deal with inputs more complex than one sentence, then sure, I guess I agree. But this post goes to show that if you require specificity in your desired visual output, then you need more than one sentence's worth of complexity, and therefore the current generation of AIs are not yet broadly usable.
It's about illustrating the current limitations. This post is not implying that the technology is a failure or that it isn't enormous progress.
> For a human, it's a simple description. You or I could draw it freehand correctly given 5 minutes, with no training or preparation.
In the blog post, the humans drew it incorrectly as well (although they got closer). If it was as simple as you say it is, i would not expect the humans to err as well.
> If you're saying they're not designed to deal with inputs more complex than one sentence, then sure, I guess I agree.
Indeed. I would further say its not designed for someone to use it as text directed paintbrush. This is not surprising since human graphic artists dont work that way either, or at least get very pissed off when they are micro managed in that fashion.
That said i think its also fairly obvious that these systems are also not replacements for graphic artists in general. The human element is important for a lot of reasons; graphic artists dont just "draw pictures". I dont think people seriously familiar with these systems have ever seriously suggested it was a full replacement for graphic artists, although in fairness random internet commentators certainly have been having a moral panic over it.
Not to mention its entirely possible that an AI more designed for this task would do better.
> But this post goes to show that if you require specificity in your desired visual output, then you need more than one sentence's worth of complexity, and therefore the current generation of AIs are not yet broadly usable.
I don't really agree that this post showed that, but i would agree that these AIs are not the best tools if you have very specific objective requirements.
AIs are tools not magic, there are things they are good at, but they aren't good at all the things and still require to be used with thought.
> It's about illustrating the current limitations. This post is not implying that the technology is a failure or that it isn't enormous progress.
I think the objection is that this article doesn't really demonstrate a meaningful limitation that wasn't obvious. It feels like a strawman. If dall-e or stable diffusion actually succeded at the task, i would be very impressed and consider it much more impressive than most of the pretty pictures everyone shows off.
If he didn't deliberately to make DALL-E look bad than he did it out of ignorance of what DALL-E's strengths and weaknesses are. Your evaluation of what is "simple" and what is "more complex" aren't in line with what DALL-E is capable of.
DALL-E isn't good with symbols like letters and numbers. It can't do even very much logical / mathematical reasoning. So a graph is one of the worst choices.
What it can do is make aesthetically pleasing images that match basic descriptions. So there are more "complex" images that DALL-E can produce than basic graphs.
I think it's just that it's such a strange comparison to make, like making an article entitled, "Who's better at doing donuts in the parking lot: helicopters or planes?"
The image generation models weren't trained on chart images, everyone already knows they're gonna be bad at that. Fiverr artists will obviously be better, though even then, who the hell is paying people on fiverr to draw generic charts?
If you wanted to compare them, it would make more sense to compare them based on how they're actually used (especially in the case of the AI models): to make art.
Though if your title was more specific, ala "DALL-E 2 vs $10 Fiverr Commissions: Who's Better at Charts?" you'd probably get somewhat fewer complaints. Having the title be generic implies that you're gonna be looking at common/primary use cases.
What you said was that they weren't trained on chart images, not that they weren't the focus:
> The image generation models weren't trained on chart images, everyone already knows they're gonna be bad at that.
I have no idea how you could even know what was, or wasn't in those models training sets. Yet you posted with conviction as if you were sure you knew. What's the point of that?
Edit - Also, what do you mean "it obviously wasn't the focus"? The focus of what? The focus of training, or the focus of presenting the results on social media?
This is absurdly silly. These data sets contain millions of images at a bare minimum from web crawls, often billions, so of course there will be a non-zero number of charts in them. If you want to be pedantic about it be my guest I guess.
You could probably find a few driver's ed teachers who taught their students to do doughnuts too, but saying "driver's ed teachers don't teach their students to do doughnuts" would nonetheless be largely accurate.
Silly yourself. If there were simply a "non-zero" number of charts in them, the model wouldn't have, you know, modelled them. That the model can reproduce graphs is clear evidence that it saw enough graphs to reproduce them.
And don't call me silly just because you used imprecise language to try to make a vague point with great conviction as if you absolutely knew what you're talking about, when you absolutely didn't. Show some respect to the intellect of your interlocutor, will you?
And, seriously, you haven't answered my question: the focus of what? What do you mean by "it obviously wasn't the focus"?
I think you were emboldened by the downvoting of my comment and assumed you don't need to make sense, but I think the downvoters were downvoting something else than what you refuse to answer.
It's still stupid. This is like asking DALL-E to generate an image that solves a math equation step by step. Of course this is easier for a human to do.
Try getting a landscape in the style of vincent van gogh for 10$ on fiver though. AI will give you that in seconds easily, and that's what's amazing about it.
I was in a meeting on Cognitive AI at the Royal Society in London last week where a gentleman from Stanford presented work where GPT-3 was prompted to solve math equations step-by-step and did well (better than I would have expected). Point being, if GPT-3 can do it, DALL-E should also be able do it, and testing whether that is the case is not stupid, but interesting.
The big question with systems like those image generation models is to what extent their generation can be controlled, and how much sense it makes. This is exactly the kind of testing that has to be done to answer such questions. Just flooding social media with cherry-picked successes doesn't help answer any questions at all. Because cherry-picking never does.
To be honest, I don't get the defensiveness of the comments in this thread. Half the comments are trying to call foul by invoking some rule they made up on the spot, according to which "that's not how you should use it". The other half pretend they knew all along what the result would be, and yet they're still upset that someone went and tried it, and posted about it. That kind of reaction is not coming from a place of inquisitiveness, or curiosity, that is for sure. It's just some kind of sclerotic reaction to novelty, people throwing their toys because someone went and did something they hadn't thought about.
> Try getting a landscape in the style of vincent van gogh for 10$ on fiver though.
In another comment posted in this thread I tried to get Stable Diffusion to give me a graph with three lines in the style of van Gogh and other famous artists. I'd be very curious to see what that would look like and I can't imagine it easily. I'm left wondering, because Stable Diffusion can't do it. Maybe I should ask someone on fiverr.
> Okay, well maybe it was impossible for anyone to deduce what I was trying to convey.
Don't know if this is evidence of "framing for more engagement," but this line irks me. The latent diffusion models are pretty powerful, but I don't think there's anyone claiming that today's diffusion models are able to interpret complicated queries better than humans. The interesting part of diffusion models is that they can produce good results at all, not that they are better than humans. We're not in AGI territory. Even text models are still limited in many ways, and latent diffusion is highly reliant on the text model to produce good results. Even simpler queries can run into quite a lot of problems, that's exactly why a lot of people have been trying to figure out the best prompts to improve results.
You could have asked those tools to create images like the ones found in AI catalogs like https://lexica.art/ and https://www.krea.ai/ and then compared with what you can get for $10. This would be a comparison more favorable to AI
That’s not a fair analogy. If anything, you could say “this is like asking a caricature artists to draw a bridge”. Sure their bridge might not end up being architecturally or structurally correct, but it will mostly look like a bridge.
These image generation tools are being discussed as something that could replace graphic designers (didn’t OpenAI refuse to open source DALLE-2 at least partially due to this concern?). So it is absolutely a reasonable idea to compare image generation vs a human designer.
Saying that, the prompt the author chose to use was hard to parse even to humans, I am not surprised the tools failed so badly.
> didn’t OpenAI refuse to open source DALLE-2 at least partially due to this concern?
If they did claim this concern I think we can safely assume that was a lie. Their business model depend on having the models closed so they can more easily charge for access
Right? It seemed like the author's premise was that the AI generators should do a good job with this prompt, but my expectation is solidly that they wouldn't. So then when the results met my expectation, the only confusing thing was the author's tone about it.
We already know what those models are good at. Everybody keeps posting their cherry-picked good results.
Why not get the chance to see some failures, too? Isn't it interesting to know what those models are bad at? There's too few examples of that around so that is definitely a very thing to know.
DALL-E is for text-to-image, not text-to-art. This experiment is valid in terms of benchmarking the extent to which LLM understands a series of texts. For an AI researcher, this gives more fuel for the next iteration.
197 comments
[ 4.8 ms ] story [ 241 ms ] threadIt's kinda sad that Google desperately wants the cool points for having its own DL models but you can only see it in the form of a store window display.
They probably also remember signing an NDA, and maybe taking some training about how not all of the world's information should be made universally accessible and useful to everyone. For instance, the contents of a user's inbox.
Getting 3 of something 1/6 of the time doesn't really sound like it groks the request.
I would not use these models for graphs yet, but for cool look tarot inspired "clipart" or background images, I think they are already usable.
The faces often look very good and they also have symmetric complexity and individual elements that come in a specific quantity (2 eyes, etc). Lower quality models do generate fly-like multi eye faces, but newer ones are so much more precise!
It's not at all surprising that an AI is bad at drawing graphs, and it is also not surprising that even a non-artist human can draw graphs pretty well.
You could equally say "it's not surprising that DALL-E can't draw words"... except that Imagen seems to be pretty good at it.
I think the real reason it's not surprising to you is that you've already seen enough DALL-E results to understand its limitations. It's not surprising that DALL-E can't draw graphs.
Why are so many people overthinking this?
From reading the comments here, they're overthinking it because they seem to be taking this as a pre-planned "attack" on AI art generation, rather than just an interesting anecdote on the limitations of these tools.
As someone who has not played with said tools, it was an outcome I found interesting to know: DALL-E et al. can't do specific graphs or even specific logical things very well a lot of the time. That's good to know, and I didn't previously!
Still found the post really interesting as it explores a very realistic use case. A client needs something simple designed for a blog post. Should they use AI or a human designer?
I read somewhere in the comments here that these tools are very bad at counting. Which is an interesting limitation with far reaching implications.
It may not make much difference, but it's not so much that they're bad at counting, as that they don't even try. The way the prompt is parsed and diffused doesn't allow for that sort of logic.
All a "two people" prompt or some such provides, is a hint to push the AI towards that section of latent space where training-set images titled "two people" exists.
That's not "counting", and it would be truly amazing if any sense of math emerged de-novo from this training process. Doesn't mean it can't be done — it means we aren't trying.
It'll be pretty exciting times once we do!
The author could also compare how well DALL·E draws text, but what would be the point of that? Is not being a scientific article a good defense for posting nonsense?
If I want to convey happy emotions in the style of Rembrandt, SD or DALL-E will do brilliantly. If I want an apple BELOW a table, or worse, a geometric shape like a triangle, they'll crash-and-burn.
GPT-3 is also really empathetic, but struggles simple logic (and especially mathematics).
Graphs are like the horror case for these.
I can think of ways to make them better at this, but it's not a weekend of hacking.
I've been using Imagen/Parti/Stable Diffusion already as a replacement for "clip art" because it takes ~15 seconds to get results and they are free. Fiverr takes at least 100 times that long and costs $10.
For tasks where the exact content isn't important and you can't invest more than a few seconds or wait a more than a few seconds for results, generative models are already a great solution.
I agree that they were trained more on artistic images, but I was still surprised with how bad they generalized to a more theoretical(?) context.
[0] https://garymarcus.substack.com/p/horse-rides-astronaut
https://twitter.com/Plinz/status/1529013919682994176
https://www.bdaddik.com/en/comics-collectible-postcards/2967...
Modulo s/Lucky Luke/astronaut/g
Note that the image above should be in Dall-E's training set. So it's seen how a horse rides a human. No excuses there.
I bet you could write a few copilot prompts to generate code which would draw a graph like this, though.
Here's the results of three attempts with slightly different prompts:
https://imgur.com/a/FqQT2Mk
As you'll see, Stable Diffusion
a) is perfectly capable of drawing graphs, and,
b) completely incapable of drawing a simple graph with three lines as prompted.
Yes, this will be the end for some artists but not for others. DALLE2 et al. are merely new tools for new generation of artists. And, we are still figuring out how to use these tools effectively.
In other words: The “AI” is a tool that we humans will use to get things done faster/better etc. Nothing less, but also not much more.
I always thought in my head that this level of creativity would remain our domain for centuries. Even as of like two or three years ago I thought that.
It’s insane to me that today some artists feel they’re going to be replaced soon. The idea of centuries is completely shattered for me and now I don’t know if we’re a year or 50 years away from AI replacing humans entirely in the creative domain. I spent the other day completely in an existential crisis, tbh.
(The main instigator on Twitter is a guy who draws “realistic Pokemon” and hates that an AI may have stolen the art he already stole from The Pokemon Company.)
From what I've seen these networks are rehashing learning set images into something matching some criteria to produce visually pleasing results. Not to belittle the results - it's impressive - but the stuff I'm not seeing here is understanding of generated material - nonsensical z-order, scale/proportions, configuration.
Fantasy images are an easy target because it's all about visually pleasing nonsense.
But then that's sort of a self-limiting factor: it means theres still space for human creativity in creating new things, new styles (as not every style exists yet!) -- at least until said new style gets loaded into the model, I suppose?
Fascinating stuff, really.
This isn't art. It's a graph meant to represent data.
Now, if you're actually Wizards of the Coast, you probably wanna spend the money with real artists anyway, but for any smaller teams, I can see the appeal of just using AI for that kind of use case now.
Bear in mind that the training data for these models has been mostly images and their alt text, scraped off the web. There is a good chance that there's nothing remotely like the examples given here in the training data. (People don't caption their graphs like that.) These models are undoubtably good at doing what they have been trained to do - but I think no-one disagrees that there's plenty of room for improvement.
(And bear in mind that these text2image models only released this year, and that this tech in general has only been invented in the last couple of years, so it's very early days...)
Let’s show all the ways that these AI obliterate $10 Fiverr non-AI commissions, thats what people want to see
https://blog.barac.at/a-business-experiment-in-data-dignity
* vs the current trend of training diffusion models on 400M images from the Internet (many of them being garbage) with mixed licenses and letting the user take responsibility of the generated images licensing issues.
It’s just the wrong tool for the job.
Yes, we all have seen badly generated graphs from DALLE-2 before, so it feels like this is an obviously limitation of AI image generation tools. But why should this be such an obvious thing to absolutely everyone?
It’s just amazing that non design people like me can just conjure up decent looking, and usable stuff with AI. I will definitely use DALL E much more going forward for creative work
The logos are a bit noisy and need redrawing in a proper vector tool but its a great starting point to try out different ideas immediately
(The results: https://twitter.com/dvcrn/status/1578710631838289922)
https://imgur.com/lVyUQFb
https://imgur.com/KJRPHi9
But it's still not perfect, a few more examples in a grid, as you can see hands are still a problem:
https://imgur.com/ugpvE4a
It seems fastest to just draw it yourself; even the pencil drawing was already decent; and you can buy color pens for less than $12.
Wouldn’t be surprised if this is already possible with today’s tech, and just waiting to be built.
edit: just tried OP’s prompt with Codex and Colab and generated this image: https://i.imgur.com/OyxJCbz.png
Not quite accurate, but shows the potential for a better language model or some prompt engineering to encourage fidelity to the prompt
In fact, my first urge was to ask you to just draw the dang thing already, so I am very glad you included the sketch later!
This might say more about me than your prompt, though, but I thought I'd share the data point.
Perhaps I would have been more successful if I read the instructions with pencil in hand, sketching it out as I went along instead of trying to fit the whole instructions in my head first and then visualize it.
I have a lot of fun treating AI as an absurdist philosophical visualizer. Feeding it very abstract prompts and getting back bizarre results that somehow make sense!
Please do this again with a better prompt.
It's funny that what we don't see is a shorter prompt. If you ran this experiment with just "A graph with 3 slightly wavy lines", maybe the difference between AI and human results would be closer. Maybe that's the basis for a legitimate research project, but it's frustrating that the author takes the ball to the 80-yard-line and just gives up.
1) The prompt uses fairly complex grammar which is incompatible with a token-based parser. In particular, symbolic references like "The third […] starts below the second, and generally follows the second" are going to be lost on it.
2) The prompt includes details which a generative network is spectacularly unlikely to be able to handle, like asking for text labels with words like "prosecution" which are unlikely to be present in its training material. (Generally speaking, image generation models can only output short words which they've seen many times, like "STOP" or "PIZZA", and even those can be iffy.)
3) Speaking of training material, most of the training material given to image generation models consists of photographs and artwork. Technical diagrams are much less common, and when they do encounter those images, they're unlikely to be paired with the sorts of detailed descriptions that would be required to produce them on demand.
And it’s like a five minute job in Inkscape where he could’ve just done the paper drawing in that and be done.
A far more interesting blog post would have been looking at Fivr artists vs AI when it came to producing unique character artwork for games, or logos, or almost anything except what was done instead.
I get the point the author is trying to make, but I really wish the example felt less contrived.
I can see how it felt contrived, but I hoped to make an apples-to-apples comparison on a real use case. Then to reduce the complexity I tried it on a much simpler prompt.
Am I being engaged right now, was your comment also to generate engagement.hm.
I don't think OP chose graphs because they're "obviously" going to make AI look bad; I think he chose it because it's an incredibly simple image - extremely so. If the AI can't do this, how can you trust it to generate something complex? If it literally can't yet draw basic lines as described, how can it illustrate a story or any form of media where specifics matter?
And I don't think his post title implies that he was going to use some complex art prompt, either. Not in any way.
Its also an entirely different task than the one these AIs were designed to solve. Its like judging a fish by its ability to fly.
I don't see how this isn't the task these AIs are supposed to solve. They are meant to take a text description and output a corresponding visual result. This just demonstrates the narrow limits on the complexity of the input they can take.
If you're saying they're not designed to deal with inputs more complex than one sentence, then sure, I guess I agree. But this post goes to show that if you require specificity in your desired visual output, then you need more than one sentence's worth of complexity, and therefore the current generation of AIs are not yet broadly usable.
It's about illustrating the current limitations. This post is not implying that the technology is a failure or that it isn't enormous progress.
In the blog post, the humans drew it incorrectly as well (although they got closer). If it was as simple as you say it is, i would not expect the humans to err as well.
> If you're saying they're not designed to deal with inputs more complex than one sentence, then sure, I guess I agree.
Indeed. I would further say its not designed for someone to use it as text directed paintbrush. This is not surprising since human graphic artists dont work that way either, or at least get very pissed off when they are micro managed in that fashion.
That said i think its also fairly obvious that these systems are also not replacements for graphic artists in general. The human element is important for a lot of reasons; graphic artists dont just "draw pictures". I dont think people seriously familiar with these systems have ever seriously suggested it was a full replacement for graphic artists, although in fairness random internet commentators certainly have been having a moral panic over it.
Not to mention its entirely possible that an AI more designed for this task would do better.
> But this post goes to show that if you require specificity in your desired visual output, then you need more than one sentence's worth of complexity, and therefore the current generation of AIs are not yet broadly usable.
I don't really agree that this post showed that, but i would agree that these AIs are not the best tools if you have very specific objective requirements.
AIs are tools not magic, there are things they are good at, but they aren't good at all the things and still require to be used with thought.
> It's about illustrating the current limitations. This post is not implying that the technology is a failure or that it isn't enormous progress.
I think the objection is that this article doesn't really demonstrate a meaningful limitation that wasn't obvious. It feels like a strawman. If dall-e or stable diffusion actually succeded at the task, i would be very impressed and consider it much more impressive than most of the pretty pictures everyone shows off.
DALL-E isn't good with symbols like letters and numbers. It can't do even very much logical / mathematical reasoning. So a graph is one of the worst choices.
What it can do is make aesthetically pleasing images that match basic descriptions. So there are more "complex" images that DALL-E can produce than basic graphs.
The image generation models weren't trained on chart images, everyone already knows they're gonna be bad at that. Fiverr artists will obviously be better, though even then, who the hell is paying people on fiverr to draw generic charts?
If you wanted to compare them, it would make more sense to compare them based on how they're actually used (especially in the case of the AI models): to make art.
Though if your title was more specific, ala "DALL-E 2 vs $10 Fiverr Commissions: Who's Better at Charts?" you'd probably get somewhat fewer complaints. Having the title be generic implies that you're gonna be looking at common/primary use cases.
Stable Diffusion was trained on images of charts and graphs. It knows what a powerpoint presentation and even an excel spreadsheet look like.
Here:
https://imgur.com/a/V4a6W4I
It just doesn't know how to generate a graph like the one it's asked to.
> The image generation models weren't trained on chart images, everyone already knows they're gonna be bad at that.
I have no idea how you could even know what was, or wasn't in those models training sets. Yet you posted with conviction as if you were sure you knew. What's the point of that?
Edit - Also, what do you mean "it obviously wasn't the focus"? The focus of what? The focus of training, or the focus of presenting the results on social media?
You could probably find a few driver's ed teachers who taught their students to do doughnuts too, but saying "driver's ed teachers don't teach their students to do doughnuts" would nonetheless be largely accurate.
And don't call me silly just because you used imprecise language to try to make a vague point with great conviction as if you absolutely knew what you're talking about, when you absolutely didn't. Show some respect to the intellect of your interlocutor, will you?
And, seriously, you haven't answered my question: the focus of what? What do you mean by "it obviously wasn't the focus"?
I think you were emboldened by the downvoting of my comment and assumed you don't need to make sense, but I think the downvoters were downvoting something else than what you refuse to answer.
Try getting a landscape in the style of vincent van gogh for 10$ on fiver though. AI will give you that in seconds easily, and that's what's amazing about it.
The big question with systems like those image generation models is to what extent their generation can be controlled, and how much sense it makes. This is exactly the kind of testing that has to be done to answer such questions. Just flooding social media with cherry-picked successes doesn't help answer any questions at all. Because cherry-picking never does.
To be honest, I don't get the defensiveness of the comments in this thread. Half the comments are trying to call foul by invoking some rule they made up on the spot, according to which "that's not how you should use it". The other half pretend they knew all along what the result would be, and yet they're still upset that someone went and tried it, and posted about it. That kind of reaction is not coming from a place of inquisitiveness, or curiosity, that is for sure. It's just some kind of sclerotic reaction to novelty, people throwing their toys because someone went and did something they hadn't thought about.
> Try getting a landscape in the style of vincent van gogh for 10$ on fiver though.
In another comment posted in this thread I tried to get Stable Diffusion to give me a graph with three lines in the style of van Gogh and other famous artists. I'd be very curious to see what that would look like and I can't imagine it easily. I'm left wondering, because Stable Diffusion can't do it. Maybe I should ask someone on fiverr.
Don't know if this is evidence of "framing for more engagement," but this line irks me. The latent diffusion models are pretty powerful, but I don't think there's anyone claiming that today's diffusion models are able to interpret complicated queries better than humans. The interesting part of diffusion models is that they can produce good results at all, not that they are better than humans. We're not in AGI territory. Even text models are still limited in many ways, and latent diffusion is highly reliant on the text model to produce good results. Even simpler queries can run into quite a lot of problems, that's exactly why a lot of people have been trying to figure out the best prompts to improve results.
These image generation tools are being discussed as something that could replace graphic designers (didn’t OpenAI refuse to open source DALLE-2 at least partially due to this concern?). So it is absolutely a reasonable idea to compare image generation vs a human designer.
Saying that, the prompt the author chose to use was hard to parse even to humans, I am not surprised the tools failed so badly.
If they did claim this concern I think we can safely assume that was a lie. Their business model depend on having the models closed so they can more easily charge for access
Why not get the chance to see some failures, too? Isn't it interesting to know what those models are bad at? There's too few examples of that around so that is definitely a very thing to know.