Right now everyone is coming out with video models, and they're all very similar.
Stability, Meta, Google, Nvidia, tencent, Alibaba all have released either demos or code for Stable Diffusion based video generation. They're all fairly similar but slightly different architecture.
The fact that we can synthesize stuff like this from text makes me believe the real world obeys some laws of physics we still haven't figured out.
Specifically, the fact that when we interpolate in embedding space, the interpolated point still represents something almost always physically plausible
(not always though, witness the surfing cat that doesn't get wet underwater).
But the embedding space is deliberately and laboriously constructed such that it represents realistic things at as many points as possible. That's what the training is, right?
Eh? It's automated statistical learning. Yes, it's very cool, but it is far from AGI, let alone magic.
I am playing around with SD right now - pulling sleepless nights because it's so fascinating, but I do get to see the patterns.
Generating unreal, magical stuff is actually a lot of fun, but if you want it to generate something that makes sense - good luck. It is really hard.
I can get Elizabeth Olsen into a Halo suit in a war zone on an alien world, but getting her to sit in a car like a normal human being has been impossible. It's always some twisted horror show. The predictive model does not know what to do with it and how it all fits. That's after you clean up all the anatomical nightmare fuel like the fingers situation.
This is why most of the SD artwork you see are magical elf girls in the foreground or some unearthly monsters.
A well-defined subject in the foreground (a human-ish one), it will do VERY well. If you really want to get your imagination going with every little detail being near-perfect, you will be busy.
Oh, and the text - it obviously cannot do text. It's in the right place, but it looks funny.
This is amazing tech, but it's not about to go sentient and kill me at night.
> obeys some laws of physics we still haven't figured out.
That statement might have been over the top, but I think they're right with respect to the information. Spatial logic, optics, signal processing, reasoning. It's beginning to feel like there's an interesting representation that cross-cuts these spaces in a powerful, generalizable way.
Agreed. I've generated tens of thousands of images with AI between Midjourney, Dall-E, and Stable Diffusion and yeah. The patterns almost ruin it for me. It's super interesting and a really cool tech but yeah at this point I can pretty much look at a picture and recognize it's AI generated just by the "vibe." Like you're saying, it's always this over focused single-subject image with a generic background.
Like you said, as well, it's impossible to do some of the stuff that seems super simple but based in reality where completely dream-like images are generally pretty automatic. It's cool and, when I have another project that I need it for, I'll go nuts with it again but I kind of lost interest in just generating images to experiment because the experimentation stopped netting interesting results. It was just always the same stuff.
> The fact that we can synthesize stuff like this from text
We can't synthesize stuff like that from text. It's synthesized from a(n obscenely large) combination of annotated still images and video clips. The text guides the synthesis (or perhaps better to call it reconstitution) process.
Seems gruel to even say this as the tech is pure magic, but sad that these videos often get that ultra processed stockfootage look. I would be more impressed to just have bland everyday videos with muted colors, shaky cam and uninteresting people doing everyday stuff.
What they are asking for is to have these machine-generated "photorealistic" outputs look more like real photos/videos, not to turn every normal looking photo into a lurid dream-scene monstrosity with completely made up details.
OT: re "a spaceship destroyer approaching Earth" (example in the first row of the second set) - I cannot for the life of me generate an SD image of a two-person space ship (a shuttle, but more Star Wars X-Wing than USS Enterprise shuttle) flying towards the viewer.
I was chatting with a studio manager at a post f/x house when he received a demo reel that immediately was relocated to the trash without viewing it. I called him on it about not even looking at it before rejecting it. He had me take it out of the trash to play it. Before it stared, he told me exactly what would be on the tape and the order of the clips, which is exactly what was on the tape. He said it was the fall semester with such and such teacher at the school, and every submission was exactly the same. They were just the class assignments made into a demo reel.
Everyone of these generative AI demo pages feels exactly like that. Nothing new other than we've done the same thing as someone else type of releases.
Now, I'm willing to say I'm jaded, so if this represents something we haven't already seen umpteengillion times already, where did they bury the lede? It's all still uncanny valley. The wheels on do not look real. The wake from the viking's vacuum does not look real. I'm not sure what the corgi is walking on, but it doesn't look like the ground.
That studio manager will be out of a job in five years if he doesn't get on the bandwagon and evolve.
This is the film to digital transition. It's going to be rough for a bit, then take over everything all at once.
The upside for these editors is that they'll be able to be single-person studios and pursue their own vision without meddling from studios or massive capital, equipment, and personnel needs.
Ha! I took a phone call back when I was in the shiny round disc business that said based on the prices we were quoting, that we'd be out of business in a year because Apple released DVD Studio Pro. Instead, we gained even more business from people that tried to have their projects done in DVDSP, but then ran into road blocks making it impossible to do the job. They'd then come back to us, and receive an even higher quote since now there is even less time before their deadlines. cie la vie.
You say out of business. I say, mediocrity all the way down from here.
> You say out of business. I say, mediocrity all the way down from here.
Feels "Old man yells at cloud." There will be lots of garbage, for sure, but more people than ever before will have access to creation.
Becoming a director means getting access to a very limited playing field. Very few domestic productions get greenlit per year due to the intense financial, logistical, and personnel requirements. Very few people are able to make the opportunity cost sacrifice to learn the tools and people network.
For all the people that write fan fiction, web comics, or have a DeviantArt, Itch.io, or Newgrounds, this is going to give them access to their own personal Miyazaki/Scorsese tooling.
It's going to be awesome seeing passionate people finally have access.
It much more like, old man has sat through many a product demo and seen the brochures, but what's in the box rarely matches. If this is what's being demoed as the examples they've culled together, the tech is lacking.
Anyone with a few hundred dollars can buy mics, steady cams, action cams and more off of amazon that would give them quality above the black and white or 16mm film that some directors started out on.
Anyone could go and make their own clerks or reservoir dogs and some people are going out and trying. Where are all the incredible movies?
Even forgetting the tools where are all the incredible scripts since everyone has access to books on script writing and a computer?
You were wrong last time you debated me on AI porn. It's here [1, 2].
Movies are next.
> Anyone with a few hundred dollars can buy mics, steady cams, action cams and more off of amazon that would give them quality above the black and white or 16mm film that some directors started out on.
I've been down this path, and frankly it sucks.
The task: tell a visual story.
The current process: script, edit, break down with AD, location scout, secure filming rights to a location, go to a rental house and get equipment, get insurance, find camera crew that can handle it, rent lights, get gels, go to a prop house, pick props, rent props, transport props to set, set dec, run electrical, light set, do blocking, rehearse, film, retake, film, reset, capture photons, smear pixels around with an editing package for three months.
This ceremony freaking sucks. It's so far divorced from the objective at hand, it's more primitive than punch cards, and nothing is reusable. I absolutely hate it. And I've always hated it. It needs to die. These steps have nothing to do with storytelling. It's vestigial and is an artifact of our primitive technologies.
The task is to convey visual stories. Not to play dress up, logistics, and awkward gridded photon capture.
Anyway, movies are next. I'll eat my hat if they're not. The current filmmaking process is as dumb as bloodletting. And it feels about the same.
Your links have nothing to do with what I'm asking. Also you predicted that websites that host porn would somehow go out of business which never happened.
The current process:
This is not the current process. Anyone can buy cameras, steady cam rigs and wireless mics from amazon then edit on a pc.
Where are you getting this idea that anyone can't buy stuff on amazon and edit on their computer?
Why aren't you answering the questions of why there isn't a glut of high quality scripts, sitcoms, tv shows and independent movies? At some point you will have to accept that these things are difficult and it isn't just about how easy it is to make images. Anyone can write a book or start a youtube channel, but that doesn't mean that anyone can do it well. The best people are still the best.
> Your links have nothing to do with what I'm asking. Also you predicted that websites that host porn would somehow go out of business which never happened.
Just wait. Generating > hosting.
> Where are you getting this idea that anyone can't buy stuff on amazon and edit on their computer?
It's fucking difficult.
> Why aren't you answering the questions of why there isn't a glut of high quality scripts, sitcoms, tv shows and independent movies?
Because the US market can only bear a few thousand theatrical and television productions a year. There are more than a thousand directors that would love a shot to be greenlit, but the capital, personnel, and logistics inputs are too high. Compare to the number of software engineers, fan fiction authors, web comic authors, etc.
> At some point you will have to accept that these things are difficult and it isn't just about how easy it is to make images.
Momentary technological limitation. At some point in the future, our daydreams will become a live canvas we share with others in real time. We're getting there.
That might be a long way off, but some of the next steps along the way include easy to use generative film tech.
> The best people are still the best.
You'll never know who could have been the best. The best director might have been bound by opportunity cost to earn money for their family working two jobs.
You said your predictions came true in your last comment, now you're saying "just wait". If "generating > hosting" (whatever that means) why are all the companies making the most money the ones that host?
It's fucking difficult.
Anyone can buy stuff off of amazon and set it up. We can see that by how many people have youtube channels. Anyone could make their own movie and put it on youtube but we don't see much of that at all.
You'll never know who could have been the best.
You haven't confronted why we don't see the same stuff in scripts, novels, plays or other forms that don't take the same amount of effort.
The truth is that it takes practice. It isn't about some innate ability as if there is a good will hunting cinematic genius sitting doing nothing because he doesn't have generative AI.
Didn't you say "hollywood" would collapse in a year? (forgetting for a second that hollywood is a city and paramount is the only movie studio actually in hollywood)
Agree... they'll definitely evolve and get better, we're in the floppy disk era, but so far they suck. Nothing but weird motion, a shitty echo of mainstream / stock footage.
I also get the sense that in the early days where data curation matters, research teams need to hire curators that can siphon good from bad examples. E.g. don't train on shit stock footage, pick up a book on fine art photographers, train on that shit + good films and you'll get a pretty good model.
In the future I suspect curation will matter less and less because in the limit you should train on literally the entire internet.
I'm definitely excited for where it might go though. If it ever develops to a sophisticated point, which it most certainly will, we'll be able to do stuff with it that is just impossible by typical production means.
Still a long way until clients will be able to play "pixel peeping" and "what's wrong with this shot" games with AI. Chores like matte/roto and matchmove are closer to danger. Distant mattes, procedural fill (even textures) will be supercharged with it though.
So what's the complaint here? That researchers research?
These folks wanted to make their own thing so they made their own thing. Eventually someone will make a thing that's far better than anything we have today.
To add, the very first video, the parrot doesn't have an eye. A few videos down the line, the horse's legs meld together and split again as it's walking.
Agreed, this is no more impressive than Stable Video or any of the other things like this we've seen.
I think this comment is a bit premature and better reserved for a finished product, not a research demo. :) Video generation is still in its infancy, it obviously has even longer way to go than the other types of generative AI.
Other than that, these demos will eventually develop into tools. You get to decide what to make with the tools, they won't do it for you.
I would love to see something longer than half a second, one day. This is the 50th video demo and it is still the same. Extremely short blurry gifs, always single angle and something morphing. Ai's video generation progress seems to be not going forward.
48 comments
[ 2.9 ms ] story [ 115 ms ] threadA line of sight, however disappointing is still better than no line of sight.
Maybe it will turn the open source efforts onto these things that are announced.
2. ???
3. Profit
https://github.com/thecooltechguy/ComfyUI-Stable-Video-Diffu...
Right now everyone is coming out with video models, and they're all very similar.
Stability, Meta, Google, Nvidia, tencent, Alibaba all have released either demos or code for Stable Diffusion based video generation. They're all fairly similar but slightly different architecture.
The fact that we can synthesize stuff like this from text makes me believe the real world obeys some laws of physics we still haven't figured out.
Specifically, the fact that when we interpolate in embedding space, the interpolated point still represents something almost always physically plausible (not always though, witness the surfing cat that doesn't get wet underwater).
I am playing around with SD right now - pulling sleepless nights because it's so fascinating, but I do get to see the patterns.
Generating unreal, magical stuff is actually a lot of fun, but if you want it to generate something that makes sense - good luck. It is really hard.
I can get Elizabeth Olsen into a Halo suit in a war zone on an alien world, but getting her to sit in a car like a normal human being has been impossible. It's always some twisted horror show. The predictive model does not know what to do with it and how it all fits. That's after you clean up all the anatomical nightmare fuel like the fingers situation.
This is why most of the SD artwork you see are magical elf girls in the foreground or some unearthly monsters.
A well-defined subject in the foreground (a human-ish one), it will do VERY well. If you really want to get your imagination going with every little detail being near-perfect, you will be busy.
Oh, and the text - it obviously cannot do text. It's in the right place, but it looks funny.
This is amazing tech, but it's not about to go sentient and kill me at night.
> obeys some laws of physics we still haven't figured out.
That statement might have been over the top, but I think they're right with respect to the information. Spatial logic, optics, signal processing, reasoning. It's beginning to feel like there's an interesting representation that cross-cuts these spaces in a powerful, generalizable way.
Like you said, as well, it's impossible to do some of the stuff that seems super simple but based in reality where completely dream-like images are generally pretty automatic. It's cool and, when I have another project that I need it for, I'll go nuts with it again but I kind of lost interest in just generating images to experiment because the experimentation stopped netting interesting results. It was just always the same stuff.
We can't synthesize stuff like that from text. It's synthesized from a(n obscenely large) combination of annotated still images and video clips. The text guides the synthesis (or perhaps better to call it reconstitution) process.
https://magnific.ai/
Which seems to have been endorsed by one of the best retouchers I know of (https://www.youtube.com/@pratiknaikedu)
https://github.com/xinntao/Real-ESRGAN
Here is an example I made yesterday, probably around 5 minutes of computing on my 3080ti: https://imgur.com/a/4dPv1hC
It's pretty much the standard comfyui svd workflow with extra upscaling.
this would be good enough for promos of those!
Any helpful tips?
Also can't say no to a cat video (even if it doesn't really remind of Van Gogh).
Everyone of these generative AI demo pages feels exactly like that. Nothing new other than we've done the same thing as someone else type of releases.
Now, I'm willing to say I'm jaded, so if this represents something we haven't already seen umpteengillion times already, where did they bury the lede? It's all still uncanny valley. The wheels on do not look real. The wake from the viking's vacuum does not look real. I'm not sure what the corgi is walking on, but it doesn't look like the ground.
This is the film to digital transition. It's going to be rough for a bit, then take over everything all at once.
The upside for these editors is that they'll be able to be single-person studios and pursue their own vision without meddling from studios or massive capital, equipment, and personnel needs.
You say out of business. I say, mediocrity all the way down from here.
Feels "Old man yells at cloud." There will be lots of garbage, for sure, but more people than ever before will have access to creation.
Becoming a director means getting access to a very limited playing field. Very few domestic productions get greenlit per year due to the intense financial, logistical, and personnel requirements. Very few people are able to make the opportunity cost sacrifice to learn the tools and people network.
For all the people that write fan fiction, web comics, or have a DeviantArt, Itch.io, or Newgrounds, this is going to give them access to their own personal Miyazaki/Scorsese tooling.
It's going to be awesome seeing passionate people finally have access.
It much more like, old man has sat through many a product demo and seen the brochures, but what's in the box rarely matches. If this is what's being demoed as the examples they've culled together, the tech is lacking.
Anyone could go and make their own clerks or reservoir dogs and some people are going out and trying. Where are all the incredible movies?
Even forgetting the tools where are all the incredible scripts since everyone has access to books on script writing and a computer?
You were wrong last time you debated me on AI porn. It's here [1, 2].
Movies are next.
> Anyone with a few hundred dollars can buy mics, steady cams, action cams and more off of amazon that would give them quality above the black and white or 16mm film that some directors started out on.
I've been down this path, and frankly it sucks.
The task: tell a visual story.
The current process: script, edit, break down with AD, location scout, secure filming rights to a location, go to a rental house and get equipment, get insurance, find camera crew that can handle it, rent lights, get gels, go to a prop house, pick props, rent props, transport props to set, set dec, run electrical, light set, do blocking, rehearse, film, retake, film, reset, capture photons, smear pixels around with an editing package for three months.
This ceremony freaking sucks. It's so far divorced from the objective at hand, it's more primitive than punch cards, and nothing is reusable. I absolutely hate it. And I've always hated it. It needs to die. These steps have nothing to do with storytelling. It's vestigial and is an artifact of our primitive technologies.
The task is to convey visual stories. Not to play dress up, logistics, and awkward gridded photon capture.
Anyway, movies are next. I'll eat my hat if they're not. The current filmmaking process is as dumb as bloodletting. And it feels about the same.
[1] https://news.ycombinator.com/item?id=32711614
[2] https://news.ycombinator.com/item?id=35523656
The current process:
This is not the current process. Anyone can buy cameras, steady cam rigs and wireless mics from amazon then edit on a pc.
Where are you getting this idea that anyone can't buy stuff on amazon and edit on their computer?
Why aren't you answering the questions of why there isn't a glut of high quality scripts, sitcoms, tv shows and independent movies? At some point you will have to accept that these things are difficult and it isn't just about how easy it is to make images. Anyone can write a book or start a youtube channel, but that doesn't mean that anyone can do it well. The best people are still the best.
Just wait. Generating > hosting.
> Where are you getting this idea that anyone can't buy stuff on amazon and edit on their computer?
It's fucking difficult.
> Why aren't you answering the questions of why there isn't a glut of high quality scripts, sitcoms, tv shows and independent movies?
Because the US market can only bear a few thousand theatrical and television productions a year. There are more than a thousand directors that would love a shot to be greenlit, but the capital, personnel, and logistics inputs are too high. Compare to the number of software engineers, fan fiction authors, web comic authors, etc.
> At some point you will have to accept that these things are difficult and it isn't just about how easy it is to make images.
Momentary technological limitation. At some point in the future, our daydreams will become a live canvas we share with others in real time. We're getting there.
That might be a long way off, but some of the next steps along the way include easy to use generative film tech.
> The best people are still the best.
You'll never know who could have been the best. The best director might have been bound by opportunity cost to earn money for their family working two jobs.
You said your predictions came true in your last comment, now you're saying "just wait". If "generating > hosting" (whatever that means) why are all the companies making the most money the ones that host?
It's fucking difficult.
Anyone can buy stuff off of amazon and set it up. We can see that by how many people have youtube channels. Anyone could make their own movie and put it on youtube but we don't see much of that at all.
You'll never know who could have been the best.
You haven't confronted why we don't see the same stuff in scripts, novels, plays or other forms that don't take the same amount of effort.
The truth is that it takes practice. It isn't about some innate ability as if there is a good will hunting cinematic genius sitting doing nothing because he doesn't have generative AI.
Didn't you say "hollywood" would collapse in a year? (forgetting for a second that hollywood is a city and paramount is the only movie studio actually in hollywood)
I also get the sense that in the early days where data curation matters, research teams need to hire curators that can siphon good from bad examples. E.g. don't train on shit stock footage, pick up a book on fine art photographers, train on that shit + good films and you'll get a pretty good model.
In the future I suspect curation will matter less and less because in the limit you should train on literally the entire internet.
I'm definitely excited for where it might go though. If it ever develops to a sophisticated point, which it most certainly will, we'll be able to do stuff with it that is just impossible by typical production means.
Except you don't have to hire anyone and you get results in two minutes, not two days later.
These folks wanted to make their own thing so they made their own thing. Eventually someone will make a thing that's far better than anything we have today.
Agreed, this is no more impressive than Stable Video or any of the other things like this we've seen.
Other than that, these demos will eventually develop into tools. You get to decide what to make with the tools, they won't do it for you.