if people weren’t so repressed, this could also be used to severely reduce exploitation in the porn industry. what’s the point in making and selling exploitative porn when it can be auto-generated at will?
These things still feel a bit like e.g. Google/GCP services to me: Super appealing at first glance, quite close to what you want, but somehow never quite there. Maybe they'll asymptotically get there, eventually? Perhaps that statistical model can't really make it to the level we want it to?
It may be that it's the deep learning tech which will never quite get there. GPT-3 has similar shortcomings in its mimicry. We're 95% there, I guess, but may never quite reach 100%.
I get the same feeling as well. This approach may well be eternal demo-ware, and you'll actually need AGI (or manual direction by a real human) to get to 100%.
Nah, the current issues are just because we're trying to do everything in one step. Because we've built tools that have so much of a stimulus-response approach, few efforts have been made toward interfaces that ask for clarification ('when you say X, do you mean XYZ or XXX?').
Image-to-image and tuning already addresses many of these issues; just as inpainting works really well, it won't be long before we have select-and-repair, where you add an additional prompt like 'improve this part - the ice cream is fine, just work on the dog's muzzle.'
The mistakes the AI makes are too numerous and hard-to-define for this to work I think. They could perhaps be addressed by having two different models trained differently, each fixing the errors of the other. When humans draw a realistic artwork, it's not 'single-pass'; they have to iterate on the details to get it right.
I’ve found that replacing the bad parts with new ones, like Dalle Outpainting, can remove the worst parts of the image, like the hands here… doesn’t make it perfect, but certainly removes the worst offenders that instantly bring attention to themselves.
The hands throw me off. The same with the cat holding the remote... never thought that hands on animals would be able to trigger my uncanny valley response, but here we are
This looks like the video equivalent of Dall-E 1. Hard to believe how far we've come so quickly.
The paper talks about "pseudo 3D attention layers" that are used in place of temporal attention layers for each dimension due to memory consumption. It seems like AI research is vastly outpacing GPU development.
Indeed - it's not hard from a research point of view - it's hard from a compute perspective because adding one more dimension requires hundreds of times more compute.
Even then, these videos are only like 50 frames long - and a real movie you would want to be hundreds of thousands of frames long.
So you need to optimise on compressed version, not the whole thing. What they’re doing right now is akin to a human trying to hold an entire picture - or entire movie - in their head all at once.
We can’t do it. AIs can sort of do it.
Latent diffusion models already demonstrated that operating on a compressed representation gives far better results, faster, but I don’t think we’re anywhere near the limit for what’s possible there. It’s no coincidence that this is how humans work.
I am curious if there has been any research on temporal attention in humans. I'm not sure how you'd quantify it. But in myself I know that I'm constantly predicting where something will be or what it will look like based on how it did a second ago. It's probably the root of reflexes.
They put an eye tracker on someone and captured their motion when walking in some rough terrain. You can sort of see that the person is focusing on the most likely place their foot will go next.
I think that we will discover that there is a more efficient way to encode temporal relationships, which appears to be "just throw transformers at it." My guess is that it will be in a more conceptual latent space that this attention will be applied.
True, but the attention layers still need to be able to look at all the shots - for example to make sure the background of a room shown at the start of the movie is the same as the background of the same room at the end.
Obviously you could do 'human assisted' movie making where humans decide the storyboard and make directions for each shot, and then that isn't necessary.
Hardware was probably always lagging behind cutting edge research, just consider video games, they pushed hardware limitations very hard since Pong.
It's a good thing to be fair, forcing research teams to optimize their projects is beneficial and creates a competition for limited resources. This gets a bit skewed when we consider a university research team vs. a MANGA type company, but the team behind Stable diffusion proved that innovation can come from unexpected places.
I'm rooting for this tech. Hopefully this will get modern movies out of their low risk reboot loop since it will be cheaper to make a movie that have new story lines that are commercially untested. I'd be happy to watch a movie that doesn't look AAA, but has compelling writing and makes me think. Or maybe I'll just stick to books.
I don't think modern movies are stuck in a "low risk reboot loop" because of the cost to produce, it's because of the potential profit.
Why spend money on a film with new IP and ideas that you're not sure will be popular when the data science team has already worked with marketing to figure out exactly what movie will sell well?
Good luck finding your movie with compelling and thought provoking writing in the big pile of movies produced by comittee to show up above yours in discovery algorithms.
With the future version of this you could theoretically prototype movies way faster and try ideas with test audiences without requiring to actually film them
In the future version of this, the end user asks for a movie and gets a custom movie generated just for them.
You want a thought-provoking Bourne-style action thriller with hints of Jane Austen and a bollywood dance sequence? How about a Matrix sequel that lives up to the first one, but ends just the way you like it? Just ask.
Xenomorph cop. It's a Dirty Harry style movie except the lead is a xenomorph. The other characters are only vaguely aware that he's a xenomorph.
Or how about the movie Clue with the three endings except an infinite sequence of "or maybe it happened this way" sequences. I mean how else are we going to get a sequence where Darth Vader and Tim Curry reenact the "No I am your father" scene while Martin Mull dies in the background due to a heart attack.
This could certainly be entertaining. The trick will be for the studios to continue to brand it as recognizable, yet have it be unique. It begs to ask what part of the experience will be a shared experience since it could be radically different for all of us. Would we then share the story created for us with friends? Will there be an Oscar for best mind to have a video created for them?
I think it would be great for people who write scripts for a movie they imagined.
You could conceivably write a script and feed it into a machine and have a decent 1080p rendition of the movie with consistent characters and voice acting which you could use to better pitch your movie idea to people, or get to watch a movie you created between you and your friends even if no one else ever gets to see it.
Having written several feature length scripts, I'm certainly looking forward to trying this. But with an explosion of cheaply produced content, people will probably spend less time watching movies and TV shows. I'm a lot more selective now than I used to be because there's just too much stuff to watch even if I spent the rest of my life on the couch.
The current trend of remaking movies as 10 hour miniseries (and then making more and more seasons) is Not Great. Whereas I could be fascinated by a quirky-but-compelling original movie, I'm less attracted to 10+ hours of hyper-polished content. Sometimes I've watched a series and thought 'that was good, but it could have been a better movie.'
Films today require institutional capital. There are less than a thousand directors a year that get greenlit. There isn't enough conceptual or directorial diversity, and it sucks.
In the future, kids will be making their own Star Wars movies from home. All kinds of people from all kinds of backgrounds will make novel films that would ordinarily never be made, such as "Steampunk Vampires of Venus", starring John Wayne, a young Betty White, and Samuel L. Jackson. This is absolutely the future.
I'm working on building this. I'm sure lots of others are too.
I was just thinking about this for the long tail problem in recommendation engines for stuff like video games. I blame the algorithms themselves.
We're in a situation where the very best algorithms (like the one used by Netflix), doing exactly what they're designed to do, create inequality and the vacuous economy of influencers we have today. Look at Steam or any other marketplace: they're all the same, with 1% of the players getting 99% of the prizes. In a very real way, the only winning move is not to play.
I would suggest that this tendency of capitalism (economic evolution) is unstoppable, and that it must be attacked from a different angle. If we don't want to inevitably end up in late-stage capitalism that looks like neofeudalism, then there has to be some form of redistribution or people spend the entirety of their lives running the rat race to make rent. Traditionally that was high taxes on the winners, but UBI would probably work better. Unfortunately, the very same people who win are the ones most resistant to any notion of a level playing field or social safety net.
So I feel like there may be no solution coming. We're probably looking at long slow decline for the next 15 years or so until AI reaches a level where economics don't really make sense anymore, since economic systems by definition control the distribution of resources under scarcity. Without scarcity, they're pointless. And we moved into the age of artificial scarcity sometime after WWII, probably in the late 1960s, but certainly no later than 1990 with the fall of the USSR and the rise of straw man enemies like terrorism, using divisive politics as the primary means of controlling the population. Noam Chomsky saw this coming before most of us were born.
In other words, when anyone can wish for anything by turning sentences into 3D-printed manifestations of their dreams, then artificial scarcity quickly loses its luster. Because the systems of control around dependency no longer work. Then a new fear-based enemy comes along to fill the void, probably aliens. I wish I was joking.
Not sure we're quite there yet. A real movie needs a lot of dialogue, speech, sound effects, music, etc. Even the best LLM's don't do really coherent storytelling yet and a script for a movie is just the absolute barebones.
Please don't post snarky comments, or shallow dismissals, to HN. You may not owe BigCo better but you owe this community better if you're participating in it:
I would guess that it didn't take long running on Meta's servers, but if you were doing this at home with a 3090 it would take at least 6 minutes per 15 second video, assuming your settings take 15 seconds to render a single frame and there are 24 frames per second in the video.
A lot of people saying it's over for traditional movie-making - lmao. I look at these and see nothing but uncanny valley artifacts, and I don't think it will improve much from here.
It's like self-driving cars. They use almost very effective statistical models, certainly better than our previous models, but they never seem to shake off that "almost" and become truly effective.
The first thing I thought was the exact opposite. This isn't very good, but it's only version 1. Motion pictures are less than 150 years old. In another 150 years I bet virtual filmmaking will progress a lot.
Case in point, just compare the results of Stable Diffusion / Dall-E / midjourney to papers from 5-10 years ago (here's a random one I found http://proceedings.mlr.press/v48/reed16.pdf). Remarkable progress in that time, and as I understand it, a good amount (but certainly not all) of the improvement comes “free” from being able to scale up to more weights.
I don't agree with the prediction but I will say that at least in America, we've been overly optimistic about almost everything since world war II. Is particularly painful when you look at the imagination of science fiction versus the reality of science in the realm of say space travel, especially interstellar travel. This imagination effect gets the public behind your new invention in part because it seems to be just the beginning the start of something new. The tip of the iceberg. But sometimes it's really just a tiny bit of ice.
For the record, I'm actually rather bullish on self-driving cars. There's nothing physically impossible about solving the problem, but I'm not surprised it's harder than it sounds. But I don't see humans being fundamentally prevented from solving the problem in the same way that humans are fundamentally prevented from ever engaging in everyday space travel.
Absolutely, particularly if you see this as simply another tool in the belt, like how Jurassic Park's thirty year old special effects absolutely stand the test of time because of them being a convincing mix of early CGI and puppetry.
It's not hard to imagine that this kind of thing could end up doing a lot of the heavy lifting for things like background scenes in the future, opening up the kind of stuff we saw in The Mandalorian, Game of Thrones, and the LOTR film trilogy to increasingly lower and lower budget productions.
Most of the AI generated images are good enough to be first drafts or sketches, and with some editing, they can be coherent images. This makes AI generators a good tool. Not sure the same is true for video though. It feels like you'll need reliable still image creation before you can get to a video generator that's useful since editing video takes a bit more than a still image.
The first Jurassic Park holds up unusually well. They used CGI for things they couldn't convincingly do any other way, and used practical effects for almost everything else. And for the most part they hid the CGI well, they took a lot of care to mix it up with close ups of practical effects, the way they framed the shots to direct audience attention, etc.
I think in a lot of modern stuff they go too far. They now use CGI for things that could easily be practical effects, but they go with CGI because it's simply cheaper or because they want to A/B test different colors of wall paneling behind a character in post production, or who knows what. The end result is apparently good enough to make a billion dollars, but I can't stand it. Movies don't feel authentic anymore. It's hard to describe rationally, but the word 'soulless' sums up how movies made the new way make me feel. Even the scenes which are wholly practical/real get degraded; the excessive CGI and compositing used in the rest of the movie cast a miasma of unrealness across the entire movie.
Haha, why did I know this was going to be a Critical Drinker video.
For real though, I think the best CGI has always been when it's a light touch, providing only slight enhancements to an otherwise mostly-practical scene. And that's also when it's most invisible— so while superhero movies are obvious CGI-fests and can be clearly said to have been "ruined" by it, I think the most interesting modern CGI use is in lower-budget productions like TV shows, enabling the insertion of fantasy- or period-themed backdrops that would never be possible if you had to actually come up with all those props and extras in real life.
And I think to the extent that this is already happening, it's a lot harder to track because it's so much more subtle than the bombastic in-your-face effects of a Marvel movie showdown.
It would be interesting to see a revival of a show like Drive [0] as that was pretty ambitious and expensive for its time, but might be a lot more possible to do now.
I remember listening to the DVD commentary track for X2 (from 2003), and loving that scene near the beginning where Nightcrawler is hiding out in a cathedral but losing control of his teleportation powers because he's being remotely controlled/possessed by the villain of the story. Bryan Singer talks on the track about how great it was to fling the character (can't remember if it it was actually Alan Cumming or a stuntman) from the rafters, and how much better and more weighty it looked to have a "real" shot and just have to remove wires/harnessing vs filming it as a blank canvas and having to insert the character digitally.
The key to predicting technological advancement, that is so often misunderstood by technologists, is the S-curve.
Laypeople tend to think everything is linear, technologists tend to think everything is exponential; more often than not, reality tends to be sigmoidal. Technologies have exponential takeoffs followed by logarithmic plateaus. We're clearly well into the exponential phase of deep-learning ML, but it's only a matter of time before this approach hits its logarithmic phase.
Of course, the hard part of sigmoidal prediction is determining where in the curve we are. Does the current paradigm have an even steeper part ahead of it? Maybe. And yet, we could just as easily be right in the middle of the function, with a leveling off coming as the state of the art gives way to incremental improvements.
The leap from "generating a frame" to "generating a video" is not as big as the leap from where we are now to a 'perfect' image synthesis engine(which would be required to get rid of artifacts).
The much more likely scenario IMO is that people get used to the artifacts and notice them less.
There is something with the AI needing to have a ‘sense’ for the world the scene exists in so longer videos can be created that are coherent. Currently we’ve only seen long videos that have no consistency and jump around a lot like an acid trip.
A good path forward is to fuse these image-element compositing tools with some of the 3d scene inference ones. So you start out with 'giant fish riding in a golf cart, using its tail to steer', then give that as ground truth to a modeling tool that figures out a fish and a wheeled vehicle well enough to reference some canonical examples with detailed shape and structure, the idea of weight etc. Then you build a new model with those and do some physics simulation (while maintaining a localized constraint of unreality that allows the fish to somehow stay in the seat of the golf cart).
We already have films that use "similar" technology (for example, adding Leia in the new SW movies), and this is just another step, another improvement into the direction of autogenerated videos. Are we far from generating complete movies? Sure! Are we progressing? Yes too!
> A lot of people saying it's over for traditional movie-making - lmao.
The groups supporting such absurd claims largely boil down to:
- Money-driven researchers in the applied AI field. Those are the people that spam popular ML conferences with barely novel contributions other than some minor tweaks on code from their previous papers.
- People unable to critically think and evaluate the significant limitations of SOTA methods occasionally marketed as AGI breakthroughs.
Last is virtually the entire HN userbase and the one that needs to be taken the least serious.
The first category is much more troubling however, since they can significantly influence research directions due to the broken citation system in academia (more citations --> higher quality contributions).
On your second point, a few years ago the SOTA was Google's Deep Dream.
I agree with your deeper criticism, though preferential attachment/ranking is very much How Humans Do Things. You could do a much improved citation system by expanding the time dimension and looking at papers that were unpopular at first and then attracted wide interest later.
Of course, academics also have a tendency to over-cite (because they don't want to be rejected for inadequate literature review), so there are incentives to cite a bunch of research whose premises or conclusions you hope to overturn.
I don't think AI will be able to create a movie anytime soon, but I think it will become "good enough" to serve as inspiration for creatives, or to replace simple stock footage (Much like SD and DALLE-2 is now).
If this technology (or its future iterations) is not enough you can always combine this with your movie cuts. This lowers the resources/bar needs for creative people and obviously also for the uncreative ones.
This tech went from nothing to beating a human artist in an art competition in a few years, and yet you say "I don't think it will improve much". So, I disagree.
It's not over for traditional moving-making. It would be decades before the software and hardware could surpass. But it will improve tremendously, just like computers do for nearly everything.
This is a good example of the most abject type of Luddism: telling yourself that a fast-developing tech you happen to dislike is already at or near the limit of its potential and that defects you notice now will be there forever.
It's not name calling to make a critique of an argument. Luddism is a real term that describes a particular philosophical tendency that places itself in opposition to what it views as a naive technophilia.
The Luddites were a labor rights movement reacting to the new shifting power dynamic of centralized production owned by a few, in an era with virtually no safety and worker rights regulations. How would you feel if your career evaporated and you were forced to choose between starvation and sending your children to work in a textile factory where they get maimed by the machines they're told to crawl into and repair? You probably wouldn't find much comfort in the popular retort of "but shirts are really cheap now!"
That Luddites have been successfully maligned as irrationally anti-technology crazy people is a propaganda victory by industrialist factory owners and their friends, the newspaper men.
Whatever their motivation, their ire is misguided and selfish. Is the world supposed to just sit and around and never innovate or try to become more productive? So people can have a job doing work better done by machines? I don't think so. The Luddites and any such analogs today are focused on the wrong thing - they should not attack new technology or the companies and individuals using them, but rather campaign for better social safety net from the government.
Maybe new technology should be restricted by default until regulation can catch up with it. I'm not sure, but the 'cheap shirts benefit everybody too much to pump the brakes' argument for unfettered innovation leaves me deeply unimpressed.
IMHO the best argument for unfettered innovation is the impossibility of slowing innovation globally; it can be slowed in one country, but that country can't force all the rest to get with that program and will eventually be overtaken by technologically superior foes.
Their video training set is still small (10M + 10M). A lot of the interpolation artifacts seems come from the model haven't acquired enough real-world understanding of "natural" movements (looking at the horse running and the sail-boat examples). I suspect scaling this up to 10x would have much less artifacts.
Reading the paper, it seems to be the "right" approach (separating temporal / spatial for both convolution and attention). Thus, I am optimistic what remains is to scale it up.
The horse leg motion in these examples is really poor. But to be fair, horse legs are very difficult for human animators too, generally necessitating the use of reference photos to get it right. The first time somebody got a series of photographs showing a horse in motion caused quite a stir: https://en.wikipedia.org/wiki/The_Horse_in_Motion
It will definitely improve from here onward. On the other hand, I agree that it's a ridiculous thing to say that traditional movie-making is over. There's so much involved in making a movie, so much happenstance from the actors performances in a specific environment. You will never be able to get this in AI.. you might get some sort of mimicry, but the comparison is futile.. a movie isn't just a sequence of changing pixels.
What you’re going to see is a race to the bottom, the same as with claymation films and 3d.
It will suddenly require a lot less (expensive, highly trained) people to make the same films.
You’ll still need expensive highly trained people to do it, but with different skills and a lot less of them can do a lot more a lot more quickly.
…and that means that some studios will make bad, low grade films… and some studios will make amazing films using hybrid techniques (like 3d printed faces for claymation).
…but overall, the people funding movies will expect to get more for less, and that will mean a downsizing of the number of people employed currently in certain roles.
Traditional film making over? Hm… it’s complicated. Is it over if the entire industry changes, but people are still making films? Or is that just the “traditional” part of it which is over?
It’s definitely going to change the industry.
People will still definitely film things.
…but, I wager, less people will be doing highly payed skilled manual work, which will replaced by a few people doing a different type of AI assisted work.
…and we’ll see some really amazing indy films, of small highly technical teams producing content with very little physical filming.
> A lot of people saying it's over for traditional movie-making
While these early versions are primitive, as a traditional filmmaker I think in a couple years these technologies will creatively empower visual storytellers in exciting new ways. The key will be developing interfaces which allow us to engage, direct and constrain the AI to help us achieve specific goals.
I'm not sure I'd agree with the "it won't improve much from here" sentiment, but I am a little confused by the sharp disagreements with this comment. They seem to contain a tacit assumption that the complexity curve for this problem is smoothe, but I could easily imagine a very sharp elbow in that curve, making any progress come to a seemingly grinding halt.
The only way it will be over is that tiktok/instagram will attract more eyeballs and hurt their bottom line. Ultimately this is for amateurs to create really short form content for the foreseeable future.
You are hilariously wrong if you think it will not improve. Combination of prompt engineering, better and bigger datasets, better architectures, bigger models, artifact-fixing tools and sheer human creativity will seriously challenge traditional movie-making within two years.
You'll see an excellent mostly- or all-AI feature within 5-10 years. There will be terrible ones before that, maybe 2-3 years. The first really good one will be enjoyed on its own merits, ad the artificiality of it will come to light after it has gained popularity.
I would imagine you’ll see one or a few that are barely AI-produced as soon as marketing teams get wind of this. “come and see the first ever AI-produced* blockbuster!
⠀
⠀
⠀
*produced, assisted, or discussed on set one time during lunchbreak”
I don’t want to think reality is a simulation, but wouldn’t our everyday experience being generated by a neural network be far far simpler to achieve than a universe of infinite minuscule atoms all interacting.. like ozcam’s razor is pointing towards our lives being a realistic dream.
Like in 10 years you could plug this tech into high end VR and get a prompted reality dynamically generated that would be indistinguishable from our own.
That what we're seeing is really reality is the simpler explanation. Because the alternative is that what we see is a simulation ... inside another reality that has full complexity. Which increases the overall complexity. Thus, Occam's Razor says what we see is likely real and not a simulation.
The ‘outside’ reality only needs to be the size of a data center, not the entire universe. And maybe the outside reality only needs to be a trained 100gb model. A much simpler outside reality than we have now. You don’t even need to sim 8 billion people, just 1, you.
Occans razor in this case applies to the complexity giving rise to a situation here though, not the complexity of a described system.
Conventional physics giving birth to our universe is currently the model with the fewest assumptions.
What would have to take place to give rise to a universe the size of a data center, running an AI model of a human? It feels like we have to bake in assumptions of stable physics, a rise of a stable system for that data center, and some path towards creating it and modelling a human.
That said, if we believe we're capable of running billions of believable simulations, then we're more likely to be in such a simulation than ground reality. But a datacenter pocket dimension bakes in a lot of assumptions that make it less likely than our own universe.
The outside universe doesn’t require our complex particles or physics - all that is overkill to run a neural net. Also think of all the data we can store in a fingernail, stable diffusion is like 4gb, what could a 4tb model produce?
You only need a universe the size of a data center with nothing in it but a bit of vacuum fluctuation that causes particles to appear sometimes. Then just wait.
Check out the original simulation argument paper[0]. The issue is, if we think we are heading to a world in which we can do simulations, it becomes increasingly likely that we are in one of those (presumably very many) worlds, rather than in the one world that existed before the advent of such simulations.
Sounds good because it seems like it explains what our reality is, but it really doesn't. It just pushes the fundamental problem up X simulations into the supposed "real reality". It also assumes that simply simulating a universe would generate consciouss beings which nobody knows the answer to, but my guess is that it would not.
These particle physics are still needed. We generally assume the dream you have at night does not take place within a whole universe, yet it is still governt by some rules - for example you may experience gravity within your dream and understand that to move around when actually there isn't any gravity but it seems like it is.
It's equally important to understand and study the rules of this waking dream you are having while reading, not because there are actually atoms out there but because it behaves like there are. You can do physics without metaphysics, altough the two usually get entangled somewhat.
The neural net itself needs something to run in (some universe). If the argument is that it needs a smaller or simpler universe than ours: first of all it is simulating our universe so all complexity of ours is included anyway, but also, maybe a neural net in a big chaotic universe is more likely than a tiny universe designed to perfectly fit one neural net
Neural nets as an abstract concept don’t necessarily need to be built with atoms, just the substance of the external universe which may be far simpler.
I also didn't imply this neural network's universe needs atoms, but it needs to run the computation somehow, the computation itself is the complexity, no matter with what physics or other manifestation it shows up
And if it is simulating our universe's atoms, then that part of it basically is our atoms. But is a neural net doing that really simpler than the atoms just running with a more direct mathematical model?
You don’t need atoms, just a universe with the minimal blocks to build a turning machine. Like building a computer in Minecraft they could conceivably run these neural nets without any sense of ‘atoms’. The point is the outside universe could have extremely simple design.
ozcam’s razor is a human made concept, the universe doesn't obey human laws, it's the opposite
> Like in 10 years you could plug this tech into high end VR and get a prompted reality dynamically generated that would be indistinguishable from our own.
They said that 10 years ago about VR and it still is dog shit
Exactly. The simulation theory proponents almost always seem to presume the existence of some sort of being that deliberately created the simulation. What if the simulation is a complete accident that has arisen from the complex interaction of brainless bacteria competing for scarce resources on the surface of a big rock? An accidental simulation emerging from randomly initialized 'cellular automata' on one rock in a host universe containing trillions of such rocks.
The usual presumption of the simulation being a deliberate construction of a conscious being makes the whole thing seem like nu-religion for people who reject supernatural things. With the presumption of a being deliberately creating the situation, you pull in these notions: We're special, we exist on purpose, we are probably being examined and judged. This reeks of religion.
Because even if our reality came from that rock bacteria, we're on the cusp of creating simulations within and statistically beings may still expect to be in a deliberate simulation, if we think we'd eventually simulate more beings than exist.
The rules of the presumed host universe are unknown, but which do you guess is more common in ours? trillions upon trillions of rocks, may of them probably covered in organic goo interacting with itself, or intellectually sophisticated computer scientists deliberately designing simulations? I think it's got to be the goo.
If you're assuming each atom is independent of other atoms then yes. But if you believe the universe is deterministic and atoms are just following the laws of physics then you can think of the universe as nothing more than a computer with electrons instead of atoms just floating around.
In the sim theory, the external neural net tells us our brain is made of atoms.. you could theoretically feed your senses a Minecraft reality and you'd think your head is full of interacting blocks - which is still capable of performing the same NN functions as a wet brain... So I don't know, the brain could be made out of anything that fits the requirements of running the computation.
That's assuming there even really is one - senses can be hijacked, just like in dreams, so we may only think we have a brain - so strange.
Why then program the brain to try to understand that it's in a simulation when the goal was to create a realistic simulation in the first place? That'd be a rookie mistake on part of the simulation developers.
Ever wonder why you weren't born a medieval peasant?
Well, from the outside reality's perspective, it's helpful for people to spend the first few decades of their lives in an early 21st century simulation, just so they can gradually acclimate themselves to all this technology.
Yeah, my grandfather once said that he lived in the most exciting possible time, having been born before the first automobile, and having lived to see a man on the moon.
But even so, this era feels like it could be a singular phase shift. Maybe.
What I want to see is a model that can generate 3D models for use in applications such as Blender. It would provide a good starting point for someone with talent to make beautiful. Or just save people like me time for making games.
I've been looking into some of the 3D model generators this past week, and there is some work happening in that field. See the following non-exhaustive list:
Have you by chance tried out NeROIC? I'm a 3D printing enthusiast, mostly video game stuff, and it seems like like it would be excellent for that purpose.
I actually have been trying it out this week, and in fact it's currently trying to process the video generation, like their example shows. While I was able to follow their steps for training using their dataset, and generate the lighting/depth maps for the milkcarton example, the video generation is taking a long time (over 24 hours so far, using a 3070Ti with 8GB VRAM).
From what I understand with NeROIC, it's not particularly meant to be able to generate an 3D model that can be imported into Blender (or other software). It requires more work to take the meshes it generates to do something with it. See https://github.com/snap-research/NeROIC/issues/10
I too was looking into it to generate 3D models for some software I've been working on.
Thanks! I imagine one could use something like ZBrush's dynamesh to create a usable mesh from the output. Shame the library doesn't provide it by default, though.
Funnily enough, you might not even need a 3D model if the txt2video is good enough. Whatever you wanted to render in Blender could just be rendered via text prompt (when this becomes 100x better).
I'm very interested in what will come out of this new (sub)medium. By virtue of video being a collaborative medium, I never feel like I'm getting a message from a singular consciousness like I do from less resource-intensive mediums like books (I know that book editors exist, but the medium has less filters to pass through compared to large products like movies). I could see this substantively lowering the barrier of entry for video and enabling a lot of new stories to be told.
You can view all their example videos at https://make-a-video.github.io (warning: all the ~95 videos are webp and are loaded at once on the page so it may take some time to load)
"Our research takes the following steps to reduce the creation of harmful, biased, or misleading content."
"Our goal is to eventually make this technology available to the public, but for now we will continue to analyze, test, and trial Make-A-Video to ensure that each step of release is safe and intentional."
Are they really going to do a replay of OpenAI and Stable Diffusion? Deja vu coming soon.
What does "intentional" mean here? Does it mean that the person intends to type the text that they do? My cat sometimes walks across the keyboard, but for the most part I pretty much intend to type everything I type. Am I missing something?
It's referring to the release of the software. I read it as "released with minimal embarassing unintended features". Like, as a rough made-up example, a "boy saluting" prompt producing a video of a Hitler youth giving a Nazi salute wouldn't make for great PR.
Wonder if all these things will bring about a kind of cambrian explosion of creativity.
Imagine a future of Prompt Wizards, who are able to coax the AI to generate things in a very specific way.
Although we would probably need a much greater level of human curation. The way algorithms curate on youtube and spotify just doesn't really hit the spot.
Perhaps stability and Dall-e already kind of showed that the value is not so much in the physical act of creating, but more-so in the ability to express something that the AI can represent and which can connect with you.
We already have evidence that the explosion in availability decreases the overall quality, even if the quantity of high-quality productions also increases greatly.
Every creative platform is going to be flooded with the equivalent of a Reddit comment.
> Wonder if all these things will bring about a kind of cambrian explosion of creativity.
Maybe, if memes are the peak of creativity. (And who am I to judge?)
If we look at static images, the bar for distribution is zero and bar for creation is near zero since the arrival of these new AI tools – though it was circling zero before that.
And what seems to be most widely shared is memes, in my feeds anyway. When Stable Diffusion landed, people giggled about the president of my country rendered in the style of Grand Theft Auto. After a week of that my feeds went right back to memes.
Every human with an internet connection can draw like Picasso now, but it doesn't seem to matter. Because what we mostly seem to want is to take part in a conversation and get some validation, it seems to me.
Think about 25 or 30 or 50 years down the line. More interesting than the next couple of years. Speculate a bit.
After Neuralink and a few other companies torture enough poor monkeys, they eventually figure out how to create high bandwidth brain computer interfaces.
We go through a few more paradigm shifts in computing and get 10000-1000000 X or more performance increase. Metaverse protocols have advanced to allow for seamless integration of simulated persons and environments across multiple clusters.
The software continues to improve.
What you could get is a simulated realm with simulated AI characters living their own lives. But groups of real people are plugged directly in to the simulation and can influence it with their thoughts. There may be some sort of rules to ensure a certain level of stability. But basically you just think "there should be a storm today" and maybe visualize some strong winds. And then it happens in their world.
So at that point we become Gods.
I am probably getting carried away because I am tired.
>Our goal is to eventually make this technology available to the public, but for now we will continue to analyze, test, and trial Make-A-Video to ensure that each step of release is safe and intentional.
I remember the threads about StableDiffusion less than 4 weeks ago where people were confidently saying this kind of thing is 6 to 12 months away. Turns out it was a month. Singularity soon?
398 comments
[ 2.9 ms ] story [ 303 ms ] thread> A golden retriever eating ice cream on a beautiful tropical beach at sunset, high resolution
example is terrifying.
But actually, this technology is super exciting. Imagine a future where movies and games are choose your own adventure.
Besides just textural content, it's intriguing to consider the possibilities of full-3d roguelikes.
https://makeavideo.studio/assets/a_golden_retriever_eating_i... (webp)
That grasp though.
These things still feel a bit like e.g. Google/GCP services to me: Super appealing at first glance, quite close to what you want, but somehow never quite there. Maybe they'll asymptotically get there, eventually? Perhaps that statistical model can't really make it to the level we want it to?
Never say never, we've come a long way since GPT-2! All this was unthinkable back then
Image-to-image and tuning already addresses many of these issues; just as inpainting works really well, it won't be long before we have select-and-repair, where you add an additional prompt like 'improve this part - the ice cream is fine, just work on the dog's muzzle.'
[1] https://makeavideo.studio/assets/A_knight_riding_on_a_horse_...
The paper talks about "pseudo 3D attention layers" that are used in place of temporal attention layers for each dimension due to memory consumption. It seems like AI research is vastly outpacing GPU development.
Even then, these videos are only like 50 frames long - and a real movie you would want to be hundreds of thousands of frames long.
We can’t do it. AIs can sort of do it.
Latent diffusion models already demonstrated that operating on a compressed representation gives far better results, faster, but I don’t think we’re anywhere near the limit for what’s possible there. It’s no coincidence that this is how humans work.
They put an eye tracker on someone and captured their motion when walking in some rough terrain. You can sort of see that the person is focusing on the most likely place their foot will go next.
[1] https://www.youtube.com/watch?v=ph6uUHq3a-g
I think that we will discover that there is a more efficient way to encode temporal relationships, which appears to be "just throw transformers at it." My guess is that it will be in a more conceptual latent space that this attention will be applied.
Yes, but consider that most films are made up of many different shots, each of which are often just seconds long.
Obviously you could do 'human assisted' movie making where humans decide the storyboard and make directions for each shot, and then that isn't necessary.
It's a good thing to be fair, forcing research teams to optimize their projects is beneficial and creates a competition for limited resources. This gets a bit skewed when we consider a university research team vs. a MANGA type company, but the team behind Stable diffusion proved that innovation can come from unexpected places.
Though I must admit that if I didn't have friends holding my hand through the minefield of modern cinema, I would also just stick to books.
Why spend money on a film with new IP and ideas that you're not sure will be popular when the data science team has already worked with marketing to figure out exactly what movie will sell well?
Good luck finding your movie with compelling and thought provoking writing in the big pile of movies produced by comittee to show up above yours in discovery algorithms.
You want a thought-provoking Bourne-style action thriller with hints of Jane Austen and a bollywood dance sequence? How about a Matrix sequel that lives up to the first one, but ends just the way you like it? Just ask.
Or how about the movie Clue with the three endings except an infinite sequence of "or maybe it happened this way" sequences. I mean how else are we going to get a sequence where Darth Vader and Tim Curry reenact the "No I am your father" scene while Martin Mull dies in the background due to a heart attack.
You could conceivably write a script and feed it into a machine and have a decent 1080p rendition of the movie with consistent characters and voice acting which you could use to better pitch your movie idea to people, or get to watch a movie you created between you and your friends even if no one else ever gets to see it.
The current trend of remaking movies as 10 hour miniseries (and then making more and more seasons) is Not Great. Whereas I could be fascinated by a quirky-but-compelling original movie, I'm less attracted to 10+ hours of hyper-polished content. Sometimes I've watched a series and thought 'that was good, but it could have been a better movie.'
In the future, kids will be making their own Star Wars movies from home. All kinds of people from all kinds of backgrounds will make novel films that would ordinarily never be made, such as "Steampunk Vampires of Venus", starring John Wayne, a young Betty White, and Samuel L. Jackson. This is absolutely the future.
I'm working on building this. I'm sure lots of others are too.
We're in a situation where the very best algorithms (like the one used by Netflix), doing exactly what they're designed to do, create inequality and the vacuous economy of influencers we have today. Look at Steam or any other marketplace: they're all the same, with 1% of the players getting 99% of the prizes. In a very real way, the only winning move is not to play.
I would suggest that this tendency of capitalism (economic evolution) is unstoppable, and that it must be attacked from a different angle. If we don't want to inevitably end up in late-stage capitalism that looks like neofeudalism, then there has to be some form of redistribution or people spend the entirety of their lives running the rat race to make rent. Traditionally that was high taxes on the winners, but UBI would probably work better. Unfortunately, the very same people who win are the ones most resistant to any notion of a level playing field or social safety net.
So I feel like there may be no solution coming. We're probably looking at long slow decline for the next 15 years or so until AI reaches a level where economics don't really make sense anymore, since economic systems by definition control the distribution of resources under scarcity. Without scarcity, they're pointless. And we moved into the age of artificial scarcity sometime after WWII, probably in the late 1960s, but certainly no later than 1990 with the fall of the USSR and the rise of straw man enemies like terrorism, using divisive politics as the primary means of controlling the population. Noam Chomsky saw this coming before most of us were born.
In other words, when anyone can wish for anything by turning sentences into 3D-printed manifestations of their dreams, then artificial scarcity quickly loses its luster. Because the systems of control around dependency no longer work. Then a new fear-based enemy comes along to fill the void, probably aliens. I wish I was joking.
https://news.ycombinator.com/newsguidelines.html
It's like self-driving cars. They use almost very effective statistical models, certainly better than our previous models, but they never seem to shake off that "almost" and become truly effective.
Any more specific reason why you feel this way? Curious
That's a bold prediction. Why do you think that?
The first thing I thought was the exact opposite. This isn't very good, but it's only version 1. Motion pictures are less than 150 years old. In another 150 years I bet virtual filmmaking will progress a lot.
For the record, I'm actually rather bullish on self-driving cars. There's nothing physically impossible about solving the problem, but I'm not surprised it's harder than it sounds. But I don't see humans being fundamentally prevented from solving the problem in the same way that humans are fundamentally prevented from ever engaging in everyday space travel.
It's not hard to imagine that this kind of thing could end up doing a lot of the heavy lifting for things like background scenes in the future, opening up the kind of stuff we saw in The Mandalorian, Game of Thrones, and the LOTR film trilogy to increasingly lower and lower budget productions.
I think in a lot of modern stuff they go too far. They now use CGI for things that could easily be practical effects, but they go with CGI because it's simply cheaper or because they want to A/B test different colors of wall paneling behind a character in post production, or who knows what. The end result is apparently good enough to make a billion dollars, but I can't stand it. Movies don't feel authentic anymore. It's hard to describe rationally, but the word 'soulless' sums up how movies made the new way make me feel. Even the scenes which are wholly practical/real get degraded; the excessive CGI and compositing used in the rest of the movie cast a miasma of unrealness across the entire movie.
https://64.media.tumblr.com/248d25e2185a58bf827d329490480fb9...
This video does a good job of explaining why, I think: https://www.youtube.com/watch?v=DY-zg8Oo8p4
For real though, I think the best CGI has always been when it's a light touch, providing only slight enhancements to an otherwise mostly-practical scene. And that's also when it's most invisible— so while superhero movies are obvious CGI-fests and can be clearly said to have been "ruined" by it, I think the most interesting modern CGI use is in lower-budget productions like TV shows, enabling the insertion of fantasy- or period-themed backdrops that would never be possible if you had to actually come up with all those props and extras in real life.
And I think to the extent that this is already happening, it's a lot harder to track because it's so much more subtle than the bombastic in-your-face effects of a Marvel movie showdown.
It would be interesting to see a revival of a show like Drive [0] as that was pretty ambitious and expensive for its time, but might be a lot more possible to do now.
[0]: https://en.wikipedia.org/wiki/Drive_(2007_TV_series)
I remember listening to the DVD commentary track for X2 (from 2003), and loving that scene near the beginning where Nightcrawler is hiding out in a cathedral but losing control of his teleportation powers because he's being remotely controlled/possessed by the villain of the story. Bryan Singer talks on the track about how great it was to fling the character (can't remember if it it was actually Alan Cumming or a stuntman) from the rafters, and how much better and more weighty it looked to have a "real" shot and just have to remove wires/harnessing vs filming it as a blank canvas and having to insert the character digitally.
Laypeople tend to think everything is linear, technologists tend to think everything is exponential; more often than not, reality tends to be sigmoidal. Technologies have exponential takeoffs followed by logarithmic plateaus. We're clearly well into the exponential phase of deep-learning ML, but it's only a matter of time before this approach hits its logarithmic phase.
Of course, the hard part of sigmoidal prediction is determining where in the curve we are. Does the current paradigm have an even steeper part ahead of it? Maybe. And yet, we could just as easily be right in the middle of the function, with a leveling off coming as the state of the art gives way to incremental improvements.
The much more likely scenario IMO is that people get used to the artifacts and notice them less.
A good path forward is to fuse these image-element compositing tools with some of the 3d scene inference ones. So you start out with 'giant fish riding in a golf cart, using its tail to steer', then give that as ground truth to a modeling tool that figures out a fish and a wheeled vehicle well enough to reference some canonical examples with detailed shape and structure, the idea of weight etc. Then you build a new model with those and do some physics simulation (while maintaining a localized constraint of unreality that allows the fish to somehow stay in the seat of the golf cart).
The groups supporting such absurd claims largely boil down to:
Last is virtually the entire HN userbase and the one that needs to be taken the least serious.The first category is much more troubling however, since they can significantly influence research directions due to the broken citation system in academia (more citations --> higher quality contributions).
I agree with your deeper criticism, though preferential attachment/ranking is very much How Humans Do Things. You could do a much improved citation system by expanding the time dimension and looking at papers that were unpopular at first and then attracted wide interest later.
Of course, academics also have a tendency to over-cite (because they don't want to be rejected for inadequate literature review), so there are incentives to cite a bunch of research whose premises or conclusions you hope to overturn.
Ok? It's a nascent technology. Look at the original DALL-E blog post from last year [1]. Now compare it to DALL-E 2 and Stable Diffusion.
[1]: https://openai.com/blog/dall-e/
I don't think AI will be able to create a movie anytime soon, but I think it will become "good enough" to serve as inspiration for creatives, or to replace simple stock footage (Much like SD and DALLE-2 is now).
It's not over for traditional moving-making. It would be decades before the software and hardware could surpass. But it will improve tremendously, just like computers do for nearly everything.
It's just the first version and seeing Stable Diffusion come out while openai's tools were coming out were something to remember and think about.
https://news.ycombinator.com/newsguidelines.html
That Luddites have been successfully maligned as irrationally anti-technology crazy people is a propaganda victory by industrialist factory owners and their friends, the newspaper men.
IMHO the best argument for unfettered innovation is the impossibility of slowing innovation globally; it can be slowed in one country, but that country can't force all the rest to get with that program and will eventually be overtaken by technologically superior foes.
Reading the paper, it seems to be the "right" approach (separating temporal / spatial for both convolution and attention). Thus, I am optimistic what remains is to scale it up.
What you’re going to see is a race to the bottom, the same as with claymation films and 3d.
It will suddenly require a lot less (expensive, highly trained) people to make the same films.
You’ll still need expensive highly trained people to do it, but with different skills and a lot less of them can do a lot more a lot more quickly.
…and that means that some studios will make bad, low grade films… and some studios will make amazing films using hybrid techniques (like 3d printed faces for claymation).
…but overall, the people funding movies will expect to get more for less, and that will mean a downsizing of the number of people employed currently in certain roles.
Traditional film making over? Hm… it’s complicated. Is it over if the entire industry changes, but people are still making films? Or is that just the “traditional” part of it which is over?
It’s definitely going to change the industry.
People will still definitely film things.
…but, I wager, less people will be doing highly payed skilled manual work, which will replaced by a few people doing a different type of AI assisted work.
…and we’ll see some really amazing indy films, of small highly technical teams producing content with very little physical filming.
While these early versions are primitive, as a traditional filmmaker I think in a couple years these technologies will creatively empower visual storytellers in exciting new ways. The key will be developing interfaces which allow us to engage, direct and constrain the AI to help us achieve specific goals.
Exactly. Every step forward in creative technology is additional leverage for the artist with a vision.
I thought the same way you did about speech-to-text and image search, back in the day. boy was I wrong.
who is saying that?
You'll see an excellent mostly- or all-AI feature within 5-10 years. There will be terrible ones before that, maybe 2-3 years. The first really good one will be enjoyed on its own merits, ad the artificiality of it will come to light after it has gained popularity.
*produced, assisted, or discussed on set one time during lunchbreak”
Like in 10 years you could plug this tech into high end VR and get a prompted reality dynamically generated that would be indistinguishable from our own.
Conventional physics giving birth to our universe is currently the model with the fewest assumptions.
What would have to take place to give rise to a universe the size of a data center, running an AI model of a human? It feels like we have to bake in assumptions of stable physics, a rise of a stable system for that data center, and some path towards creating it and modelling a human.
That said, if we believe we're capable of running billions of believable simulations, then we're more likely to be in such a simulation than ground reality. But a datacenter pocket dimension bakes in a lot of assumptions that make it less likely than our own universe.
[0] https://www.simulation-argument.com/simulation.pdf
Remember in this case the simulation is not running a model of physical reality - it’s only running a neural net that is fed into your senses.
In that case there are no ‘atoms’ it’s just a concept fed into your mind. Just like in dream it’s hard to question reality, you just go along with it.
It's equally important to understand and study the rules of this waking dream you are having while reading, not because there are actually atoms out there but because it behaves like there are. You can do physics without metaphysics, altough the two usually get entangled somewhat.
And if it is simulating our universe's atoms, then that part of it basically is our atoms. But is a neural net doing that really simpler than the atoms just running with a more direct mathematical model?
> Like in 10 years you could plug this tech into high end VR and get a prompted reality dynamically generated that would be indistinguishable from our own.
They said that 10 years ago about VR and it still is dog shit
The original Oculus was a step change, and current VR is a step change from that.
8k VR is coming and will be close to indistinguishable from reality visually at least.
At the end of the day you're still sitting/standing out there with two tv screens 2cm from your eye balls
If the screen is wide enough with eye tracking and divested rendering, how could you tell the difference?
The usual presumption of the simulation being a deliberate construction of a conscious being makes the whole thing seem like nu-religion for people who reject supernatural things. With the presumption of a being deliberately creating the situation, you pull in these notions: We're special, we exist on purpose, we are probably being examined and judged. This reeks of religion.
The neural net itself is built on a much simpler substrate in an external universe.
That's assuming there even really is one - senses can be hijacked, just like in dreams, so we may only think we have a brain - so strange.
Well, from the outside reality's perspective, it's helpful for people to spend the first few decades of their lives in an early 21st century simulation, just so they can gradually acclimate themselves to all this technology.
/folly
But even so, this era feels like it could be a singular phase shift. Maybe.
https://twitter.com/EMostaque/status/1539881228773429249
https://github.com/snap-research/NeROIC
https://github.com/threedle/text2mesh
https://github.com/AutodeskAILab/Clip-Forge
https://nv-tlabs.github.io/GET3D/
From what I understand with NeROIC, it's not particularly meant to be able to generate an 3D model that can be imported into Blender (or other software). It requires more work to take the meshes it generates to do something with it. See https://github.com/snap-research/NeROIC/issues/10
I too was looking into it to generate 3D models for some software I've been working on.
[0] https://www.kaedim3d.com/
Let the AI generate random blender files.
Render them.
Train on source => render mapping (which is 1:many)
Repeat.
You should be able to get an incredibly high-fidelity decoder to go from image to blender source.
Then you're in luck today as this was just submitted: DreamFusion: Text-to-3D using 2D Diffusion
https://news.ycombinator.com/item?id=33025446
I didn't know you could be drifting with a horse.
"Our goal is to eventually make this technology available to the public, but for now we will continue to analyze, test, and trial Make-A-Video to ensure that each step of release is safe and intentional."
Are they really going to do a replay of OpenAI and Stable Diffusion? Deja vu coming soon.
Imagine a future of Prompt Wizards, who are able to coax the AI to generate things in a very specific way.
Although we would probably need a much greater level of human curation. The way algorithms curate on youtube and spotify just doesn't really hit the spot.
Perhaps stability and Dall-e already kind of showed that the value is not so much in the physical act of creating, but more-so in the ability to express something that the AI can represent and which can connect with you.
It's AI written blogspam, but for images and video too. The signal to noise ratio is getting worse and worse
Every creative platform is going to be flooded with the equivalent of a Reddit comment.
(yes I am aware of the irony)
Maybe, if memes are the peak of creativity. (And who am I to judge?)
If we look at static images, the bar for distribution is zero and bar for creation is near zero since the arrival of these new AI tools – though it was circling zero before that.
And what seems to be most widely shared is memes, in my feeds anyway. When Stable Diffusion landed, people giggled about the president of my country rendered in the style of Grand Theft Auto. After a week of that my feeds went right back to memes.
Every human with an internet connection can draw like Picasso now, but it doesn't seem to matter. Because what we mostly seem to want is to take part in a conversation and get some validation, it seems to me.
After Neuralink and a few other companies torture enough poor monkeys, they eventually figure out how to create high bandwidth brain computer interfaces.
We go through a few more paradigm shifts in computing and get 10000-1000000 X or more performance increase. Metaverse protocols have advanced to allow for seamless integration of simulated persons and environments across multiple clusters.
The software continues to improve.
What you could get is a simulated realm with simulated AI characters living their own lives. But groups of real people are plugged directly in to the simulation and can influence it with their thoughts. There may be some sort of rules to ensure a certain level of stability. But basically you just think "there should be a storm today" and maybe visualize some strong winds. And then it happens in their world.
So at that point we become Gods.
I am probably getting carried away because I am tired.
Examples: https://make-a-video.github.io/
Demo site: https://makeavideo.studio/
I am told live demo and open model are on the way.
This will absolutely snowball easy videos all over social media such as TikTok, Insta and YouTube.
Meta is falling fast imo