It’s a new take on what we called a camera lucida, which was a trick with optics that let you superimpose a scene over your view of a piece of paper. You could get the proportions right and then do your magic with color and technique. Not unlike how animators work.
If you’ve seen anything on animation prior to computers you’ve seen the classic technique of putting one sheet of cello over the last thing you drew and flipping between them to copy the shared details before changing tone thing for the next frame.
There are several instances of matching motion in older Disney features; it seems likely that instead of rotoscoping 3rd generation animation off of 2nd etc. they're all 2nd gen rotoscopings off an internal 1st generation live action reference library.
> Rotoscoping is the process of tracing over live-action footage frame-by-frame to create accurate and detailed mattes or masks.. ensuring a seamless blend between the foreground and background.. requires a keen eye for detail and patience, as it can be a time-consuming process.. Rotoscoping plays a crucial role in integrating visual effects seamlessly into live-action footage. Artists meticulously trace elements like explosions, fire, or magical effects to match the movements and perspectives of the scene. This process ensures the VFX elements appear natural and believably interact with the real-world elements.
> ..automatically synthetise retro-style 2D animations from videos. The method approaches the challenge mainly as an object segmentation and tracking problem. Video frames are processed with the Segment Anything Model (SAM) and the resulting masks are tracked through subsequent frames with DeAOT, a method of hierarchical propagation for semi-supervised video object segmentation. The geometry of the masks' contours is simplified with the Douglas-Peucker algorithm. Finally, facial traits, pixelation and a basic shadow effect can be optionally added. The results show that the method exhibits an excellent temporal consistency and can correctly process videos with different poses and appearances, dynamic shots, partial shots and diverse backgrounds.
One thing that they kind gloss over is that rotoscoping is used a lot in films to separate objects.
For example, I did an exam on prince caspian (https://eu.southcoasttoday.com/story/entertainment/local/200...) to separate his hair from the forest. They didn't have green screen, and even if they did, it would obliterate his hair. (hair is translucent so glows green, and is removed by the software.)
I failed the rotoscoping part, but passed the painting section (where you remove markers, crew or other features not wanted in the final image.)
Whilst "a scanner darkly" is a brilliant example of rotoscoping, its not the example. if you look at this showreel towards the end you see captain marvel, https://www.youtube.com/watch?v=_sYWzFNobEo look at her hair, every individual wisp of hair that extends past her neck is individually masked out.
Good rotoscoping is often the building block of "expensive" looking VFX.
For computer vision its also important so you can get ground truth on tracking various object and training models like segment anything
You are mixing up rotoscoping to do natural image matting with rotoscoping to paint over a frame. The original meaning was to match animation to a plate frame by frame as a sort of poor man's motion capture but that seems to be lost here.
For your second example it's impossible to find a YouTube video of Chika dance that is available in every country. Try this: https://streamable.com/0hrdh
It's remarkable how smooth the animation is. And how natural the character's movements are. This scene jumps over the uncanny valley straight into being enjoyable.
And I think there's an AI paper from a Chinese researcher using this video as a demo.
Rotoscoping is great but pretty labor intensive and not really creative in itself, though it can obviously be part of a very creative process. That makes it a perfect candidate for AI/ML tools to accelerate and I hope they do (or rather I hope they continue to). Tons of animators and filmmakers love the roto look but don't want to deal with the roto process. Get the outlines and motion tracking done 90% with the AI and everyone can focus on the important stuff.
21 comments
[ 2.5 ms ] story [ 52.9 ms ] threadRotoscoped Animation of Filmed Parkour - https://news.ycombinator.com/item?id=12380112 - Aug 2016 (49 comments)
https://news.ycombinator.com/item?id=41068176
If you’ve seen anything on animation prior to computers you’ve seen the classic technique of putting one sheet of cello over the last thing you drew and flipping between them to copy the shared details before changing tone thing for the next frame.
There is nothing new here, all this stuff is a century old at least.
what we called a camera lucida
Who is 'we' here? That technique is like a reverse overhead projector to see something real in front of you projected down on to a piece of paper.
That isn't necessary with film, you can project it from overhead or however you want.
https://en.wikipedia.org/wiki/Camera_lucida
> Rotoscoping is the process of tracing over live-action footage frame-by-frame to create accurate and detailed mattes or masks.. ensuring a seamless blend between the foreground and background.. requires a keen eye for detail and patience, as it can be a time-consuming process.. Rotoscoping plays a crucial role in integrating visual effects seamlessly into live-action footage. Artists meticulously trace elements like explosions, fire, or magical effects to match the movements and perspectives of the scene. This process ensures the VFX elements appear natural and believably interact with the real-world elements.
Feb 2024, https://arxiv.org/abs/2402.09883
> ..automatically synthetise retro-style 2D animations from videos. The method approaches the challenge mainly as an object segmentation and tracking problem. Video frames are processed with the Segment Anything Model (SAM) and the resulting masks are tracked through subsequent frames with DeAOT, a method of hierarchical propagation for semi-supervised video object segmentation. The geometry of the masks' contours is simplified with the Douglas-Peucker algorithm. Finally, facial traits, pixelation and a basic shadow effect can be optionally added. The results show that the method exhibits an excellent temporal consistency and can correctly process videos with different poses and appearances, dynamic shots, partial shots and diverse backgrounds.
Apr 2023, [SAM 1] "Segment Anything Model and the hard problems of computer vision", 52 comments, https://news.ycombinator.com/item?id=35558522
Jul 2024, "SAM 2: Segment Anything in Images and Videos", 147 comments, https://news.ycombinator.com/item?id=41104523
For example, I did an exam on prince caspian (https://eu.southcoasttoday.com/story/entertainment/local/200...) to separate his hair from the forest. They didn't have green screen, and even if they did, it would obliterate his hair. (hair is translucent so glows green, and is removed by the software.)
I failed the rotoscoping part, but passed the painting section (where you remove markers, crew or other features not wanted in the final image.)
Whilst "a scanner darkly" is a brilliant example of rotoscoping, its not the example. if you look at this showreel towards the end you see captain marvel, https://www.youtube.com/watch?v=_sYWzFNobEo look at her hair, every individual wisp of hair that extends past her neck is individually masked out.
Good rotoscoping is often the building block of "expensive" looking VFX.
For computer vision its also important so you can get ground truth on tracking various object and training models like segment anything
Flowers of Evil https://www.youtube.com/watch?v=Toc9x19Cmkg
Kaguya-sama Love is War scene: https://www.youtube.com/watch?v=Toc9x19Cmkg
It's remarkable how smooth the animation is. And how natural the character's movements are. This scene jumps over the uncanny valley straight into being enjoyable.
And I think there's an AI paper from a Chinese researcher using this video as a demo.
https://www.youtube.com/watch?v=eqjFmsZGBSc