I worked at a studio in London, I was basically a data wrangler, writing python tools to automate everything.
The render farm consisted of 250 nodes, each had 48 cores and 256GB of RAM. It was connected to a large storage array, total storage was approx 5PB.
In the evenings the desktop machines that the modellers worked on were turned into render nodes. These machines were given lighter scenes to render since we wanted them to complete overnight.
The longest frame I have seen has taken a 48 core node 60 hours to render a single frame.
The scheduling software (Qube) is responsible for retrying failed jobs. A frame render can fail for many reasons (e.g. Incorrectly packaged assets, missing textures, out of memory, bad lighting info, missing plugins, random SIGHUPs - rarer) - so if something screws up then Qube will retry N times if it can find available hardware.
From the artists point of view, they don't render a frame once, they continuously render and progressively increase the level of detail until they're happy with the scene .. this means the render farm is almost always busy.
I worked in VFX - so we were delivered the edited scenes and the artists would render on top of them ... theres very little post-processing done, what comes off the farm is fit for the client.
Although the farm schedules individual frames, they are combined into scenes (short clips from 2 - 30 seconds) and the artists get the scenes back. The scene file format is proprietary (and viewed with proprietary software), but I had a sneaky look inside and it seems to be losslessly compressed TGA frames. Scenes are usually short 2-6 seconds, and filesize would be 2-3 GB
There are two main bits of VFX developer work: tools and RnD.
Tools developers are there to allow artists to find, convert, import and send assets to a render queue. Most of this is written in python, as thats the embedded language of choice for most VFX packages (maya, nuke, $other)
Then there is RnD: this is most of the stuff you see at sigraph. We need to implement a new type of water caustic that doesn't suck, Model snow more realistically, or trying to make hair systems not an utter pain in the arse to use.
If you want to be in the tools department you'll need python, and some experience with maya/nuke/renderman/arnold.
If you want to be doing RnD you'll need to figure out what you want to play with and either apply to a company that trying to build their own renderer (MPC) or join a company that is already building one (disney, pixar, the foundry, autodesk)
Also, beware that its an industry thats in the second part of a massive globalisation heave. Jobs are clustered in vancouver & london. However there are large changes that mean lots of work is starting to gravitate to china and india. At the moment its low end stuff like paint and roto, but thats where the next generation comes from.
I also forgot to mention; at a well established shop everything is automated & integrated.
most workstations will have a network home partition, which means that your home directory is everwhere you log in. Its easier and miles faster to write something out to an NFS share than it is to use HTTP/database
Ten gig switches get saturated without trying too hard (Place I worked pulled a constant 15 gigabytes a second from storage. bursting to 35. thats bytes, not bits)
Render farms have been using cgroups for many years. you'll start to worry about scheduling rate of tasks.
The only things that are virtualised are things like the intranet and the wiki. everything else is on real steel
rendernodes are big, there are loads of them, but never enough. They are treated like cattle.
There are large data pipes. however if you deal with marvel, you'll not have internet access at your desk, it'll be through some horrid remote desktop system. If you're lucky you might get one way copy and paste.
Fluid sims have caches that are in the order of 10s of terabytes
So Marvel are wonderfully paranoid about footage leaks.
It didn't help that someone got drunk on a plane and left wolverine on an unencrypted laptop. (which wasn't a VFX person because it had sound, titles and non VFX shots)
They specify that there must be an air gap between "work side" and the internet.
This means that there must be two/three networks, A workstation network which has no internet, a Data IO network for sending and receiving assets from other companies and finally the remote desktop server for internet access.
Other things like USB disks are disabled on all non dataIO workstations.
I've heard that some places were being forced to disallow email access as well. However I think that was successfully argued against.
The hardest part is not having access to download libraries.
At all of the studios where I've interviewed for either tools jobs or R&D jobs, C++ seems to be lingua franca. Knowing Python is basically a plus, but not taken seriously.
For example, I used to work on portfolio construction systems that were written in a mix of Python and Cython. So while I typically list Python as my main programming language, this is far from just the dynamic typing stuff. I know more than I ever expected to about the CPython implementation and using the CPython API from Cython, writing extremely performant extension types and extension classes, wrapping pre-existing C libraries and exposing them, with no performance degradation, through easy to use Python APIs, and a number of related skills.
This has universally been a complete non-starter. I even studied computer vision in grad school and have some specialized experience with bi-directional reflectance functions, GPU optical flow estimation algorithms, and face detection.
Having no previous job experience in C++ has always been a literally instantaneous deal-breaker regardless. It's actually caused me to give up hope and just stop applying to VFX roles all together.
14 comments
[ 3.3 ms ] story [ 34.9 ms ] threadI worked at a studio in London, I was basically a data wrangler, writing python tools to automate everything.
The render farm consisted of 250 nodes, each had 48 cores and 256GB of RAM. It was connected to a large storage array, total storage was approx 5PB.
In the evenings the desktop machines that the modellers worked on were turned into render nodes. These machines were given lighter scenes to render since we wanted them to complete overnight.
The longest frame I have seen has taken a 48 core node 60 hours to render a single frame.
What else would you like to know?
From the artists point of view, they don't render a frame once, they continuously render and progressively increase the level of detail until they're happy with the scene .. this means the render farm is almost always busy.
Although the farm schedules individual frames, they are combined into scenes (short clips from 2 - 30 seconds) and the artists get the scenes back. The scene file format is proprietary (and viewed with proprietary software), but I had a sneaky look inside and it seems to be losslessly compressed TGA frames. Scenes are usually short 2-6 seconds, and filesize would be 2-3 GB
Do you think that someone having experience with backend / web development with Python would do well in this kind of job?
Thanks
Thanks
There are two main bits of VFX developer work: tools and RnD.
Tools developers are there to allow artists to find, convert, import and send assets to a render queue. Most of this is written in python, as thats the embedded language of choice for most VFX packages (maya, nuke, $other)
Then there is RnD: this is most of the stuff you see at sigraph. We need to implement a new type of water caustic that doesn't suck, Model snow more realistically, or trying to make hair systems not an utter pain in the arse to use.
If you want to be in the tools department you'll need python, and some experience with maya/nuke/renderman/arnold.
If you want to be doing RnD you'll need to figure out what you want to play with and either apply to a company that trying to build their own renderer (MPC) or join a company that is already building one (disney, pixar, the foundry, autodesk)
Also, beware that its an industry thats in the second part of a massive globalisation heave. Jobs are clustered in vancouver & london. However there are large changes that mean lots of work is starting to gravitate to china and india. At the moment its low end stuff like paint and roto, but thats where the next generation comes from.
most workstations will have a network home partition, which means that your home directory is everwhere you log in. Its easier and miles faster to write something out to an NFS share than it is to use HTTP/database
Ten gig switches get saturated without trying too hard (Place I worked pulled a constant 15 gigabytes a second from storage. bursting to 35. thats bytes, not bits)
Render farms have been using cgroups for many years. you'll start to worry about scheduling rate of tasks.
The only things that are virtualised are things like the intranet and the wiki. everything else is on real steel
rendernodes are big, there are loads of them, but never enough. They are treated like cattle.
There are large data pipes. however if you deal with marvel, you'll not have internet access at your desk, it'll be through some horrid remote desktop system. If you're lucky you might get one way copy and paste.
Fluid sims have caches that are in the order of 10s of terabytes
It didn't help that someone got drunk on a plane and left wolverine on an unencrypted laptop. (which wasn't a VFX person because it had sound, titles and non VFX shots)
They specify that there must be an air gap between "work side" and the internet.
This means that there must be two/three networks, A workstation network which has no internet, a Data IO network for sending and receiving assets from other companies and finally the remote desktop server for internet access.
Other things like USB disks are disabled on all non dataIO workstations.
I've heard that some places were being forced to disallow email access as well. However I think that was successfully argued against.
The hardest part is not having access to download libraries.
For example, I used to work on portfolio construction systems that were written in a mix of Python and Cython. So while I typically list Python as my main programming language, this is far from just the dynamic typing stuff. I know more than I ever expected to about the CPython implementation and using the CPython API from Cython, writing extremely performant extension types and extension classes, wrapping pre-existing C libraries and exposing them, with no performance degradation, through easy to use Python APIs, and a number of related skills.
This has universally been a complete non-starter. I even studied computer vision in grad school and have some specialized experience with bi-directional reflectance functions, GPU optical flow estimation algorithms, and face detection.
Having no previous job experience in C++ has always been a literally instantaneous deal-breaker regardless. It's actually caused me to give up hope and just stop applying to VFX roles all together.