We use this in production every day. We've hit a couple bugs, but it's a surprisingly stable piece of software.
We hit an issue with user namespaces, and it not correctly chowning things when "exporting" them into Docker containers. I honestly forget how we solved this, but I believe it was just an upgrade.
We hit an issue with the first run of a Docker container on a system being extremely slow, to the point where downstream systems would give up, and retry. The issue was fixed by starting a container that leverages the driver at boot time, "warming" it up.
It's mainly the speed of creating a huge number of small files on disk. When unpacking a docker image into an AUFS file system, it has to create every file in the shadow system. A robust Ubuntu install has around 100000 files. On AWS with default EBS storage, this can take a few minutes. At OpenAI we pre-built AMI instances with file systems pre-unpacked.
Given the description of your workaround, you may be running into an issue of long GPU driver load times for the nvidia driver. If that is the root cause the recommended solution for this is nvidia-persistenced, though it seems you may have rolled your own equivalent.
Would this be useful for people who wanted to leverage GPUs for deep learning, but didn't have the wherewithal or the willingness to set up all the dependencies on the host machine?
There's another unexpected bonus to nvidia-docker; CNTK leaks memory if you abort training prematurely, causing the GPU to go OOM. But since the memory is owned by the nvidia-docker process, killing-and-restarting the container fixes the issue without having to restart the entire system.
At FloydHub, we use nvidia-docker in production for running DL jobs. It has been very solid for the past 6 months or so. We have also built a collection of open source DL docker images for various frameworks and we actively maintain them.
Check out VirtuaGL and TurboVNC. It is possible to run these two things inside a docker container, install a DE and accelerate a GUI app. I've been doing it for quite a while.
Also for all those checking the OPs original lib out. It does work and it is great and also probably the most mature option, but my understanding is that future focus for GPU inside containers is with this newer lib, also looked after by Nvidia:
I'm fairly familiar with this stuff, though not an expert, for my startup http://realityzero.one. But, I've not been paying that much attention to this area of late, so if I've got this wrong then please feel free to correct me. Perhaps they're for different things. I know there's been some discussion from the k8n crowd about the newer lib in k8n's github issues.
In a nutshell, you can give the container access to the NVIDIA device file via the `--device` flag, so all you need is a container with the NVIDIA drivers. The added benefit of this is that you can use different versions of the drivers side-by-side (in my understanding).
I thought this might be relevant as some people might not want to use the `docker-nvidia` CLI to run containers (I'm not sure how you would use this via the Docker API for example).
> The added benefit of this is that you can use different versions of the drivers side-by-side (in my understanding).
No, you can only have one driver version, the one that correspond to the loaded kernel modules. Installing the driver inside a Docker image makes it non-portable.
One of our engineers recently shared an intro-level talk at the Docker meetup about a ~year of (happy) experiences with this: https://www.meetup.com/Docker-Santa-Clara/events/240641246/ . It should be recorded somewhere, and may be helpful if you're considering production use, not just personal DL stuff.
docker has been great for this in the lab: only one person now goes through the pain to get latest framework + hack + model tweak work and then the rest can reuse.
If you're using DC/OS you can also run docker containers that leverage GPUs. We worked with Nvidia to mimic the same functionality provided by nvidia-docker. Here's a talk I gave at this years Nvidia GTC conference explaining how it works:
I use NVidia-Docker extensively in my Open Source project Deep Video Analytics [1] when combined with TensorFlow (which allows explicit GPU memory allocation) its unbeatable in running multiple inference models on a single GPU in a reliable manner. Combining this setup with docker volumes on AWS EFS allows simple multi machine deployments.
Certain algorithms/models are implemented in PyTorch or Caffe and typically it's huge amount of work to convert them to TensorFlow while ensuring correctness / Parity. Also I personally like design of PyTorch.
Isn't AMD leaving a lot of potential revenue on the table without a comparable amdgpu-docker?
If I'm building an ML cluster, I'm going to go with the vendor that's easiest to containerize and deploy, and right now it looks like nvidia has a commanding advantage in software.
Why doesn't AMD throw a few hundred thousand dollars at some developers to get containerization parity?
34 comments
[ 5.2 ms ] story [ 85.0 ms ] threadWe hit an issue with user namespaces, and it not correctly chowning things when "exporting" them into Docker containers. I honestly forget how we solved this, but I believe it was just an upgrade.
We hit an issue with the first run of a Docker container on a system being extremely slow, to the point where downstream systems would give up, and retry. The issue was fixed by starting a container that leverages the driver at boot time, "warming" it up.
I've observed the same thing but it seemed to be IO related, not CPU or GPU related.
https://github.com/NVIDIA/nvidia-persistenced
Docs here http://docs.nvidia.com/deploy/driver-persistence/index.html
There's another unexpected bonus to nvidia-docker; CNTK leaks memory if you abort training prematurely, causing the GPU to go OOM. But since the memory is owned by the nvidia-docker process, killing-and-restarting the container fixes the issue without having to restart the entire system.
They actually created this service using that (a Deep Learning "as a service"): https://www.floydhub.com/
[1]: https://hub.docker.com/r/floydhub/ [2]: https://github.com/floydhub/dockerfiles
Check out VirtuaGL and TurboVNC. It is possible to run these two things inside a docker container, install a DE and accelerate a GUI app. I've been doing it for quite a while.
Also for all those checking the OPs original lib out. It does work and it is great and also probably the most mature option, but my understanding is that future focus for GPU inside containers is with this newer lib, also looked after by Nvidia:
https://github.com/NVIDIA/libnvidia-container
I'm fairly familiar with this stuff, though not an expert, for my startup http://realityzero.one. But, I've not been paying that much attention to this area of late, so if I've got this wrong then please feel free to correct me. Perhaps they're for different things. I know there's been some discussion from the k8n crowd about the newer lib in k8n's github issues.
https://stackoverflow.com/questions/25185405/using-gpu-from-...
In a nutshell, you can give the container access to the NVIDIA device file via the `--device` flag, so all you need is a container with the NVIDIA drivers. The added benefit of this is that you can use different versions of the drivers side-by-side (in my understanding).
I thought this might be relevant as some people might not want to use the `docker-nvidia` CLI to run containers (I'm not sure how you would use this via the Docker API for example).
> The added benefit of this is that you can use different versions of the drivers side-by-side (in my understanding).
No, you can only have one driver version, the one that correspond to the loaded kernel modules. Installing the driver inside a Docker image makes it non-portable.
https://github.com/umautobots/driving-in-the-matrix
docker has been great for this in the lab: only one person now goes through the pain to get latest framework + hack + model tweak work and then the rest can reuse.
https://drive.google.com/file/d/0B7vZqCY-AJrpMEYyVmZkVGlyOEE...
https://docs.google.com/presentation/d/1q6brajoQkFZVtSocMPGg...
Floydhub also curates most frameworks as nvidia-docker images: https://hub.docker.com/r/floydhub/
[1] https://github.com/AKSHAYUBHAT/DeepVideoAnalytics
Tensorflow is apache licensed. I think in general, the perception is that it is far safer to stay away from caffe.
If I'm building an ML cluster, I'm going to go with the vendor that's easiest to containerize and deploy, and right now it looks like nvidia has a commanding advantage in software.
Why doesn't AMD throw a few hundred thousand dollars at some developers to get containerization parity?