I've been following podman for rootless containers. It works fairly well on Fedora. RHEL 8 works, but has older version of packages compared to Fedora.
The big annoying one in HPC is NFS/GPFS, but that will actually work just fine if you stay as root inside the container, which maps to your uid outside the container.
I'm a little confused by this statement, since at my current employer and previous employer we used NFS mounted home dirs and I would and currently do need to explicitly change my ~/.config/containers/storage.conf to point to a local mount point, such as /var/tmp/containers/$USER in order to perform anything rootless container related. In my case that's with Podman on RHEL 7 and RHEL 8.
I had a derp-out moment yesterday, I totally misinterpreted “as root in the container”. The topic was about rootless containers, so I was completely mentally blacklisting the idea of starting a container from the root user account.
Words might not be clicking with me today, I’m trying to specifically decipher this part:
> but that will actually work just fine if you stay as root inside the container, which maps to your uid outside the container
The context of the statement isn’t clicking with me. You start with “it’s a problem with these filesystems” which I’m on board with but then follow with “but it’s fine if you do this” and I’m not seeing the correct correlation.
With user namespaces you can get different uids inside the container. They are usually mapped to something above 65535, in blocks of 65535, according to /etc/subuid. When you run as rootless on podman with uid=1000, you end up at 66535, or 1000 above your start range in /etc/subuid, for example. This is what NFS/GPFS doesn’t like. Locally, in other file systems on the node, it’s fine.
When you run as uid=0 in same the container, you are actually mapped to your uid, since the range starts to be applicable for uid=1. If you can stay as root, you can write to NFS/GPFS just fine.
The issue with this is that lots of available containers have moved away from using things as root. For example, you couldn’t start the vanilla redis container with a data directory on NFS for snapshots because redis runs as user 999.
> When you run as uid=0 in same the container, you are actually mapped to your uid, since the range starts to be applicable for uid=1. If you can stay as root, you can write to NFS/GPFS just fine.
Oh oh oh, you were just saying run the container as root, like literally. That’s what threw me off, because I was in the mindset of rootless. Wasn’t allowing myself to think outside of that environment. Sorry for the confusion on that. Long day yesterday.
Changing uids is something you don't want for HPC use anyhow. Charliecloud is a simple alternative approach specifically for HPC.
Actually, a lot of the time you can get away without namespacing anyhow, just by settings paths to point into an unpacked image. That does depend on what you want to run being sufficiently configurable, though.
> Slirp is a well-known project that was originally and mainly used for networking in QEMU (aka Quick Emulator).
Good grief, that's off by almost 15 years. Slirp was originally a SLIP endpoint you could run on a remote dial-in shell account to connect your local system to the internet. As it happens that's basically the same problem needed to get (something like) IP to work out of a userspace context in general, so it got appropriated by the container community and used pervasively. But it's roots go much deeper.
SLiRP was written by Danny Gasparovski from Canberra.
Actual SLIP needed privs on the host to configure everything. slirp did it in usermode, allowing lowly undergrads to get real IP connections at home from the terminal services universities provided. Then you could run NSCA Mosaic at home :-) And ftp things directly, instead of downloading to your shell account tmp dir and using zmodem to transfer it the last hop.
It went away as real ISPs began to appear, offering PPP connections, and universities dropped their dial-in services. Strange and glorious to see it back again.
Thank you for pointing that out. You are correct. I'll ask the post author to rectify that sentence. Regarding QEMU, I was also surprised, but I did find some resources using this name. Not sure where it originates though.
Singularity containers go some way towards running without root, though it defaults to setuid setup at boot and dropping privs before execution.
Singularity makes some bad design choices though, like eliminating layers, which results in proliferation of huge container images cluttering everywhere, with no idea what they do. It's really designed for HPC administrators who don't like end users.
Ironically, I keep getting told I have to switch over from docker because of security. Plenty of IT security folk have their knickers in a twist at the idea of non-sysadmin priviledged users. They would rather run a giant insecure system (Linux or Windows AD) with lots of LPE and lateral movement than smaller more isolated systems with more trusted users and less LPE. Dev(Sec)Ops concepts still haven't penetrated very deep.
Lack of CVEs for security issues is part of the problem with it. I think SUSE rejected it on the basis of auditing.
While it would be better to run a capability micro-kernel-ish system, you're stuck with Linux for things like HPC. There's no excuse for known exploitable privilege escalations, though and, yes, introducing more opportunity with setuid programs doesn't help. I agree the way such systems are run is unfortunate.
Several DOE sites won't even consider Singularity, including NERSC, who created Shifter (aka Docker + patching - networking) around the same time as development at Singularity started at LBNL (NERSC's parent lab). LANL created CharlieCloud (https://hpc.github.io/charliecloud/) because they needed a hard focus on security first.
There is quite a bit of distrust of Singularity places that value security. I'm hoping podman rootless can eventually come through and render all the bespoke implementations moot, but it will probably be a few years.
How does it relate, security wise, to mirageos? I know it's apples and oranges, but only on security dimention scale, is one far better than the other or hard to say?
30 comments
[ 3.1 ms ] story [ 79.4 ms ] threadI thought, when i used that tool for the first time, that is some small company building it but nope its paloaltonetworks.
There are some fairly well documented shortcomings to consider/work-around. You can find that list at https://github.com/containers/libpod/blob/master/rootless.md
> but that will actually work just fine if you stay as root inside the container, which maps to your uid outside the container
In short, you don't remap when running as root in the container, thus no uid/gid remapping.
> but that will actually work just fine if you stay as root inside the container, which maps to your uid outside the container
The context of the statement isn’t clicking with me. You start with “it’s a problem with these filesystems” which I’m on board with but then follow with “but it’s fine if you do this” and I’m not seeing the correct correlation.
When you run as uid=0 in same the container, you are actually mapped to your uid, since the range starts to be applicable for uid=1. If you can stay as root, you can write to NFS/GPFS just fine.
The issue with this is that lots of available containers have moved away from using things as root. For example, you couldn’t start the vanilla redis container with a data directory on NFS for snapshots because redis runs as user 999.
Oh oh oh, you were just saying run the container as root, like literally. That’s what threw me off, because I was in the mindset of rootless. Wasn’t allowing myself to think outside of that environment. Sorry for the confusion on that. Long day yesterday.
Actually, a lot of the time you can get away without namespacing anyhow, just by settings paths to point into an unpacked image. That does depend on what you want to run being sufficiently configurable, though.
Good grief, that's off by almost 15 years. Slirp was originally a SLIP endpoint you could run on a remote dial-in shell account to connect your local system to the internet. As it happens that's basically the same problem needed to get (something like) IP to work out of a userspace context in general, so it got appropriated by the container community and used pervasively. But it's roots go much deeper.
Also... "Quick Emulator"?!
https://en.wikipedia.org/wiki/QEMU → "QEMU (short for Quick EMUlator)[2] is a free and open-source emulator that performs hardware virtualization."
However, the citation points to an article from The Register circa 2019: https://www.theregister.com/2019/03/11/windows_10_on_kvm/
So who knows?
Actual SLIP needed privs on the host to configure everything. slirp did it in usermode, allowing lowly undergrads to get real IP connections at home from the terminal services universities provided. Then you could run NSCA Mosaic at home :-) And ftp things directly, instead of downloading to your shell account tmp dir and using zmodem to transfer it the last hop.
https://www.linuxjournal.com/article/1174
It went away as real ISPs began to appear, offering PPP connections, and universities dropped their dial-in services. Strange and glorious to see it back again.
Singularity makes some bad design choices though, like eliminating layers, which results in proliferation of huge container images cluttering everywhere, with no idea what they do. It's really designed for HPC administrators who don't like end users.
https://sylabs.io/guides/3.5/user-guide/security.html
Ironically, I keep getting told I have to switch over from docker because of security. Plenty of IT security folk have their knickers in a twist at the idea of non-sysadmin priviledged users. They would rather run a giant insecure system (Linux or Windows AD) with lots of LPE and lateral movement than smaller more isolated systems with more trusted users and less LPE. Dev(Sec)Ops concepts still haven't penetrated very deep.
While it would be better to run a capability micro-kernel-ish system, you're stuck with Linux for things like HPC. There's no excuse for known exploitable privilege escalations, though and, yes, introducing more opportunity with setuid programs doesn't help. I agree the way such systems are run is unfortunate.
There is quite a bit of distrust of Singularity places that value security. I'm hoping podman rootless can eventually come through and render all the bespoke implementations moot, but it will probably be a few years.
In contrast, gvisor, needs ptrace or kvm.
Both nanovms and gvisor aim to provide an abstraction through emulated system calls.
device controller is present since kernel 4.15.
> Adoption status
Docker/Moby supports FUSE-OverlayFS and cgroup2 as well on master. Planned to be released as v20.0X.
> spearheaded by Podman and LXC.
Yes w.r.t. cgroup, but the network stuffs (slirp4netns & RootlessKit) were originally written for Docker/Moby :)
> the truth is that container engines run Slirp without the seccomp support
Untrue. https://github.com/containers/libpod/blob/d4a3c05c0fcd0c53fa... https://github.com/moby/moby/blob/b47e74255811b2ead92b222541...