CPU also causes a similar problem - for instance Java (and lots of runtime Java libraries) use the number of available CPUs to decide how many threads to use for various tasks, which can cause terrible performance if you're giving a container 1 CPU of a 48 CPU server and it thinks all 48 are available - we encountered this at work recently and eventually bodged around it, but definitely a gotcha that this stuff isn't handled properly by runtimes yet.
java9/hotspot has -XX:+UseCGroupMemoryLimitForHeap to deal with the memory part and I think the method to get the number of CPUs also takes cgroups/taskset into account now.
Actually the `-XX:+UseCGroupMemoryLimitForHeap` option is also in Java 8 since Java 8u131, so it's available for nearly everyone right now. It doesn't solve a lot of problems though: by default the JVM will use only 1/4th of the memory (cgroup) limit as its heap size, which wastes most of the container's memory. It's thus attractive to use `-XX:+UseCGroupMemoryLimitForHeap` together with `-XX:MaxRAMFraction=1` to get maximum heap size given your container limits. However, because total JVM memory is heap+permgen+threads*stacksize, you'll leave no memory for threads this way. If you're running an average web server, it will spawn dozens of threads and get your application above the cgroup memory limit (=OOM killed) pretty fast.
The only way out is to configure memory limits by hand, taking into account the maximum number of threads your application will use. That usually isn't trivial.
They're changing MaxRAMFraction to take non-integer values, so in some upcoming java release (yay 6month release cycle) that problem should go away too.
Anyone who is running important production java programs without carefully considering and testing the JVM’s command line arguments is headed for trouble.
If you want an OS environment virtualize. If you want a jail containerize. But containers are basically misused as lightweight vms so this type of conversation is inevitable.
I don't see the point of complicating operations monitoring based on a solutions model that involves containers as virtual machines. This also ignores the fundamental issue between host<->guest that exists in most hypervisors where monitoring must be dual scoped and integrated.
10 comments
[ 3.1 ms ] story [ 24.9 ms ] threadYou can use the lxcfs fuse filesystem to get that and a little more: https://github.com/lxc/lxcfs/blob/master/README.md
The only way out is to configure memory limits by hand, taking into account the maximum number of threads your application will use. That usually isn't trivial.
Are java processes usually deployed in (1 server to 1 java process ) configuration ?
Seems a bit strange for runtime to assume certain deployment configurations.
I don't see the point of complicating operations monitoring based on a solutions model that involves containers as virtual machines. This also ignores the fundamental issue between host<->guest that exists in most hypervisors where monitoring must be dual scoped and integrated.