For the Java OOM analyzer, did you look at creating a heapdump automatically (using -XX:+HeapDumpOnOutOfMemoryError), and then using the Memory Analyzer Toolkit (http://www.eclipse.org/mat/)?
I can see that your approach is (presumably) much faster because it avoids the need to dump all the memory, but I've found heapdump + MAT to be incredibly powerful.
We did, but for basically anything over 1GB we found heap dumps to be prohibitively slow. Then, instead of failing for one request we'd basically end up killing the whole server.
Definitely hear you on that one! It only really works if you can put up with a service going down for a while (e.g. not customer facing, or one back-end in a web cluster with a load balancer in front). I can see how that wouldn't work for Greplin. Looks very useful now that I better understand the context.
I've had good experience with 2GB heap dumps. Actually dumping it was, in my experience, pretty quick. It was essentially as fast as the machine could do that much sequential IO. You get a lot more detail, like what was in those Strings that took up the whole heap. That is usually invaluable for diagnosing what's really wrong.
Actually analyzing a multi-GB heap dump takes 2-3x as much heap for the analyzer. The jhat tool that ships with the JVM is perfectly capable of analyzing a 2 GB heap if it has 6 GB of its own heap to play with.
Thank you! The Nagios utils look very cool.
I write all my Nagios checks in Python locally, and the startup/stop through NRPE is getting to be a problem.
I'd been thinking "Hrmm.. I bet I could use Tornado to fire these off inline.." and Bam, here it is ;)
9 comments
[ 5.0 ms ] story [ 38.7 ms ] threadI can see that your approach is (presumably) much faster because it avoids the need to dump all the memory, but I've found heapdump + MAT to be incredibly powerful.
Actually analyzing a multi-GB heap dump takes 2-3x as much heap for the analyzer. The jhat tool that ships with the JVM is perfectly capable of analyzing a 2 GB heap if it has 6 GB of its own heap to play with.
I'd been thinking "Hrmm.. I bet I could use Tornado to fire these off inline.." and Bam, here it is ;)
Seriously awesome.