I've used this a few times to debug running processes that were having small issues that weren't bad enough to restart the system to put in a proper manhole, but bad enough to be affecting users. Worked perfectly to figure out what was going on, and to patch the naughty method.
It's one of those tools that you'll be glad exists when the need arises, but you'll feel a little dirty using it.
Why "dirty"? Attaching and inspecting a live process without cooperation is something that should be expected.
pdb is quite limited in what it can debug if you have a stuck process which is very hard to reproduce. I've resorted to gdb + python extensions too many times to count (which basically allows you do what pyrasite does, though at a lower level).
It's basically this: attach with GDB, get GIL, run commands (your session), release GIL. This is pretty dangerous as the GIL release is not guaranteed.
This is why I wrote https://pypi.python.org/pypi/manhole - I couldn't afford hosing up (deadlocking) my processes, and it happen pretty often with pyrasite in my usecases. Have in mind that manhole is not that flexible as it uses different technique.
I had some "memory leak" issues a few months ago with the api hosts of the service I'm working on. As the code changes made during the days before didn't seem to have affected anything that could be related to this leak I realized it was something about the requests we were receiving but I wasn't able to replicate the same issues in my dev host nor in staging.
Eventually I dumped the memory of a running process using Pyrasite and after hacking a little bit with meliae (https://launchpad.net/meliae) I was able to find the issue and solve it.
Very useful.
By the way is worth noticing that you won't probably be able to attach the interactive console to a running web server as the output is usually handled by the supervisor process, at least I wasn't able to do that in my first tentative and the memory dump was good enough for me.
Wow, live inspection of running code without prior instrumentation is a great feature of Erlang which I always missed in Python. I used `rconsole` from rfoo package (https://code.google.com/p/rfoo/) before, but this looks like a cleaner solution because you don't have to include server spawning code in the code you want to inspect. Very nice.
11 comments
[ 4.6 ms ] story [ 43.8 ms ] threadIt's one of those tools that you'll be glad exists when the need arises, but you'll feel a little dirty using it.
pdb is quite limited in what it can debug if you have a stuck process which is very hard to reproduce. I've resorted to gdb + python extensions too many times to count (which basically allows you do what pyrasite does, though at a lower level).
[1] http://aosabook.org/en/gdb.html [2] https://docs.python.org/2/c-api/veryhigh.html
This is why I wrote https://pypi.python.org/pypi/manhole - I couldn't afford hosing up (deadlocking) my processes, and it happen pretty often with pyrasite in my usecases. Have in mind that manhole is not that flexible as it uses different technique.
Very useful.
By the way is worth noticing that you won't probably be able to attach the interactive console to a running web server as the output is usually handled by the supervisor process, at least I wasn't able to do that in my first tentative and the memory dump was good enough for me.
Check this sample to dump the memory out of a running process: http://pyrasite.readthedocs.org/en/latest/Payloads.html#dump...