I built this library to deal with OpenOffice's PyUNO service hanging, leaking memory, and not being thread-safe. I think it could be quite useful for people running into similar problems.
I'm not familiar with PyUNO so forgive me if this is way out there, but would placing PyUNO behind a ZMQ gateway then using ZMQ for communication with your frontend work?
The somewhat weird use-case that I'm solving for PyUno is that the OpenOffice service leaks memory like a sieve. With the hooks built into TCPLock, I can periodically kill OpenOffice and hold off an accepting more connections until the service is available on port 8100 again.
I don't know enough about ZMQ to speak to it, but I'm sure I could wire something together to solve this problem using any number of approaches. A proxy in Node.js was tempting because it's simple, lightweight, and does the trick quite nicely :)
I had misunderstood. I thought you were using the semaphore to allow only one request at a time so as to avoid/mitigate the memory leak, not pend all requests during a restart of OO.
Why not run a couple PyUNO workers on EC2 and pull conversion jobs out of a MQ system where they get queued up by whatever system does the conversion requests? The workers pull a job out of the queue, load the files from wherever they are (S3 for example) and then put the converted files back when done. With a little scripting the PyUNO workers can be monitored and restarted after <x> conversion jobs, or simply when the conversion doesn't happen within a timeout. And the whole thing can be scaled up simply by running more EC2 conversion workers. This is the sort of thing AWS was made for.
That's fairly similar to our architecture. We run N crawling boxes, each consuming from a queue and running an OpenOffice process. TCPLock runs on localhost of these boxes and handles the logic around keeping OO up and running.
10 comments
[ 2.6 ms ] story [ 31.0 ms ] threadI don't know enough about ZMQ to speak to it, but I'm sure I could wire something together to solve this problem using any number of approaches. A proxy in Node.js was tempting because it's simple, lightweight, and does the trick quite nicely :)
http://zguide.zeromq.org/page:all#header-15