Debugging a Long-Running Apache Spark Application: A War Story (tech.channable.com)

Excellent write up! Running a single long running Spark Context that accepts and runs jars compiled from user input is a curse I thought we were suffering alone at my company. We too have fought the ContextCleaner, though for us it was G1GC and `-XX:+ExplicitGCInvokesConcurrent` on the driver that saved us.

Is this dynamic class execution a common use case? How frequently are you broadcasting new bytecode to your workers?