Ask HN: How do you run long computations without interruption?

1. Spend some time working on the verbose mode of your program so that you can use the output for debugging.

2. Consider setting up another computer on your local network to run your simulation. I often use workflows where to run my program, I execute a makefile that Rsyncs the project directory and runs the program on a remote machine over SSH.

What is long in this context? 1 day? 1 week? 3 months ?

Checkpointing - saving the state of the computation at intermediate points - can help a lot with this.

BUT, beware that checkpointing can also be unreliable if the system being simulated exhibits chaotic dynamics; truncating off "unimportant" decimal places during checkpointing can produce results that diverge rapidly between the "original" and "restarted from checkpoint" versions.

Have the simulation do periodic dumps of its state to disk and have the ability to resume the simulation from those dumps if interrupted.

At some point your "data set" becomes a "data base" and you might look at what actual DBMS things can do for you. The idea of transactions can be very helpful if your simulation runs in steps; might get rollback for free.

I have run weeks-long computations in a Jupyter cell! The only issue was with OSX updates-- I disabled those.

I put computing clusters on big UPS's to help keep them running during temporary power glitches. Make sure to put all the related equipment on UPS's also (switches, routers, hubs, KVM, monitor). Access control and scheduling computing time is important also, you don't want multiple people running computations at the same time because you might run out of RAM for example and in many experiments knowing the run time on a fully utilized CPU is important. Batch scripts can help; if you are running a series of computations that are not dependent on each other run them from a script, then if some fail it is easy to comment out the others and rerun the ones that failed. There are cluster management software programs that can automate much of this for you. Security measures are important, you don't want ransomware to hit during a run. Write good code for your computation that has sensible error handling so the computation can recover from errors, or at least give you data on what caused them. It can be difficult to debug long running software, it is similar to debugging server software. Sometimes you need the ability to only start generating logs after X hours of running or when the computation reaches a particular stage or to capture state continuously but only dump recent state history when an error occurs. Reliable air conditioning and power are essential. Physical access control to the hardware keeps anyone from bumping the hardware or accidentally unplugging it or otherwise messing with it. Sometimes it may make sense to have some spare machines that a task can migrate to if there is a hardware failure. Critical management functions you might run on more than one machine so that there is always at least one machine alive to control the systems. There are ways to lock circuit breakers so that service people can not turn them off (or on) by mistake without a key. Write your code to remove dependencies on an internet connection (e.g., use IP addresses instead of DNS, keep data local, have a local DNS server, send error reports using a local mail server that will keep retrying.) The computer location should keep insects, rodents, animals, mold, and water at bay. Vibration from nearby mechanical systems can be an issue (e.g., construction going on in a nearby room. Make sure facilities people have to get permission from you.) Remote access can let you check that things are running smoothly, a dashboard showing a variety of measures can be useful, though these are also a security risk. Keep automatic upgrades/updates turned off, only manually upgrade/update. Stop unnecessary services from running. Once you start thinking about it you'll find lots of things that could potentially go wrong; the world is extremely perverse and sneaky in this respect. If you find yourself running long term tasks over many years your list of mitigation measures will always be growing. Maybe you didn't consider the RF environment but a new local cell tower or radio tower starts messing with your systems. A pile driver a half mile away at a construction site can create strong vibrations and impulses in your computers. A solar storm can increase RF noise and reduce the quality of the AC power. Earthquakes happen. Med labs can have radiation sources. Fire is destructive, fire suppression systems can destroy your computers even if there is no fire. Someone may fire a gun through the walls if the public can get close to your computer room. Wireless networking is a security risk.

Get extensive code out of notebooks into testable libraries. Confirm it works before running a big computation.

We try to keep our notebooks slim and factor anything complex into a Python library. We can experiment with this libraries implementation, write unit tests on smaller data sets to confirm it works, before doing a bigger experiment. We can also deploy the library to prod and ensure users get the exact same thing we experimented with.

I would recommend the talk “I don’t like notebooks” https://m.youtube.com/watch?v=7jiPeIFXb6U

(There’s also the alternative nbdev that creates a full software engineering and experimentation setup in 100% notebooks. Not something I particularly like, but it might be another option.)