Unscheduled downtime Helma+Fritz 23.06.
- 23.06. 13:00: There is an interruption of the cold water supply for the building housing Helma and Fritz, so we had to shut down most of the Helma and the Fritz cluster without prior warning. Almost all jobs running on Helma and Fritz at that time were teminated ungently.
- 23.06. 14:00: Luckily, this does not seem to be a bigger hardware problem, just a control error that lead to the coolers on the roof going into emergency shutdown. The technicians are on it. Cold water is already partially back, but not stable yet. We currently expect normal batch processing to resume later today.
- 23.06. 14:45: We just got the information that the cold water supply is now considered stable again. We’re in the progress of turning the clusters back on and then resuming batch processing. This will happen step by step, i.e. more and more of the cluster nodes will come online again over the next hours.
- 23.06. 15:05: Regular batch operation for Fritz is resumed.
- 23.06. 15:35: Regular batch operation has been resumed on Helma.
- 23.06. 15:35: End of unscheduled downtime, all systems operating normally again. If your jobs were terminated through the ungentle shutdown, you will have to requeue them.