Crusoe’s Connor Guerrero, Young Jeong and Nikhil Gupta describe a managed infrastructure design combining Slurm scheduling with Kubernetes orchestration for large GPU training jobs. They explain the operational burden of node health, failures and observability.
The demonstration injects a GPU error, replaces the affected node and requeues training from an application checkpointCheckpointing saves a program's state so work can resume after an interruption.. The presenters report timing for their example and describe their managed service; the results are attributed to them.
Watch on YouTube
