AWS adds training checks and stalled-job recovery to HyperPod

AWS announced expanded training-health checks and workload recovery for SageMaker HyperPod on November 18, targeting failures that ordinary hardware monitoring can miss.
The announcement describes three capabilities. MultiNodeTrainingValidation runs a small training workload on individual instances and across matching instances, flagging invalid loss values or timeouts. It supports EKS-managed clusters using eligible EFA-enabled NVIDIA GPU instances.
A Kubernetes Resource Monitoring Agent watches infrastructure components and identifies the pods and nodes involved when training fails. A separate hung-job detector watches Slurm workloads for stalled progress and can cancel a job so Slurm requeues it.
The Slurm feature requires a HyperPod machine image released on or after November 10. AWS recommends starting with notifications while tuning detection rules before enabling cancellation. Hung-job detection for supported EKS training jobs already existed.
These additions extend monitoring beyond whether an individual GPU reports a fault. For teams running expensive distributed training jobs, earlier detection can reduce time spent holding compute capacity without useful training progress.
Sources and citations
Revision history
Published .