A Guth Labs publication ↗
Back to News

AWS adds training checks and stalled-job recovery to HyperPod

Three green glass server modules linked by a gold recovery path, one lifted above a cream spare pedestal
AI-generated editorial illustration

AWS announced expanded training-health checks and workload recovery for SageMaker HyperPod on November 18, targeting failures that ordinary hardware monitoring can miss.

The announcement describes three capabilities. MultiNodeTrainingValidation runs a small training workload on individual instances and across matching instances, flagging invalid loss values or timeouts. It supports EKS-managed clusters using eligible EFA-enabled NVIDIA GPU instances.

A Kubernetes Resource Monitoring Agent watches infrastructure components and identifies the pods and nodes involved when training fails. A separate hung-job detector watches Slurm workloads for stalled progress and can cancel a job so Slurm requeues it.

The Slurm feature requires a HyperPod machine image released on or after November 10. AWS recommends starting with notifications while tuning detection rules before enabling cancellation. Hung-job detection for supported EKS training jobs already existed.

These additions extend monitoring beyond whether an individual GPU reports a fault. For teams running expensive distributed training jobs, earlier detection can reduce time spent holding compute capacity without useful training progress.

Sources and citations

Revision history

Published .