AWS adds GPU anomaly detection for SageMaker HyperPod training jobs

Executive Briefing
“AWS has announced new observability features for SageMaker HyperPod to help teams locate...”
Key Takeaways
- 1.AWS says enhanced observability is available in every AWS Region that supports SageMaker HyperPod.
- 2.The diagnostic summary uses CloudWatch’s DetectGroupOutliers API to compare GPU utilization, memory use, temperature, power draw and clock frequency within a job. It flags substantial deviations and works with both EKS and Slurm.
- 3.AWS also provides rules for known failure patterns, including GPU ECC memory errors and XID faults. For a detected issue, the system identifies the affected job, cluster, node, accelerator and its rank within the job.
- 4.AWS says the automatically enabled metrics cover cluster health, host resources, GPU utilization and temperature, and per-job resource use. They are collected every 30 seconds, with an estimated overhead of 1–2% CPU and 250–300 MB of memory.
- 5.CloudWatch is the new default metric store, while Amazon Managed Service for Prometheus remains an option. Advanced metrics are opt-in because they increase telemetry volume and can add storage costs.
AWS has announced new observability features for SageMaker HyperPod to help teams locate GPUs or infrastructure components involved in model-training problems. The features support jobs orchestrated with Amazon EKS or Slurm.
Product Features & Key Highlights
AWS says enhanced observability is available in every AWS Region that supports SageMaker HyperPod.
The diagnostic summary uses CloudWatch’s DetectGroupOutliers API to compare GPU utilization, memory use, temperature, power draw and clock frequency within a job. It flags substantial deviations and works with both EKS and Slurm.
AWS also provides rules for known failure patterns, including GPU ECC memory errors and XID faults. For a detected issue, the system identifies the affected job, cluster, node, accelerator and its rank within the job.
AWS says the automatically enabled metrics cover cluster health, host resources, GPU utilization and temperature, and per-job resource use. They are collected every 30 seconds, with an estimated overhead of 1–2% CPU and 250–300 MB of memory.
CloudWatch is the new default metric store, while Amazon Managed Service for Prometheus remains an option. Advanced metrics are opt-in because they increase telemetry volume and can add storage costs.
Practical Assessment & Evaluation
Comparing GPUs within a job could help operators narrow their search before inspecting individual dashboards. An alert would still need to be checked against incident data to establish the cause.
For large training clusters, choosing advanced metrics involves a trade-off between the detail needed to investigate faults and the additional telemetry cost.
Pricing, Availability & Next Steps
AWS provides no comparative measurement of how much the new features shorten diagnosis in real training jobs.
The supplied information is insufficient to calculate the cost of advanced metrics for a particular cluster configuration.







