AWS – Introducing Advanced Scaling in Amazon EMR Managed Scaling

We are excited to announce Advanced Scaling, a new capability in Amazon EMR Managed Scaling which provides customers increased flexibility to control the performance and resource utilization of their Amazon EMR on EC2 clusters. With Advanced Scaling, customers will be able to configure the desired resource utilization or performance levels for their cluster, and Amazon EMR Managed Scaling will leverage the customers intent to intelligently scale the cluster and optimize cluster compute resources.

Customers appreciate the simplicity of Amazon EMR Managed Scaling. However, there are instances where the default Amazon EMR Managed Scaling algorithm might lead to cluster under-utilization for specific customer’s workload. For instance, clusters running multiple tasks of relatively short duration (task runtime of 10 seconds or less), Amazon EMR Managed Scaling by default scales up the cluster aggressively and conservatively scale it down to avoid negative impact to job run times. While this is the right approach for SLA-sensitive workloads, it might not be optimal for cost sensitive workloads. With Advanced Scaling, customer can now configure Amazon EMR Managed Scaling behavior suitable for their workload type and we will apply tailored optimization to intelligently add or remove nodes from the clusters.

To get started with Advanced Scaling, you can set the ScalingStrategy and UtilizationPerformanceIndex parameters either when creating a new Managed Scaling policy, or updating an existing Managed Scaling policy. Advanced Scaling is available with Amazon EMR release 7.0 and later and is available in all regions where Amazon EMR Managed Scaling is available. For more details, please refer to our Advanced Scaling documentation.

Read More for the details.