GCP – Cloud Composer 3: The next generation of data pipeline orchestration

As any data team can tell you, managing Apache Airflow is often a complex undertaking, with many hours spent wrestling with reliability, security, and efficient resource scaling. This valuable time could be better spent deriving insights and driving business value.

Today, we’re announcing the general availability of Cloud Composer 3, the latest version of our fully managed Apache Airflow service.This release represents a significant advancement in data pipeline orchestration, enabling data teams to streamline workflows, reduce operational overhead, and accelerate time-to-value.

Cloud Composer: A leading managed Airflow offering

Cloud Composer 3 empowers data teams to build, run, and manage data pipelines with greater efficiency and control. Its simplified user experience, enhanced performance, and robust security features make it a great platform for accelerating data-driven initiatives. Cloud Composer remains a top choice for many customers for data and analytics workflow orchestration and is widely used to orchestrate AI/ML workloads, supporting MLOps activities that are crucial for organizations in their AI efforts.

What’s new in Cloud Composer 3?

Cloud Composer 3 introduces a host of powerful features and improvements designed to simplify data pipeline management and boost efficiency:  

• Simplified networking: Easily configure network settings with streamlined options, reducing complexity and management overhead.  

• Evergreen versioning: Stay up-to-date with the latest Cloud Composer releases and gain access to new features, security updates, and performance improvements.  

• Hidden infrastructure: Focus on your data pipelines, not infrastructure management. Cloud Composer 3 handles the underlying infrastructure, allowing you to concentrate on building and running your DAGs.  

• Enhanced performance and reliability: Experience improved performance and reliability with optimized infrastructure and enhanced resource management.  

• Per-task CPU and memory control: Fine-tune resource allocation for individual tasks, optimizing performance and cost efficiency.  

• Strengthen your security posture: Cloud Composer 3’s hidden infrastructure lets Google proactively manage the environment’s security posture using industry best practices.

• and many more….

Benefits for data teams

Cloud Composer 3 delivers tangible benefits for data engineers, data scientists, and data architects:

• Increased productivity: Streamlined workflows and simplified environment management free up valuable time for data teams to focus on core tasks.  

• Reduced operational overhead: Automated infrastructure management and evergreen versioning minimize operational burden, allowing teams to focus on innovation.  

• Faster time-to-value: Enhanced performance and scalability accelerate data pipeline execution, enabling faster insights and quicker time-to-market for data-driven initiatives.  

Upgrade to Cloud Composer 3 today

Ready to experience the next generation of data pipeline orchestration? Visit our documentation for detailed instructions and join the many enterprise customers using Cloud Composer today.