GCP – 6 tips for database pros to adapt to cloud data warehouses

Data warehouse modernization affects both people and technology. For IT professionals like database administrators (DBAs) who run legacy architectures, using a cloud data warehouse means you no longer have to spend the bulk of your time on tedious activities like testing, patching, upgrading, and managing backups. It can also free you from the impossible task of playing constant catch-up with technology designed in the 1990s, when you’re faced with modern business demands for better, faster data insights. 

Instead, DBAs, data engineers, and other data pros can carve out strategic and highly visible roles in a serverless world. Here are some tips on how you can adapt and excel. 

1. Partner with business leaders before migration
Enterprise data warehouse environments are notoriously complex, making cloud migration a challenge. As the only people capable of navigating these sprawling systems and the work that depends on them, DBAs can lead their organizations through planning, execution, and beyond. You know the daily ins and outs of your data environment like no one else, and that information is valuable. Even if your company hasn’t begun to modernize, you can prepare for the future by evaluating different serverless data warehouse solutions, performing an initial TCO analysis, and choosing or suggesting an appropriate migration approach.

The next step is identifying and prioritizing the use cases to migrate first, along with the datasets, processes, pipelines, and applications that underpin each one. During this phase, you should work closely with leaders across the company to understand the value that data is bringing to the business and develop shared goals for the data warehouse modernization project. These conversations can form the basis of a longer outreach initiative aimed at reframing the way people think about data. It’s important to help business teams recognize data as the organization’s biggest strategic asset—not just a bunch of numbers and tables. Now is the time for you to establish yourself not only as the guardian of this key resource, but also as the expert on unlocking its full potential.

2. Empower your colleagues to solve problems 
In legacy environments, a DBA’s work takes place largely behind the scenes, removed from—and unrecognized by—the countless people who rely on functioning databases to do their jobs. (Until the on-call pager goes off at 2 in the morning, that is.) Sometimes, the DBA’s role as the protector of vital information systems can earn you and your team a reputation for enforcing excessively high standards or standing in the way of change. In fact, you’re just being realistic about the limits of your legacy environment: The next hardware refresh cycle may be several years away, making it impossible to meet growing demand for access to data and insights.

We hear from our users that serverless data warehouses like Google Cloud’s BigQuery can change this model. They automate away most routine maintenance and configuration tasks, so you’re not constantly stuck in the server room, putting out fires. The built-in elasticity of the cloud makes it a lot easier to scale resources when business teams need them, which in turn means you can say “yes” a little more often without fearing an impending sleepless night.

3. Make strategic decisions about how to allocate resources
A serverless cloud data warehouse can eliminate the need for a database pro to buy and set up new hardware, add more nodes, provision resources, and predict future capacity needs years in advance. This lightens the load for IT while reducing system downtime and letting organizations promptly seize new business opportunities. But depending on your platform, opening the floodgates to limitless usage can result in unexpected query costs, taking away budget dollars from mission-critical applications with strict SLAs.

Even in a serverless world, organizations still need to control costs and manage workloads, and DBAs already excel at these tasks. BigQuery includes several tools and features that can help, including a calculator for pricing query costs and the ability to create custom quotas at the project or user level. If your company has opted for BigQuery’s flat-rate pricing model, in which slots of dedicated querying capacity are purchased in monthly or yearly commitments, you can partition your available resources into reservations and give priority to specific teams, projects, or applications. Idle capacity is distributed across the organization, ensuring that resources don’t go to waste. 

4. Take a proactive approach to performance optimization
In on-prem environments, DBAs are always in survival mode, spending a lot of time on routine activities to keep the data warehouse up and running. Serverless solutions let you focus less on maintaining the status quo and more on continuously improving performance to cut costs, speed up product delivery, and boost customer satisfaction. Instead of simply making data available, you can help the organization harness it as effectively as possible.

There’s another reason why solutions like BigQuery help admins take their optimization game to the next level. Audit logs capture every query in real time, providing a comprehensive picture of data usage patterns across the organization. This lets you find and address inefficiencies before critical applications start lagging. For example, if a query is scanning an unnecessarily large amount of data, you can reduce the number of bytes processed by requiring users to specify partition filters. You can also proactively advise teams on ways to enhance their workflows, like by creating materialized views or querying public datasets.

5. Establish and enforce security best practices
Even in a serverless world, DBAs are responsible for protecting corporate and customer information from malicious attacks and accidental data loss. Solutions like BigQuery cut out some of the grunt work with automatic data replication and encryption, but you can still customize the security and governance foundation to meet business needs and maintain compliance. For instance, you can grant access to BigQuery resources down to the cell level, create service perimeters, detect and mask personally identifiable information (PII), and manage your own encryption keys just like in an on-prem environment.

Since security in the cloud works a bit differently, it helps to familiarize yourself with key principles and best practices before migrating your data warehouse. Armed with this knowledge, you can take a leading role in developing a new, cloud-ready framework of policies and procedures for the entire organization.

6. Share data insights with everyone
Lots of businesses are working to transform the data they collect and analyze into faster decisions, smarter predictions, and better customer experiences. Not everyone is a data scientist, though, so demand has grown for simple, accessible tools that let more people use business intelligence and machine learning. Some of these solutions work right inside serverless data warehouses like BigQuery, minimizing latency and eliminating the need to move data. For example, BigQuery ML lets users create and execute machine learning models in BigQuery using standard SQL queries, and BigQuery BI Engine is an in-memory analysis service for interactive visual analytics.

Of course, business teams often need help deciding on the right tools and data sources, building a system that meets their needs, and putting the resulting insights into action. There’s also a risk that different departments will choose and adopt solutions independently, leading to duplicate efforts, siloed data, and security issues. You can take control by offering tailored guidance based on specific questions and problems, directing teams to the most relevant tools and information, and making sure that all data projects support the overall business strategy. Along the way, you can play matchmaker, helping connect teams for collaboration and knowledge sharing.

These are just a few examples of how data warehouse modernization elevates the roles of DBAs, data engineers, and other data professionals, helping you make your job more rewarding and engaging—not to mention less thankless. Check out the full discussion from our recent global online conference.