Tibor Kiss

Logs are an important part of troubleshooting and it’s critical to have them when you need them. When it comes to logging, Google Kubernetes Engine (GKE) is integrated with Google Cloud’s Logging service. But perhaps you’ve never investigated your GKE logs, or Cloud Logging?  Here’s an overview of how logging works in GKE, and how to configure, find, and interact effectively with the GKE logs stored in Cloud Logging.

How your GKE logs get to Cloud Logging

Any containerized code that is running in a GKE cluster—either your code or pre-packaged software—typically generates a variety of logs. These logs are usually written to standard output (‘stdout’) and standard error ‘stderr’, and include error, informational and debugging messages. 

When you set up a new GKE cluster in Google Cloud, system and app logs are enabled by default. A dedicated agent is automatically deployed and managed on the GKE node to collect logs, add helpful metadata about the container, pod and cluster and then send the logs to Cloud Logging. Both system logs and your app logs are then ingested and stored in Cloud Logging, with no additional configuration needed. 

Find your GKE logs in Cloud Logging

In order to find these logs in the Cloud Logging service, all you need to do is filter your logs by GKE-related Kubernetes resources, by clicking this link, or by running the following query in the log viewer:  

resource.type=(“k8s_container” OR “container” OR “k8s_cluster” OR “gke_cluster” OR “gke_nodepool” OR “k8s_node”)

This query surfaces logs that are related to Kubernetes resources in GKE: clusters, nodes, pods and containers. Alternatively, you can access any of your workloads in your GKE cluster and click on the container logs links in your deployment, pod or container details; this also brings you directly to your logs in the Cloud Logging console.

If no log entries return with your query, it’s time to look for reasons your logs aren’t being generated or collected into Cloud Logging.   

Make sure you’re collecting GKE logs

As mentioned above, when you create a GKE cluster, system and app logs are set to be collected by default. You can update how you configure log collection either when you create the cluster or by updating the cluster configuration. 

If you don’t see any of your logs in Cloud Logging, check whether the GKE integration with Cloud Logging is properly enabled. Follow these instructions to check the status of your cluster’s configuration.

If the GKE integration is not enabled, you can enable log collection for the cluster by editing the cluster in the Google Cloud Console, or by using the gcloud container clusters update command line.

If you have already enabled the GKE integration with Cloud Logging and Cloud Monitoring and still don’t see any of your GKE logs, check whether your logs they’ve been excluded. Logging exclusions may have been added to exclude logs from ingestion into Cloud Logging either for all or specific GKE logs. Adjusting these exclusions allows you to ingest the GKE logs that you need into Cloud Logging. 

Beginning with GKE version 1.15.7, you can configure a GKE cluster to only capture system logs. If you have already enabled the GKE integration with Cloud Logging and Cloud Monitoring and only see system logs in Cloud Logging, check whether you have selected this option. To check whether application log collection is enabled or disabled, and to then enable app log collection, follow these instructions. 

Make your GKE logs more effective

Having structured logs can help you create more effective queries. With Cloud Logging, structuring your logs means it parses your JSON object which makes it easier to build queries for your application JSON messages. GKE automatically adds structure to its log messages if your logs contain JSON objects in the log message. As a developer, you can also add specific elements in your JSON object that Cloud Logging will automatically map to the corresponding fields when stored in Cloud Logging. This may be useful to set the severity, traceId or labels for your log messages.

Using traces in conjunction with log messages is another common practice to monitor and maintain the health and performance of your app. Traces offer valuable context for every transaction in your application and thus make the troubleshooting effort significantly more effective, especially in distributed applications. If you use Cloud Trace (or any other tracing solution) to monitor distributed application tracing, another way to make your logs more useful is to include the trace id in the log message. With this connection, you can link to Cloud Trace directly from your log messages when you’re troubleshooting your app.

Reduce your GKE log usage 

GKE produces both system and application logs. While useful, sometimes the volume of logs may be higher than you expected. For example, certain log messages generated by Kubernetes such as the kublet logs on the node can be quite chatty and repetitive. These logs can be useful if you’re operating a production cluster for troubleshooting purposes, but may not be as useful in a purely development environment. If you feel you have too many logs, you can use Logging exclusions along with a specific filter to exclude log messages that you may not use. But be thoughtful about excluding logs, since you often won’t need the logs until later when you are for troubleshooting a problem. Excluding some repetitive logs (or excluding a certain percentage of them) can reduce the noise. 

Kubernetes produces many different kinds of logs and Cloud Logging can help you to make sense of them. For more details about using Cloud Logging with your GKE apps, check out our blog post.

Being responsible for business continuity isn’t easy. You must consider a wide variety of failure scenarios, including the outage of a Google region. In the event of a regional outage, you want your application and database to quickly start serving your customers in another available region if a Google Cloud region fails.

We’ve worked closely with Cloud SQL customers facing business continuity challenges to simplify the experience, and we are excited to launch Cloud SQL cross-region replication, which is available for MySQL and PostgreSQL database engines. 

What is a cross-region replica for Cloud SQL?

Cross-region replica makes it easy to create a fully managed read replica in a different region than that of the primary instance. You can create a replica in any Google Cloud region.

We’ve heard from Major League Baseball (MLB) that cross-region replicas have been useful. “We store all our important tracking information such as location of player, pitch velocity, and even the wind data on Cloud SQL for PostgreSQL,” says Greg Cain, MLB vice president, Baseball Data. We take great pride as the national pastime with millions of fans across the U.S., but we also have a large fanbase beyond that which spans all seven continents around the world. Our global audiences enjoy watching games at all times of day on MLB.com and our different consumer products. Cross-region replication was a very critical feature for us to implement to provide uninterrupted services to our fans.”

Using Cloud SQL cross-region replicas

With cross-region replica, you can: 

1. Minimize recovery point objective (RPO): A cross-region replica is a copy of the primary that reflects changes to the primary instance in almost real time, so data loss is very small in the event of a Google Cloud region failure. 

2. Minimize recovery time objective (RTO): Cross-region replica maintains an online copy of your data in another region. In the event of Google Cloud region failure, a replica can be promoted within minutes.

3. Make globally distributed applications faster: Read replicas are closer to their application in another region.

4. Migrate data between regions: Use cross-region replicas to minimize downtime when moving data between regions.

Simple and secure, by default

Cloud SQL cross-region replication reduces operational overhead and is fully integrated with Google Cloud’s Cloud SQL security and privacy features.

• Fully managed 

• Easily set up, maintain, manage, and administer replicas in any region on Google Cloud.

• Google Cloud networking

• Creating a cross-region replica requires no networking setup. Global VPC uses private IP for replication traffic between regions—eliminating the need of complex VPN and VPC configuration, which would be otherwise needed to set up cross-region networking.

• Cross-region replication traffic uses reliable, high-performing, and scalable Google Cloud networking.

• Network monitoring, verification, and optimization is simplified using proactive network operations with Network Intelligence Center.

• Cloud SQL security and privacy 

• Data at rest in replicas is encrypted using customer-managed encryption keys (CMEK).

• Cross-region replication traffic remains private, without access to and from the public internet, when a private IP option is used.

• Cross-region replicas are supported as part of Access Transparency, which represents Google’s long-term commitment to security and transparency by providing you with logs that capture the actions Google personnel take when accessing your content. 

• Connection org policy control provides centralized control of the public IP settings of Cloud SQL to reduce the security attack surface of Cloud SQL instances from the internet.

• Cloud SQL will enforce the data residency policy you define. Replicas can only be created in permitted regions.

Getting started with cross-region replica

Creating a cross-region replica is as simple as creating a read replica.

Get started with cross-region replication today.

Recently, we’ve been highlighting all the ways that Anthos, our hybrid and multi-cloud application platform, can help you modernize your Java applications and development and delivery processes. This week we’ll focus on how Migrate for Anthos, which takes your existing VM-based applications and intelligently converts them to run in containers on Google Kubernetes Engine (GKE), can also help you move your legacy Java applications. 

Whether it’s to enable new functionality, decommission an on-premises data center, or to save on maintenance costs, many organizations are actively trying to modernize legacy Java applications—preferably by running them in containers on GKE and Anthos. Unfortunately, the way that some legacy applications acquire resource configuration and usage information is incompatible with standard-issue Kubernetes, and requires some complicated workarounds. 

To help with this, the most recent release of Migrate for Anthos has a new feature to help streamline and simplify legacy application migration, automatically augmenting container resource visibility for legacy Linux-based applications, such as those that use Oracle Java SE 7 and 8 (prior to update 191). This is crucial if you want to successfully convert your legacy Java applications into containers without having to upgrade or refactor them. 

Migrate for Anthos helps you successfully move Java applications into containers by transparently and automatically implementing a userspace filesystem that addresses the limitations of the Linux filesystem. As you probably know, Linux uses cgroups to enforce container resource allocations. However, a known issue when running in Kubernetes, is that the Kubernetes node’s procfs /proc file system is mounted by default in the container, and reflects host resources rather than those allocated to the container itself. And because some legacy applications still acquire resource configuration and usage information from files like meminfo and cpuinfo in the /proc directory, rather than from cgroups files, running those applications in a container can result in errors and instability. For example, older Java versions may use the information from meminfo and cpuinfo to determine how much memory to allocate to its JVM heap, how many threads to run in parallel for garbage collection (GC), etc. Running an older Java application in a container that hasn’t been properly configured can result in processes being killed due to out-of-memory errors, which can be difficult to triage and troubleshoot. 

For legacy applications for which you cannot upgrade Java versions, Migrate for Anthos takes a common approach used in the community: it implements the LXCFS filesystem. It does this without requiring user intervention, special configuration or application rebuild. Our goal is to help you migrate all your applications—not just the easy ones—quickly and effectively, so you can make progress on your modernization goals.

The sample legacy Java web application

Let’s take a look at the difference in behaviors of a legacy Java application migrated with and without Migrate for Anthos. 

For this test, we’re using a JBOSS 8.2.1 server using an older version of Oracle Java SE 7 update 80. You can download this version from Oracle’s Java SE 7 Archives. We package it in two ways: as a regular Docker container image, and as a server VM from which we have migrated the application to a container using Migrate for Anthos. 

For the application, we use a sample JBoss node-info application with some additional lines of code to simulate memory pressure for each request served. The following modifications were applied:

src/main/java/pl/goldmann/work/helloworld/NodeInfoServlet.java

Testing the application on GKE

When deploying the two application containers, we apply the following resource restrictions in the GKE Pod spec, allocating 1 vCPU and 1 GiB of RAM, on a GKE node that has 4 vCPU and 16 GiB of RAM:

We then run the two application instances. First let’s check basic application output by directing a web browser to the application URL.

Here’s what happens on the standard container:

But here’s what happens on the Migrate for Anthos migrated container:

You can immediately see a difference between the results. In the standard container, as already reported in many such tests, Java reports resource values from the host node, and not from the container resource allocations. In the standard container, the reported maximum heap size is derived from Java 7’s sizing algorithm, which, by default, is one quarter of the host’s physical memory. However, in this case of the Migrate for Anthos migrated container, the values are reported correctly.

You can see a similar impact when querying the Java Garbage Collection (GC) threading plan. Connect to shell, and run:

java -XX:+PrintFlagsFinal -version | grep ParallelGCThreads

On the standard container, you get:

   uintx ParallelGCThreads                         = 4               {product}

But on the migrated workload container, you get:

   uintx ParallelGCThreads                         = 0               {product}

So here as well, you see the correct concurrency from the Migrate for Anthos container, but not in the standard container.

Now let’s see the impact of these differences under load. We generate application load using Hey. For example, the following command generates application load for two minutes, with a request concurrency of 50:

 ./hey_linux_amd64 -z 2m http://##.###.###.###:8080/node-info/

Here are the test results with the standard container:

Status code distribution:
  [200] 332 responses
  [404] 8343 responses
Error distribution:
  [29]  Get http://##.###.###.###:8080/node-info/: EOF
  [10116]       Get http://##.###.###.###:8080/node-info/: dial tcp ##.###.###.###:8080: connect: connection refused
  [91]  Get http://##.###.###.###:8080/node-info/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers

This is a clear indication that the service is not handling the load correctly, and indeed when inspecting the container logs, we see multiple occurrences of 

*** JBossAS process (79) received KILL signal ***

This is due to an out-of-memory (OOM) error. The Kubernetes deployment took care of automatically restarting the OOM-killed container, during which time the service was unavailable. The reason for this is a miscalculated Java heap size from considering the host resources, instead of the container resource constraints. When not calculated right, Java tries to allocate more memory than available and therefore gets killed, disrupting the app.

In contrast, executing the same load test on the container migrated with Migrate for Anthos results in:

Status code distribution:
 [200] 1676 responses
 [202] 76 responses

This indicates the application handled the load successfully even when memory pressure was high.

Unlock the power of containers for your legacy apps

We showed how Migrate for Anthos automatically augments a known container resource visibility issue in Kubernetes. This helps ensure that  legacy applications that run on older Java versions behave correctly after being migrated, without having to manually tune or reconfigure them to fit dynamic constraints applied through the Kubernetes Pod specs. We also demonstrated how the legacy application remains stable and responsive under memory load, without experiencing errors or restarts. 

With this feature, Migrate for Anthos can help you harness the benefits of containerization and container orchestration with Kubernetes, to modernize your operations and management of legacy applications. You’ll be able to leverage the power of CI/CD with image-based management, non-disruptive rolling updates, and unified policy and application performance management across cloud native and legacy applications, without requiring access to source code or application rewrite. 

For more information, see our original release blog that outlines support for day-two operations and more or fill out this form for more info (please mention ‘Migrate for Anthos’ in the comment box).

Finding the right cloud talent and expertise is important for virtually all businesses today. Almost 70 percent of IT leaders report that hiring was somewhat or extremely difficult in 2019 and that finding qualified cloud computing talent was a top challenge. Our Google Cloud training allows you to build cloud skills while our certifications help organizations grow the expert cloud talent they need to effectively transform their business and help individuals elevate their IT careers with validated cloud skills. 

As thousands of technical professionals are looking for ways to stay productive and keep their skills current, we have seen a spike in demand for Google Cloud learning resources. In the month of April, enrollments in Google Cloud training on Coursera increased by more than 500% year-over-year. Google Cloud training completions more than doubled year-over-year across all platforms and partners.

We’re committed to developing cloud professionals across all stages of their career as well as enabling enterprises who need cloud expertise to respond, adapt, and transform to dynamic market pressures. To help, we continue to grow our training and certification programs to empower both learners and employers.

Today, we’re excited to announce three new initiatives. Firstly, we are introducing Google Cloud skill badges which will recognize and help employers identify those of you with demonstrated Google Cloud technology skills. For experienced professionals, we’ve created new six-week learning paths to guide you through the certification preparation journey. Lastly, in response to overwhelming market demand, we’ve made remote certification exams available, so you can take your exam from home. 

Show your growing cloud skillset with skill badges 

Demonstrate your growing Google Cloud-recognized skillset to employers and share your progress with your network through exclusive digital Google Cloud skill badges. The digital badges are earned through completing labs and a rigorous hands-on skill test on Qwiklabs. We’ve made Qwiklabs available at no cost for 30 days through the end of 2020, making hands-on practice and skill building accessible to anyone interested in starting a career in cloud. We have several skill badges available now for beginners, such as deploying and managing cloud environments and performing foundational data, machine learning and AI tasks. Get started with free access to Qwiklabs and skill badges here.

Start your certification preparation journey 

These six-week learning paths will outline recommended Google Cloud training to help experienced cloud professionals prepare for either the Associate Cloud Engineer, Professional Data Engineer, or the Professional Cloud Architect certifications. Through our training partner, Pluralsight, the first month of training will be available at no cost as well as 30 days of unlimited Qwiklabs access at no cost. To start, choose either the Associate Cloud Engineer, Professional Data Engineer, or Professional Cloud Architect certification, and we’ll send you a dedicated learning path with relevant training offers. If you’d like to prepare for certifications not covered by the learning paths, you can check out our latest offers for training courses here. 

If you’re interested in mastering the ability to deploy applications, monitor operations, and manage enterprise solutions, the Associate Cloud Engineer learning path is for you. We recommend at least six months of hands on experience with Google Cloud before attempting the Associate Cloud Engineer exam. 

To earn the Professional Data Engineer certification, you have to prove your expertise in designing, building, operationalizing, securing, and monitoring data processing systems. We recommend you have three or more years of cloud industry experience and at least one year of experience designing and managing solutions using Google Cloud prior to preparing for this certification. 

The Google Cloud Professional Cloud Architect certification, which was ranked the highest-paying IT certification for the second year in a row in the US, is recommended for individuals with three or more years cloud industry experience, including at least one year of Google Cloud experience. The learning path for this certification will guide you through how to securely design, develop, and manage robust, scalable, highly available, and dynamic solutions to drive business objectives. 

Take your certification exam  

We know many of you have already prepared and are ready to take your certification exams but are unable to do so because exam centers around the world are closed in response to COVID-19. To help, we’re making online proctored testing available for three of our exams; Associate Cloud Engineer, Professional Data Engineer, and Professional Cloud Architect. You can learn more about online proctored testing here and register for exams here.

You can start earning your Google Cloud skill badges here. Already have some cloud experience and ready to begin your certification preparation journey? Sign up here to get started with your six-week learning plan.