Article

Logs are an important part of troubleshooting and it’s critical to have them when you need them. When it comes to logging, Google Kubernetes Engine (GKE) is integrated with Google Cloud’s Logging service. But perhaps you’ve never investigated your GKE logs, or Cloud Logging?  Here’s an overview of how logging works in GKE, and how to configure, find, and interact effectively with the GKE logs stored in Cloud Logging.

How your GKE logs get to Cloud Logging

Any containerized code that is running in a GKE cluster—either your code or pre-packaged software—typically generates a variety of logs. These logs are usually written to standard output (‘stdout’) and standard error ‘stderr’, and include error, informational and debugging messages. 

When you set up a new GKE cluster in Google Cloud, system and app logs are enabled by default. A dedicated agent is automatically deployed and managed on the GKE node to collect logs, add helpful metadata about the container, pod and cluster and then send the logs to Cloud Logging. Both system logs and your app logs are then ingested and stored in Cloud Logging, with no additional configuration needed. 

Find your GKE logs in Cloud Logging

In order to find these logs in the Cloud Logging service, all you need to do is filter your logs by GKE-related Kubernetes resources, by clicking this link, or by running the following query in the log viewer:  

resource.type=(“k8s_container” OR “container” OR “k8s_cluster” OR “gke_cluster” OR “gke_nodepool” OR “k8s_node”)

This query surfaces logs that are related to Kubernetes resources in GKE: clusters, nodes, pods and containers. Alternatively, you can access any of your workloads in your GKE cluster and click on the container logs links in your deployment, pod or container details; this also brings you directly to your logs in the Cloud Logging console.

If no log entries return with your query, it’s time to look for reasons your logs aren’t being generated or collected into Cloud Logging.   

Make sure you’re collecting GKE logs

As mentioned above, when you create a GKE cluster, system and app logs are set to be collected by default. You can update how you configure log collection either when you create the cluster or by updating the cluster configuration. 

If you don’t see any of your logs in Cloud Logging, check whether the GKE integration with Cloud Logging is properly enabled. Follow these instructions to check the status of your cluster’s configuration.

If the GKE integration is not enabled, you can enable log collection for the cluster by editing the cluster in the Google Cloud Console, or by using the gcloud container clusters update command line.

If you have already enabled the GKE integration with Cloud Logging and Cloud Monitoring and still don’t see any of your GKE logs, check whether your logs they’ve been excluded. Logging exclusions may have been added to exclude logs from ingestion into Cloud Logging either for all or specific GKE logs. Adjusting these exclusions allows you to ingest the GKE logs that you need into Cloud Logging. 

Beginning with GKE version 1.15.7, you can configure a GKE cluster to only capture system logs. If you have already enabled the GKE integration with Cloud Logging and Cloud Monitoring and only see system logs in Cloud Logging, check whether you have selected this option. To check whether application log collection is enabled or disabled, and to then enable app log collection, follow these instructions. 

Make your GKE logs more effective

Having structured logs can help you create more effective queries. With Cloud Logging, structuring your logs means it parses your JSON object which makes it easier to build queries for your application JSON messages. GKE automatically adds structure to its log messages if your logs contain JSON objects in the log message. As a developer, you can also add specific elements in your JSON object that Cloud Logging will automatically map to the corresponding fields when stored in Cloud Logging. This may be useful to set the severity, traceId or labels for your log messages.

Using traces in conjunction with log messages is another common practice to monitor and maintain the health and performance of your app. Traces offer valuable context for every transaction in your application and thus make the troubleshooting effort significantly more effective, especially in distributed applications. If you use Cloud Trace (or any other tracing solution) to monitor distributed application tracing, another way to make your logs more useful is to include the trace id in the log message. With this connection, you can link to Cloud Trace directly from your log messages when you’re troubleshooting your app.

Reduce your GKE log usage 

GKE produces both system and application logs. While useful, sometimes the volume of logs may be higher than you expected. For example, certain log messages generated by Kubernetes such as the kublet logs on the node can be quite chatty and repetitive. These logs can be useful if you’re operating a production cluster for troubleshooting purposes, but may not be as useful in a purely development environment. If you feel you have too many logs, you can use Logging exclusions along with a specific filter to exclude log messages that you may not use. But be thoughtful about excluding logs, since you often won’t need the logs until later when you are for troubleshooting a problem. Excluding some repetitive logs (or excluding a certain percentage of them) can reduce the noise. 

Kubernetes produces many different kinds of logs and Cloud Logging can help you to make sense of them. For more details about using Cloud Logging with your GKE apps, check out our blog post.

Being responsible for business continuity isn’t easy. You must consider a wide variety of failure scenarios, including the outage of a Google region. In the event of a regional outage, you want your application and database to quickly start serving your customers in another available region if a Google Cloud region fails.

We’ve worked closely with Cloud SQL customers facing business continuity challenges to simplify the experience, and we are excited to launch Cloud SQL cross-region replication, which is available for MySQL and PostgreSQL database engines. 

What is a cross-region replica for Cloud SQL?

Cross-region replica makes it easy to create a fully managed read replica in a different region than that of the primary instance. You can create a replica in any Google Cloud region.

We’ve heard from Major League Baseball (MLB) that cross-region replicas have been useful. “We store all our important tracking information such as location of player, pitch velocity, and even the wind data on Cloud SQL for PostgreSQL,” says Greg Cain, MLB vice president, Baseball Data. We take great pride as the national pastime with millions of fans across the U.S., but we also have a large fanbase beyond that which spans all seven continents around the world. Our global audiences enjoy watching games at all times of day on MLB.com and our different consumer products. Cross-region replication was a very critical feature for us to implement to provide uninterrupted services to our fans.”

Using Cloud SQL cross-region replicas

With cross-region replica, you can: 

1. Minimize recovery point objective (RPO): A cross-region replica is a copy of the primary that reflects changes to the primary instance in almost real time, so data loss is very small in the event of a Google Cloud region failure. 

2. Minimize recovery time objective (RTO): Cross-region replica maintains an online copy of your data in another region. In the event of Google Cloud region failure, a replica can be promoted within minutes.

3. Make globally distributed applications faster: Read replicas are closer to their application in another region.

4. Migrate data between regions: Use cross-region replicas to minimize downtime when moving data between regions.

Simple and secure, by default

Cloud SQL cross-region replication reduces operational overhead and is fully integrated with Google Cloud’s Cloud SQL security and privacy features.

• Fully managed 

• Easily set up, maintain, manage, and administer replicas in any region on Google Cloud.

• Google Cloud networking

• Creating a cross-region replica requires no networking setup. Global VPC uses private IP for replication traffic between regions—eliminating the need of complex VPN and VPC configuration, which would be otherwise needed to set up cross-region networking.

• Cross-region replication traffic uses reliable, high-performing, and scalable Google Cloud networking.

• Network monitoring, verification, and optimization is simplified using proactive network operations with Network Intelligence Center.

• Cloud SQL security and privacy 

• Data at rest in replicas is encrypted using customer-managed encryption keys (CMEK).

• Cross-region replication traffic remains private, without access to and from the public internet, when a private IP option is used.

• Cross-region replicas are supported as part of Access Transparency, which represents Google’s long-term commitment to security and transparency by providing you with logs that capture the actions Google personnel take when accessing your content. 

• Connection org policy control provides centralized control of the public IP settings of Cloud SQL to reduce the security attack surface of Cloud SQL instances from the internet.

• Cloud SQL will enforce the data residency policy you define. Replicas can only be created in permitted regions.

Getting started with cross-region replica

Creating a cross-region replica is as simple as creating a read replica.

Get started with cross-region replication today.

Recently, we’ve been highlighting all the ways that Anthos, our hybrid and multi-cloud application platform, can help you modernize your Java applications and development and delivery processes. This week we’ll focus on how Migrate for Anthos, which takes your existing VM-based applications and intelligently converts them to run in containers on Google Kubernetes Engine (GKE), can also help you move your legacy Java applications. 

Whether it’s to enable new functionality, decommission an on-premises data center, or to save on maintenance costs, many organizations are actively trying to modernize legacy Java applications—preferably by running them in containers on GKE and Anthos. Unfortunately, the way that some legacy applications acquire resource configuration and usage information is incompatible with standard-issue Kubernetes, and requires some complicated workarounds. 

To help with this, the most recent release of Migrate for Anthos has a new feature to help streamline and simplify legacy application migration, automatically augmenting container resource visibility for legacy Linux-based applications, such as those that use Oracle Java SE 7 and 8 (prior to update 191). This is crucial if you want to successfully convert your legacy Java applications into containers without having to upgrade or refactor them. 

Migrate for Anthos helps you successfully move Java applications into containers by transparently and automatically implementing a userspace filesystem that addresses the limitations of the Linux filesystem. As you probably know, Linux uses cgroups to enforce container resource allocations. However, a known issue when running in Kubernetes, is that the Kubernetes node’s procfs /proc file system is mounted by default in the container, and reflects host resources rather than those allocated to the container itself. And because some legacy applications still acquire resource configuration and usage information from files like meminfo and cpuinfo in the /proc directory, rather than from cgroups files, running those applications in a container can result in errors and instability. For example, older Java versions may use the information from meminfo and cpuinfo to determine how much memory to allocate to its JVM heap, how many threads to run in parallel for garbage collection (GC), etc. Running an older Java application in a container that hasn’t been properly configured can result in processes being killed due to out-of-memory errors, which can be difficult to triage and troubleshoot. 

For legacy applications for which you cannot upgrade Java versions, Migrate for Anthos takes a common approach used in the community: it implements the LXCFS filesystem. It does this without requiring user intervention, special configuration or application rebuild. Our goal is to help you migrate all your applications—not just the easy ones—quickly and effectively, so you can make progress on your modernization goals.

The sample legacy Java web application

Let’s take a look at the difference in behaviors of a legacy Java application migrated with and without Migrate for Anthos. 

For this test, we’re using a JBOSS 8.2.1 server using an older version of Oracle Java SE 7 update 80. You can download this version from Oracle’s Java SE 7 Archives. We package it in two ways: as a regular Docker container image, and as a server VM from which we have migrated the application to a container using Migrate for Anthos. 

For the application, we use a sample JBoss node-info application with some additional lines of code to simulate memory pressure for each request served. The following modifications were applied:

src/main/java/pl/goldmann/work/helloworld/NodeInfoServlet.java

Testing the application on GKE

When deploying the two application containers, we apply the following resource restrictions in the GKE Pod spec, allocating 1 vCPU and 1 GiB of RAM, on a GKE node that has 4 vCPU and 16 GiB of RAM:

We then run the two application instances. First let’s check basic application output by directing a web browser to the application URL.

Here’s what happens on the standard container:

But here’s what happens on the Migrate for Anthos migrated container:

You can immediately see a difference between the results. In the standard container, as already reported in many such tests, Java reports resource values from the host node, and not from the container resource allocations. In the standard container, the reported maximum heap size is derived from Java 7’s sizing algorithm, which, by default, is one quarter of the host’s physical memory. However, in this case of the Migrate for Anthos migrated container, the values are reported correctly.

You can see a similar impact when querying the Java Garbage Collection (GC) threading plan. Connect to shell, and run:

java -XX:+PrintFlagsFinal -version | grep ParallelGCThreads

On the standard container, you get:

   uintx ParallelGCThreads                         = 4               {product}

But on the migrated workload container, you get:

   uintx ParallelGCThreads                         = 0               {product}

So here as well, you see the correct concurrency from the Migrate for Anthos container, but not in the standard container.

Now let’s see the impact of these differences under load. We generate application load using Hey. For example, the following command generates application load for two minutes, with a request concurrency of 50:

 ./hey_linux_amd64 -z 2m http://##.###.###.###:8080/node-info/

Here are the test results with the standard container:

Status code distribution:
  [200] 332 responses
  [404] 8343 responses
Error distribution:
  [29]  Get http://##.###.###.###:8080/node-info/: EOF
  [10116]       Get http://##.###.###.###:8080/node-info/: dial tcp ##.###.###.###:8080: connect: connection refused
  [91]  Get http://##.###.###.###:8080/node-info/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers

This is a clear indication that the service is not handling the load correctly, and indeed when inspecting the container logs, we see multiple occurrences of 

*** JBossAS process (79) received KILL signal ***

This is due to an out-of-memory (OOM) error. The Kubernetes deployment took care of automatically restarting the OOM-killed container, during which time the service was unavailable. The reason for this is a miscalculated Java heap size from considering the host resources, instead of the container resource constraints. When not calculated right, Java tries to allocate more memory than available and therefore gets killed, disrupting the app.

In contrast, executing the same load test on the container migrated with Migrate for Anthos results in:

Status code distribution:
 [200] 1676 responses
 [202] 76 responses

This indicates the application handled the load successfully even when memory pressure was high.

Unlock the power of containers for your legacy apps

We showed how Migrate for Anthos automatically augments a known container resource visibility issue in Kubernetes. This helps ensure that  legacy applications that run on older Java versions behave correctly after being migrated, without having to manually tune or reconfigure them to fit dynamic constraints applied through the Kubernetes Pod specs. We also demonstrated how the legacy application remains stable and responsive under memory load, without experiencing errors or restarts. 

With this feature, Migrate for Anthos can help you harness the benefits of containerization and container orchestration with Kubernetes, to modernize your operations and management of legacy applications. You’ll be able to leverage the power of CI/CD with image-based management, non-disruptive rolling updates, and unified policy and application performance management across cloud native and legacy applications, without requiring access to source code or application rewrite. 

For more information, see our original release blog that outlines support for day-two operations and more or fill out this form for more info (please mention ‘Migrate for Anthos’ in the comment box).