GCP – Easier log management for multi-tenancy through new routing features

At Google Cloud, we believe you should have full control over your log data. That’s why we created the Log Router, which gives you the flexibility to choose which logs are stored in Cloud Logging, sent to other Google Cloud products like Cloud Storage, or even sent to your favorite third-party product. Customers tell us that this flexibility is really important, especially when managing log data from a large number of teams in their organization. Today we’re announcing the general availability of our latest log sink destination: a Google Cloud Project, to provide greater flexibility for routing logs.

Using a Google Cloud Project as a log sink destination lets you route logs from one project to the Log Router of another project, as though the logs had actually been received by the second project. This gives the team that owns the destination project a whole new level of control over these logs.

What’s new?

Up until now, you could route logs from one project directly only into the log bucket of another project. Centralizing logs directly in log storage is very useful for secops and audit purposes, but it means that the destination project has no control over these logs. Any processing in the destination project, such as logs-based metrics, real-time log based alerting, or log sinks does not apply to those logs. Using a project as a log sink destination changes this ownership model giving valuable controls to the team who owns the destination project:

Increased insights — The team can create log-based metrics and log-based alerts from the logs that were routed directly to the destination project. Error Reporting will also work on logs routed to the destination project so that teams can identify and troubleshoot errors more quickly (coming soon!).Cost control — The team that owns the destination project now has full control over the logs routed to their projects. This means they can set up log sinks with inclusion and exclusion filters to control which logs go to Cloud Logging log storage buckets — especially useful if you want each team to manage the cost of their logs, since logs that are excluded from Cloud Logging are not charged.Access to the Google Cloud ecosystem — The team that owns the destination project can now route these logs to other supported destinations like Cloud Storage, BigQuery, and Pub/Sub, even if these logs are not retained in Cloud Logging.

Using a project as a log sink destination is ideal for you if you are managing log data for lots of different teams in your organization and want to give those teams the ability to independently manage their own logs. This can also improve accountability and transparency, as individual teams will have visibility into their own log data costs.

You can start routing your logs to a new project with just a few easy steps:

1. From Cloud Logging’s Log Router page, click CREATE SINK.

2. Enter the sink name, and click NEXT.

3. In Sink Destination, select Google Cloud project as the sink service.

4. To set the destination project, either manually enter the project id or click BROWSE to search for the project. Click NEXT when you are finished.

5. Set your inclusion/exclusion filters in step 3 and 4. Click CREATE SINK when you are finished.

Once you’ve created the sink, logs that match the filters will be routed to the destination project, giving you the same power of choice and control to the cross-project logs. You can then use the logs in the destination project to create metrics, alerts, use Error Reporting (coming soon), and route that log data to additional supported destinations of your choice.

Try routing your logs to a new project in your organization today— Cloud Logging Log Router.

To learn more, check out the documentation:

For more information about routing your logs to other projects and supported destinations, see Route logs to supported destinations.For an example of how to use this feature to create a system where GKE tenant logs are distributed to tenant projects, see Multi-tenant logging on GKE.