Logging

Cloud Logging

Cloud Logging allows you to store, search, analyze, monitor, and alert on logging data and events from Google Cloud runtime environments and also any other on-premises or Cloud environments.

Enable API

gcloud services enable logging.googleapis.com

Logging API is usually enabled by default for your project.

Centralized Logging

There are a couple of ways to send log messages to Google Cloud.

• If you are running in a Kubernetes Engine, App Engine, Cloud Run, Cloud Functions, then logs to STDOUT or STDERR are automatically sent to Cloud Logging.

• If you are running in Compute Engine, then you can install a Logging Agent.

• If you are running outside of Google Cloud runtime environment, e.g., from on-premise datacenter, or another cloud, you can:

  • Use the Cloud Logging API to send log entries to Cloud Logging

  • Use a Logging Agent

  • Use a Fluend adapter

Once the log is collected by Cloud Logging, you can see:

• Search the logs

• Create metrics from logs based on criteria, to see in Cloud Monitoring, or create alerts

• Stream log entries to BigQuery, Pub/Sub, or Cloud Storage for further analysis

Error Reporting

Google Cloud will automatically identify exceptions and in Error Reporting console, list recently occurring errors, in order of frequency. You can quickly identify new errors, frequent errors, and dig into details through Centralized Logging.

Error Reporting showing an new exception in identified from the logs

Severity Level

Cloud Logging has 9 different log severity levels the log entries can associate with.

However, in all the runtime environments where logs printed STDOUT and STDERR are sent to Cloud Logging, original log entry's severity level is not retained:

• Log entries printed to STDOUT will have a severity level of INFO regardless of the original log entry level.

• Log entries printed to STDERR will have a severity level of WARNING regardless of the original log entry level.

Different runtime environments have different ways of associating the log level properly.

Environment	Preferred Logging
Cloud Function	Use Java Logging API (JUL)
App Engine Standard	Output Structured Logs in JSON format
Cloud Run	Output Structured Logs in JSON format
Compute Engine	Install Logging Agent, or use Cloud Logging API
Kubernetes Engine	Output Structured Logs in JSON format

In Cloud Logging dashboard, you can see graphs with segmentation on the Severity levels:

Log / Trace Correlation

When your log messages are also associated with the same trace ID and span ID as the ones sent to Cloud Trace, then the Trace console can display the logs along side of the trace/spans views when you click Show Logs:

Request Log Grouping

For a HTTP-based application, it's useful to see all of the log messages related to a single request grouped together. The Log Viewer can do this if your log messages meet the following criteria:

• A "request" log entry that contains the httpRequest information that contains request information such as the URL, response code, latency, etc. This is usually produced by Google Cloud HTTP Load Balancer.

• Associate each log message with the same Trace ID. This is usually generated by the Google Cloud HTTP Load Balancer.

When these conditions are met, then the Log Viewer can group these log entries together, with the top-level log that contains the httpRequest information:

Log entries are grouped under the top-level request log

When using Google Cloud HTTP Load Balancer (default if you are running in App Engine or Cloud Run), the Load Balancer will automatically:

• Produce the request log with the httpRequest information.

• Generate a Trace ID and it's propagated to your application via the X-Cloud-Trace-Context HTTP header.

You can use Spring Cloud GCP Trace starter to automatically read and use this trace header. In addition, use the Spring Cloud GCP Logging starter to automatically associate log entries with the Trace ID.

Log entries are grouped under the top-level request log

If you are not using a Google Cloud HTTP Load Balancer, then you can produce the httpRequest log manually. See Other Loggers' JSON Logging section.

Logback

Spring Boot uses Slf4J logging API and Logback logger by default. You can user Spring Cloud GCP's Logging Starter to use pre-configured Logback appenders to produce Structured JSON logs, or send the log via the Cloud Logging API.

Dependency

Add the Spring Cloud GCP Trace starter:

Maven

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-gcp-starter-logging</artifactId>
</dependency>

Gradle

compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter-logging'

Configuration

Configure Logback to use the additional appenders, by adding a logback-spring.xml file, and import the appender configuration:

logback-spring.xml

<configuration>
  <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
  <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
  <include resource="org/springframework/cloud/gcp/logging/logback-appender.xml"/>

  ...
</configuration>

Log with Cloud Logging API

logback-spring.xml

<configuration>
  <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
  <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
  <include resource="org/springframework/cloud/gcp/logging/logback-appender.xml"/>

  <root level="INFO">
    <appender-ref ref="CONSOLE"/>
    <appender-ref ref="STACKDRIVER"/>
  </root>
</configuration>

Notice that there is no explicit configuration for username/password. Cloud Logging authentication uses the GCP credential (either your user credential, or Service Account credential), and authorization is configured via Identity Access Management (IAM).

Log with Structured JSON Logging

logback-spring.xml

<configuration>
  <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
  <include resource="org/springframework/cloud/gcp/logging/logback-json-appender.xml"/>

  <root level="INFO">
    <appender-ref ref="CONSOLE_JSON"/>
  </root>
</configuration>

Use Different Appenders with Profile

It's useful to configure different appenders when running in Spring Boot profiles. For example, in local/dev environments, simply output regular log entries to STDOUT, in staging/production environments, use Structured JSON Logging.

See Spring Boot Logging documentation and Spring Boot profiles for more details.

For example, to configure default profile to use regular logging, and higher environments with Structured JSON Logging:

logback-spring.xml

<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml" />
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />

    <springProfile name="qa | staging | prod">
        <include resource="org/springframework/cloud/gcp/logging/logback-json-appender.xml"/>
        <root level="INFO">
            <appender-ref ref="CONSOLE_JSON"/>
        </root>
    </springProfile>
    <springProfile name="default | dev">
        <root level="INFO">
            <appender-ref ref="CONSOLE"/>
        </root>
    </springProfile>
</configuration>

This sample application allows you to:

• If no profile is specified, then the default profile is used, then use the default CONSOLE appender.

• If you specify dev profile, then use the CONSOLE appender

• If you specify qa, staging, prod profile, then it'll output to Structured JSON Logging.

Alternatively, you can also mix and match the profiles with more generic profiles:

logback-spring.xml

<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml" />
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />

    <springProfile name="logging-json">
        <include resource="org/springframework/cloud/gcp/logging/logback-json-appender.xml"/>
        <root level="INFO">
            <appender-ref ref="CONSOLE_JSON"/>
        </root>
    </springProfile>
    <springProfile name="logging-api">
        <include resource="org/springframework/cloud/gcp/logging/logback-appender.xml"/>
        <root level="INFO">
            <appender-ref ref="STACKDRIVER"/>
        </root>
    </springProfile>
    <springProfile name="logging-console | default">
        <root level="INFO">
            <appender-ref ref="CONSOLE"/>
        </root>
    </springProfile>
</configuration>

This sample application allows you to:

• If no profile is specified, then the default profile is used, then use the default CONSOLE appender.

• If you specify logging-json profile, it'll output to Structured JSON Logging.

• If you specify logging-api profile, it'll send the logs via the API.

• If you speicfy default and logging-api profiles, then it'll use the default CONSOLE appender and send the logs via the API.

Log / Trace Correlation

When using Structured JSON Logging or logging using the API, then Spring Cloud Sleuth's trace context (Trace ID, Span ID) are automatically added to the log metadata. If you explore the log message in the Cloud Logging Console, you can see the trace attribute and the spanId attribute are both populated with the correct values:

In the Cloud Trace console, check Show Logs, then you can then see the logs alongside the trace itself:

Request Log Grouping

In addition to Trace / Log Correlation, if the application is running in Cloud Run, App Engine, or any environment that's fronted by a GCP's HTTP load balancer, then the log entries can be grouped into the top level load balancer produced request log.

Log entries are grouped under the top-level request log

Samples

• Spring Cloud GCP Logging sample

Other Loggers

It's highly recommended that you use the default logger (Logback) with Spring Boot, to take advantage of Spring Cloud GCP features. If you do use other Loggers, you may be able to configure logging to API with different appenders/handlers.

JSON Logging

The official Structured Logging documentation suggests that you output the JSON format according to the LogEntry. Rather than producing the entire LogEntry, you can produce a more simplified JSON payload:

{
  "message": "My log message",
  "severity": "WARN"
}

Cloud Logging agents will automatically extrapolate the severity attribute, and also fill in the rest of the LogEntry fields so that you don't need to.

If you want to add the trace ID or span ID, you can do so by adding Special Fields to the JSON payload. These special fields will be automatically extrapolated to the LogEntry.

{
  "message": "My log message",
  "severity": "WARN",
  "logging.googleapis.com/trace": "projects/PROJECT_ID/traces/TRACE_ID",
  "logging.googleapis.com/spanId": "SPAN_ID"
}

If you want to associate HTTP request information (especially if you are not using a Google Cloud Load Balancer), then you can also add the httpRequest field:

{
  "message": "My log message",
  "severity": "WARN",
  "logging.googleapis.com/trace": "projects/PROJECT_ID/traces/TRACE_ID",
  "logging.googleapis.com/spanId": "SPAN_ID",
  "httpRequest": {
    ...
  }
}

API Logging

Java Logging API (JUL)

See Cloud Logging handler for Java Logging API.

Apache Commons Logging (JCL)

There is no ready-to-use appender to Cloud Logging. But you can bridge it to Slf4J, or bridge it to Java Logging API.

Log4J 2

There is no ready-to-use appender to Cloud Logging. But you can bridge it to Slf4J.

Learn More

• Troubleshooting and Debugging Microservices in Kubernetes