# Logging

## Cloud Logging

Cloud Logging allows you to store, search, analyze, monitor, and alert on logging data and events from Google Cloud runtime environments and also any other on-premises or Cloud environments.

### Enable API

```bash
gcloud services enable logging.googleapis.com
```

{% hint style="info" %}
Logging API is usually enabled by default for your project.
{% endhint %}

## Centralized Logging

There are a couple of ways to send log messages to Google Cloud.

* If you are running in a Kubernetes Engine,  App Engine, Cloud Run, Cloud Functions, then logs to `STDOUT` or `STDERR` are automatically sent to Cloud Logging.
* If you are running in Compute Engine, then you can install a [Logging Agent](https://cloud.google.com/logging/docs/agent/installation).
* If you are running outside of Google Cloud runtime environment, e.g., from on-premise datacenter, or another cloud, you can:
  * Use the Cloud Logging API to send log entries to Cloud Logging
  * Use a [Logging Agent](https://cloud.google.com/logging/docs/agent/installation)
  * Use a [Fluend adapter](https://github.com/GoogleCloudPlatform/google-fluentd)

Once the log is collected by Cloud Logging, you can see:

* Search the logs
* Create metrics from logs based on criteria, to see in Cloud Monitoring, or create alerts
* Stream log entries to BigQuery, Pub/Sub, or Cloud Storage for further analysis

![](https://3412348858-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-L_Laqs9uSAihPmRemDj%2F-MCtNZsrlRG_0ScjZ_0_%2F-MCtO9eSTqMLbv936Tf4%2Fimage.png?alt=media\&token=c58cd514-8728-435b-8171-9d50e7415701)

## Error Reporting

Google Cloud will automatically identify exceptions and in Error Reporting console, list recently occurring errors, in order of frequency. You can quickly identify new errors, frequent errors, and dig into details through Centralized Logging.

![Error Reporting showing an new exception in identified from the logs](https://3412348858-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-L_Laqs9uSAihPmRemDj%2F-MCtLMTjF79-HZzUCPZ4%2F-MCtNT4ul8ZwDTWrTh26%2Fimage.png?alt=media\&token=c763c0db-c9fc-4242-9bc0-7a2b41d6e959)

## Severity Level

Cloud Logging has [9 different log severity levels](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity) the log entries can associate with.

However, in all the runtime environments where logs printed `STDOUT` and `STDERR` are sent to Cloud Logging, original log entry's severity level is not retained:

* Log entries printed to `STDOUT` will have a severity level of `INFO` regardless of the original log entry level.
* Log entries printed to `STDERR` will have a severity level of `WARNING` regardless of the original log entry level.

Different runtime environments have different ways of associating the log level properly.

| Environment         | Preferred Logging                                                                                           |
| ------------------- | ----------------------------------------------------------------------------------------------------------- |
| Cloud Function      | [Use Java Logging API (JUL)](https://cloud.google.com/functions/docs/concepts/java-logging)                 |
| App Engine Standard | [Output Structured Logs in JSON format](https://cloud.google.com/logging/docs/structured-logging)           |
| Cloud Run           | [Output Structured Logs in JSON format](https://cloud.google.com/logging/docs/structured-logging)           |
| Compute Engine      | [Install Logging Agent](https://cloud.google.com/logging/docs/agent/installation), or use Cloud Logging API |
| Kubernetes Engine   | [Output Structured Logs in JSON format](https://cloud.google.com/logging/docs/structured-logging)           |

In Cloud Logging dashboard, you can see graphs with segmentation on the Severity levels:

![](https://3412348858-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-L_Laqs9uSAihPmRemDj%2F-MCtOdkpo5CA-G9Ot_uy%2F-MCtOyPGMfESCJ4tRRbx%2Fimage.png?alt=media\&token=a4ea3b46-cd74-4890-8271-940e4a7b2d9a)

## Log / Trace Correlation

When your log messages are also associated with the same trace ID and span ID as the ones sent to Cloud Trace, then the Trace console can display the logs along side of the trace/spans views when you click **Show Logs**:

![](https://lh3.googleusercontent.com/O6u214GgMO_GD-xNUkHVj8KTOBH6pf8-_SJP1x17QhdT9Fle3D30gjV-wuTOSSYDHWnjMqFyZmymAIroBTrxNRJGXrT6JqWRQYGVyZE0DMXRDCR4IkNxBCoAwKGnzyctcJMk7-PPBQ)

## Request Log Grouping

For a HTTP-based application, it's useful to see all of the log messages related to a single request grouped together. The Log Viewer can do this if your log messages meet the following criteria:

* A "request" log entry that contains the `httpRequest` information that contains request information such as the URL, response code, latency, etc. This is usually produced by Google Cloud HTTP Load Balancer.
* Associate each log message with the same Trace ID. This is usually generated by the Google Cloud HTTP Load Balancer.

When these conditions are met, then the Log Viewer can group these log entries together, with the top-level log that contains the `httpRequest` information:

![Log entries are grouped under the top-level request log](https://3412348858-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-L_Laqs9uSAihPmRemDj%2F-MHlz54XMOCc8uCqGDgu%2F-MHm-wfRMa-PPzXsOTgt%2Fimage.png?alt=media\&token=c637e20d-5cee-434a-8436-046c7d828b36)

When using Google Cloud HTTP Load Balancer (default if you are running in App Engine or Cloud Run), the Load Balancer will automatically:

* Produce the request log with the `httpRequest` information.
* Generate a Trace ID and it's propagated to your application via the `X-Cloud-Trace-Context` HTTP header.

You can use [Spring Cloud GCP Trace starter](https://spring-gcp.saturnism.me/app-dev/trace#cloud-trace) to automatically read and use this trace header. In addition, use the [Spring Cloud GCP Logging starter](#logback) to automatically associate log entries with the Trace ID.

![Log entries are grouped under the top-level request log](https://3412348858-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-L_Laqs9uSAihPmRemDj%2F-MHlz54XMOCc8uCqGDgu%2F-MHm-wfRMa-PPzXsOTgt%2Fimage.png?alt=media\&token=c637e20d-5cee-434a-8436-046c7d828b36)

If you are not using a Google Cloud HTTP Load Balancer, then you can produce the `httpRequest` log manually. See [Other Loggers' JSON Logging](#json-logging) section.

## Logback

Spring Boot uses [Slf4J](http://www.slf4j.org/) logging API and [Logback](http://logback.qos.ch/) logger by default. You can user [Spring Cloud GCP's Logging Starter](https://cloud.spring.io/spring-cloud-static/spring-cloud-gcp/current/reference/html/#stackdriver-logging) to use pre-configured Logback appenders to produce Structured JSON logs, or send the log via the Cloud Logging API.

### Dependency

Add the Spring Cloud GCP Trace starter:

{% tabs %}
{% tab title="Maven" %}

```markup
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-gcp-starter-logging</artifactId>
</dependency>
```

{% endtab %}

{% tab title="Gradle" %}

```groovy
compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter-logging'
```

{% endtab %}
{% endtabs %}

### Configuration

Configure Logback to use the additional appenders, by adding a `logback-spring.xml` file, and import the appender configuration:

{% code title="logback-spring.xml" %}

```markup
<configuration>
  <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
  <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
  <include resource="org/springframework/cloud/gcp/logging/logback-appender.xml"/>

  ...
</configuration>
```

{% endcode %}

### Log with Cloud Logging API

{% code title="logback-spring.xml" %}

```markup
<configuration>
  <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
  <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
  <include resource="org/springframework/cloud/gcp/logging/logback-appender.xml"/>

  <root level="INFO">
    <appender-ref ref="CONSOLE"/>
    <appender-ref ref="STACKDRIVER"/>
  </root>
</configuration>
```

{% endcode %}

{% hint style="info" %}
Notice that there is no explicit configuration for username/password. Cloud Logging authentication uses the GCP credential (either your user credential, or Service Account credential), and authorization is configured via Identity Access Management (IAM).
{% endhint %}

### Log with Structured JSON Logging

{% code title="logback-spring.xml" %}

```markup
<configuration>
  <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
  <include resource="org/springframework/cloud/gcp/logging/logback-json-appender.xml"/>

  <root level="INFO">
    <appender-ref ref="CONSOLE_JSON"/>
  </root>
</configuration>
```

{% endcode %}

#### Use Different Appenders with Profile

It's useful to configure different appenders when running in [Spring Boot profiles](https://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#boot-features-profiles). For example, in local/dev environments, simply output regular log entries to `STDOUT`, in staging/production environments, use Structured JSON Logging.

{% hint style="info" %}
See [Spring Boot Logging documentation](https://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#profile-specific-configuration) and Spring Boot profiles for more details.
{% endhint %}

For example, to configure default profile to use regular logging, and higher environments with Structured JSON Logging:

{% code title="logback-spring.xml" %}

```markup
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml" />
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />

    <springProfile name="qa | staging | prod">
        <include resource="org/springframework/cloud/gcp/logging/logback-json-appender.xml"/>
        <root level="INFO">
            <appender-ref ref="CONSOLE_JSON"/>
        </root>
    </springProfile>
    <springProfile name="default | dev">
        <root level="INFO">
            <appender-ref ref="CONSOLE"/>
        </root>
    </springProfile>
</configuration>
```

{% endcode %}

This sample application allows you to:

* If no profile is specified, then the `default` profile is used, then use the default `CONSOLE` appender.
* If you specify `dev` profile, then use the `CONSOLE` appender
* If you specify `qa`, `staging`, `prod` profile, then it'll output to Structured JSON Logging.

Alternatively, you can also mix and match the profiles with more generic profiles:

{% code title="logback-spring.xml" %}

```markup
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml" />
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />

    <springProfile name="logging-json">
        <include resource="org/springframework/cloud/gcp/logging/logback-json-appender.xml"/>
        <root level="INFO">
            <appender-ref ref="CONSOLE_JSON"/>
        </root>
    </springProfile>
    <springProfile name="logging-api">
        <include resource="org/springframework/cloud/gcp/logging/logback-appender.xml"/>
        <root level="INFO">
            <appender-ref ref="STACKDRIVER"/>
        </root>
    </springProfile>
    <springProfile name="logging-console | default">
        <root level="INFO">
            <appender-ref ref="CONSOLE"/>
        </root>
    </springProfile>
</configuration>
```

{% endcode %}

This sample application allows you to:

* If no profile is specified, then the `default` profile is used, then use the default `CONSOLE` appender.
* If you specify `logging-json` profile, it'll output to Structured JSON Logging.
* If you specify `logging-api` profile, it'll send the logs via the API.
* If you speicfy `default` and `logging-api` profiles, then it'll use the default `CONSOLE` appender and send the logs via the API.

### Log / Trace Correlation

When using Structured JSON Logging or logging using the API, then Spring Cloud Sleuth's trace context (Trace ID, Span ID) are automatically added to the log metadata. If you explore the log message in the Cloud Logging Console, you can see the `trace` attribute and the `spanId` attribute are both populated with the correct values:

![](https://lh3.googleusercontent.com/4Z0u20hq8WiwuueOU-DqDG58hqbs2m6IG3jCOZpmrNTu8vJjN8sfcjBbbhiDfmQI1MZf_IeJ9x8tnLSUapoiR8kM5M8fqu7avXucQ4JgU3FoWEWu_NbzL8nd1l7kbXdfzqJkiAYfeA)

In the Cloud Trace console, check **Show Logs**, then you can then see the logs alongside the trace itself:

![](https://lh3.googleusercontent.com/O6u214GgMO_GD-xNUkHVj8KTOBH6pf8-_SJP1x17QhdT9Fle3D30gjV-wuTOSSYDHWnjMqFyZmymAIroBTrxNRJGXrT6JqWRQYGVyZE0DMXRDCR4IkNxBCoAwKGnzyctcJMk7-PPBQ)

### Request Log Grouping

In addition to Trace / Log Correlation, if the application is running in Cloud Run, App Engine, or any environment that's fronted by a GCP's HTTP load balancer, then the log entries can be grouped into the top level load balancer produced request log.

![Log entries are grouped under the top-level request log](https://3412348858-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-L_Laqs9uSAihPmRemDj%2F-MHlz54XMOCc8uCqGDgu%2F-MHm-wfRMa-PPzXsOTgt%2Fimage.png?alt=media\&token=c637e20d-5cee-434a-8436-046c7d828b36)

### Samples

* [Spring Cloud GCP Logging sample](https://github.com/spring-cloud/spring-cloud-gcp/tree/master/spring-cloud-gcp-samples/spring-cloud-gcp-logging-sample)

## Other Loggers

It's highly recommended that you use the default logger (Logback) with Spring Boot, to take advantage of Spring Cloud GCP features. If you do use other Loggers, you may be able to configure logging to API with different appenders/handlers.

### JSON Logging

The official [Structured Logging documentation](https://cloud.google.com/logging/docs/structured-logging) suggests that you output the JSON format according to the [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry). Rather than producing the entire `LogEntry`, you can produce a more simplified JSON payload:

```javascript
{
  "message": "My log message",
  "severity": "WARN"
}
```

Cloud Logging agents will automatically extrapolate the `severity` attribute, and also fill in the rest of the `LogEntry` fields so that you don't need to.

If you want to add the trace ID or span ID, you can do so by adding [Special Fields](https://cloud.google.com/logging/docs/agent/configuration#special-fields) to the JSON payload. These special fields will be automatically extrapolated to the `LogEntry`.

```bash
{
  "message": "My log message",
  "severity": "WARN",
  "logging.googleapis.com/trace": "projects/PROJECT_ID/traces/TRACE_ID",
  "logging.googleapis.com/spanId": "SPAN_ID"
}
```

If you want to associate HTTP request information (especially if you are not using a Google Cloud Load Balancer), then you can also add the [`httpRequest`](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#HttpRequest) field:

```bash
{
  "message": "My log message",
  "severity": "WARN",
  "logging.googleapis.com/trace": "projects/PROJECT_ID/traces/TRACE_ID",
  "logging.googleapis.com/spanId": "SPAN_ID",
  "httpRequest": {
    ...
  }
}
```

### API Logging

#### Java Logging API (JUL)

See [Cloud Logging handler for Java Logging API](https://cloud.google.com/logging/docs/setup/java#the_javautillogging_handler).

#### Apache Commons Logging (JCL)

There is no ready-to-use appender to Cloud Logging. But you can [bridge it to Slf4J](http://www.slf4j.org/legacy.html), or [bridge it to Java Logging API](http://commons.apache.org/proper/commons-logging/apidocs/org/apache/commons/logging/impl/Jdk14Logger.html).

#### Log4J 2

There is no ready-to-use appender to Cloud Logging. But you can [bridge it to Slf4J](https://logging.apache.org/log4j/log4j-2.2/log4j-to-slf4j/index.html).

## Learn More

* [Troubleshooting and Debugging Microservices in Kubernetes](https://saturnism.me/talk/troubleshooting-debugging-microservices/)