Debugging

Cloud Debugger

Cloud Debugger lets you inspect the state of an application, at any code location, without stopping or slowing down the running application.

Cloud Debugger is supported in all Google Cloud runtime environments (except for Cloud Functions) and can also be used when running applications in non-Google Cloud environments (on-premises, other clouds).

Enable API

gcloud services enable clouddebugger.googleapis.com

Snapshot

A Snapshot can introspect the context information on a given line of code as user go through the code flow.

Example of a Snapshot

Logpoint

A Logpoint can add additional log messages to a running application without modifying the code nor redeploying the code.

Example of a Logpoint

Conditions

In both Snapshot and Logpoint, you can specify conditionals so you can capture specific information for a specific request (e.g., match against a session ID, or request ID).

Java Agent

Cloud Debugger works by adding a Java agent to your JVM startup argument, and the agent can communicate with the Cloud Debugger service in the Cloud. Through the Cloud Console, you can then instruct your JVM instances to take a Snapshot of the application state at a specific line of code, or to add an additional log message on a specific line.

Agent Files

There are 2 types of Cloud Debugger Java agents that authenticates with Cloud Debugger service differently:

Type	When to use?	Latest Version	Versioned URL
Machine Credentials	Google Cloud runtime environments	Download	https://storage.googleapis.com/cloud-debugger/archive/java/${VERSION}/cdbg_java_agent_gce.tar.gz
Service Account Key	Non-Google Cloud environments	Download	https://storage.googleapis.com/cloud-debugger/archive/java/${VERSION}/cdbg_java_agent_service_account.tar.gz

You can find all the versions in the cloud-debug-java GitHub repository. For example, Cloud Debugger agent version 2.25 using Machine Credentials can be downloaded with URL: https://storage.googleapis.com/cloud-debugger/archive/java/2.25/cdbg_java_agent_gce.tar.gz

Agent Configurations

Agent Path

To use the agent, you'll need to configure the JVM command line using the standard -agentpath , e.g.:

java -agentpath:/opt/cdbg/cdbg_java_agent.so \
  -jar ...

Rather than hard coding the startup command line, you can also configure it with the JAVA_TOOL_OPTIONS environmental variable:

JAVA_TOOL_OPTIONS="-agentpath:/opt/cdbg/cdbg_java_agent.so"
java -jar ...

System Properties

There are additional flags you can pass to the Java agent using Java's system properties.

System Properties	Description	Required
com.google.cdbg .module	The name of your application.	Not required for Cloud Run or App Engine.
com.google.cdbg .version	The version of your application.	Not required for Cloud Run or App Engine.
com.google.cdbg .breakpoints .enable_canary	true or false.Whether to turn on debugger for a subset of the running instances. See Canary snapshots and logpoints documentation.	Not required and defaults to false.
com.google.cdbg .auth.serviceaccount.enable	true or false. Whether to authenticate with a Service Account key file.	Required when running outside of Google Cloud.
com.google.cdbg.auth .serviceaccount.jsonfile	File path to the Service Account key file.	Required when running outside of Google Cloud.

For example, you can enable the snapshot using the system property:

JAVA_TOOL_OPTIONS="-agentpath:/opt/cdbg/cdbg_java_agent.so \
  -Dcom.google.cdbg.breakpoints.enable_canary=true"

Logging

By default the Cloud Debugger agent writes its logs to cdbg_java_agent.INFO file in the default logging directory. You can overwrite the log file path:

java -agentpath:/opt/cdbg/cdbg_java_agent.so=--log_dir=/tmp/cdbg.log \
  -jar ...

Alternatively you can make the Java Cloud Debugger log to stderr:

java -agentpath:/opt/cdbg/cdbg_java_agent.so=--logtostderr=1 \
  -jar ...

See Setting Up Cloud Debugger for Java documentation for more information.

Runtime Configuration

App Engine

Follow App Engine Hello World! instructions to deploy an application to App Engine.

Cloud Debugger agent is automatically added to your App Engine application.

In Cloud Debugger console, you can see the Default service in the drop down:

Cloud Run

Add the Cloud Debugger Java agent to the container, and configure the agent in the startup command line.

Clone

# Clone the sample repository manually
git clone https://github.com/saturnism/jvm-helloworld-by-example
cd jvm-helloworld-by-example/helloworld-springboot-tomcat

Containerize with a Dockerfile

In the Dockerfile, download the Cloud Debugger and build it as part of the container image:

Dockerfile

FROM openjdk:11

# Create a directory for the Debugger. Add and unzip the agent in the directory.
RUN mkdir /opt/cdbg && \
     wget -qO- https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbg_java_agent_gce.tar.gz | \
     tar xvz -C /opt/cdbg

COPY target/helloworld.jar /app.jar

ENTRYPOINT ["java", "-jar", "/app.jar"]

Then build and push the container:

mvn package

PROJECT_ID=$(gcloud config get-value project)
docker build -t gcr.io/${PROJECT_ID}/helloworld .
docker push gcr.io/${PROJECT_ID}/helloworld

Containerize with Jib

Download the Cloud Debugger Java agent into src/main/jib directory so that Jib can include the agent files as part of the container image:

# Make a directory to store the Java agent
mkdir -p src/main/jib/opt/cdbg

# Download and extract the Java agent to the directory
wget -qO- https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbg_java_agent_gce.tar.gz | \
  tar xvz -C src/main/jib/opt/cdbg

Create the container image with Jib:

PROJECT_ID=$(gcloud config get-value project)
mvn compile com.google.cloud.tools:jib-maven-plugin:2.4.0:build \
  -Dimage=gcr.io/${PROJECT_ID}/helloworld

Deploy

Deploy to Cloud Run with Debugger Enabled using the environmental variable:

gcloud run deploy helloworld \
  --region=us-central1 \
  --platform=managed \
  --allow-unauthenticated \
  --set-env-vars='JAVA_TOOL_OPTIONS="-agentpath:/opt/cdbg/cdbg_java_agent.so=--logtostderr=1"' \
  --image=gcr.io/${PROJECT_ID}/helloworld

In Cloud Debugger console, you can see the helloworld service in the drop down:

Kubernetes Engine

Add the Cloud Debugger Java agent to the container, and configure the agent in the startup command line.

Clone

# Clone the sample repository manually
git clone https://github.com/saturnism/jvm-helloworld-by-example
cd jvm-helloworld-by-example/helloworld-springboot-tomcat

Containerize with a Dockerfile

In the Dockerfile, download the Cloud Debugger and build it as part of the container image:

Dockerfile

FROM openjdk:11

# Create a directory for the Debugger. Add and unzip the agent in the directory.
RUN mkdir /opt/cdbg && \
    wget -qO- https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbg_java_agent_gce.tar.gz | \
    tar xvz -C /opt/cdbg

COPY target/helloworld.jar /app.jar

ENTRYPOINT ["java", "-jar", "/app.jar"]

Then build and push the container:

mvn package

PROJECT_ID=$(gcloud config get-value project)
docker build -t gcr.io/${PROJECT_ID}/helloworld .
docker push gcr.io/${PROJECT_ID}/helloworld

Containerize with Jib

Download the Cloud Debugger Java agent into src/main/jib directory so that Jib can include the agent files as part of the container image:

# Make a directory to store the Java agent
mkdir -p src/main/jib/opt/cdbg

# Download and extract the Java agent to the directory
wget -qO- https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbg_java_agent_gce.tar.gz | \
  tar xvz -C src/main/jib/opt/cdbg

Create the image with Jib:

PROJECT_ID=$(gcloud config get-value project)
mvn compile com.google.cloud.tools:jib-maven-plugin:2.4.0:build \
  -Dimage=gcr.io/${PROJECT_ID}/helloworld

Deploy

Deploy to Kubernetes Engine with Debugger Enabled using the environmental variable using a Deployment YAML:

# Make a directory to store Kubernetes YAMLs
mkdir k8s/

Create a Deployment YAML file and configure the environmental variable:

k8s/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: helloworld
  name: helloworld
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - image: gcr.io/YOUR_PROJECT_ID/helloworld
        name: helloworld
        env:
        - name: JAVA_TOOL_OPTIONS
          value: "-agentpath:/opt/cdbg/cdbg_java_agent.so=--logtostderr=1 -Dcom.google.cdbg.module=helloworld-gke -Dcom.google.cdbg.version=1.0"

Deploy the YAML file:

kubectl apply -f k8s/deployment.yaml

In Cloud Debugger console, you can see the helloworld-gke service in the drop down:

Compute Engine

Follow the Compute Engine Hello World! to deploy an application in Compute Engine.

SSH into the Compute Engine instance:

gcloud compute ssh helloworld

From the Compute Engine instance, download the Java agent:

sudo mkdir -p /opt/cdbg
curl -s -o- https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbg_java_agent_gce.tar.gz \
  | sudo tar xvz -C /opt/cdbg

Run the Java application with the Cloud Debugger agent:

java -agentpath:/opt/cdbg/cdbg_java_agent.so=--logtostderr=1 \
    -Dcom.google.cdbg.module=helloworld-gce \
    -Dcom.google.cdbg.version=1.0 \
    -jar helloworld.jar

In Cloud Debugger console, you can see the helloworld-gce service in the drop down:

Non-Google Cloud Environment

You can attach the Cloud Debugger agent to any Java application even if it runs outside of the Google Cloud environment (whether it's in a container, or on your local laptop, or in another cloud). Authentication has to be done using Service Account key file rather than using the Machine Credentials.

This only works on a Linux x86 based system.

Clone

# Clone the sample repository manually
git clone https://github.com/saturnism/jvm-helloworld-by-example
cd jvm-helloworld-by-example/helloworld-springboot-tomcat

Build

mvn package

Download Agent

sudo mkdir -p /opt/cdbg
curl -s -o- https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbg_java_agent_gce.tar.gz \
  | sudo tar xvz -C /opt/cdbg

Create a Service Account

PROJECT_ID=$(gcloud config get-value project)
gcloud iam service-accounts create helloworld-app

Add IAM Permission

PROJECT_ID=$(gcloud config get-value project)
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member serviceAccount:helloworld-app@${PROJECT_ID}.iam.gserviceaccount.com \
  --role roles/clouddebugger.agent

Create a Service Account Key File

PROJECT_ID=$(gcloud config get-value project)
gcloud iam service-accounts keys create \
  $HOME/helloworld-app-sa.json \
  --iam-account helloworld-app@${PROJECT_ID}.iam.gserviceaccount.com

Use Service Account Cloud Debugger Agent

java -agentpath:/opt/cdbg/cdbg_java_agent.so=--logtostderr=1 \
    -Dcom.google.cdbg.module=helloworld-local \
    -Dcom.google.cdbg.version=1.0 \
    -Dcom.google.cdbg.auth.serviceaccount.enable=true \
    -Dcom.google.cdbg.auth.serviceaccount.jsonfile=$HOME/hellworld-app-sa.json \
    -jar target/helloworld.jar

Source Code

Cloud Debugger needs to have access to the application's source code in order for you to add a Snapshot or Logpoint from the Cloud Debugger console. There are severals ways to associating the source code:

• Existing Git Repository

• Source code capture / upload

• Git repository reference from git.properties

• IntelliJ Cloud Code plugin

Git Repository

From Cloud Debugger console, navigate to Deployed Files > Add source code.

Cloud Debugger console and find "Add source code"

Choose an Alternative source code.

Alternative source code choices

For example, using an existing GitHub repository:

Select source from GitHub.com

Once selected, the contents of the Git repository will be available to navigate.

Upload

Upload from Browser

From Cloud Debugger console, navigate to Deployed Files > Add source code.

Cloud Debugger console and find "Add source code"

Choose an Alternative source code.

Click on Local files's Select Source, then simply select the folder/directory that contains the source code.

Upload from Command Line

You can use gcloud CLI to upload the source code into a Source Captures repository.

Create a Source Captures repository:

# Enable API
gcloud services enable sourcerepo.googleapis.com

# Create a source capture repository
gcloud source repos create google-source-captures

In the Alternative source code choices, scroll to the very bottom is Upload a source code capture to Google servers.

Do not click on Select source yet.

Alternative source code choices

Use the command line to upload the source code (for example, if you deployed the Helloworld Application):

# Clone the sample repository manually
git clone https://github.com/GoogleCloudPlatform/java-docs-samples
cd java-docs-samples/appengine-java11/springboot-helloworld

# Upload just the `src/` directory.
# Note that the `branch` value is important and you must use the same value
# that's shown in the UI
gcloud beta debug source upload \
  --project=<FROM THE UI> \
  --branch=<FROM THE UI> \
  src/

Once uploaded, click Select source.

Use git.properties

You can associate a Git repository using the git-commit-plugin that generates a git.properties file, which contains the information to the Git repository. This only works if the repository is publicly accessible.

<plugin>
  <groupId>pl.project13.maven</groupId>
  <artifactId>git-commit-id-plugin</artifactId>
  <version>4.0.1</version>
  <executions>
    <execution>
      <goals>
        <goal>revision</goal>
      </goals>
    </execution>
	</executions>
</plugin>

Cloud Debugger service will automatically examine this file, and clone the code, and checkout the corresponding revision.

IntelliJ with Cloud Code

You can use the Cloud Code plugin to directly add a Snapshot point without using the Cloud Debugger console.

Navigate to Tools > Cloud Code > Attach Cloud Debugger.

Once configured the IntelliJ profile, you can add Snapshot to source code directly from the IDE.

Debug in the Cloud

Learn More

• Troubleshooting and Debugging Microservices in Kubernetes