Storage

Cloud Storage
Cloud Storage provides globally unified, scalable, and highly durable object storage. You can store files in Cloud Storage without having to worry about running out of space, or managing your own filesystems.
Cloud Storage buckets can be configured to store files in a single region, dual-region, or multi-region.
Location
Use Case
Region
Use a region to help optimize latency and network bandwidth for data consumers, such as analytics pipelines, that are grouped in the same region.
Dual-Region
Use a dual-region when you want similar performance advantages as regions, but also want the higher availability that comes with being geo-redundant.
Multi-Region
Use a multi-region when you want to serve content to data consumers that are outside of the Google network and distributed across large geographic areas, or when you want the higher availability that comes with being geo-redundant.
See Cloud Storage Bucket locations for more information.
Cloud Storage buckets can also be configured with a different Storage Class, for different access patterns to optimize cost.
Storage Class
Use Case
Standard
Standard Storage is best for data that is frequently accessed ("hot" data) and/or stored for only brief periods of time.
Nearline
Nearline Storage is a low-cost, highly durable storage service for storing infrequently accessed data. Nearline Storage is a better choice than Standard Storage in scenarios where slightly lower availability, a 30-day minimum storage duration.
Coldline
Coldline Storage is a very-low-cost, highly durable storage service for storing infrequently accessed data. Coldline Storage is a better choice than Standard Storage or Nearline Storage in scenarios where slightly lower availability, a 90-day minimum storage duration.
Archive
Archive Storage is the lowest-cost, highly durable storage service for data archiving, online backup, and disaster recovery. Unlike the "coldest" storage services offered by other Cloud providers, your data is available within milliseconds, not hours or days.
See Cloud Storage Classes for more information.
Enable API
1
gcloud services enable storage-component.googleapis.com
Copied!
Create a Bucket
A bucket is the top-level directory that you can add additional files and sub-directories into. A bucket name must be globally unique. For example, create a bucket with the same name as your Google Cloud project.
1
PROJECT_ID=$(gcloud config get-value project)
2
gsutil mb gs://$PROJECT_ID
Copied!
Copy a File
1
echo "Hello World" > hello.txt
2
gsutil cp hello.txt gs://$PROJECT_ID
Copied!
Delete a File
1
gsutil rm gs://$PROJECT_ID/hello.txt
Copied!
Spring Resource
With Spring Cloud GCP, you can use Spring Resource to perform store and retrieve file from Cloud Storage.
Dependency
Add the Spring Cloud GCP Storage starter:
Maven
Gradle
1
<dependency>
2
    <groupId>org.springframework.cloud</groupId>
3
    <artifactId>spring-cloud-gcp-starter-storage</artifactId>
4
</dependency>
Copied!
1
​
2
compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter-storage'
Copied!
Configuration
There is no explicit configuration required if you use the automatic authentication and project ID detection. I.e., if you already logged in locally with gcloud command line, then it'll automatically access buckets that you have access to.
Notice that there is no explicit configuration for username/password. Cloud Storage authentication uses the GCP credential (either your user credential, or Service Account credential), and authorization is configured via Identity Access Management (IAM).
Storage Client
The starter automatically creates a pre-configured Storage bean that provides raw-access to Google Cloud Storage.
1
@Bean
2
ApplicationRunner storageRunner(Storage storage, GcpProjectIdProvider projectIdProvider) {
3
  return (args) -> {
4
    Page<Blob> list = storage.list(projectIdProvider.getProjectId());
5
    list.iterateAll().forEach(blob -> System.out.println(blob.getName()));
6
  };
7
}
Copied!
Resource URI
You can address a Cloud Storage file by using the resource URI prefixed with gs://. The fully qualified URI is of the form: gs://project-id/path/to/file.
Read a file
You can open the Resource using ApplicationContext. Then read the content from InputStream. You must close the stream when you are done (or wrap with try-with-resource since it's auto-closeable).
1
@Bean
2
ApplicationRunner runner(ApplicationContext ctx, GcpProjectIdProvider projectIdProvider) {
3
  return (args) -> {
4
    WritableResource resource = (WritableResource) ctx
5
        .getResource(String.format("gs://%s/hello.txt", projectIdProvider.getProjectId()));
6
    try (PrintWriter writer = new PrintWriter(resource.getOutputStream())) {
7
      writer.println("Hello World!");
8
    }
9
  };
10
}
Copied!
Write a file
You can open the Resource using ApplicationContext. Then cast the Resource to a WritableResource, and then use the OutputStream to write the content. Lastly, you must close the stream in order for the file to write (or wrap with try-with-resource since it's auto-closeable).
1
@Bean
2
ApplicationRunner readRunner(ApplicationContext ctx, GcpProjectIdProvider projectIdProvider) {
3
  return (args) -> {
4
    Resource resource = ctx
5
        .getResource(String.format("gs://%s/hello.txt", projectIdProvider.getProjectId()));
6
    try (InputStreamReader reader = new InputStreamReader(resource.getInputStream())) {
7
      BufferedReader bufferedReader = new BufferedReader(reader);
8
      bufferedReader.lines().forEach(System.out::println);
9
    }
10
  };
11
}
Copied!
Samples
​Spring Cloud GCP Storage sample​
Spring Integration
You can channel adapters for Google Cloud Storage to read and write files to Google Cloud Storage through MessageChannels.
Dependency
Add both the Spring Cloud GCP Storage starter, and Spring Integration File component.
Maven
Gradle
1
<dependency>
2
    <groupId>org.springframework.cloud</groupId>
3
    <artifactId>spring-cloud-gcp-starter-storage</artifactId>
4
</dependency>
5
<dependency>
6
    <groupId>org.springframework.integration</groupId>
7
    <artifactId>spring-integration-file</artifactId>
8
</dependency>
Copied!
1
​
2
compile group: 'org.springframework.cloud', name: 'spring-cloud-gcp-starter-storage'
3
compile group: 'org.springframework.integration', name: 'spring-integration-file'
Copied!
Inbound Channel Adapter
Inbound File Synchronizer
File-based integration with files typically requires polling a directory that contains the new files.  In Spring Integration, this is configured through an InboundFileSynchronizer. Use GcsInboundFileSyncronizer to create a MessageSource and adapt it to a MessageChannel.
The files are temporarily stored in a directory in the local file system.
1
@Bean
2
public MessageChannel gcsInputChannel() {
3
  return MessageChannels.direct().get();
4
}
5
​
6
@Bean
7
@InboundChannelAdapter(channel = "gcsInputChannel", poller = @Poller(fixedDelay = "5000"))
8
public MessageSource<File> (Storage gcs, GcpProjectIdProvider projectIdProvider)
9
    throws IOException {
10
  GcsInboundFileSynchronizer synchronizer = new GcsInboundFileSynchronizer(gcs);
11
  synchronizer.setRemoteDirectory(projectIdProvider.getProjectId());
12
​
13
  GcsInboundFileSynchronizingMessageSource messageSource =
14
          new GcsInboundFileSynchronizingMessageSource(synchronizer);
15
  File localDirectory = Files.createTempDirectory("gcs");
16
  messageSource.setLocalDirectory(localDirectory);
17
​
18
  return messageSource;
19
}
Copied!
Streaming Message Source
For most use cases, you should use the streaming message source, which does not require files to be stored in the file system.
1
@Bean
2
public MessageChannel gcsInputChannel() {
3
  return MessageChannels.direct().get();
4
}
5
​
6
@Bean
7
@InboundChannelAdapter(channel = "gcsInputChannel", poller = @Poller(fixedDelay = "5000"))
8
public MessageSource<InputStream> streamingAdapter(Storage gcs, GcpProjectIdProvider projectIdProvider) {
9
  GcsStreamingMessageSource adapter =
10
          new GcsStreamingMessageSource(new GcsRemoteFileTemplate(new GcsSessionFactory(gcs)));
11
  adapter.setRemoteDirectory(projectIdProvider.getProjectId());
12
  return adapter;
13
}
Copied!
Outbound Channel Adapter
The outbound channel adapter allows files to be written to Google Cloud Storage. When it receives a Message containing a payload of type File, it writes that file to the Google Cloud Storage bucket specified in the adapter.
1
@Bean
2
@ServiceActivator(inputChannel = "gcsOutputChannel")
3
public MessageHandler outboundChannelAdapter(Storage gcs, GcpProjectIdProvider projectIdProvider) {
4
  GcsMessageHandler outboundChannelAdapter = new GcsMessageHandler(new GcsSessionFactory(gcs));
5
  outboundChannelAdapter.setRemoteDirectoryExpression(new ValueExpression<>(projectIdProvider.getProjectId()));
6
​
7
  return outboundChannelAdapter;
8
}
Copied!
Samples
​Spring Cloud Integration with Cloud Storage Sample​