com.google.cloud.dataflow.sdk.io.bigtable

Class BigtableIO

java.lang.Object

com.google.cloud.dataflow.sdk.io.bigtable.BigtableIO

@Experimental
public class BigtableIO
extends Object

A bounded source and sink for Google Cloud Bigtable.

For more information, see the online documentation at Google Cloud Bigtable.

Reading from Cloud Bigtable

The Bigtable source returns a set of rows from a single table, returning a PCollection<Row>.

To configure a Cloud Bigtable source, you must supply a table id and a BigtableOptions or builder configured with the project and other information necessary to identify the Bigtable cluster. A RowFilter may also optionally be specified using BigtableIO.Read.withRowFilter(com.google.bigtable.v1.RowFilter). For example:

 BigtableOptions.Builder optionsBuilder =
     new BigtableOptions.Builder()
         .setProjectId("project")
         .setClusterId("cluster")
         .setZoneId("zone");

 Pipeline p = ...;

 // Scan the entire table.
 p.apply("read",
     BigtableIO.read()
         .withBigtableOptions(optionsBuilder)
         .withTableId("table"));

 // Scan a subset of rows that match the specified row filter.
 p.apply("filtered read",
     BigtableIO.read()
         .withBigtableOptions(optionsBuilder)
         .withTableId("table")
         .withRowFilter(filter));
 

Writing to Cloud Bigtable

The Bigtable sink executes a set of row mutations on a single table. It takes as input a PCollection<KV<ByteString, Iterable<Mutation>>>, where the ByteString is the key of the row being mutated, and each Mutation represents an idempotent transformation to that row.

To configure a Cloud Bigtable sink, you must supply a table id and a BigtableOptions or builder configured with the project and other information necessary to identify the Bigtable cluster, for example:

 BigtableOptions.Builder optionsBuilder =
     new BigtableOptions.Builder()
         .setProjectId("project")
         .setClusterId("cluster")
         .setZoneId("zone");

 PCollection<KV<ByteString, Iterable<Mutation>>> data = ...;

 data.apply("write",
     BigtableIO.write()
         .withBigtableOptions(optionsBuilder)
         .withTableId("table"));
 

Experimental

This connector for Cloud Bigtable is considered experimental and may break or receive backwards-incompatible changes in future versions of the Cloud Dataflow SDK. Cloud Bigtable is in Beta, and thus it may introduce breaking changes in future revisions of its service or APIs.

Permissions

Permission requirements depend on the PipelineRunner that is used to execute the Dataflow job. Please refer to the documentation of corresponding PipelineRunners for more details.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BigtableIO.Read A PTransform that reads from Google Cloud Bigtable.
static class	BigtableIO.Write A PTransform that writes to Google Cloud Bigtable.

Method Summary

Modifier and Type	Method and Description
static BigtableIO.Read	read() Creates an uninitialized BigtableIO.Read.
static BigtableIO.Write	write() Creates an uninitialized BigtableIO.Write.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

read

@Experimental
public static BigtableIO.Read read()

Creates an uninitialized BigtableIO.Read. Before use, the Read must be initialized with a BigtableOptions that specifies the source Cloud Bigtable cluster, and a tableId that specifies which table to read. A RowFilter may also optionally be specified using BigtableIO.Read.withRowFilter(com.google.bigtable.v1.RowFilter).

write

@Experimental
public static BigtableIO.Write write()

Creates an uninitialized BigtableIO.Write. Before use, the Write must be initialized with a BigtableOptions that specifies the destination Cloud Bigtable cluster, and a tableId that specifies which table to write.