Package com.google.cloud.dataflow.sdk.io

Defines transforms for reading and writing common storage formats, including AvroIO, BigQueryIO, and TextIO.

See: Description

Class Summary

Class	Description
AvroIO	Transforms for reading and writing Avro files.
AvroIO.Read	A root PTransform that reads from an Avro file (or multiple Avro files matching a pattern) and returns a PCollection containing the decoding of each record.
AvroIO.Read.Bound<T>	A PTransform that reads from an Avro file (or multiple Avro files matching a pattern) and returns a bounded PCollection containing the decoding of each record.
AvroIO.Write	A root PTransform that writes a PCollection to an Avro file (or multiple Avro files matching a sharding pattern).
AvroIO.Write.Bound<T>	A PTransform that writes a bounded PCollection to an Avro file (or multiple Avro files matching a sharding pattern).
BigQueryIO	Transformations for reading and writing BigQuery tables.
BigQueryIO.Read	A PTransform that reads from a BigQuery table and returns a PCollection<TableRow> containing each of the rows of the table.
BigQueryIO.Read.Bound	A PTransform that reads from a BigQuery table and returns a bounded PCollection<TableRow>.
BigQueryIO.Write	A PTransform that writes a PCollection<TableRow> containing rows to a BigQuery table.
BigQueryIO.Write.Bound	A PTransform that can write either a bounded or unbounded PCollection<TableRow>s to a BigQuery table.
DatastoreIO	Transforms for reading and writing Google Cloud Datastore entities.
DatastoreIO.Read	A PTransform that reads from a Datastore query and returns a PCollection<Entity> containing each of the rows of the table.
DatastoreIO.Read.Bound	A PTransform that reads from a Datastore query and returns a bounded PCollection<Entity>.
DatastoreIO.Write	A PTransform that writes a PCollection<Entity> containing entities to a Datastore kind.
DatastoreIO.Write.Bound	A PTransform that writes a bounded PCollection<Entities> to a Datastore.
PubsubIO	[Whitelisting Required] Read and Write transforms for Pub/Sub streams.
PubsubIO.Read	A PTransform that continuously reads from a Pubsub stream and returns a PCollection<String> containing the items from the stream.
PubsubIO.Read.Bound	A PTransform that reads from a PubSub source and returns a unbounded PCollection containing the items from the stream.
PubsubIO.Validator	Utility class to validate topic and subscription names.
PubsubIO.Write	A PTransform that continuously writes a PCollection<String> to a Pubsub stream.
PubsubIO.Write.Bound	A PTransfrom that writes a unbounded PCollection<String> to a PubSub stream.
ShardNameTemplate	Standard shard naming templates.
TextIO	Transforms for reading and writing text files.
TextIO.Read	A root PTransform that reads from a text file (or multiple text files matching a pattern) and returns a PCollection containing the decoding of each of the lines of the text file(s).
TextIO.Read.Bound<T>	A root PTransform that reads from a text file (or multiple text files matching a pattern) and returns a bounded PCollection containing the decoding of each of the lines of the text file(s).
TextIO.Write	A PTransform that writes a PCollection to a text file (or multiple text files matching a sharding pattern), with each PCollection element being encoded into its own line.
TextIO.Write.Bound<T>	A PTransform that writes a bounded PCollection to a text file (or multiple text files matching a sharding pattern), with each PCollection element being encoded into its own line.

Enum Summary

Enum	Description
BigQueryIO.Write.CreateDisposition	An enumeration type for the BigQuery create disposition strings publicly documented as CREATE_NEVER, and CREATE_IF_NEEDED.
BigQueryIO.Write.WriteDisposition	An enumeration type for the BigQuery write disposition strings publicly documented as WRITE_TRUNCATE, WRITE_APPEND, and WRITE_EMPTY.

Package com.google.cloud.dataflow.sdk.io Description

Defines transforms for reading and writing common storage formats, including AvroIO, BigQueryIO, and TextIO.

The classes in this package provide Read transforms which create PCollections from existing storage:

PCollection<TableRow> inputData = pipeline.apply(
     BigQueryIO.Read.named("Read")
                    .from("clouddataflow-readonly:samples.weather_stations");
 

and Write transforms which persist PCollections to external storage:

 PCollection<Integer> numbers = ...;
 numbers.apply(TextIO.Write.named("WriteNumbers")
                           .to("gs://my_bucket/path/to/numbers"));