Package com.google.cloud.dataflow.sdk.io

Defines transforms for reading and writing common storage formats, including AvroIO, BigQueryIO, and TextIO.

See: Description

Interface Summary

Interface	Description
Source.Reader<T>	The interface which readers of custom input sources must implement.

Class Summary

Class	Description
AvroIO	Transforms for reading and writing Avro files.
AvroIO.Read	A root PTransform that reads from an Avro file (or multiple Avro files matching a pattern) and returns a PCollection containing the decoding of each record.
AvroIO.Read.Bound<T>	A PTransform that reads from an Avro file (or multiple Avro files matching a pattern) and returns a bounded PCollection containing the decoding of each record.
AvroIO.Write	A root PTransform that writes a PCollection to an Avro file (or multiple Avro files matching a sharding pattern).
AvroIO.Write.Bound<T>	A PTransform that writes a bounded PCollection to an Avro file (or multiple Avro files matching a sharding pattern).
BigQueryIO	Transformations for reading and writing BigQuery tables.
BigQueryIO.Read	A PTransform that reads from a BigQuery table and returns a PCollection<TableRow> containing each of the rows of the table.
BigQueryIO.Read.Bound	A PTransform that reads from a BigQuery table and returns a bounded PCollection<TableRow>.
BigQueryIO.Write	A PTransform that writes a PCollection<TableRow> containing rows to a BigQuery table.
BigQueryIO.Write.Bound	A PTransform that can write either a bounded or unbounded PCollection<TableRow>s to a BigQuery table.
ByteOffsetBasedSource<T>	A source that uses byte offsets to define starting and ending positions.
ByteOffsetBasedSource.ByteOffsetBasedReader<T>	A reader that implements code common to readers of all ByteOffsetBasedSources.
DatastoreIO	Transforms for reading and writing Google Cloud Datastore entities.
DatastoreIO.DatastoreReader	A reader over the records from a query of the datastore.
DatastoreIO.Sink	A PTransform that writes a PCollection<Entity> containing entities to a Datastore kind.
DatastoreIO.Source	A source that reads the result rows of a Datastore query as Entity objects.
FileBasedSource<T>	A common base class for all file-based Sources.
FileBasedSource.FileBasedReader<T>	A reader that implements code common to readers of FileBasedSources.
PubsubIO	Read and Write transforms for Pub/Sub streams.
PubsubIO.Read	A PTransform that continuously reads from a Pubsub stream and returns a PCollection<String> containing the items from the stream.
PubsubIO.Read.Bound	A PTransform that reads from a PubSub source and returns a unbounded PCollection containing the items from the stream.
PubsubIO.Validator	Utility class to validate topic and subscription names.
PubsubIO.Write	A PTransform that continuously writes a PCollection<String> to a Pubsub stream.
PubsubIO.Write.Bound	A PTransfrom that writes a unbounded PCollection<String> to a PubSub stream.
ReadSource	The PTransform for reading from a Source.
ReadSource.Bound<T>	Implementation of the ReadSource PTransform builder.
ShardNameTemplate	Standard shard naming templates.
Source<T>	Base class for defining input formats, with custom logic for splitting the input into bundles (parts of the input, each of which may be processed on a different worker) and creating a Source for reading the input.
TextIO	Transforms for reading and writing text files.
TextIO.Read	A root PTransform that reads from a text file (or multiple text files matching a pattern) and returns a PCollection containing the decoding of each of the lines of the text file(s).
TextIO.Read.Bound<T>	A root PTransform that reads from a text file (or multiple text files matching a pattern) and returns a bounded PCollection containing the decoding of each of the lines of the text file(s).
TextIO.Write	A PTransform that writes a PCollection to a text file (or multiple text files matching a sharding pattern), with each PCollection element being encoded into its own line.
TextIO.Write.Bound<T>	A PTransform that writes a bounded PCollection to a text file (or multiple text files matching a sharding pattern), with each PCollection element being encoded into its own line.

Enum Summary

Enum	Description
BigQueryIO.Write.CreateDisposition	An enumeration type for the BigQuery create disposition strings publicly documented as CREATE_NEVER, and CREATE_IF_NEEDED.
BigQueryIO.Write.WriteDisposition	An enumeration type for the BigQuery write disposition strings publicly documented as WRITE_TRUNCATE, WRITE_APPEND, and WRITE_EMPTY.
FileBasedSource.Mode	A given FileBasedSource represents a file resource of one of these types.
TextIO.CompressionType	Possible text file compression types.

Package com.google.cloud.dataflow.sdk.io Description

Defines transforms for reading and writing common storage formats, including AvroIO, BigQueryIO, and TextIO.

The classes in this package provide Read transforms which create PCollections from existing storage:

PCollection<TableRow> inputData = pipeline.apply(
     BigQueryIO.Read.named("Read")
                    .from("clouddataflow-readonly:samples.weather_stations");
 

and Write transforms which persist PCollections to external storage:

 PCollection<Integer> numbers = ...;
 numbers.apply(TextIO.Write.named("WriteNumbers")
                           .to("gs://my_bucket/path/to/numbers"));