com.google.cloud.dataflow.sdk.io

Class AvroIO.Read

java.lang.Object

com.google.cloud.dataflow.sdk.io.AvroIO.Read

Enclosing class:

AvroIO

public static class AvroIO.Read
extends Object

A root PTransform that reads from an Avro file (or multiple Avro files matching a pattern) and returns a PCollection containing the decoding of each record.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroIO.Read.Bound<T> A PTransform that reads from an Avro file (or multiple Avro files matching a pattern) and returns a bounded PCollection containing the decoding of each record.

Constructor Summary

Constructors

Constructor and Description
Read()

Method Summary

Modifier and Type	Method and Description
static AvroIO.Read.Bound<org.apache.avro.generic.GenericRecord>	from(String filepattern) Returns an AvroIO.Read PTransform that reads from the file(s) with the given name or pattern.
static AvroIO.Read.Bound<org.apache.avro.generic.GenericRecord>	named(String name) Returns an AvroIO.Read PTransform with the given step name.
static AvroIO.Read.Bound<org.apache.avro.generic.GenericRecord>	withoutValidation() Returns a AvroIO.Read PTransform that has GCS path validation on pipeline creation disabled.
static <T> AvroIO.Read.Bound<T>	withSchema(Class<T> type) Returns an AvroIO.Read PTransform that reads Avro file(s) containing records whose type is the specified Avro-generated class.
static AvroIO.Read.Bound<org.apache.avro.generic.GenericRecord>	withSchema(org.apache.avro.Schema schema) Returns an AvroIO.Read PTransform that reads Avro file(s) containing records of the specified schema.
static AvroIO.Read.Bound<org.apache.avro.generic.GenericRecord>	withSchema(String schema) Returns an AvroIO.Read PTransform that reads Avro file(s) containing records of the specified schema in a JSON-encoded string form.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Read

public Read()

Method Detail

named

public static AvroIO.Read.Bound<org.apache.avro.generic.GenericRecord> named(String name)

Returns an AvroIO.Read PTransform with the given step name.

from

public static AvroIO.Read.Bound<org.apache.avro.generic.GenericRecord> from(String filepattern)

Returns an AvroIO.Read PTransform that reads from the file(s) with the given name or pattern. This can be a local filename or filename pattern (if running locally), or a Google Cloud Storage filename or filename pattern of the form "gs://<bucket>/<filepath>") (if running locally or via the Google Cloud Dataflow service). Standard Java Filesystem glob patterns ("*", "?", "[..]") are supported.

withSchema

public static <T> AvroIO.Read.Bound<T> withSchema(Class<T> type)

Returns an AvroIO.Read PTransform that reads Avro file(s) containing records whose type is the specified Avro-generated class.

Type Parameters:

T - the type of the decoded elements, and the elements of the resulting PCollection

withSchema

public static AvroIO.Read.Bound<org.apache.avro.generic.GenericRecord> withSchema(org.apache.avro.Schema schema)

Returns an AvroIO.Read PTransform that reads Avro file(s) containing records of the specified schema.

withSchema

public static AvroIO.Read.Bound<org.apache.avro.generic.GenericRecord> withSchema(String schema)

Returns an AvroIO.Read PTransform that reads Avro file(s) containing records of the specified schema in a JSON-encoded string form.

withoutValidation

public static AvroIO.Read.Bound<org.apache.avro.generic.GenericRecord> withoutValidation()

Returns a AvroIO.Read PTransform that has GCS path validation on pipeline creation disabled.

This can be useful in the case where the GCS input location does not exist at the pipeline creation time, but is expected to be available at execution time.