com.google.cloud.dataflow.sdk.io

## Class AvroSource<T>

- java.lang.Object
- - com.google.cloud.dataflow.sdk.io.Source<T>
  - - com.google.cloud.dataflow.sdk.io.BoundedSource<T>
    - - com.google.cloud.dataflow.sdk.io.OffsetBasedSource<T>
      - - com.google.cloud.dataflow.sdk.io.FileBasedSource<T>
        - - com.google.cloud.dataflow.sdk.io.BlockBasedSource<T>
          - - com.google.cloud.dataflow.sdk.io.AvroSource<T>

Type Parameters:
  `T` - The type of records to be read from the source.

All Implemented Interfaces:
  HasDisplayData, Serializable

    @Experimental(value=SOURCE_SINK)
    public class AvroSource<T>
    extends BlockBasedSource<T>

A `FileBasedSource` for reading Avro files.

To read a `PCollection` of objects from one or more Avro files, use `from(java.lang.String)` to specify the path(s) of the files to read. The `AvroSource` that is returned will read objects of type `GenericRecord` with the schema(s) that were written at file creation. To further configure the `AvroSource` to read with a user-defined schema, or to return records of a type other than `GenericRecord`, use `withSchema(Schema)` (using an Avro `Schema`), `withSchema(String)` (using a JSON schema), or `withSchema(Class)` (to return objects of the Avro-generated class specified).

An `AvroSource` can be read from using the `Read` transform. For example:

     AvroSource<MyType> source = AvroSource.from(file.toPath()).withSchema(MyType.class);
     PCollection<MyType> records = Read.from(mySource);

The `readFromFileWithClass(String, Class)` method is a convenience method that returns a read transform. For example:

     PCollection<MyType> records = AvroSource.readFromFileWithClass(file.toPath(), MyType.class));

This class's implementation is based on the Avro 1.7.7 specification and implements parsing of some parts of Avro Object Container Files. The rationale for doing so is that the Avro API does not provide efficient ways of computing the precise offsets of blocks within a file, which is necessary to support dynamic work rebalancing. However, whenever it is possible to use the Avro API in a way that supports maintaining precise offsets, this class uses the Avro API.

Avro Object Container files store records in blocks. Each block contains a collection of records. Blocks may be encoded (e.g., with bzip2, deflate, snappy, etc.). Blocks are delineated from one another by a 16-byte sync marker.

An `AvroSource` for a subrange of a single file contains records in the blocks such that the start offset of the block is greater than or equal to the start offset of the source and less than the end offset of the source.

To use XZ-encoded Avro files, please include an explicit dependency on `xz-1.5.jar`, which has been marked as optional in the Maven `sdk/pom.xml` for Google Cloud Dataflow:

     <dependency>
       <groupId>org.tukaani</groupId>
       <artifactId>xz</artifactId>
       <version>1.5</version>
     </dependency>

### Permissions

Permission requirements depend on the `PipelineRunner` that is used to execute the Dataflow job. Please refer to the documentation of corresponding `PipelineRunner`s for more details.

See Also:
  Serialized Form

### Nested Class Summary

Nested Classes
`static class` `AvroSource.AvroReader<T>` A `BlockBasedSource.BlockBasedReader` for reading blocks from Avro files.

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.BlockBasedSource

`BlockBasedSource.Block<T>, BlockBasedSource.BlockBasedReader<T>`

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.FileBasedSource

`FileBasedSource.FileBasedReader<T>, FileBasedSource.Mode`

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.OffsetBasedSource

`OffsetBasedSource.OffsetBasedReader<T>`

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

`BoundedSource.BoundedReader<T>`

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

`Source.Reader<T>`

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`BlockBasedSource<T>` `createForSubrangeOfFile(String fileName, long start, long end)` Creates a `BlockBasedSource` for the specified range in a single file.
`protected BlockBasedSource.BlockBasedReader<T>` `createSingleFileReader(PipelineOptions options)` Creates a `BlockBasedReader`.
`static AvroSource<GenericRecord>` `from(String fileNameOrPattern)` Creates an `AvroSource` that reads from the given file name or pattern ("glob").
`AvroCoder<T>` `getDefaultOutputCoder()` Returns the default `Coder` to use for the data read from this source.
`String` `getSchema()`
`boolean` `producesSortedKeys(PipelineOptions options)` Whether this source is known to produce key/value pairs sorted by lexicographic order on the bytes of the encoded key.
`static <T> Read.Bounded<T>` `readFromFileWithClass(String filePattern, Class<T> clazz)` Creates a `Read` transform that will read from an `AvroSource` that is configured to read records of the given type from a file pattern.
`void` `validate()` Checks that this source is valid, before it can be used in a pipeline.
`AvroSource<T>` `withMinBundleSize(long minBundleSize)` Returns an `AvroSource` that's like this one but uses the supplied minimum bundle size.
`<X> AvroSource<X>` `withSchema(Class<X> clazz)` Returns an `AvroSource` that's like this one but reads files containing records of the type of the given class.
`AvroSource<GenericRecord>` `withSchema(Schema schema)` Returns an `AvroSource` that's like this one but reads files containing records that conform to the given schema.
`AvroSource<GenericRecord>` `withSchema(String schema)` Returns an `AvroSource` that's like this one but reads files containing records that conform to the given schema.

### Methods inherited from class com.google.cloud.dataflow.sdk.io.FileBasedSource

`createReader, createSourceForSubrange, expandFilePattern, getEstimatedSizeBytes, getFileOrPatternSpec, getMaxEndOffset, getMode, isSplittable, populateDisplayData, splitIntoBundles, toString`

### Methods inherited from class com.google.cloud.dataflow.sdk.io.OffsetBasedSource

`allowsDynamicSplitting, getBytesPerOffset, getEndOffset, getMinBundleSize, getStartOffset`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Method Detail

#### readFromFileWithClass

    public static <T> Read.Bounded<T> readFromFileWithClass(String filePattern,
                                                            Class<T> clazz)

Creates a `Read` transform that will read from an `AvroSource` that is configured to read records of the given type from a file pattern.

#### from

    public static AvroSource<GenericRecord> from(String fileNameOrPattern)

Creates an `AvroSource` that reads from the given file name or pattern ("glob"). The returned source can be further configured by calling `withSchema(java.lang.String)` to return a type other than `GenericRecord`.

#### withSchema

    public AvroSource<GenericRecord> withSchema(String schema)

Returns an `AvroSource` that's like this one but reads files containing records that conform to the given schema.

Does not modify this object.

#### withSchema

    public AvroSource<GenericRecord> withSchema(Schema schema)

Returns an `AvroSource` that's like this one but reads files containing records that conform to the given schema.

Does not modify this object.

#### withSchema

    public <X> AvroSource<X> withSchema(Class<X> clazz)

Returns an `AvroSource` that's like this one but reads files containing records of the type of the given class.

Does not modify this object.

#### withMinBundleSize

    public AvroSource<T> withMinBundleSize(long minBundleSize)

Returns an `AvroSource` that's like this one but uses the supplied minimum bundle size. Refer to `OffsetBasedSource` for a description of `minBundleSize` and its use.

Does not modify this object.

#### validate

    public void validate()

Description copied from class: `Source`
Checks that this source is valid, before it can be used in a pipeline.

It is recommended to use `Preconditions` for implementing this method.

Overrides:
  `validate` in class `FileBasedSource<T>`

#### createForSubrangeOfFile

    public BlockBasedSource<T> createForSubrangeOfFile(String fileName,
                                                       long start,
                                                       long end)

Description copied from class: `BlockBasedSource`
Creates a `BlockBasedSource` for the specified range in a single file.

Specified by:
  `createForSubrangeOfFile` in class `BlockBasedSource<T>`
Parameters:
  `fileName` - file backing the new `FileBasedSource`.
  `start` - starting byte offset of the new `FileBasedSource`.
  `end` - ending byte offset of the new `FileBasedSource`. May be Long.MAX_VALUE, in which case it will be inferred using `FileBasedSource.getMaxEndOffset(com.google.cloud.dataflow.sdk.options.PipelineOptions)`.

#### createSingleFileReader

    protected BlockBasedSource.BlockBasedReader<T> createSingleFileReader(PipelineOptions options)

Description copied from class: `BlockBasedSource`
Creates a `BlockBasedReader`.

Specified by:
  `createSingleFileReader` in class `BlockBasedSource<T>`

#### producesSortedKeys

    public boolean producesSortedKeys(PipelineOptions options)
                               throws Exception

Description copied from class: `BoundedSource`
Whether this source is known to produce key/value pairs sorted by lexicographic order on the bytes of the encoded key.

Specified by:
  `producesSortedKeys` in class `BoundedSource<T>`
Throws:
  `Exception`

#### getDefaultOutputCoder

    public AvroCoder<T> getDefaultOutputCoder()

Description copied from class: `Source`
Returns the default `Coder` to use for the data read from this source.

Specified by:
  `getDefaultOutputCoder` in class `Source<T>`

#### getSchema

    public String getSchema()