com.google.cloud.dataflow.sdk.io

## Class CompressedSource<T>

- java.lang.Object
- - com.google.cloud.dataflow.sdk.io.Source<T>
  - - com.google.cloud.dataflow.sdk.io.BoundedSource<T>
    - - com.google.cloud.dataflow.sdk.io.OffsetBasedSource<T>
      - - com.google.cloud.dataflow.sdk.io.FileBasedSource<T>
        - - com.google.cloud.dataflow.sdk.io.CompressedSource<T>

Type Parameters:
  `T` - The type to read from the compressed file.

All Implemented Interfaces:
  HasDisplayData, Serializable

    @Experimental(value=SOURCE_SINK)
    public class CompressedSource<T>
    extends FileBasedSource<T>

A Source that reads from compressed files. A `CompressedSources` wraps a delegate `FileBasedSource` that is able to read the decompressed file format.

For example, use the following to read from a gzip-compressed XML file:

     XmlSource mySource = XmlSource.from(...);
     PCollection<T> collection = p.apply(Read.from(CompressedSource
         .from(mySource)
         .withDecompression(CompressedSource.CompressionMode.GZIP)));

Supported compression algorithms are `CompressedSource.CompressionMode.GZIP` and `CompressedSource.CompressionMode.BZIP2`. User-defined compression types are supported by implementing `CompressedSource.DecompressingChannelFactory`.

By default, the compression algorithm is selected from those supported in `CompressedSource.CompressionMode` based on the file name provided to the source, namely `".bz2"` indicates `CompressedSource.CompressionMode.BZIP2` and `".gz"` indicates `CompressedSource.CompressionMode.GZIP`. If the file name does not match any of the supported algorithms, it is assumed to be uncompressed data.

See Also:
  Serialized Form

### Nested Class Summary

Nested Classes
`static class` `CompressedSource.CompressedReader<T>` Reader for a `CompressedSource`.
`static class` `CompressedSource.CompressionMode` Default compression types supported by the `CompressedSource`.
`static interface` `CompressedSource.DecompressingChannelFactory` Factory interface for creating channels that decompress the content of an underlying channel.

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.FileBasedSource

`FileBasedSource.FileBasedReader<T>, FileBasedSource.Mode`

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.OffsetBasedSource

`OffsetBasedSource.OffsetBasedReader<T>`

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

`BoundedSource.BoundedReader<T>`

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

`Source.Reader<T>`

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`protected FileBasedSource<T>` `createForSubrangeOfFile(String fileName, long start, long end)` Creates a `CompressedSource` for a subrange of a file.
`protected FileBasedSource.FileBasedReader<T>` `createSingleFileReader(PipelineOptions options)` Creates a `FileBasedReader` to read a single file.
`static <T> CompressedSource<T>` `from(FileBasedSource<T> sourceDelegate)` Creates a `CompressedSource` from an underlying `FileBasedSource`.
`CompressedSource.DecompressingChannelFactory` `getChannelFactory()`
`Coder<T>` `getDefaultOutputCoder()` Returns the delegate source's default output coder.
`protected boolean` `isSplittable()` Determines whether a single file represented by this source is splittable.
`void` `populateDisplayData(DisplayData.Builder builder)` Register display data for the given transform or component.
`boolean` `producesSortedKeys(PipelineOptions options)` Returns whether the delegate source produces sorted keys.
`static <T> Read.Bounded<T>` `readFromSource(FileBasedSource<T> sourceDelegate, CompressedSource.DecompressingChannelFactory channelFactory)` Creates a `Read` transform that reads from that reads from the underlying `FileBasedSource` `sourceDelegate` after decompressing it with a `CompressedSource.DecompressingChannelFactory`.
`void` `validate()` Validates that the delegate source is a valid source and that the channel factory is not null.
`CompressedSource<T>` `withDecompression(CompressedSource.DecompressingChannelFactory channelFactory)` Return a `CompressedSource` that is like this one but will decompress its underlying file with the given `CompressedSource.DecompressingChannelFactory`.

### Methods inherited from class com.google.cloud.dataflow.sdk.io.FileBasedSource

`createReader, createSourceForSubrange, expandFilePattern, getEstimatedSizeBytes, getFileOrPatternSpec, getMaxEndOffset, getMode, splitIntoBundles, toString`

### Methods inherited from class com.google.cloud.dataflow.sdk.io.OffsetBasedSource

`allowsDynamicSplitting, getBytesPerOffset, getEndOffset, getMinBundleSize, getStartOffset`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Method Detail

#### readFromSource

    public static <T> Read.Bounded<T> readFromSource(FileBasedSource<T> sourceDelegate,
                                                     CompressedSource.DecompressingChannelFactory channelFactory)

Creates a `Read` transform that reads from that reads from the underlying `FileBasedSource` `sourceDelegate` after decompressing it with a `CompressedSource.DecompressingChannelFactory`.

#### from

    public static <T> CompressedSource<T> from(FileBasedSource<T> sourceDelegate)

Creates a `CompressedSource` from an underlying `FileBasedSource`. The type of compression used will be based on the file name extension unless explicitly configured via `withDecompression(com.google.cloud.dataflow.sdk.io.CompressedSource.DecompressingChannelFactory)`.

#### withDecompression

    public CompressedSource<T> withDecompression(CompressedSource.DecompressingChannelFactory channelFactory)

Return a `CompressedSource` that is like this one but will decompress its underlying file with the given `CompressedSource.DecompressingChannelFactory`.

#### validate

    public void validate()

Validates that the delegate source is a valid source and that the channel factory is not null.

Overrides:
  `validate` in class `FileBasedSource<T>`

#### createForSubrangeOfFile

    protected FileBasedSource<T> createForSubrangeOfFile(String fileName,
                                                         long start,
                                                         long end)

Creates a `CompressedSource` for a subrange of a file. Called by superclass to create a source for a single file.

Specified by:
  `createForSubrangeOfFile` in class `FileBasedSource<T>`
Parameters:
  `fileName` - file backing the new `FileBasedSource`.
  `start` - starting byte offset of the new `FileBasedSource`.
  `end` - ending byte offset of the new `FileBasedSource`. May be Long.MAX_VALUE, in which case it will be inferred using `FileBasedSource.getMaxEndOffset(com.google.cloud.dataflow.sdk.options.PipelineOptions)`.

#### isSplittable

    protected final boolean isSplittable()
                                  throws Exception

Determines whether a single file represented by this source is splittable. Returns true if we are using the default decompression factory and and it determines from the requested file name that the file is not compressed.

Overrides:
  `isSplittable` in class `FileBasedSource<T>`
Throws:
  `Exception`

#### createSingleFileReader

    protected final FileBasedSource.FileBasedReader<T> createSingleFileReader(PipelineOptions options)

Creates a `FileBasedReader` to read a single file.

Uses the delegate source to create a single file reader for the delegate source. Utilizes the default decompression channel factory to not wrap the source reader if the file name does not represent a compressed file allowing for splitting of the source.

Specified by:
  `createSingleFileReader` in class `FileBasedSource<T>`

#### producesSortedKeys

    public final boolean producesSortedKeys(PipelineOptions options)
                                     throws Exception

Returns whether the delegate source produces sorted keys.

Specified by:
  `producesSortedKeys` in class `BoundedSource<T>`
Throws:
  `Exception`

#### populateDisplayData

    public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: `Source`
Register display data for the given transform or component.

`populateDisplayData(DisplayData.Builder)` is invoked by Pipeline runners to collect display data via `DisplayData.from(HasDisplayData)`. Implementations may call `super.populateDisplayData(builder)` in order to register display data in the current namespace, but should otherwise use `subcomponent.populateDisplayData(builder)` to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:
  `populateDisplayData` in interface `HasDisplayData`
Overrides:
  `populateDisplayData` in class `FileBasedSource<T>`
Parameters:
  `builder` - The builder to populate with display data.
See Also:
  `HasDisplayData`

#### getDefaultOutputCoder

    public final Coder<T> getDefaultOutputCoder()

Returns the delegate source's default output coder.

Specified by:
  `getDefaultOutputCoder` in class `Source<T>`

#### getChannelFactory

    public final CompressedSource.DecompressingChannelFactory getChannelFactory()