com.google.cloud.dataflow.sdk.io

Class FileBasedSource<T>

java.lang.Object

com.google.cloud.dataflow.sdk.io.Source<T>

com.google.cloud.dataflow.sdk.io.ByteOffsetBasedSource<T>

com.google.cloud.dataflow.sdk.io.FileBasedSource<T>

Type Parameters:

T - Type of records represented by the source.

All Implemented Interfaces:

java.io.Serializable

public abstract class FileBasedSource<T>
extends ByteOffsetBasedSource<T>

A common base class for all file-based Sources. Extend this class to implement your own file-based custom source.

A file-based Source is a Source backed by a file pattern defined as a Java glob, a single file, or a offset range for a single file. See ByteOffsetBasedSource for semantics of offset ranges.

This source stores a String that is an IOChannelFactory specification for a file or file pattern. There should be an IOChannelFactory defined for the file specification provided. Please refer to IOChannelUtils and IOChannelFactory for more information on this.

In addition to the methods left abstract from Source, subclasses must implement methods to create a sub-source and a reader for a range of a single file - createForSubrangeOfFile(java.lang.String, long, long) and createSingleFileReader(com.google.cloud.dataflow.sdk.options.PipelineOptions, com.google.cloud.dataflow.sdk.coders.Coder<T>, com.google.cloud.dataflow.sdk.util.ExecutionContext).

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileBasedSource.FileBasedReader<T> A reader that implements code common to readers of FileBasedSources.
static class	FileBasedSource.Mode A given FileBasedSource represents a file resource of one of these types.

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.ByteOffsetBasedSource

ByteOffsetBasedSource.ByteOffsetBasedReader<T>

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

Source.Reader<T>

Constructor Summary

Constructors

Constructor and Description
FileBasedSource(boolean isFilePattern, java.lang.String fileOrPatternSpec, long minBundleSize, long startOffset, long endOffset) Create a FileBasedSource based on a file or a file pattern specification.

Method Summary

Methods

Modifier and Type	Method and Description
protected Source.Reader<T>	createBasicReader(PipelineOptions options, Coder<T> coder, com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext) Creates a basic (non-windowed) reader for this source.
abstract FileBasedSource<T>	createForSubrangeOfFile(java.lang.String fileName, long start, long end) Creates and returns a new FileBasedSource of the same type as the current FileBasedSource backed by a given file and an offset range.
abstract FileBasedSource.FileBasedReader<T>	createSingleFileReader(PipelineOptions options, Coder<T> coder, com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext) Creates and returns an instance of a FileBasedReader implementation for the current source assuming the source represents a single file.
FileBasedSource<T>	createSourceForSubrange(long start, long end) Returns a ByteOffsetBasedSource for a subrange of the current source.
long	getEstimatedSizeBytes(PipelineOptions options) An estimate of the total size (in bytes) of the data that would be read from this source.
java.lang.String	getFileOrPatternSpec()
long	getMaxEndOffset(PipelineOptions options) Returns the exact ending offset of the current source.
FileBasedSource.Mode	getMode()
java.util.List<? extends FileBasedSource<T>>	splitIntoBundles(long desiredBundleSizeBytes, PipelineOptions options) Splits the source into bundles.
java.lang.String	toString()
void	validate() Checks that this source is valid, before it can be used in a pipeline.

Methods inherited from class com.google.cloud.dataflow.sdk.io.ByteOffsetBasedSource

getEndOffset, getMinBundleSize, getStartOffset

Methods inherited from class com.google.cloud.dataflow.sdk.io.Source

createWindowedReader, getDefaultOutputCoder, producesSortedKeys

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

FileBasedSource

public FileBasedSource(boolean isFilePattern,
               java.lang.String fileOrPatternSpec,
               long minBundleSize,
               long startOffset,
               long endOffset)

Create a FileBasedSource based on a file or a file pattern specification.

See ByteOffsetBasedSource for detailed descriptions of minBundleSize, startOffset, and endOffset.

Parameters:

isFilePattern - if true provided fileOrPatternSpec may be a file pattern and FileBasedSource will try to expand the file pattern, if false provided fileOrPatternSpec will be considered a single file and will be used verbatim.

fileOrPatternSpec - IOChannelFactory specification of file or file pattern represented by the FileBasedSource.

minBundleSize - minimum bundle size in bytes.

startOffset - starting byte offset.

endOffset - ending byte offset. If the specified value >= #getMaxEndOffset() it implies #getMaxEndOffSet().

Method Detail

getFileOrPatternSpec

public final java.lang.String getFileOrPatternSpec()

getMode

public final FileBasedSource.Mode getMode()

createSourceForSubrange

public final FileBasedSource<T> createSourceForSubrange(long start,
                                         long end)

Description copied from class: ByteOffsetBasedSource

Returns a ByteOffsetBasedSource for a subrange of the current source. [start, end) will be within the range [startOffset, endOffset] of the current source.

Specified by:

createSourceForSubrange in class ByteOffsetBasedSource<T>

createForSubrangeOfFile

public abstract FileBasedSource<T> createForSubrangeOfFile(java.lang.String fileName,
                                         long start,
                                         long end)

Creates and returns a new FileBasedSource of the same type as the current FileBasedSource backed by a given file and an offset range. When current source is being split, this method is used to generate new sub-sources. When creating the source subclasses must call the constructor of FileBasedSource with exactly the same start and end values passed here.

Parameters:

fileName - file backing the new FileBasedSource.

start - starting byte offset of the new FileBasedSource.

end - ending byte offset of the new FileBasedSource. May be Long.MAX_VALUE, in which case it will be inferred using getMaxEndOffset(com.google.cloud.dataflow.sdk.options.PipelineOptions).

createSingleFileReader

public abstract FileBasedSource.FileBasedReader<T> createSingleFileReader(PipelineOptions options,
                                                        Coder<T> coder,
                                                        com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext)

Creates and returns an instance of a FileBasedReader implementation for the current source assuming the source represents a single file. File patterns will be handled by FileBasedSource implementation automatically.

getEstimatedSizeBytes

public final long getEstimatedSizeBytes(PipelineOptions options)
                                 throws java.lang.Exception

Description copied from class: Source

An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before any decompression or other processing done by the reader.

Specified by:

getEstimatedSizeBytes in class Source<T>

Throws:

java.lang.Exception

splitIntoBundles

public final java.util.List<? extends FileBasedSource<T>> splitIntoBundles(long desiredBundleSizeBytes,
                                                            PipelineOptions options)
                                                                    throws java.lang.Exception

Description copied from class: Source

Splits the source into bundles.

PipelineOptions can be used to get information such as credentials for accessing an external storage.

Overrides:

splitIntoBundles in class ByteOffsetBasedSource<T>

Throws:

java.lang.Exception

createBasicReader

protected final Source.Reader<T> createBasicReader(PipelineOptions options,
                                 Coder<T> coder,
                                 com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext)
                                            throws java.io.IOException

Description copied from class: Source

Creates a basic (non-windowed) reader for this source. If you override this method, each value returned by this reader will be wrapped into the global window.

Overrides:

createBasicReader in class Source<T>

Throws:

java.io.IOException

toString

public java.lang.String toString()

Overrides:

toString in class ByteOffsetBasedSource<T>

validate

public void validate()

Description copied from class: Source

Checks that this source is valid, before it can be used in a pipeline.

It is recommended to use Preconditions for implementing this method.

Overrides:

validate in class ByteOffsetBasedSource<T>

getMaxEndOffset

public final long getMaxEndOffset(PipelineOptions options)
                           throws java.lang.Exception

Description copied from class: ByteOffsetBasedSource

Returns the exact ending offset of the current source. This will be used if the source was constructed with an endOffset value Long.MAX_VALUE.

Specified by:

getMaxEndOffset in class ByteOffsetBasedSource<T>

Throws:

java.lang.Exception