com.google.cloud.dataflow.sdk.runners.worker

Class TextReader<T>

java.lang.Object

java.util.Observable

com.google.cloud.dataflow.sdk.util.common.worker.Reader<T>

com.google.cloud.dataflow.sdk.runners.worker.FileBasedReader<T>

com.google.cloud.dataflow.sdk.runners.worker.TextReader<T>

Type Parameters:

T - the type of the elements read from the source

public class TextReader<T>
extends FileBasedReader<T>

A source that reads text files.

Nested Class Summary

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.runners.worker.FileBasedReader

FileBasedReader.DecompressingStreamFactory, FileBasedReader.FileBasedIterator, FileBasedReader.FilenameBasedStreamFactory

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Reader

Reader.AbstractReaderIterator<T>, Reader.DynamicSplitRequest, Reader.DynamicSplitResult, Reader.DynamicSplitResultWithPosition, Reader.Position, Reader.Progress, Reader.ReaderIterator<T>

Field Summary

Fields inherited from class com.google.cloud.dataflow.sdk.runners.worker.FileBasedReader

BUF_SIZE, coder, endPosition, filename, startPosition, useDefaultBufferSize

Fields inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Reader

stateSampler, stateSamplerOperationName

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	TextReader(String filename, boolean stripTrailingNewlines, Long startPosition, Long endPosition, Coder<T> coder, boolean useDefaultBufferSize, TextIO.CompressionType compressionType)
	TextReader(String filename, boolean stripTrailingNewlines, Long startPosition, Long endPosition, Coder<T> coder, TextIO.CompressionType compressionType)

Method Summary

Modifier and Type	Method and Description
protected Reader.ReaderIterator<T>	newReaderIteratorForFiles(IOChannelFactory factory, Collection<String> files) Returns a new iterator for elements in the given files.
protected Reader.ReaderIterator<T>	newReaderIteratorForRangeInFile(IOChannelFactory factory, String oneFile, long startPosition, Long endPosition) Returns a new iterator for elements in the given range in the given file.

Methods inherited from class com.google.cloud.dataflow.sdk.runners.worker.FileBasedReader

iterator

Methods inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Reader

notifyElementRead, setStateSamplerAndOperationName, supportsRestart

Methods inherited from class java.util.Observable

addObserver, clearChanged, countObservers, deleteObserver, deleteObservers, hasChanged, notifyObservers, notifyObservers, setChanged

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TextReader

public TextReader(String filename,
                  boolean stripTrailingNewlines,
                  @Nullable
                  Long startPosition,
                  @Nullable
                  Long endPosition,
                  Coder<T> coder,
                  TextIO.CompressionType compressionType)

TextReader

protected TextReader(String filename,
                     boolean stripTrailingNewlines,
                     @Nullable
                     Long startPosition,
                     @Nullable
                     Long endPosition,
                     Coder<T> coder,
                     boolean useDefaultBufferSize,
                     TextIO.CompressionType compressionType)

Method Detail

newReaderIteratorForRangeInFile

protected Reader.ReaderIterator<T> newReaderIteratorForRangeInFile(IOChannelFactory factory,
                                                                   String oneFile,
                                                                   long startPosition,
                                                                   @Nullable
                                                                   Long endPosition)
                                                            throws IOException

Description copied from class: FileBasedReader

Returns a new iterator for elements in the given range in the given file. If the range starts in the middle an element, this element is skipped as it is considered part of the previous range; if the last element that starts in the range finishes beyond the end position, it is still considered part of this range. In other words, the start position and the end position are "rounded up" to element boundaries.

Specified by:

newReaderIteratorForRangeInFile in class FileBasedReader<T>

endPosition - offset of the end position; null means end-of-file

Throws:

IOException

newReaderIteratorForFiles

protected Reader.ReaderIterator<T> newReaderIteratorForFiles(IOChannelFactory factory,
                                                             Collection<String> files)
                                                      throws IOException

Description copied from class: FileBasedReader

Returns a new iterator for elements in the given files. Caller must ensure that the file collection is not empty.

Specified by:

newReaderIteratorForFiles in class FileBasedReader<T>

Throws:

IOException