com.google.cloud.dataflow.sdk.runners.worker

Class FileBasedReader.FileBasedIterator

java.lang.Object

com.google.cloud.dataflow.sdk.util.common.worker.Reader.AbstractReaderIterator<T>

com.google.cloud.dataflow.sdk.util.common.worker.AbstractBoundedReaderIterator<T>

com.google.cloud.dataflow.sdk.runners.worker.FileBasedReader.FileBasedIterator

All Implemented Interfaces:

Reader.ReaderIterator<T>, AutoCloseable

Enclosing class:

FileBasedReader<T>

protected abstract class FileBasedReader.FileBasedIterator
extends AbstractBoundedReaderIterator<T>

Abstract base class for file-based source iterators.

Field Summary

Fields

Modifier and Type	Field and Description
protected FileBasedReader.DecompressingStreamFactory	compressionStreamFactory
protected ByteArrayOutputStream	nextElement
protected long	offset
protected ProgressTracker<Integer>	progressTracker
protected OffsetRangeTracker	rangeTracker
protected com.google.cloud.dataflow.sdk.runners.worker.CopyableSeekableByteChannel	seeker
protected PushbackInputStream	stream

Method Summary

Modifier and Type	Method and Description
void	close()
Reader.Progress	getProgress() Returns a representation of how far this iterator is through the source.
protected boolean	hasNextImpl()
protected T	nextImpl()
protected abstract ByteArrayOutputStream	readElement() Reads the next element.
Reader.DynamicSplitResult	requestDynamicSplit(Reader.DynamicSplitRequest splitRequest) Attempts to split the input in two parts: the "primary" part and the "residual" part.

Methods inherited from class com.google.cloud.dataflow.sdk.util.common.worker.AbstractBoundedReaderIterator

hasNext, next

Methods inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Reader.AbstractReaderIterator

copy

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

seeker

protected final com.google.cloud.dataflow.sdk.runners.worker.CopyableSeekableByteChannel seeker

stream

protected final PushbackInputStream stream

rangeTracker

protected final OffsetRangeTracker rangeTracker

offset

protected long offset

progressTracker

protected final ProgressTracker<Integer> progressTracker

nextElement

protected ByteArrayOutputStream nextElement

compressionStreamFactory

protected FileBasedReader.DecompressingStreamFactory compressionStreamFactory

Method Detail

readElement

protected abstract ByteArrayOutputStream readElement()
                                              throws IOException

Reads the next element.

Returns:

a ByteArrayOutputStream containing the contents of the element, or null if the end of the stream has been reached.

Throws:

IOException - if an I/O error occurs

hasNextImpl

protected boolean hasNextImpl()
                       throws IOException

Specified by:

hasNextImpl in class AbstractBoundedReaderIterator<T>

Throws:

IOException

nextImpl

protected T nextImpl()
              throws IOException

Specified by:

nextImpl in class AbstractBoundedReaderIterator<T>

Throws:

IOException

getProgress

public Reader.Progress getProgress()

Description copied from interface: Reader.ReaderIterator

Returns a representation of how far this iterator is through the source.

Specified by:

getProgress in interface Reader.ReaderIterator<T>

Overrides:

getProgress in class Reader.AbstractReaderIterator<T>

Returns:

the progress, or null if no progress measure can be provided (implementors are discouraged from throwing UnsupportedOperationException in this case).

requestDynamicSplit

public Reader.DynamicSplitResult requestDynamicSplit(Reader.DynamicSplitRequest splitRequest)

Description copied from interface: Reader.ReaderIterator

Attempts to split the input in two parts: the "primary" part and the "residual" part. The current Reader.ReaderIterator keeps processing the primary part, while the residual part will be processed elsewhere (e.g. perhaps on a different worker).

The primary and residual parts, if concatenated, must represent the same input as the current input of this Reader.ReaderIterator before this call.

The boundary between the primary part and the residual part is specified in a framework-specific way using Reader.DynamicSplitRequest: e.g., if the framework supports the notion of positions, it might be a position at which the input is asked to split itself (which is not necessarily the same position at which it will split itself); it might be an approximate fraction of input, or something else.

Reader.DynamicSplitResult encodes, in a framework-specific way, the information sufficient to construct a description of the resulting primary and residual inputs. For example, it might, again, be a position demarcating these parts, or it might be a pair of fully-specified input descriptions, or something else.

After a successful call to Reader.ReaderIterator.requestDynamicSplit(com.google.cloud.dataflow.sdk.util.common.worker.Reader.DynamicSplitRequest), subsequent calls should be interpreted relative to the new primary.

This call should not affect the range of input represented by the Reader that produced this Reader.ReaderIterator.

Specified by:

requestDynamicSplit in interface Reader.ReaderIterator<T>

Overrides:

requestDynamicSplit in class Reader.AbstractReaderIterator<T>

Returns:

null if the Reader.DynamicSplitRequest cannot be honored (in that case the input represented by this Reader.ReaderIterator stays the same), or a Reader.DynamicSplitResult describing how the input was split into a primary and residual part.

close

public void close()
           throws IOException

Specified by:

close in interface Reader.ReaderIterator<T>

Specified by:

close in interface AutoCloseable

Overrides:

close in class Reader.AbstractReaderIterator<T>

Throws:

IOException