com.google.cloud.dataflow.sdk.util.common.worker

Interface Reader.ReaderIterator<T>

All Superinterfaces:

AutoCloseable

All Known Implementing Classes:

AbstractBoundedReaderIterator, FileBasedReader.FileBasedIterator, Reader.AbstractReaderIterator

Enclosing class:

Reader<T>

public static interface Reader.ReaderIterator<T>
extends AutoCloseable

A stateful iterator over the data in a Reader.

Partially thread-safe: methods hasNext(), next(), close(), getProgress() are called serially, but requestDynamicSplit(com.google.cloud.dataflow.sdk.util.common.worker.Reader.DynamicSplitRequest) can be called asynchronously to those. There will not be multiple concurrent calls to requestDynamicSplit(com.google.cloud.dataflow.sdk.util.common.worker.Reader.DynamicSplitRequest)).

Method Summary

Modifier and Type	Method and Description
void	close()
Reader.ReaderIterator<T>	copy() Copies the current ReaderIterator.
Reader.Progress	getProgress() Returns a representation of how far this iterator is through the source.
boolean	hasNext() Returns whether the source has any more elements.
T	next() Returns the next element.
Reader.DynamicSplitResult	requestDynamicSplit(Reader.DynamicSplitRequest request) Attempts to split the input in two parts: the "primary" part and the "residual" part.

Method Detail

hasNext

boolean hasNext()
         throws IOException

Returns whether the source has any more elements. Some sources, such as GroupingShuffleReader, invalidate the return value of the previous next() call during the call to hasNext().

Throws:

IOException

next

T next()
throws IOException,
       NoSuchElementException

Returns the next element.

Throws:

IOException - if attempting to access an element involves IO that fails

NoSuchElementException - if there are no more elements

copy

Reader.ReaderIterator<T> copy()
                       throws IOException

Copies the current ReaderIterator.

Throws:

UnsupportedOperationException - if the particular implementation does not support copy

IOException - if copying the iterator involves IO that fails

close

void close()
    throws IOException

Specified by:

close in interface AutoCloseable

Throws:

IOException

getProgress

Reader.Progress getProgress()

Returns a representation of how far this iterator is through the source.

Returns:

the progress, or null if no progress measure can be provided (implementors are discouraged from throwing UnsupportedOperationException in this case).

requestDynamicSplit

Reader.DynamicSplitResult requestDynamicSplit(Reader.DynamicSplitRequest request)

Attempts to split the input in two parts: the "primary" part and the "residual" part. The current Reader.ReaderIterator keeps processing the primary part, while the residual part will be processed elsewhere (e.g. perhaps on a different worker).

The primary and residual parts, if concatenated, must represent the same input as the current input of this Reader.ReaderIterator before this call.

The boundary between the primary part and the residual part is specified in a framework-specific way using Reader.DynamicSplitRequest: e.g., if the framework supports the notion of positions, it might be a position at which the input is asked to split itself (which is not necessarily the same position at which it will split itself); it might be an approximate fraction of input, or something else.

Reader.DynamicSplitResult encodes, in a framework-specific way, the information sufficient to construct a description of the resulting primary and residual inputs. For example, it might, again, be a position demarcating these parts, or it might be a pair of fully-specified input descriptions, or something else.

After a successful call to requestDynamicSplit(com.google.cloud.dataflow.sdk.util.common.worker.Reader.DynamicSplitRequest), subsequent calls should be interpreted relative to the new primary.

This call should not affect the range of input represented by the Reader that produced this Reader.ReaderIterator.

Returns:

null if the Reader.DynamicSplitRequest cannot be honored (in that case the input represented by this Reader.ReaderIterator stays the same), or a Reader.DynamicSplitResult describing how the input was split into a primary and residual part.