com.google.cloud.dataflow.sdk.runners.worker

Class ConcatReader<T>

java.lang.Object

java.util.Observable

com.google.cloud.dataflow.sdk.util.common.worker.Reader<T>

com.google.cloud.dataflow.sdk.runners.worker.ConcatReader<T>

Type Parameters:

T - Type of the elements read by the Readers.

public class ConcatReader<T>
extends Reader<T>

A Reader that reads elements from a given set of encoded Sources. Creates Readers for sources lazily, i.e. only when elements from the particular Reader are about to be read.

This class does does not cache Readers and creates a new Reader every time a new ReaderIterator has to be created. Because of this, multiple iterators created using the same ConcatReader will not be able to share any state between each other. This design was chosen since keeping a large number of Reader objects alive within a single ConcatReader could be highly memory consuming.

Nested Class Summary

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Reader

Reader.AbstractReaderIterator<T>, Reader.DynamicSplitRequest, Reader.DynamicSplitResult, Reader.DynamicSplitResultWithPosition, Reader.Position, Reader.Progress, Reader.ReaderIterator<T>

Field Summary

Fields

Modifier and Type	Field and Description
static String	SOURCE_NAME

Fields inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Reader

stateSampler, stateSamplerOperationName

Constructor Summary

Constructors

Constructor and Description
ConcatReader(PipelineOptions options, ExecutionContext executionContext, List<com.google.api.services.dataflow.model.Source> sources) Create a ConcatReader using a given list of encoded Sources.

Method Summary

Modifier and Type	Method and Description
Iterator<com.google.api.services.dataflow.model.Source>	getSources()
Reader.ReaderIterator<T>	iterator() Returns a ReaderIterator that allows reading from this source.

Methods inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Reader

notifyElementRead, setStateSamplerAndOperationName, supportsRestart

Methods inherited from class java.util.Observable

addObserver, clearChanged, countObservers, deleteObserver, deleteObservers, hasChanged, notifyObservers, notifyObservers, setChanged

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SOURCE_NAME

public static final String SOURCE_NAME

See Also:

Constant Field Values

Constructor Detail

ConcatReader

public ConcatReader(PipelineOptions options,
                    ExecutionContext executionContext,
                    List<com.google.api.services.dataflow.model.Source> sources)

Create a ConcatReader using a given list of encoded Sources.

Method Detail

getSources

public Iterator<com.google.api.services.dataflow.model.Source> getSources()

iterator

public Reader.ReaderIterator<T> iterator()
                                  throws IOException

Description copied from class: Reader

Returns a ReaderIterator that allows reading from this source.

Specified by:

iterator in class Reader<T>

Throws:

IOException