@ParametersAreNonnullByDefault

Package com.google.cloud.dataflow.sdk.runners.worker

Implementation of the harness that runs on each Google Compute Engine instance to coordinate execution of Pipeline code.

See: Description

Interface Summary

Interface	Description
FileBasedReader.DecompressingStreamFactory	Factory interface for creating a decompressing InputStream.
ParDoFnFactory	Creates a ParDoFn from a CloudObject spec.

Class Summary

Class	Description
ApplianceShuffleReader	ApplianceShuffleReader reads chunks of data from a shuffle dataset for a position range.
ApplianceShuffleWriter	ApplianceShuffleWriter writes chunks of data to a shuffle dataset.
AvroByteReader<T>	A source that reads Avro files.
AvroByteSink<T>	A sink that writes Avro files.
AvroReader<T>	A source that reads Avro files.
AvroReaderFactory	Creates an AvroReader from a CloudObject spec.
AvroSink<T>	A sink that writes Avro files.
AvroSinkFactory	Creates an AvroSink from a CloudObject spec.
BigQueryReader	A source that reads a BigQuery table and yields TableRow objects.
BigQueryReaderFactory	Creates a BigQueryReader from a CloudObject spec.
ByteArrayShufflePosition	Represents a position of a GroupingShuffleReader as an opaque array of bytes, encoded in a way such that lexicographic ordering of the bytes is consistent with the inherent ordering of GroupingShuffleReader positions.
ConcatReader<T>	A Reader that reads elements from a given set of encoded Sources.
ConcatReaderFactory	Creates an ConcatReader from a CloudObject spec.
DataflowExecutionContext	Extensions to BatchModeExecutionContext specific to the Dataflow worker.
DataflowOutputCounter	A Dataflow-specific version of ElementCounter, which specifies the object counter name differently as PhysicalElementCount.
DataflowSideInputReader	A simple side input reader that re-reads a side input from its iterable each time it is requested.
DataflowWorker	This is a semi-abstract harness for executing WorkItem tasks in Java workers.
DataflowWorker.WorkUnitClient	Abstract base class describing a client for WorkItem work units.
DataflowWorkerHarness	This is a harness for executing WorkItem tasks in Java workers.
DataflowWorkProgressUpdater	DataflowWorkProgressUpdater implements the WorkProgressUpdater interface for the Cloud Dataflow system.
FileBasedReader<T>	Abstract base class for sources that read from files.
FileBasedReader.FilenameBasedStreamFactory	Factory for creating decompressing input streams based on a filename and a TextIO.CompressionType.
GroupingShuffleRangeTracker	A RangeTracker for positions used by GroupingShuffleReader (ByteArrayShufflePosition).
GroupingShuffleReader<K,V>	A source that reads from a shuffled dataset and yields key-grouped data.
GroupingShuffleReaderFactory	Creates a GroupingShuffleReader from a CloudObject spec.
InMemoryReader<T>	A source that yields a set of precomputed elements.
InMemoryReaderFactory	Creates an InMemoryReader from a CloudObject spec.
MapTaskExecutorFactory	Creates a MapTaskExecutor from a MapTask definition.
MapTaskExecutorFactory.CoderSizeEstimator<T>	Implements PGBKOp.SizeEstimator via Coder.
MapTaskExecutorFactory.ElementByteSizeObservableCoder<T>	Adapts a Coder to the ElementByteSizeObservable interface.
MapTaskExecutorFactory.PairInfo	Implements PGBKOp.PairInfo via KVs.
MapTaskExecutorFactory.ValueCombiner<K,InputT,AccumT,OutputT>	Implements PGBKOp.Combiner via Combine.KeyedCombineFn.
MapTaskExecutorFactory.WindowingCoderGroupingKeyCreator<K>	Implements PGBKOp.GroupingKeyCreator via Coder.
MetricTrackingWindmillServerStub	Wrapper around a WindmillServerStub that tracks metrics for the number of in-flight requests.
OrderedCode	This module provides routines for encoding a sequence of typed entities into a byte array.
ParDoFnBase	A base class providing simple set up, processing, and tear down for a wrapped DoFn.
ParDoFnFactory.DefaultFactory	A factory that dispatches to all known factories in the Dataflow SDK based on the value of CloudObject.getClassName() for the specified DoFn.
PartitioningShuffleReader<K,V>	A source that reads from a key-sharded dataset, and returns KVs without any values grouping.
PartitioningShuffleReaderFactory	Creates a PartitioningShuffleReader from a CloudObject spec.
ReaderFactory	Constructs a Reader from a Dataflow API Source definition.
ShuffleSink<T>	A sink that writes to a shuffle dataset.
ShuffleSinkFactory	Creates a ShuffleSink from a CloudObject spec.
SideInputUtils	Utilities for working with side inputs.
SinkFactory	Constructs a Sink from a Dataflow service protocol Sink definition.
SourceFormatFactory	Creates SourceFormat objects from Source.
SourceOperationExecutor	An executor for a source operation, defined by a SourceOperationRequest.
SourceOperationExecutorFactory	Creates a SourceOperationExecutor from a SourceOperation.
SourceTranslationUtils	Utilities for representing input-specific objects using Dataflow model protos.
StreamingDataflowWorker	Implements a Streaming Dataflow worker.
TextReader<T>	A source that reads text files.
TextReaderFactory	Creates a TextReader from a CloudObject spec.
TextSink<T>	A sink that writes text files.
TextSinkFactory	Creates a TextSink from a CloudObject spec.
UngroupedShuffleReader<T>	A source that reads from a shuffled dataset, without any key grouping.
UngroupedShuffleReaderFactory	Creates an UngroupedShuffleReader from a CloudObject spec.

Exception Summary

Exception	Description
StreamingDataflowWorker.KeyTokenInvalidException	Indicates that the key token was invalid when data was attempted to be fetched.

Package com.google.cloud.dataflow.sdk.runners.worker Description

Implementation of the harness that runs on each Google Compute Engine instance to coordinate execution of Pipeline code.