Package com.google.cloud.dataflow.sdk.runners

Defines runners for executing Pipelines in different modes, including DirectPipelineRunner and DataflowPipelineRunner.

See: Description

Interface Summary

Interface	Description
DataflowPipelineTranslator.TransformTranslator<PT extends PTransform>	A translator of a PTransform.
DataflowPipelineTranslator.TranslationContext	The interface provided to registered callbacks for interacting with the DataflowPipelineRunner, including reading and writing the values of PCollections and side inputs (PCollectionViews).
DirectPipelineRunner.EvaluationContext	The interface provided to registered callbacks for interacting with the DirectPipelineRunner, including reading and writing the values of PCollections and PCollectionViews.
DirectPipelineRunner.EvaluationResults	The interface provided to registered callbacks for interacting with the DirectPipelineRunner, including reading and writing the values of PCollections and PCollectionViews.
DirectPipelineRunner.TransformEvaluator<PT extends PTransform>	An evaluator of a PTransform.
PipelineRunnerRegistrar	PipelineRunner creators have the ability to automatically have their PipelineRunner registered with this SDK by creating a ServiceLoader entry and a concrete implementation of this interface.

Class Summary

Class	Description
BlockingDataflowPipelineRunner	A PipelineRunner that's like DataflowPipelineRunner but that waits for the launched job to finish.
BlockingDataflowPipelineRunner.PipelineJobState	Holds the status of a run request.
DataflowPipeline	A DataflowPipeline, which returns a DataflowPipelineJob subclass of PipelineResult from Pipeline.run().
DataflowPipelineJob	A DataflowPipelineJob represents a job submitted to Dataflow using DataflowPipelineRunner.
DataflowPipelineRegistrar	Contains the PipelineOptionsRegistrar and PipelineRunnerRegistrar for the DataflowPipeline.
DataflowPipelineRegistrar.Options	Register the DataflowPipelineOptions and BlockingDataflowPipelineOptions.
DataflowPipelineRegistrar.Runner	Register the DataflowPipelineRunner and BlockingDataflowPipelineRunner.
DataflowPipelineRunner	A PipelineRunner that executes the operations in the pipeline by first translating them to the Dataflow representation using the DataflowPipelineTranslator and then submitting them to a Dataflow service for execution.
DataflowPipelineRunnerHooks	An instance of this class can be passed to the DataflowPipeline runner to add user defined hooks to be invoked at various times during pipeline execution.
DataflowPipelineTranslator	DataflowPipelineTranslator knows how to translate Pipeline objects into Dataflow API Jobs.
DirectPipeline	A DirectPipeline, which returns a DirectPipelineRunner.EvaluationResults subclass of PipelineResult from Pipeline.run().
DirectPipelineRegistrar	Contains the PipelineOptionsRegistrar and PipelineRunnerRegistrar for the DirectPipeline.
DirectPipelineRegistrar.Options	Register the DirectPipelineRunner.
DirectPipelineRegistrar.Runner	Register the DirectPipelineOptions.
DirectPipelineRunner	Executes the operations in the pipeline directly, in this process, without any optimization.
DirectPipelineRunner.TestCombineDoFn<K,VI,VA,VO>	The implementation may split the KeyedCombineFn into ADD, MERGE and EXTRACT phases (see CombineValuesFn).
DirectPipelineRunner.ValueWithMetadata<V>	An immutable (value, timestamp) pair, along with other metadata necessary for the implementation of DirectPipelineRunner.
PipelineRunner<Results extends PipelineResult>	A PipelineRunner can execute, translate, or otherwise process a Pipeline.
RecordingPipelineVisitor	Provides a simple PipelineVisitor which records the transformation tree.
TransformHierarchy	Captures information about a collection of transformations and their associated PValues.
TransformTreeNode	Provides internal tracking of transform relationships with helper methods for initialization and ordered visitation.

Package com.google.cloud.dataflow.sdk.runners Description

Defines runners for executing Pipelines in different modes, including DirectPipelineRunner and DataflowPipelineRunner.

DirectPipelineRunner executes a Pipeline locally, without contacting the Dataflow service. DataflowPipelineRunner submits a Pipeline to the Dataflow service, which executes it on Dataflow-managed Compute Engine instances. DataflowPipelineRunner returns as soon as the Pipeline has been submitted. Use BlockingDataflowPipelineRunner to have execution updates printed to the console.

The runner is specified as part PipelineOptions.