com.google.cloud.dataflow.sdk.runners

Class DirectPipelineRunner

java.lang.Object

com.google.cloud.dataflow.sdk.runners.PipelineRunner<DirectPipelineRunner.EvaluationResults>

com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner

public class DirectPipelineRunner
extends PipelineRunner<DirectPipelineRunner.EvaluationResults>

Executes the operations in the pipeline directly, in this process, without any optimization. Useful for small local execution and tests.

Throws an exception from run(com.google.cloud.dataflow.sdk.Pipeline) if execution fails.

Permissions

When reading from a Dataflow source or writing to a Dataflow sink using DirectPipelineRunner, the Cloud Platform account that you configured with the gcloud executable will need access to the corresponding source/sink.

Please see Google Cloud Dataflow Security and Permissions for more details.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	DirectPipelineRunner.EvaluationContext The interface provided to registered callbacks for interacting with the DirectPipelineRunner, including reading and writing the values of PCollections and PCollectionViews.
static interface	DirectPipelineRunner.EvaluationResults The interface provided to registered callbacks for interacting with the DirectPipelineRunner, including reading and writing the values of PCollections and PCollectionViews.
static class	DirectPipelineRunner.TestCombineDoFn<K,InputT,AccumT,OutputT> The implementation may split the Combine.KeyedCombineFn into ADD, MERGE and EXTRACT phases ( see com.google.cloud.dataflow.sdk.runners.worker.CombineValuesFn).
static interface	DirectPipelineRunner.TransformEvaluator<TransformT extends PTransform> An evaluator of a PTransform.
static class	DirectPipelineRunner.ValueWithMetadata<V> An immutable (value, timestamp) pair, along with other metadata necessary for the implementation of DirectPipelineRunner.

Method Summary

Modifier and Type	Method and Description
<OutputT extends POutput,InputT extends PInput> OutputT	apply(PTransform<InputT,OutputT> transform, InputT input) Applies a transform to the given input, returning the output.
static DirectPipelineRunner	createForTest() Constructs a runner with default properties for testing.
static DirectPipelineRunner	fromOptions(PipelineOptions options) Constructs a DirectPipelineRunner from the given options.
DirectPipelineOptions	getPipelineOptions()
<TransformT extends PTransform<?,?>> DirectPipelineRunner.TransformEvaluator<TransformT>	getTransformEvaluator(Class<TransformT> transformClass) Returns the TransformEvaluator to use for instances of the specified PTransform class, or null if none registered.
static <TransformT extends PTransform<?,?>> void	registerDefaultTransformEvaluator(Class<TransformT> transformClass, DirectPipelineRunner.TransformEvaluator<? super TransformT> transformEvaluator) Records that instances of the specified PTransform class should be evaluated by default by the corresponding TransformEvaluator.
<TransformT extends PTransform<?,?>> void	registerTransformEvaluator(Class<TransformT> transformClass, DirectPipelineRunner.TransformEvaluator<TransformT> transformEvaluator) Records that instances of the specified PTransform class should be evaluated by the corresponding TransformEvaluator.
DirectPipelineRunner.EvaluationResults	run(Pipeline pipeline) Processes the given Pipeline, returning the results.
String	toString()
DirectPipelineRunner	withEncodabilityTesting(boolean enable) Enable runtime testing to verify that all values can be encoded.
DirectPipelineRunner	withSerializabilityTesting(boolean enable) Enable runtime testing to verify that all functions and Coder instances can be serialized.
DirectPipelineRunner	withUnorderednessTesting(boolean enable) Enable runtime testing to verify that functions do not depend on order of the elements.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

registerDefaultTransformEvaluator

public static <TransformT extends PTransform<?,?>> void registerDefaultTransformEvaluator(Class<TransformT> transformClass,
                                                                                          DirectPipelineRunner.TransformEvaluator<? super TransformT> transformEvaluator)

Records that instances of the specified PTransform class should be evaluated by default by the corresponding TransformEvaluator.

registerTransformEvaluator

public <TransformT extends PTransform<?,?>> void registerTransformEvaluator(Class<TransformT> transformClass,
                                                                            DirectPipelineRunner.TransformEvaluator<TransformT> transformEvaluator)

Records that instances of the specified PTransform class should be evaluated by the corresponding TransformEvaluator. Overrides any bindings specified by registerDefaultTransformEvaluator(java.lang.Class<TransformT>, com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.TransformEvaluator<? super TransformT>).

getTransformEvaluator

public <TransformT extends PTransform<?,?>> DirectPipelineRunner.TransformEvaluator<TransformT> getTransformEvaluator(Class<TransformT> transformClass)

Returns the TransformEvaluator to use for instances of the specified PTransform class, or null if none registered.

fromOptions

public static DirectPipelineRunner fromOptions(PipelineOptions options)

Constructs a DirectPipelineRunner from the given options.

createForTest

public static DirectPipelineRunner createForTest()

Constructs a runner with default properties for testing.

Returns:

The newly created runner.

withSerializabilityTesting

public DirectPipelineRunner withSerializabilityTesting(boolean enable)

Enable runtime testing to verify that all functions and Coder instances can be serialized.

Enabled by default.

This method modifies the DirectPipelineRunner instance and returns itself.

withEncodabilityTesting

public DirectPipelineRunner withEncodabilityTesting(boolean enable)

Enable runtime testing to verify that all values can be encoded.

Enabled by default.

This method modifies the DirectPipelineRunner instance and returns itself.

withUnorderednessTesting

public DirectPipelineRunner withUnorderednessTesting(boolean enable)

Enable runtime testing to verify that functions do not depend on order of the elements.

This is accomplished by randomizing the order of elements.

Enabled by default.

This method modifies the DirectPipelineRunner instance and returns itself.

apply

public <OutputT extends POutput,InputT extends PInput> OutputT apply(PTransform<InputT,OutputT> transform,
                                                                     InputT input)

Description copied from class: PipelineRunner

Applies a transform to the given input, returning the output.

The default implementation calls PTransform.apply(input), but can be overridden to customize behavior for a particular runner.

Overrides:

apply in class PipelineRunner<DirectPipelineRunner.EvaluationResults>

run

public DirectPipelineRunner.EvaluationResults run(Pipeline pipeline)

Description copied from class: PipelineRunner

Processes the given Pipeline, returning the results.

Specified by:

run in class PipelineRunner<DirectPipelineRunner.EvaluationResults>

getPipelineOptions

public DirectPipelineOptions getPipelineOptions()

toString

public String toString()

Overrides:

toString in class Object