com.google.cloud.dataflow.sdk.runners

Class BlockingDataflowPipelineRunner

java.lang.Object

com.google.cloud.dataflow.sdk.runners.PipelineRunner<DataflowPipelineJob>

com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner

public class BlockingDataflowPipelineRunner
extends PipelineRunner<DataflowPipelineJob>

A PipelineRunner that's like DataflowPipelineRunner but that waits for the launched job to finish.

Prints out job status updates and console messages while it waits.

Returns the final job state, or throws an exception if the job fails or cannot be monitored.

Permissions

When reading from a Dataflow source or writing to a Dataflow sink using BlockingDataflowPipelineRunner, the Google cloudservices account and the Google compute engine service account of the GCP project running the Dataflow Job will need access to the corresponding source/sink.

Please see Google Cloud Dataflow Security and Permissions for more details.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BlockingDataflowPipelineRunner(DataflowPipelineRunner internalRunner, BlockingDataflowPipelineOptions options)

Method Summary

Modifier and Type	Method and Description
<OutputT extends POutput,InputT extends PInput> OutputT	apply(PTransform<InputT,OutputT> transform, InputT input) Applies a transform to the given input, returning the output.
static BlockingDataflowPipelineRunner	fromOptions(PipelineOptions options) Constructs a runner from the provided options.
DataflowPipelineJob	run(Pipeline p) Processes the given Pipeline, returning the results.
void	setHooks(DataflowPipelineRunnerHooks hooks) Sets callbacks to invoke during execution see DataflowPipelineRunnerHooks.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

BlockingDataflowPipelineRunner

protected BlockingDataflowPipelineRunner(DataflowPipelineRunner internalRunner,
                                         BlockingDataflowPipelineOptions options)

Method Detail

fromOptions

public static BlockingDataflowPipelineRunner fromOptions(PipelineOptions options)

Constructs a runner from the provided options.

run

public DataflowPipelineJob run(Pipeline p)

Processes the given Pipeline, returning the results.

Specified by:

run in class PipelineRunner<DataflowPipelineJob>

Throws:

DataflowJobExecutionException - if there is an exception during job execution.

DataflowServiceException - if there is an exception retrieving information about the job.

apply

public <OutputT extends POutput,InputT extends PInput> OutputT apply(PTransform<InputT,OutputT> transform,
                                                                     InputT input)

Description copied from class: PipelineRunner

Applies a transform to the given input, returning the output.

The default implementation calls PTransform.apply(input), but can be overridden to customize behavior for a particular runner.

Overrides:

apply in class PipelineRunner<DataflowPipelineJob>

setHooks

@Experimental
public void setHooks(DataflowPipelineRunnerHooks hooks)

Sets callbacks to invoke during execution see DataflowPipelineRunnerHooks.

toString

public String toString()

Overrides:

toString in class Object