com.google.cloud.dataflow.sdk.runners

Class DataflowPipelineRunner

java.lang.Object

com.google.cloud.dataflow.sdk.runners.PipelineRunner<DataflowPipelineJob>

com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner

public class DataflowPipelineRunner
extends PipelineRunner<DataflowPipelineJob>

A PipelineRunner that executes the operations in the pipeline by first translating them to the Dataflow representation using the DataflowPipelineTranslator and then submitting them to a Dataflow service for execution.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DataflowPipelineRunner.StreamingPubsubIOWrite<T> Specialized implementation for PubsubIO.Write for the Dataflow runner in streaming mode.

Method Summary

Modifier and Type	Method and Description
<OutputT extends POutput,InputT extends PInput> OutputT	apply(PTransform<InputT,OutputT> transform, InputT input) Applies the given transform to the input.
protected static List<String>	detectClassPathResourcesToStage(ClassLoader classLoader) Attempts to detect all the resources the class loader has access to.
static DataflowPipelineRunner	fromOptions(PipelineOptions options) Construct a runner from the provided options.
DataflowPipelineTranslator	getTranslator() Returns the DataflowPipelineTranslator associated with this object.
DataflowPipelineJob	run(Pipeline pipeline) Processes the given Pipeline, returning the results.
void	setHooks(DataflowPipelineRunnerHooks hooks) Sets callbacks to invoke during execution see DataflowPipelineRunnerHooks.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

fromOptions

public static DataflowPipelineRunner fromOptions(PipelineOptions options)

Construct a runner from the provided options.

Parameters:

options - Properties that configure the runner.

Returns:

The newly created runner.

apply

public <OutputT extends POutput,InputT extends PInput> OutputT apply(PTransform<InputT,OutputT> transform,
                                                                     InputT input)

Applies the given transform to the input. For transforms with customized definitions for the Dataflow pipeline runner, the application is intercepted and modified here.

Overrides:

apply in class PipelineRunner<DataflowPipelineJob>

run

public DataflowPipelineJob run(Pipeline pipeline)

Description copied from class: PipelineRunner

Processes the given Pipeline, returning the results.

Specified by:

run in class PipelineRunner<DataflowPipelineJob>

getTranslator

public DataflowPipelineTranslator getTranslator()

Returns the DataflowPipelineTranslator associated with this object.

setHooks

@Experimental
public void setHooks(DataflowPipelineRunnerHooks hooks)

Sets callbacks to invoke during execution see DataflowPipelineRunnerHooks.

toString

public String toString()

Overrides:

toString in class Object

detectClassPathResourcesToStage

protected static List<String> detectClassPathResourcesToStage(ClassLoader classLoader)

Attempts to detect all the resources the class loader has access to. This does not recurse to class loader parents stopping it from pulling in resources from the system class loader.

Parameters:

classLoader - The URLClassLoader to use to detect resources to stage.

Returns:

A list of absolute paths to the resources the class loader uses.

Throws:

IllegalArgumentException - If either the class loader is not a URLClassLoader or one of the resources the class loader exposes is not a file resource.