com.google.cloud.dataflow.sdk.runners

## Class DataflowPipelineRunner

- java.lang.Object
- - com.google.cloud.dataflow.sdk.runners.PipelineRunner<DataflowPipelineJob>
  - - com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner

    public class DataflowPipelineRunner
    extends PipelineRunner<DataflowPipelineJob>

A `PipelineRunner` that executes the operations in the pipeline by first translating them to the Dataflow representation using the `DataflowPipelineTranslator` and then submitting them to a Dataflow service for execution.

### Permissions

When reading from a Dataflow source or writing to a Dataflow sink using `DataflowPipelineRunner`, the Google cloudservices account and the Google compute engine service account of the GCP project running the Dataflow Job will need access to the corresponding source/sink.

Please see Google Cloud Dataflow Security and Permissions for more details.

### Nested Class Summary

Nested Classes
`static class` `DataflowPipelineRunner.BatchBigQueryIONativeReadTranslator` Implements BigQueryIO Read translation for the Dataflow backend.

### Field Summary

Fields
`static String` `BATCH_WORKER_HARNESS_CONTAINER_IMAGE`
`static String` `PROJECT_ID_REGEXP` Project IDs must contain lowercase letters, digits, or dashes.
`static String` `STREAMING_WORKER_HARNESS_CONTAINER_IMAGE`

### Constructor Summary

Constructors
`protected` `DataflowPipelineRunner(DataflowPipelineOptions options)`

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`<OutputT extends POutput,InputT extends PInput> OutputT` `apply(PTransform<InputT,OutputT> transform, InputT input)` Applies the given transform to the input.
`protected static List<String>` `detectClassPathResourcesToStage(ClassLoader classLoader)` Attempts to detect all the resources the class loader has access to.
`static DataflowPipelineRunner` `fromOptions(PipelineOptions options)` Construct a runner from the provided options.
`DataflowPipelineTranslator` `getTranslator()` Returns the DataflowPipelineTranslator associated with this object.
`DataflowPipelineJob` `run(Pipeline pipeline)` Processes the given Pipeline, returning the results.
`void` `setHooks(DataflowPipelineRunnerHooks hooks)` Sets callbacks to invoke during execution see `DataflowPipelineRunnerHooks`.
`String` `toString()`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Field Detail

#### BATCH_WORKER_HARNESS_CONTAINER_IMAGE

    public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE

See Also:
  Constant Field Values

#### STREAMING_WORKER_HARNESS_CONTAINER_IMAGE

    public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE

See Also:
  Constant Field Values

#### PROJECT_ID_REGEXP

    public static final String PROJECT_ID_REGEXP

Project IDs must contain lowercase letters, digits, or dashes. IDs must start with a letter and may not end with a dash. This regex isn't exact - this allows for patterns that would be rejected by the service, but this is sufficient for basic validation of project IDs.

See Also:
  Constant Field Values

### Constructor Detail

#### DataflowPipelineRunner

    protected DataflowPipelineRunner(DataflowPipelineOptions options)

### Method Detail

#### fromOptions

    public static DataflowPipelineRunner fromOptions(PipelineOptions options)

Construct a runner from the provided options.

Parameters:
  `options` - Properties that configure the runner.
Returns:
  The newly created runner.

#### apply

    public <OutputT extends POutput,InputT extends PInput> OutputT apply(PTransform<InputT,OutputT> transform,
                                                                         InputT input)

Applies the given transform to the input. For transforms with customized definitions for the Dataflow pipeline runner, the application is intercepted and modified here.

Overrides:
  `apply` in class `PipelineRunner<DataflowPipelineJob>`

#### run

    public DataflowPipelineJob run(Pipeline pipeline)

Description copied from class: `PipelineRunner`
Processes the given Pipeline, returning the results.

Specified by:
  `run` in class `PipelineRunner<DataflowPipelineJob>`

#### getTranslator

    public DataflowPipelineTranslator getTranslator()

Returns the DataflowPipelineTranslator associated with this object.

#### setHooks

    @Experimental
    public void setHooks(DataflowPipelineRunnerHooks hooks)

Sets callbacks to invoke during execution see `DataflowPipelineRunnerHooks`.

#### toString

    public String toString()

Overrides:
  `toString` in class `Object`

#### detectClassPathResourcesToStage

    protected static List<String> detectClassPathResourcesToStage(ClassLoader classLoader)

Attempts to detect all the resources the class loader has access to. This does not recurse to class loader parents stopping it from pulling in resources from the system class loader.

Parameters:
  `classLoader` - The URLClassLoader to use to detect resources to stage.
Returns:
  A list of absolute paths to the resources the class loader uses.
Throws:
  `IllegalArgumentException` - If either the class loader is not a URLClassLoader or one of the resources the class loader exposes is not a file resource.