com.google.cloud.dataflow.examples.common

Class DataflowExampleUtils

java.lang.Object

com.google.cloud.dataflow.examples.common.DataflowExampleUtils

public class DataflowExampleUtils
extends Object

The utility class that sets up and tears down external resources, starts the Google Cloud Pub/Sub injector, and cancels the streaming and the injector pipelines once the program terminates.

It is used to run Dataflow examples, such as TrafficMaxLaneFlow and TrafficRoutes.

Constructor Summary

Constructors

Constructor and Description
DataflowExampleUtils(DataflowPipelineOptions options)
DataflowExampleUtils(DataflowPipelineOptions options, boolean isUnbounded) Do resources and runner options setup.

Method Summary

Modifier and Type	Method and Description
void	mockUnboundedSource(String inputFile, PipelineResult result) Start the auxiliary injector pipeline, then wait for this pipeline to finish.
void	runInjectorPipeline(Pipeline injectorPipeline) Runs the provided pipeline to inject data into the PubSubIO input topic.
void	runInjectorPipeline(PTransform<? super PBegin,PCollection<String>> readSource, String topic, String pubsubTimestampTabelKey) Runs a batch pipeline to inject data into the PubSubIO input topic.
void	runInjectorPipeline(String inputFile, String topic) Runs a batch pipeline to inject data into the PubSubIO input topic.
void	setup() Sets up external resources that are required by the example, such as Pub/Sub topics and BigQuery tables.
void	setupBigQueryTable() Sets up the BigQuery table with the given schema.
void	setupPubsub() Sets up the Google Cloud Pub/Sub topic.
void	setupResourcesAndRunner(boolean isUnbounded) Set up external resources, and configure the runner appropriately.
void	setupRunner() Do some runner setup: check that the DirectPipelineRunner is not used in conjunction with streaming, and if streaming is specified, use the DataflowPipelineRunner.
void	startInjectorIfNeeded(String inputFile) If this is an unbounded (streaming) pipeline, and both inputFile and pubsub topic are defined, start an 'injector' pipeline that publishes the contents of the file to the given topic, first creating the topic if necessary.
void	waitToFinish(PipelineResult result) If DataflowPipelineRunner or BlockingDataflowPipelineRunner is used, waits for the pipeline to finish and cancels it (and the injector) before the program exists.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataflowExampleUtils

public DataflowExampleUtils(DataflowPipelineOptions options)

DataflowExampleUtils

public DataflowExampleUtils(DataflowPipelineOptions options,
                            boolean isUnbounded)
                     throws IOException

Do resources and runner options setup.

Throws:

IOException

Method Detail

setup

public void setup()
           throws IOException

Sets up external resources that are required by the example, such as Pub/Sub topics and BigQuery tables.

Throws:

IOException - if there is a problem setting up the resources

setupResourcesAndRunner

public void setupResourcesAndRunner(boolean isUnbounded)
                             throws IOException

Set up external resources, and configure the runner appropriately.

Throws:

IOException

setupPubsub

public void setupPubsub()
                 throws IOException

Sets up the Google Cloud Pub/Sub topic.

If the topic doesn't exist, a new topic with the given name will be created.

Throws:

IOException - if there is a problem setting up the Pub/Sub topic

setupBigQueryTable

public void setupBigQueryTable()
                        throws IOException

Sets up the BigQuery table with the given schema.

If the table already exists, the schema has to match the given one. Otherwise, the example will throw a RuntimeException. If the table doesn't exist, a new table with the given schema will be created.

Throws:

IOException - if there is a problem setting up the BigQuery table

startInjectorIfNeeded

public void startInjectorIfNeeded(String inputFile)

If this is an unbounded (streaming) pipeline, and both inputFile and pubsub topic are defined, start an 'injector' pipeline that publishes the contents of the file to the given topic, first creating the topic if necessary.

setupRunner

public void setupRunner()

Do some runner setup: check that the DirectPipelineRunner is not used in conjunction with streaming, and if streaming is specified, use the DataflowPipelineRunner. Return the streaming flag value.

runInjectorPipeline

public void runInjectorPipeline(String inputFile,
                                String topic)

Runs a batch pipeline to inject data into the PubSubIO input topic.

The injector pipeline will read from the given text file, and inject data into the Google Cloud Pub/Sub topic.

runInjectorPipeline

public void runInjectorPipeline(PTransform<? super PBegin,PCollection<String>> readSource,
                                String topic,
                                String pubsubTimestampTabelKey)

Runs a batch pipeline to inject data into the PubSubIO input topic.

The injector pipeline will read from the given source, and inject data into the Google Cloud Pub/Sub topic.

runInjectorPipeline

public void runInjectorPipeline(Pipeline injectorPipeline)

Runs the provided pipeline to inject data into the PubSubIO input topic.

mockUnboundedSource

public void mockUnboundedSource(String inputFile,
                                PipelineResult result)

Start the auxiliary injector pipeline, then wait for this pipeline to finish.

waitToFinish

public void waitToFinish(PipelineResult result)

If DataflowPipelineRunner or BlockingDataflowPipelineRunner is used, waits for the pipeline to finish and cancels it (and the injector) before the program exists.