com.google.cloud.dataflow.examples.common

Class DataflowExampleUtils

java.lang.Object

com.google.cloud.dataflow.examples.common.DataflowExampleUtils

public class DataflowExampleUtils
extends Object

The utility class that sets up and tears down external resources, starts the Google Cloud Pub/Sub injector, and cancels the streaming and the injector pipelines once the program terminates.

It is used to run Dataflow examples, such as TrafficMaxLaneFlow and TrafficRoutes.

Constructor Summary

Constructors

Constructor and Description
DataflowExampleUtils(DataflowPipelineOptions options)

Method Summary

Methods

Modifier and Type	Method and Description
void	runInjectorPipeline(Pipeline injectorPipeline) Runs the provided injector pipeline for the streaming pipeline.
void	runInjectorPipeline(String inputFile, String topic) Runs the batch injector for the streaming pipeline.
void	setup() Sets up external resources that are required by the example, such as Pub/Sub topics and BigQuery tables.
void	setupBigQueryTable() Sets up the Google Cloud Pub/Sub topic.
void	setupPubsubTopic() Sets up the BigQuery table with the given schema.
void	waitToFinish(PipelineResult result) If DataflowPipelineRunner or BlockingDataflowPipelineRunner is used, waits for the pipeline to finish and cancels it (and the injector) before the program exists.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataflowExampleUtils

public DataflowExampleUtils(DataflowPipelineOptions options)

Method Detail

setup

public void setup()
           throws IOException

Sets up external resources that are required by the example, such as Pub/Sub topics and BigQuery tables.

Throws:

IOException - if there is a problem setting up the resources

setupPubsubTopic

public void setupPubsubTopic()
                      throws IOException

Sets up the BigQuery table with the given schema.

If the table already exists, the schema has to match the given one. Otherwise, the example will throw a RuntimeException. If the table doesn't exist, a new table with the given schema will be created.

Throws:

IOException - if there is a problem setting up the BigQuery table

setupBigQueryTable

public void setupBigQueryTable()
                        throws IOException

Sets up the Google Cloud Pub/Sub topic.

If the topic doesn't exist, a new topic with the given name will be created.

Throws:

IOException - if there is a problem setting up the Pub/Sub topic

runInjectorPipeline

public void runInjectorPipeline(String inputFile,
                       String topic)

Runs the batch injector for the streaming pipeline.

The injector pipeline will read from the given text file, and inject data into the Google Cloud Pub/Sub topic.

runInjectorPipeline

public void runInjectorPipeline(Pipeline injectorPipeline)

Runs the provided injector pipeline for the streaming pipeline.

waitToFinish

public void waitToFinish(PipelineResult result)

If DataflowPipelineRunner or BlockingDataflowPipelineRunner is used, waits for the pipeline to finish and cancels it (and the injector) before the program exists.