public class TrafficMaxLaneFlow extends Object
Concepts: The batch and streaming runners, sliding windows, Google Cloud Pub/Sub topic injection, use of the AvroCoder to encode a custom class, and custom Combine transforms.
This example analyzes traffic sensor data using SlidingWindows. For each window, it finds the lane that had the highest flow recorded, for each sensor station. It writes those max values along with auxiliary info to a BigQuery table.
In batch mode, the pipeline reads traffic sensor data from --inputFile.
In streaming mode, the pipeline reads the data from a Pub/Sub topic. By default, the example will run a separate pipeline to inject the data from the default --inputFile to the Pub/Sub --pubsubTopic. It will make it available for the streaming pipeline to process. You may override the default --inputFile with the file of your choosing. You may also set --inputFile to an empty string, which will disable the automatic Pub/Sub injection, and allow you to use separate tool to control the input to this example. An example code, which publishes traffic sensor data to a Pub/Sub topic, is provided in .
The example is configured to use the default Pub/Sub topic and the default BigQuery table from the example common package (there are no defaults for a general Dataflow pipeline). You can override them by using the --pubsubTopic, --bigQueryDataset, and --bigQueryTable options. If the Pub/Sub topic or the BigQuery table do not exist, the example will try to create them.
The example will try to cancel the pipelines on the signal to terminate the process (CTRL-C) and then exits.
| Modifier and Type | Class and Description |
|---|---|
static class |
TrafficMaxLaneFlow.MaxFlow
A custom 'combine function' used with the Combine.perKey transform.
|
| Constructor and Description |
|---|
TrafficMaxLaneFlow() |
| Modifier and Type | Method and Description |
|---|---|
static void |
main(String[] args)
Sets up and starts streaming pipeline.
|
public static void main(String[] args) throws IOException
IOException - if there is a problem setting up resources