com.google.cloud.dataflow.examples

Class CombinePerKeyExamples

java.lang.Object

com.google.cloud.dataflow.examples.CombinePerKeyExamples

public class CombinePerKeyExamples
extends java.lang.Object

An example that reads the public 'Shakespeare' data, and for each word in the dataset that is over a given length, generates a string containing the list of play names in which that word appears, and saves this information to a bigquery table.

Concepts: the Combine.perKey transform, which lets you combine the values in a key-grouped Collection, and how to use an Aggregator to track information in the Monitoring UI.

Note: Before running this example, you must create a BigQuery dataset to contain your output table.

To execute this pipeline locally, specify general pipeline configuration: --project= and the BigQuery table for the output: --output=:.

To execute this pipeline using the Dataflow service, specify pipeline configuration: --project= --stagingLocation=gs:// --runner=BlockingDataflowPipelineRunner and the BigQuery table for the output: --output=:.

The BigQuery input table defaults to publicdata:samples.shakespeare and can be overridden with --input.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CombinePerKeyExamples.ConcatWords A 'combine function' used with the Combine.perKey transform.

Constructor Summary

Constructors

Constructor and Description
CombinePerKeyExamples()

Method Summary

Methods

Modifier and Type	Method and Description
static void	main(java.lang.String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CombinePerKeyExamples

public CombinePerKeyExamples()

Method Detail

main

public static void main(java.lang.String[] args)
                 throws java.lang.Exception

Throws:

java.lang.Exception