com.google.cloud.dataflow.examples.cookbook

Class CombinePerKeyExamples

java.lang.Object

com.google.cloud.dataflow.examples.cookbook.CombinePerKeyExamples

public class CombinePerKeyExamples
extends Object

An example that reads the public 'Shakespeare' data, and for each word in the dataset that is over a given length, generates a string containing the list of play names in which that word appears, and saves this information to a bigquery table.

Concepts: the Combine.perKey transform, which lets you combine the values in a key-grouped Collection, and how to use an Aggregator to track information in the Monitoring UI.

Note: Before running this example, you must create a BigQuery dataset to contain your output table.

To execute this pipeline locally, specify general pipeline configuration:

   --project=YOUR_PROJECT_ID
 
 

and the BigQuery table for the output:

   --output=YOUR_PROJECT_ID:DATASET_ID.TABLE_ID
 

To execute this pipeline using the Dataflow service, specify pipeline configuration:

   --project=YOUR_PROJECT_ID
   --stagingLocation=gs://<STAGING DIRECTORY>
   --runner=BlockingDataflowPipelineRunner
 
 

and the BigQuery table for the output:

   --output=YOUR_PROJECT_ID:DATASET_ID.TABLE_ID
 

The BigQuery input table defaults to publicdata:samples.shakespeare and can be overridden with --input.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CombinePerKeyExamples.ConcatWords A 'combine function' used with the Combine.perKey transform.

Constructor Summary

Constructors

Constructor and Description
CombinePerKeyExamples()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CombinePerKeyExamples

public CombinePerKeyExamples()

Method Detail

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception