com.google.cloud.dataflow.examples.cookbook

Class DeDupExample

java.lang.Object

com.google.cloud.dataflow.examples.cookbook.DeDupExample

public class DeDupExample
extends Object

This example uses as input Shakespeare's plays as plaintext files, and will remove any duplicate lines across all the files. (The output does not preserve any input order).

Concepts: the RemoveDuplicates transform, and how to wire transforms together. Demonstrates TextIO.Read/ RemoveDuplicates/TextIO.Write.

To execute this pipeline locally, specify general pipeline configuration: --project=YOUR_PROJECT_ID and a local output file or output prefix on GCS: --output=[YOUR_LOCAL_FILE | gs://YOUR_OUTPUT_PREFIX]

To execute this pipeline using the Dataflow service, specify pipeline configuration: --project=YOUR_PROJECT_ID --stagingLocation=gs://YOUR_STAGING_DIRECTORY --runner=BlockingDataflowPipelineRunner and an output prefix on GCS: --output=gs://YOUR_OUTPUT_PREFIX

The input defaults to gs://dataflow-samples/shakespeare/* and can be overridden with --input.

Constructor Summary

Constructors

Constructor and Description
DeDupExample()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DeDupExample

public DeDupExample()

Method Detail

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception