com.google.cloud.dataflow.examples.cookbook

## Class DeDupExample

- java.lang.Object
- - com.google.cloud.dataflow.examples.cookbook.DeDupExample

    public class DeDupExample
    extends Object

This example uses as input Shakespeare's plays as plaintext files, and will remove any duplicate lines across all the files. (The output does not preserve any input order).

Concepts: the RemoveDuplicates transform, and how to wire transforms together. Demonstrates `TextIO.Read`/ `RemoveDuplicates`/`TextIO.Write`.

To execute this pipeline locally, specify general pipeline configuration: --project=YOUR_PROJECT_ID and a local output file or output prefix on GCS: --output=[YOUR_LOCAL_FILE | gs://YOUR_OUTPUT_PREFIX]

To execute this pipeline using the Dataflow service, specify pipeline configuration: --project=YOUR_PROJECT_ID --stagingLocation=gs://YOUR_STAGING_DIRECTORY --runner=BlockingDataflowPipelineRunner and an output prefix on GCS: --output=gs://YOUR_OUTPUT_PREFIX

The input defaults to `gs://dataflow-samples/shakespeare/*` and can be overridden with `--input`.

### Constructor Summary

Constructors
Constructor and Description
`DeDupExample()`

### Method Summary

All Methods Static Methods Concrete Methods
`static void` `main(String[] args)`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### DeDupExample

    public DeDupExample()

### Method Detail

#### main

    public static void main(String[] args)
                     throws Exception

Throws:
  `Exception`