This example uses as input Shakespeare's plays as plaintext files, and will remove any
duplicate lines across all the files. (The output does not preserve any input order).
Concepts: the RemoveDuplicates transform, and how to wire transforms together.
Demonstrates TextIO.Read/RemoveDuplicates/TextIO.Write.
To execute this pipeline locally, specify general pipeline configuration:
--project=
and a local output file or output prefix on GCS:
--output=[ | gs://