com.google.cloud.dataflow.examples

## Class DebuggingWordCount

- java.lang.Object
- - com.google.cloud.dataflow.examples.DebuggingWordCount

    public class DebuggingWordCount
    extends Object

An example that verifies word counts in Shakespeare and includes Dataflow best practices.

This class, `DebuggingWordCount`, is the third in a series of four successively more detailed 'word count' examples. You may first want to take a look at `MinimalWordCount` and `WordCount`. After you've looked at this example, then see the `WindowedWordCount` pipeline, for introduction of additional concepts.

Basic concepts, also in the MinimalWordCount and WordCount examples: Reading text files; counting a PCollection; executing a Pipeline both locally and using the Dataflow service; defining DoFns.

New Concepts:

       1. Logging to Cloud Logging
       2. Controlling Dataflow worker log levels
       3. Creating a custom aggregator
       4. Testing your Pipeline via DataflowAssert

To execute this pipeline locally, specify general pipeline configuration:

       --project=YOUR_PROJECT_ID

To execute this pipeline using the Dataflow service and the additional logging discussed below, specify pipeline configuration:

       --project=YOUR_PROJECT_ID
       --stagingLocation=gs://YOUR_STAGING_DIRECTORY
       --runner=BlockingDataflowPipelineRunner
       --workerLogLevelOverrides={"com.google.cloud.dataflow.examples":"DEBUG"}

Note that when you run via `mvn exec`, you may need to escape the quotations as appropriate for your shell. For example, in `bash`:

     mvn compile exec:java ... \
       -Dexec.args="... \
         --workerLogLevelOverrides={\\\"com.google.cloud.dataflow.examples\\\":\\\"DEBUG\\\"}"

Concept #2: Dataflow workers which execute user code are configured to log to Cloud Logging by default at "INFO" log level and higher. One may override log levels for specific logging namespaces by specifying:

       --workerLogLevelOverrides={"Name1":"Level1","Name2":"Level2",...}

For example, by specifying:

       --workerLogLevelOverrides={"com.google.cloud.dataflow.examples":"DEBUG"}

when executing this pipeline using the Dataflow service, Cloud Logging would contain only "DEBUG" or higher level logs for the `com.google.cloud.dataflow.examples` package in addition to the default "INFO" or higher level logs. In addition, the default Dataflow worker logging configuration can be overridden by specifying `--defaultWorkerLogLevel=<one of TRACE, DEBUG, INFO, WARN, ERROR>`. For example, by specifying `--defaultWorkerLogLevel=DEBUG` when executing this pipeline with the Dataflow service, Cloud Logging would contain all "DEBUG" or higher level logs. Note that changing the default worker log level to TRACE or DEBUG will significantly increase the amount of logs output.

The input file defaults to `gs://dataflow-samples/shakespeare/kinglear.txt` and can be overridden with `--inputFile`.

### Nested Class Summary

Nested Classes
`static class` `DebuggingWordCount.FilterTextFn` A DoFn that filters for a specific key based upon a regular expression.
`static interface` `DebuggingWordCount.WordCountOptions` Options supported by `DebuggingWordCount`.

### Constructor Summary

Constructors
Constructor and Description
`DebuggingWordCount()`

### Method Summary

All Methods Static Methods Concrete Methods
`static void` `main(String[] args)`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### DebuggingWordCount

    public DebuggingWordCount()

### Method Detail

#### main

    public static void main(String[] args)