com.google.cloud.dataflow.examples.cookbook

Class FilterExamples

java.lang.Object

com.google.cloud.dataflow.examples.cookbook.FilterExamples

public class FilterExamples
extends Object

This is an example that demonstrates several approaches to filtering, and use of the Mean transform. It shows how to dynamically set parameters by defining and using new pipeline options, and how to use a value derived by the pipeline.

Concepts: The Mean transform; Options configuration; using pipeline-derived data as a side input; approaches to filtering, selection, and projection.

The example reads public samples of weather data from BigQuery. It performs a projection on the data, finds the global mean of the temperature readings, filters on readings for a single given month, and then outputs only data (for that month) that has a mean temp smaller than the derived global mean.

Note: Before running this example, you must create a BigQuery dataset to contain your output table.

To execute this pipeline locally, specify general pipeline configuration:

   --project=YOUR_PROJECT_ID
 
 

and the BigQuery table for the output:

   --output=YOUR_PROJECT_ID:DATASET_ID.TABLE_ID
   [--monthFilter=<month_number>]
 
 

where optional parameter --monthFilter is set to a number 1-12.

To execute this pipeline using the Dataflow service, specify pipeline configuration:

   --project=YOUR_PROJECT_ID
   --stagingLocation=gs://YOUR_STAGING_DIRECTORY
   --runner=BlockingDataflowPipelineRunner
 
 

and the BigQuery table for the output:

   --output=YOUR_PROJECT_ID:DATASET_ID.TABLE_ID
   [--monthFilter=<month_number>]
 
 

where optional parameter --monthFilter is set to a number 1-12.

The BigQuery input table defaults to clouddataflow-readonly:samples.weather_stations and can be overridden with --input.

Constructor Summary

Constructors

Constructor and Description
FilterExamples()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FilterExamples

public FilterExamples()

Method Detail

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception