com.google.cloud.dataflow.examples.complete

Class TopWikipediaSessions

java.lang.Object

com.google.cloud.dataflow.examples.complete.TopWikipediaSessions

public class TopWikipediaSessions
extends Object

An example that reads Wikipedia edit data from Cloud Storage and computes the user with the longest string of edits separated by no more than an hour within each month.

Concepts: Using Windowing to perform time-based aggregations of data.

It is not recommended to execute this pipeline locally, given the size of the default input data.

To execute this pipeline using the Dataflow service, specify pipeline configuration:

   --project=YOUR_PROJECT_ID
   --stagingLocation=gs://YOUR_STAGING_DIRECTORY
   --runner=BlockingDataflowPipelineRunner
 
 

and an output prefix on GCS:

   --output=gs://YOUR_OUTPUT_PREFIX
 

The default input is gs://dataflow-samples/wikipedia_edits/*.json and can be overridden with --input.

The input for this example is large enough that it's a good place to enable (experimental) autoscaling:

   --autoscalingAlgorithm=BASIC
   --maxNumWorkers=20
 
 

This will automatically scale the number of workers up over time until the job completes.

Constructor Summary

Constructors

Constructor and Description
TopWikipediaSessions()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TopWikipediaSessions

public TopWikipediaSessions()

Method Detail

main

public static void main(String[] args)