com.google.cloud.dataflow.examples.cookbook

Class DatastoreWordCount

java.lang.Object

com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount

public class DatastoreWordCount
extends Object

A WordCount example using DatastoreIO.

This example shows how to use DatastoreIO to read from Datastore and write the results to Cloud Storage. Note that this example will write data to Datastore, which may incur charge for Datastore operations.

To run this example, users need to use gcloud to get credential for Datastore:

 $ gcloud auth login
 

Note that the environment variable CLOUDSDK_EXTRA_SCOPES must be set to the same value when executing a Datastore pipeline, as the local auth cache is keyed by the requested scopes.

To run this pipeline locally, the following options must be provided:

   --project=YOUR_PROJECT_ID
   --dataset=YOUR_DATASET_ID
   --output=[YOUR_LOCAL_FILE | gs://YOUR_OUTPUT_PATH]
 

To run this example using Dataflow service, you must additionally provide either --stagingLocation or --tempLocation, and select one of the Dataflow pipeline runners, eg --runner=BlockingDataflowPipelineRunner.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	DatastoreWordCount.Options Options supported by DatastoreWordCount.

Constructor Summary

Constructors

Constructor and Description
DatastoreWordCount()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args) Main function.
static void	readDataFromDatastore(DatastoreWordCount.Options options) An example that creates a pipeline to do DatastoreIO.Read from Datastore.
static void	writeDataToDatastore(DatastoreWordCount.Options options) An example that creates a pipeline to populate DatastoreIO from a text input.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DatastoreWordCount

public DatastoreWordCount()

Method Detail

writeDataToDatastore

public static void writeDataToDatastore(DatastoreWordCount.Options options)

An example that creates a pipeline to populate DatastoreIO from a text input. Forces use of DirectPipelineRunner for local execution mode.

readDataFromDatastore

public static void readDataFromDatastore(DatastoreWordCount.Options options)

An example that creates a pipeline to do DatastoreIO.Read from Datastore.

main

public static void main(String[] args)

Main function. An example to demo how to use DatastoreIO. The runner here is customizable, which means users could pass either DirectPipelineRunner or DataflowPipelineRunner in PipelineOptions.