com.google.cloud.dataflow.examples.cookbook

Class DatastoreWordCount

java.lang.Object

com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount

public class DatastoreWordCount
extends Object

A WordCount example using DatastoreIO.

This example shows how to use DatastoreIO to read from Datastore and write the results to Cloud Storage. Note that this example will write data to Datastore, which may incur charge for Datastore operations.

To run this example, users need to use gcloud to get credential for Datastore:

 $ gcloud auth login
 

To run this pipeline locally, the following options must be provided:

   --project=YOUR_PROJECT_ID
   --dataset=YOUR_DATASET_ID
   --output=[YOUR_LOCAL_FILE | gs://YOUR_OUTPUT_PATH]
 

To run this example using Dataflow service, you must additionally provide either --stagingLocation or --tempLocation, and select one of the Dataflow pipeline runners, eg --runner=BlockingDataflowPipelineRunner.

Note: this example creates entities with Ancestor keys to ensure that all entities created are in the same entity group. Similarly, the query used to read from the Cloud Datastore uses an Ancestor filter. Ancestors are used to ensure strongly consistent results in Cloud Datastore. For more information, see the Cloud Datastore documentation on Structing Data for Strong Consistency.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	DatastoreWordCount.Options Options supported by DatastoreWordCount.

Constructor Summary

Constructors

Constructor and Description
DatastoreWordCount()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args) An example to demo how to use DatastoreIO.
static void	readDataFromDatastore(DatastoreWordCount.Options options) An example that creates a pipeline to do DatastoreIO.Read from Datastore.
static void	writeDataToDatastore(DatastoreWordCount.Options options) An example that creates a pipeline to populate DatastoreIO from a text input.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DatastoreWordCount

public DatastoreWordCount()

Method Detail

writeDataToDatastore

public static void writeDataToDatastore(DatastoreWordCount.Options options)

An example that creates a pipeline to populate DatastoreIO from a text input. Forces use of DirectPipelineRunner for local execution mode.

readDataFromDatastore

public static void readDataFromDatastore(DatastoreWordCount.Options options)

An example that creates a pipeline to do DatastoreIO.Read from Datastore.

main

public static void main(String[] args)

An example to demo how to use DatastoreIO. The runner here is customizable, which means users could pass either DirectPipelineRunner or DataflowPipelineRunner in the pipeline options.