Index (Google Cloud Dataflow Examples)

A B C D E F G I J L M O P R S T W

A

apply(PCollection<String>) - Method in class com.google.cloud.dataflow.examples.complete.AutoComplete.ComputeTopCompletions
apply(PCollection<KV<URI, String>>) - Method in class com.google.cloud.dataflow.examples.complete.TfIdf.ComputeTfIdf
apply(PInput) - Method in class com.google.cloud.dataflow.examples.complete.TfIdf.ReadDocuments
apply(PCollection<KV<String, KV<URI, Double>>>) - Method in class com.google.cloud.dataflow.examples.complete.TfIdf.WriteTfIdf
apply(Iterable<TrafficMaxLaneFlow.LaneInfo>) - Method in class com.google.cloud.dataflow.examples.complete.TrafficMaxLaneFlow.MaxFlow
apply(Iterable<String>) - Method in class com.google.cloud.dataflow.examples.cookbook.CombinePerKeyExamples.ConcatWords
apply(PCollection<String>) - Method in class com.google.cloud.dataflow.examples.WordCount.CountWords
apply(KV<String, Long>) - Method in class com.google.cloud.dataflow.examples.WordCount.FormatAsTextFn
AutoComplete - Class in com.google.cloud.dataflow.examples.complete: An example that computes the most popular hash tags for every prefix, which can be used for auto-completion.
AutoComplete() - Constructor for class com.google.cloud.dataflow.examples.complete.AutoComplete
AutoComplete.ComputeTopCompletions - Class in com.google.cloud.dataflow.examples.complete: A PTransform that takes as input a list of tokens and returns the most common tokens per prefix.

B

BigQueryTableFactory() - Constructor for class com.google.cloud.dataflow.examples.common.ExampleBigQueryTableOptions.BigQueryTableFactory
BigQueryTornadoes - Class in com.google.cloud.dataflow.examples.cookbook: An example that reads the public samples of weather data from BigQuery, counts the number of tornadoes that occur in each month, and writes the results to BigQuery.
BigQueryTornadoes() - Constructor for class com.google.cloud.dataflow.examples.cookbook.BigQueryTornadoes
Bound(String, String) - Constructor for class com.google.cloud.dataflow.examples.common.PubsubFileInjector.Bound

C

com.google.cloud.dataflow.examples - package com.google.cloud.dataflow.examples
com.google.cloud.dataflow.examples.common - package com.google.cloud.dataflow.examples.common
com.google.cloud.dataflow.examples.complete - package com.google.cloud.dataflow.examples.complete
com.google.cloud.dataflow.examples.cookbook - package com.google.cloud.dataflow.examples.cookbook
CombinePerKeyExamples - Class in com.google.cloud.dataflow.examples.cookbook: An example that reads the public 'Shakespeare' data, and for each word in the dataset that is over a given length, generates a string containing the list of play names in which that word appears, and saves this information to a bigquery table.
CombinePerKeyExamples() - Constructor for class com.google.cloud.dataflow.examples.cookbook.CombinePerKeyExamples
CombinePerKeyExamples.ConcatWords - Class in com.google.cloud.dataflow.examples.cookbook: A 'combine function' used with the Combine.perKey transform.
ComputeTfIdf() - Constructor for class com.google.cloud.dataflow.examples.complete.TfIdf.ComputeTfIdf
ComputeTopCompletions(int, boolean) - Constructor for class com.google.cloud.dataflow.examples.complete.AutoComplete.ComputeTopCompletions
ConcatWords() - Constructor for class com.google.cloud.dataflow.examples.cookbook.CombinePerKeyExamples.ConcatWords
CountWords() - Constructor for class com.google.cloud.dataflow.examples.WordCount.CountWords
create(PipelineOptions) - Method in class com.google.cloud.dataflow.examples.common.ExampleBigQueryTableOptions.BigQueryTableFactory
create(PipelineOptions) - Method in class com.google.cloud.dataflow.examples.common.ExamplePubsubTopicAndSubscriptionOptions.PubsubSubscriptionFactory
create(PipelineOptions) - Method in class com.google.cloud.dataflow.examples.common.ExamplePubsubTopicOptions.PubsubTopicFactory
create(PipelineOptions) - Method in class com.google.cloud.dataflow.examples.WordCount.WordCountOptions.OutputFactory

D

DataflowExampleOptions - Interface in com.google.cloud.dataflow.examples.common: Options that can be used to configure the Dataflow examples.
DataflowExampleUtils - Class in com.google.cloud.dataflow.examples.common: The utility class that sets up and tears down external resources, starts the Google Cloud Pub/Sub injector, and cancels the streaming and the injector pipelines once the program terminates.
DataflowExampleUtils(DataflowPipelineOptions) - Constructor for class com.google.cloud.dataflow.examples.common.DataflowExampleUtils
DataflowExampleUtils(DataflowPipelineOptions, boolean) - Constructor for class com.google.cloud.dataflow.examples.common.DataflowExampleUtils: Do resources and runner options setup.
DatastoreWordCount - Class in com.google.cloud.dataflow.examples.cookbook: A WordCount example using DatastoreIO.
DatastoreWordCount() - Constructor for class com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount
DatastoreWordCount.Options - Interface in com.google.cloud.dataflow.examples.cookbook: Options supported by DatastoreWordCount.
DebuggingWordCount - Class in com.google.cloud.dataflow.examples: An example that verifies word counts in Shakespeare and includes Dataflow best practices.
DebuggingWordCount() - Constructor for class com.google.cloud.dataflow.examples.DebuggingWordCount
DebuggingWordCount.FilterTextFn - Class in com.google.cloud.dataflow.examples: A DoFn that filters for a specific key based upon a regular expression.
DeDupExample - Class in com.google.cloud.dataflow.examples.cookbook: This example uses as input Shakespeare's plays as plaintext files, and will remove any duplicate lines across all the files.
DeDupExample() - Constructor for class com.google.cloud.dataflow.examples.cookbook.DeDupExample

E

ExampleBigQueryTableOptions - Interface in com.google.cloud.dataflow.examples.common: Options that can be used to configure BigQuery tables in Dataflow examples.
ExampleBigQueryTableOptions.BigQueryTableFactory - Class in com.google.cloud.dataflow.examples.common: Returns the job name as the default BigQuery table name.
ExamplePubsubTopicAndSubscriptionOptions - Interface in com.google.cloud.dataflow.examples.common: Options that can be used to configure Pub/Sub topic/subscription in Dataflow examples.
ExamplePubsubTopicAndSubscriptionOptions.PubsubSubscriptionFactory - Class in com.google.cloud.dataflow.examples.common: Returns a default Pub/Sub subscription based on the project and the job names.
ExamplePubsubTopicOptions - Interface in com.google.cloud.dataflow.examples.common: Options that can be used to configure Pub/Sub topic in Dataflow examples.
ExamplePubsubTopicOptions.PubsubTopicFactory - Class in com.google.cloud.dataflow.examples.common: Returns a default Pub/Sub topic based on the project and the job names.

F

FilterExamples - Class in com.google.cloud.dataflow.examples.cookbook: This is an example that demonstrates several approaches to filtering, and use of the Mean transform.
FilterExamples() - Constructor for class com.google.cloud.dataflow.examples.cookbook.FilterExamples
FilterTextFn(String) - Constructor for class com.google.cloud.dataflow.examples.DebuggingWordCount.FilterTextFn
FIVE_MINUTES - Static variable in class com.google.cloud.dataflow.examples.cookbook.TriggerExample
FormatAsTextFn() - Constructor for class com.google.cloud.dataflow.examples.WordCount.FormatAsTextFn

G

getBigQueryDataset() - Method in interface com.google.cloud.dataflow.examples.common.ExampleBigQueryTableOptions
getBigQuerySchema() - Method in interface com.google.cloud.dataflow.examples.common.ExampleBigQueryTableOptions
getBigQueryTable() - Method in interface com.google.cloud.dataflow.examples.common.ExampleBigQueryTableOptions
getDataset() - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
getDefaultOutputCoder() - Method in class com.google.cloud.dataflow.examples.complete.TfIdf.ReadDocuments
getInjectorNumWorkers() - Method in interface com.google.cloud.dataflow.examples.common.DataflowExampleOptions
getInput() - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
getInput() - Method in interface com.google.cloud.dataflow.examples.cookbook.TriggerExample.TrafficFlowOptions
getInputFile() - Method in interface com.google.cloud.dataflow.examples.WordCount.WordCountOptions
getKeepJobsRunning() - Method in interface com.google.cloud.dataflow.examples.common.DataflowExampleOptions
getKind() - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
getNamespace() - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
getNumShards() - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
getOutput() - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
getOutput() - Method in interface com.google.cloud.dataflow.examples.WordCount.WordCountOptions
getPubsubSubscription() - Method in interface com.google.cloud.dataflow.examples.common.ExamplePubsubTopicAndSubscriptionOptions
getPubsubTopic() - Method in interface com.google.cloud.dataflow.examples.common.ExamplePubsubTopicOptions
getWindowDuration() - Method in interface com.google.cloud.dataflow.examples.cookbook.TriggerExample.TrafficFlowOptions
getWindowSize() - Method in interface com.google.cloud.dataflow.examples.WindowedWordCount.Options

I

InsertDelays() - Constructor for class com.google.cloud.dataflow.examples.cookbook.TriggerExample.InsertDelays
isReadOnly() - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
isUnbounded() - Method in interface com.google.cloud.dataflow.examples.WindowedWordCount.Options

J

JoinExamples - Class in com.google.cloud.dataflow.examples.cookbook: This example shows how to do a join on two collections.
JoinExamples() - Constructor for class com.google.cloud.dataflow.examples.cookbook.JoinExamples

L

listInputDocuments(TfIdf.Options) - Static method in class com.google.cloud.dataflow.examples.complete.TfIdf: Lists documents contained beneath the options.input prefix/directory.

M

main(String[]) - Static method in class com.google.cloud.dataflow.examples.common.PubsubFileInjector: Sets up and starts streaming pipeline.
main(String[]) - Static method in class com.google.cloud.dataflow.examples.complete.AutoComplete
main(String[]) - Static method in class com.google.cloud.dataflow.examples.complete.StreamingWordExtract: Sets up and starts streaming pipeline.
main(String[]) - Static method in class com.google.cloud.dataflow.examples.complete.TfIdf
main(String[]) - Static method in class com.google.cloud.dataflow.examples.complete.TopWikipediaSessions
main(String[]) - Static method in class com.google.cloud.dataflow.examples.complete.TrafficMaxLaneFlow: Sets up and starts streaming pipeline.
main(String[]) - Static method in class com.google.cloud.dataflow.examples.complete.TrafficRoutes: Sets up and starts streaming pipeline.
main(String[]) - Static method in class com.google.cloud.dataflow.examples.cookbook.BigQueryTornadoes
main(String[]) - Static method in class com.google.cloud.dataflow.examples.cookbook.CombinePerKeyExamples
main(String[]) - Static method in class com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount: An example to demo how to use DatastoreIO.
main(String[]) - Static method in class com.google.cloud.dataflow.examples.cookbook.DeDupExample
main(String[]) - Static method in class com.google.cloud.dataflow.examples.cookbook.FilterExamples
main(String[]) - Static method in class com.google.cloud.dataflow.examples.cookbook.JoinExamples
main(String[]) - Static method in class com.google.cloud.dataflow.examples.cookbook.MaxPerKeyExamples
main(String[]) - Static method in class com.google.cloud.dataflow.examples.cookbook.TriggerExample
main(String[]) - Static method in class com.google.cloud.dataflow.examples.DebuggingWordCount
main(String[]) - Static method in class com.google.cloud.dataflow.examples.MinimalWordCount
main(String[]) - Static method in class com.google.cloud.dataflow.examples.WindowedWordCount
main(String[]) - Static method in class com.google.cloud.dataflow.examples.WordCount
MaxFlow() - Constructor for class com.google.cloud.dataflow.examples.complete.TrafficMaxLaneFlow.MaxFlow
MaxPerKeyExamples - Class in com.google.cloud.dataflow.examples.cookbook: An example that reads the public samples of weather data from BigQuery, and finds the maximum temperature ('mean_temp') for each month.
MaxPerKeyExamples() - Constructor for class com.google.cloud.dataflow.examples.cookbook.MaxPerKeyExamples
MinimalWordCount - Class in com.google.cloud.dataflow.examples: An example that counts words in Shakespeare.
MinimalWordCount() - Constructor for class com.google.cloud.dataflow.examples.MinimalWordCount
mockUnboundedSource(String, PipelineResult) - Method in class com.google.cloud.dataflow.examples.common.DataflowExampleUtils: Start the auxiliary injector pipeline, then wait for this pipeline to finish.

O

ONE_DAY - Static variable in class com.google.cloud.dataflow.examples.cookbook.TriggerExample
ONE_MINUTE - Static variable in class com.google.cloud.dataflow.examples.cookbook.TriggerExample
OutputFactory() - Constructor for class com.google.cloud.dataflow.examples.WordCount.WordCountOptions.OutputFactory

P

processElement(DoFn<String, Void>.ProcessContext) - Method in class com.google.cloud.dataflow.examples.common.PubsubFileInjector.Bound
processElement(DoFn<String, String>.ProcessContext) - Method in class com.google.cloud.dataflow.examples.cookbook.TriggerExample.InsertDelays
processElement(DoFn<KV<String, Long>, KV<String, Long>>.ProcessContext) - Method in class com.google.cloud.dataflow.examples.DebuggingWordCount.FilterTextFn
publish(String) - Static method in class com.google.cloud.dataflow.examples.common.PubsubFileInjector: Creates a PubsubFileInjector transform that publishes to the given output topic.
publish(String) - Method in class com.google.cloud.dataflow.examples.common.PubsubFileInjector.Unbound
pubsub - Variable in class com.google.cloud.dataflow.examples.common.PubsubFileInjector.Bound
PubsubFileInjector - Class in com.google.cloud.dataflow.examples.common: A batch Dataflow pipeline for injecting a set of GCS files into a PubSub topic line by line.
PubsubFileInjector() - Constructor for class com.google.cloud.dataflow.examples.common.PubsubFileInjector
PubsubFileInjector.Bound - Class in com.google.cloud.dataflow.examples.common: A DoFn that publishes non-empty lines to Google Cloud PubSub.
PubsubFileInjector.Unbound - Class in com.google.cloud.dataflow.examples.common: An incomplete PubsubFileInjector transform with unbound output topic.
PubsubSubscriptionFactory() - Constructor for class com.google.cloud.dataflow.examples.common.ExamplePubsubTopicAndSubscriptionOptions.PubsubSubscriptionFactory
PubsubTopicFactory() - Constructor for class com.google.cloud.dataflow.examples.common.ExamplePubsubTopicOptions.PubsubTopicFactory

R

readDataFromDatastore(DatastoreWordCount.Options) - Static method in class com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount: An example that creates a pipeline to do DatastoreIO.Read from Datastore.
ReadDocuments(Iterable<URI>) - Constructor for class com.google.cloud.dataflow.examples.complete.TfIdf.ReadDocuments
runInjectorPipeline(String, String) - Method in class com.google.cloud.dataflow.examples.common.DataflowExampleUtils: Runs a batch pipeline to inject data into the PubSubIO input topic.
runInjectorPipeline(PTransform<? super PBegin, PCollection<String>>, String, String) - Method in class com.google.cloud.dataflow.examples.common.DataflowExampleUtils: Runs a batch pipeline to inject data into the PubSubIO input topic.
runInjectorPipeline(Pipeline) - Method in class com.google.cloud.dataflow.examples.common.DataflowExampleUtils: Runs the provided pipeline to inject data into the PubSubIO input topic.

S

setBigQueryDataset(String) - Method in interface com.google.cloud.dataflow.examples.common.ExampleBigQueryTableOptions
setBigQuerySchema(TableSchema) - Method in interface com.google.cloud.dataflow.examples.common.ExampleBigQueryTableOptions
setBigQueryTable(String) - Method in interface com.google.cloud.dataflow.examples.common.ExampleBigQueryTableOptions
setDataset(String) - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
setInjectorNumWorkers(int) - Method in interface com.google.cloud.dataflow.examples.common.DataflowExampleOptions
setInput(String) - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
setInput(String) - Method in interface com.google.cloud.dataflow.examples.cookbook.TriggerExample.TrafficFlowOptions
setInputFile(String) - Method in interface com.google.cloud.dataflow.examples.WordCount.WordCountOptions
setKeepJobsRunning(boolean) - Method in interface com.google.cloud.dataflow.examples.common.DataflowExampleOptions
setKind(String) - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
setNamespace(String) - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
setNumShards(int) - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
setOutput(String) - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
setOutput(String) - Method in interface com.google.cloud.dataflow.examples.WordCount.WordCountOptions
setPubsubSubscription(String) - Method in interface com.google.cloud.dataflow.examples.common.ExamplePubsubTopicAndSubscriptionOptions
setPubsubTopic(String) - Method in interface com.google.cloud.dataflow.examples.common.ExamplePubsubTopicOptions
setReadOnly(boolean) - Method in interface com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount.Options
setUnbounded(boolean) - Method in interface com.google.cloud.dataflow.examples.WindowedWordCount.Options
setup() - Method in class com.google.cloud.dataflow.examples.common.DataflowExampleUtils: Sets up external resources that are required by the example, such as Pub/Sub topics and BigQuery tables.
setupBigQueryTable() - Method in class com.google.cloud.dataflow.examples.common.DataflowExampleUtils: Sets up the BigQuery table with the given schema.
setupPubsub() - Method in class com.google.cloud.dataflow.examples.common.DataflowExampleUtils: Sets up the Google Cloud Pub/Sub topic.
setupResourcesAndRunner(boolean) - Method in class com.google.cloud.dataflow.examples.common.DataflowExampleUtils: Set up external resources, and configure the runner appropriately.
setupRunner() - Method in class com.google.cloud.dataflow.examples.common.DataflowExampleUtils: Do some runner setup: check that the DirectPipelineRunner is not used in conjunction with streaming, and if streaming is specified, use the DataflowPipelineRunner.
setWindowDuration(Integer) - Method in interface com.google.cloud.dataflow.examples.cookbook.TriggerExample.TrafficFlowOptions
setWindowSize(Integer) - Method in interface com.google.cloud.dataflow.examples.WindowedWordCount.Options
startBundle(DoFn<String, Void>.Context) - Method in class com.google.cloud.dataflow.examples.common.PubsubFileInjector.Bound
startInjectorIfNeeded(String) - Method in class com.google.cloud.dataflow.examples.common.DataflowExampleUtils: If this is an unbounded (streaming) pipeline, and both inputFile and pubsub topic are defined, start an 'injector' pipeline that publishes the contents of the file to the given topic, first creating the topic if necessary.
StreamingWordExtract - Class in com.google.cloud.dataflow.examples.complete: A streaming Dataflow Example using BigQuery output.
StreamingWordExtract() - Constructor for class com.google.cloud.dataflow.examples.complete.StreamingWordExtract

T

TfIdf - Class in com.google.cloud.dataflow.examples.complete: An example that computes a basic TF-IDF search table for a directory or GCS prefix.
TfIdf() - Constructor for class com.google.cloud.dataflow.examples.complete.TfIdf
TfIdf.ComputeTfIdf - Class in com.google.cloud.dataflow.examples.complete: A transform containing a basic TF-IDF pipeline.
TfIdf.ReadDocuments - Class in com.google.cloud.dataflow.examples.complete: Reads the documents at the provided uris and returns all lines from the documents tagged with which document they are from.
TfIdf.WriteTfIdf - Class in com.google.cloud.dataflow.examples.complete: A PTransform to write, in CSV format, a mapping from term and URI to score.
top(int, boolean) - Static method in class com.google.cloud.dataflow.examples.complete.AutoComplete.ComputeTopCompletions
TopWikipediaSessions - Class in com.google.cloud.dataflow.examples.complete: An example that reads Wikipedia edit data from Cloud Storage and computes the user with the longest string of edits separated by no more than an hour within each month.
TopWikipediaSessions() - Constructor for class com.google.cloud.dataflow.examples.complete.TopWikipediaSessions
TrafficMaxLaneFlow - Class in com.google.cloud.dataflow.examples.complete: A Dataflow Example that runs in both batch and streaming modes with traffic sensor data.
TrafficMaxLaneFlow() - Constructor for class com.google.cloud.dataflow.examples.complete.TrafficMaxLaneFlow
TrafficMaxLaneFlow.MaxFlow - Class in com.google.cloud.dataflow.examples.complete: A custom 'combine function' used with the Combine.perKey transform.
TrafficRoutes - Class in com.google.cloud.dataflow.examples.complete: A Dataflow Example that runs in both batch and streaming modes with traffic sensor data.
TrafficRoutes() - Constructor for class com.google.cloud.dataflow.examples.complete.TrafficRoutes
TriggerExample - Class in com.google.cloud.dataflow.examples.cookbook: This example illustrates the basic concepts behind triggering.
TriggerExample() - Constructor for class com.google.cloud.dataflow.examples.cookbook.TriggerExample
TriggerExample.InsertDelays - Class in com.google.cloud.dataflow.examples.cookbook: Add current time to each record.
TriggerExample.TrafficFlowOptions - Interface in com.google.cloud.dataflow.examples.cookbook: Inherits standard configuration options.

W

waitToFinish(PipelineResult) - Method in class com.google.cloud.dataflow.examples.common.DataflowExampleUtils: If DataflowPipelineRunner or BlockingDataflowPipelineRunner is used, waits for the pipeline to finish and cancels it (and the injector) before the program exists.
WINDOW_DURATION - Static variable in class com.google.cloud.dataflow.examples.cookbook.TriggerExample
WindowedWordCount - Class in com.google.cloud.dataflow.examples: An example that counts words in text, and can run over either unbounded or bounded input collections.
WindowedWordCount() - Constructor for class com.google.cloud.dataflow.examples.WindowedWordCount
WindowedWordCount.Options - Interface in com.google.cloud.dataflow.examples: Options supported by WindowedWordCount.
withTimestampLabelKey(String) - Static method in class com.google.cloud.dataflow.examples.common.PubsubFileInjector: Creates a PubsubFileInjector transform with the given timestamp label key.
WordCount - Class in com.google.cloud.dataflow.examples: An example that counts words in Shakespeare and includes Dataflow best practices.
WordCount() - Constructor for class com.google.cloud.dataflow.examples.WordCount
WordCount.CountWords - Class in com.google.cloud.dataflow.examples: A PTransform that converts a PCollection containing lines of text into a PCollection of formatted word counts.
WordCount.FormatAsTextFn - Class in com.google.cloud.dataflow.examples: A SimpleFunction that converts a Word and Count into a printable string.
WordCount.WordCountOptions - Interface in com.google.cloud.dataflow.examples: Options supported by WordCount.
WordCount.WordCountOptions.OutputFactory - Class in com.google.cloud.dataflow.examples: Returns "gs://${YOUR_STAGING_DIRECTORY}/counts.txt" as the default destination.
writeDataToDatastore(DatastoreWordCount.Options) - Static method in class com.google.cloud.dataflow.examples.cookbook.DatastoreWordCount: An example that creates a pipeline to populate DatastoreIO from a text input.
WriteTfIdf(String) - Constructor for class com.google.cloud.dataflow.examples.complete.TfIdf.WriteTfIdf

A B C D E F G I J L M O P R S T W