com.google.cloud.dataflow.examples

Class TfIdf

java.lang.Object

com.google.cloud.dataflow.examples.TfIdf

public class TfIdf
extends java.lang.Object

An example that computes a basic TF-IDF search table for a directory or GCS prefix.

Concepts: joining data; side inputs

To execute this pipeline locally, specify general pipeline configuration: --project= and a local output file or output prefix on GCS: --output=[ | gs://

]

To execute this pipeline using the Dataflow service, specify pipeline configuration: --project= --stagingLocation=gs:// --runner=BlockingDataflowPipelineRunner and an output prefix on GCS: --output=gs://

The default input is gs://dataflow-samples/shakespeare/ and can be overridden with --input.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TfIdf.ComputeTfIdf A transform containing a basic TF-IDF pipeline.
static class	TfIdf.ReadDocuments Reads the documents at the provided uris and returns all lines from the documents tagged with which document they are from.
static class	TfIdf.WriteTfIdf A PTransform to write, in CSV format, a mapping from term and URI to score.

Constructor Summary

Constructors

Constructor and Description
TfIdf()

Method Summary

Methods

Modifier and Type	Method and Description
static java.util.Set<java.net.URI>	listInputDocuments(com.google.cloud.dataflow.examples.TfIdf.Options options) Lists documents contained beneath the options.input prefix/directory.
static void	main(java.lang.String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TfIdf

public TfIdf()

Method Detail

listInputDocuments

public static java.util.Set<java.net.URI> listInputDocuments(com.google.cloud.dataflow.examples.TfIdf.Options options)
                                                      throws java.net.URISyntaxException,
                                                             java.io.IOException

Lists documents contained beneath the options.input prefix/directory.

Throws:

java.net.URISyntaxException

java.io.IOException

main

public static void main(java.lang.String[] args)
                 throws java.lang.Exception

Throws:

java.lang.Exception