com.google.cloud.dataflow.sdk.options

Interface DataflowPipelineOptions

All Superinterfaces:

ApplicationNameOptions, BigQueryOptions, CloudDebuggerOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, DataflowWorkerLoggingOptions, GcpOptions, GcsOptions, GoogleApiDebugOptions, PipelineOptions, StreamingOptions

All Known Subinterfaces:

BlockingDataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

public interface DataflowPipelineOptions
extends PipelineOptions, GcpOptions, ApplicationNameOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, BigQueryOptions, GcsOptions, StreamingOptions, CloudDebuggerOptions, DataflowWorkerLoggingOptions

Options which can be used to configure the DataflowPipeline.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	DataflowPipelineOptions.JobNameFactory Returns a normalized job name constructed from ApplicationNameOptions.getAppName(), the local system user name (if available), and the current time.

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineDebugOptions

DataflowPipelineDebugOptions.DataflowClientFactory, DataflowPipelineDebugOptions.PathValidatorFactory, DataflowPipelineDebugOptions.StagerFactory

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineWorkerPoolOptions

DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType, DataflowPipelineWorkerPoolOptions.ClusterManagerApiType, DataflowPipelineWorkerPoolOptions.TeardownPolicy

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GcpOptions

GcpOptions.CredentialDirFactory, GcpOptions.GcpUserCredentialsFactory

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GoogleApiDebugOptions

GoogleApiDebugOptions.GoogleApiTracer

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GcsOptions

GcsOptions.ExecutorServiceFactory

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.CloudDebuggerOptions

CloudDebuggerOptions.DebuggerConfig

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowWorkerLoggingOptions

DataflowWorkerLoggingOptions.Level, DataflowWorkerLoggingOptions.WorkerLogLevelOverride

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	getJobName() The Dataflow job name is used as an idempotence key within the Dataflow service.
java.lang.String	getStagingLocation() GCS path for staging local files, e.g.
java.lang.String	getTempLocation() GCS path for temporary files, e.g.
void	setJobName(java.lang.String value)
void	setStagingLocation(java.lang.String value)
void	setTempLocation(java.lang.String value)

Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineDebugOptions

getApiRootUrl, getDataflowClient, getDataflowEndpoint, getDataflowJobFile, getExperiments, getPathValidator, getPathValidatorClass, getStager, getStagerClass, setApiRootUrl, setDataflowClient, setDataflowEndpoint, setDataflowJobFile, setExperiments, setPathValidator, setPathValidatorClass, setStager, setStagerClass

Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineWorkerPoolOptions

getAutoscalingAlgorithm, getClusterManagerApi, getDiskSizeGb, getDiskSourceImage, getFilesToStage, getMaxNumWorkers, getNumWorkers, getTeardownPolicy, getWorkerDiskType, getWorkerMachineType, getWorkerPoolType, getZone, setAutoscalingAlgorithm, setClusterManagerApi, setDiskSizeGb, setDiskSourceImage, setFilesToStage, setMaxNumWorkers, setNumWorkers, setTeardownPolicy, setWorkerDiskType, setWorkerMachineType, setWorkerPoolType, setZone

Methods inherited from interface com.google.cloud.dataflow.sdk.options.BigQueryOptions

getTempDatasetId, setTempDatasetId

Methods inherited from interface com.google.cloud.dataflow.sdk.options.StreamingOptions

isStreaming, setStreaming

Methods inherited from interface com.google.cloud.dataflow.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface com.google.cloud.dataflow.sdk.options.GcpOptions

getCredentialDir, getCredentialFactoryClass, getCredentialId, getGCloudPath, getGcpCredential, getProject, getSecretsFile, getServiceAccountKeyfile, getServiceAccountName, setCredentialDir, setCredentialFactoryClass, setCredentialId, setGCloudPath, setGcpCredential, setProject, setSecretsFile, setServiceAccountKeyfile, setServiceAccountName

Methods inherited from interface com.google.cloud.dataflow.sdk.options.GoogleApiDebugOptions

getGoogleApiTrace, setGoogleApiTrace

Methods inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

as, getRunner, setRunner

Methods inherited from interface com.google.cloud.dataflow.sdk.options.GcsOptions

getExecutorService, getGcsUtil, setExecutorService, setGcsUtil

Methods inherited from interface com.google.cloud.dataflow.sdk.options.CloudDebuggerOptions

getCdbgVersion, setCdbgVersion

Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowWorkerLoggingOptions

getDefaultWorkerLogLevel, getWorkerLogLevelOverrides, setDefaultWorkerLogLevel, setWorkerLogLevelOverrides

Method Detail

getTempLocation

java.lang.String getTempLocation()

GCS path for temporary files, e.g. gs://bucket/object

Must be a valid Cloud Storage url, beginning with the prefix "gs://"

At least one of getTempLocation() or getStagingLocation() must be set. If getTempLocation() is not set, then the Dataflow pipeline defaults to using getStagingLocation().

setTempLocation

void setTempLocation(java.lang.String value)

getStagingLocation

java.lang.String getStagingLocation()

GCS path for staging local files, e.g. gs://bucket/object

Must be a valid Cloud Storage url, beginning with the prefix "gs://"

At least one of getTempLocation() or getStagingLocation() must be set. If getTempLocation() is not set, then the Dataflow pipeline defaults to using getStagingLocation().

setStagingLocation

void setStagingLocation(java.lang.String value)

getJobName

java.lang.String getJobName()

The Dataflow job name is used as an idempotence key within the Dataflow service. If there is an existing job which is currently active, another active job with the same name will not be able to be created. Defaults to using the ApplicationName-UserName-Date.

setJobName

void setJobName(java.lang.String value)