com.google.cloud.dataflow.sdk.options

Interface DataflowPipelineOptions

All Superinterfaces:

ApplicationNameOptions, BigQueryOptions, CloudDebuggerOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, GcpOptions, GcsOptions, PipelineOptions, StreamingOptions

All Known Subinterfaces:

BlockingDataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

public interface DataflowPipelineOptions
extends PipelineOptions, GcpOptions, ApplicationNameOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, BigQueryOptions, GcsOptions, StreamingOptions, CloudDebuggerOptions

Options which can be used to configure the DataflowPipeline.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	DataflowPipelineOptions.DataflowClientFactory Returns the default Dataflow client built from the passed in PipelineOptions.
static class	DataflowPipelineOptions.JobNameFactory Returns a normalized job name constructed from ApplicationNameOptions.getAppName(), the local system user name (if available), and the current time.

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineWorkerPoolOptions

DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType, DataflowPipelineWorkerPoolOptions.ClusterManagerApiType, DataflowPipelineWorkerPoolOptions.TeardownPolicy

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GcpOptions

GcpOptions.CredentialDirFactory, GcpOptions.GcpUserCredentialsFactory

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GcsOptions

GcsOptions.ExecutorServiceFactory

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.CloudDebuggerOptions

CloudDebuggerOptions.DebuggerConfig

Method Summary

Methods

Modifier and Type	Method and Description
com.google.api.services.dataflow.Dataflow	getDataflowClient() Alternative Dataflow client.
java.lang.String	getJobName() The job name is used as an idempotence key within the Dataflow service.
java.lang.String	getStagingLocation() GCS path for staging local files.
java.lang.String	getTempLocation() GCS path for temporary files.
void	setDataflowClient(com.google.api.services.dataflow.Dataflow value)
void	setJobName(java.lang.String value)
void	setStagingLocation(java.lang.String value)
void	setTempLocation(java.lang.String value)

Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineDebugOptions

getApiRootUrl, getDataflowEndpoint, getDataflowJobFile, getExperiments, setApiRootUrl, setDataflowEndpoint, setDataflowJobFile, setExperiments

Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineWorkerPoolOptions

getAutoscalingAlgorithm, getClusterManagerApi, getDiskSizeGb, getDiskSourceImage, getFilesToStage, getMachineType, getMaxNumWorkers, getNumWorkers, getTeardownPolicy, getWorkerMachineType, getZone, setAutoscalingAlgorithm, setClusterManagerApi, setDiskSizeGb, setDiskSourceImage, setFilesToStage, setMachineType, setMaxNumWorkers, setNumWorkers, setTeardownPolicy, setWorkerMachineType, setZone

Methods inherited from interface com.google.cloud.dataflow.sdk.options.BigQueryOptions

getTempDatasetId, setTempDatasetId

Methods inherited from interface com.google.cloud.dataflow.sdk.options.StreamingOptions

isStreaming, setStreaming

Methods inherited from interface com.google.cloud.dataflow.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface com.google.cloud.dataflow.sdk.options.GcpOptions

getCredentialDir, getCredentialId, getGCloudPath, getGcpCredential, getProject, getSecretsFile, getServiceAccountKeyfile, getServiceAccountName, setCredentialDir, setCredentialId, setGCloudPath, setGcpCredential, setProject, setSecretsFile, setServiceAccountKeyfile, setServiceAccountName

Methods inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

as, getRunner, setRunner

Methods inherited from interface com.google.cloud.dataflow.sdk.options.GcsOptions

getExecutorService, getGcsUtil, setExecutorService, setGcsUtil

Methods inherited from interface com.google.cloud.dataflow.sdk.options.CloudDebuggerOptions

getCdbgVersion, setCdbgVersion

Method Detail

getTempLocation

java.lang.String getTempLocation()

GCS path for temporary files.

Must be a valid Cloud Storage url, beginning with the prefix "gs://"

At least one of getTempLocation() or getStagingLocation() must be set. If getTempLocation() is not set, then the Dataflow pipeline defaults to using getStagingLocation().

setTempLocation

void setTempLocation(java.lang.String value)

getStagingLocation

java.lang.String getStagingLocation()

GCS path for staging local files.

If getStagingLocation() is not set, then the Dataflow pipeline defaults to a staging directory within getTempLocation().

At least one of getTempLocation() or getStagingLocation() must be set.

setStagingLocation

void setStagingLocation(java.lang.String value)

getJobName

java.lang.String getJobName()

The job name is used as an idempotence key within the Dataflow service. If there is an existing job which is currently active, another job with the same name will not be able to be created.

setJobName

void setJobName(java.lang.String value)

getDataflowClient

com.google.api.services.dataflow.Dataflow getDataflowClient()

Alternative Dataflow client.

setDataflowClient

void setDataflowClient(com.google.api.services.dataflow.Dataflow value)