com.google.cloud.dataflow.sdk.runners

Interface DataflowPipelineTranslator.TranslationContext

Enclosing class:

DataflowPipelineTranslator

public static interface DataflowPipelineTranslator.TranslationContext

The interface provided to registered callbacks for interacting with the DataflowPipelineRunner, including reading and writing the values of PCollections and side inputs (PCollectionViews).

Method Summary

Modifier and Type	Method and Description
void	addCollectionToSingletonOutput(String name, PValue inputValue, PValue outputValue) Adds an output with the given name to the previously added CollectionToSingleton Dataflow step, consuming the specified input PValue and producing the specified output PValue.
void	addEncodingInput(Coder<?> value) Sets the encoding for the current Dataflow step.
void	addInput(String name, Boolean value) Adds an input with the given name and value to the current Dataflow step.
void	addInput(String name, List<? extends Map<String,Object>> elements) Adds an input that is a list of objects.
void	addInput(String name, Long value) Adds an input with the given name and value to the current Dataflow step.
void	addInput(String name, Map<String,Object> elements) Adds an input that is a dictionary of strings to objects.
void	addInput(String name, PInput value) Adds an input with the given name to the previously added Dataflow step, coming from the specified input PValue.
void	addInput(String name, String value) Adds an input with the given name and value to the current Dataflow step.
void	addOutput(String name, PValue value) Adds an output with the given name to the previously added Dataflow step, producing the specified output PValue, including its Coder if a TypedPValue.
void	addStep(PTransform<?,?> transform, String type) Adds a step to the Dataflow workflow for the given transform, with the given Dataflow step type.
void	addStep(PTransform<?,? extends PValue> transform, com.google.api.services.dataflow.model.Step step) Adds a pre-defined step to the Dataflow workflow.
void	addValueOnlyOutput(String name, PValue value) Adds an output with the given name to the previously added Dataflow step, producing the specified output PValue, including its Coder if a TypedPValue.
com.google.cloud.dataflow.sdk.util.OutputReference	asOutputReference(PValue value) Encode a PValue reference as an output reference.
String	getFullName(PTransform<?,?> transform) Returns the full name of the currently being translated transform.
<InputT extends PInput> InputT	getInput(PTransform<InputT,?> transform) Returns the input of the currently being translated transform.
<OutputT extends POutput> OutputT	getOutput(PTransform<?,OutputT> transform) Returns the output of the currently being translated transform.
DataflowPipelineOptions	getPipelineOptions() Returns the configured pipeline options.

Method Detail

getPipelineOptions

DataflowPipelineOptions getPipelineOptions()

Returns the configured pipeline options.

getInput

<InputT extends PInput> InputT getInput(PTransform<InputT,?> transform)

Returns the input of the currently being translated transform.

getOutput

<OutputT extends POutput> OutputT getOutput(PTransform<?,OutputT> transform)

Returns the output of the currently being translated transform.

getFullName

String getFullName(PTransform<?,?> transform)

Returns the full name of the currently being translated transform.

addStep

void addStep(PTransform<?,?> transform,
             String type)

Adds a step to the Dataflow workflow for the given transform, with the given Dataflow step type. This step becomes "current" for the purpose of addInput(java.lang.String, java.lang.Boolean) and addOutput(java.lang.String, com.google.cloud.dataflow.sdk.values.PValue).

addStep

void addStep(PTransform<?,? extends PValue> transform,
             com.google.api.services.dataflow.model.Step step)

Adds a pre-defined step to the Dataflow workflow. The given PTransform should be consistent with the Step, in terms of input, output and coder types.

This is a low-level operation, when using this method it is up to the caller to ensure that names do not collide.

addEncodingInput

void addEncodingInput(Coder<?> value)

Sets the encoding for the current Dataflow step.

addInput

void addInput(String name,
              Boolean value)

Adds an input with the given name and value to the current Dataflow step.

addInput

void addInput(String name,
              String value)

Adds an input with the given name and value to the current Dataflow step.

addInput

void addInput(String name,
              Long value)

Adds an input with the given name and value to the current Dataflow step.

addInput

void addInput(String name,
              PInput value)

Adds an input with the given name to the previously added Dataflow step, coming from the specified input PValue.

addInput

void addInput(String name,
              Map<String,Object> elements)

Adds an input that is a dictionary of strings to objects.

addInput

void addInput(String name,
              List<? extends Map<String,Object>> elements)

Adds an input that is a list of objects.

addOutput

void addOutput(String name,
               PValue value)

Adds an output with the given name to the previously added Dataflow step, producing the specified output PValue, including its Coder if a TypedPValue. If the PValue is a PCollection, wraps its coder inside a WindowedValueCoder.

addValueOnlyOutput

void addValueOnlyOutput(String name,
                        PValue value)

Adds an output with the given name to the previously added Dataflow step, producing the specified output PValue, including its Coder if a TypedPValue. If the PValue is a PCollection, wraps its coder inside a ValueOnlyCoder.

addCollectionToSingletonOutput

void addCollectionToSingletonOutput(String name,
                                    PValue inputValue,
                                    PValue outputValue)

Adds an output with the given name to the previously added CollectionToSingleton Dataflow step, consuming the specified input PValue and producing the specified output PValue. This step requires special treatment for its output encoding.

asOutputReference

com.google.cloud.dataflow.sdk.util.OutputReference asOutputReference(PValue value)

Encode a PValue reference as an output reference.