com.google.cloud.dataflow.sdk.transforms

Class DoFn.Context

java.lang.Object

com.google.cloud.dataflow.sdk.transforms.DoFn.Context

Direct Known Subclasses:

DoFn.ProcessContext

Enclosing class:

DoFn<I,O>

public abstract class DoFn.Context
extends java.lang.Object

Information accessible to all methods in this DoFn.

Constructor Summary

Constructors

Constructor and Description
DoFn.Context()

Method Summary

Methods

Modifier and Type	Method and Description
abstract <AI,AA,AO> Aggregator<AI>	createAggregator(java.lang.String name, Combine.CombineFn<? super AI,AA,AO> combiner) Returns an aggregator with aggregation logic specified by the CombineFn argument.
abstract <AI,AO> Aggregator<AI>	createAggregator(java.lang.String name, SerializableFunction<java.lang.Iterable<AI>,AO> combiner) Returns an aggregator with aggregation logic specified by the SerializableFunction argument.
abstract PipelineOptions	getPipelineOptions() Returns the PipelineOptions specified with the PipelineRunner invoking this DoFn.
abstract void	output(O output) Adds the given element to the main output PCollection.
abstract void	outputWithTimestamp(O output, Instant timestamp) Adds the given element to the main output PCollection, with the given timestamp.
abstract <T> void	sideOutput(TupleTag<T> tag, T output) Adds the given element to the side output PCollection with the given tag.
abstract <T> void	sideOutputWithTimestamp(TupleTag<T> tag, T output, Instant timestamp) Adds the given element to the specified side output PCollection, with the given timestamp.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DoFn.Context

public DoFn.Context()

Method Detail

getPipelineOptions

public abstract PipelineOptions getPipelineOptions()

Returns the PipelineOptions specified with the PipelineRunner invoking this DoFn. The PipelineOptions will be the default running via DoFnTester.

output

public abstract void output(O output)

Adds the given element to the main output PCollection.

If invoked from DoFn.processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.ProcessContext), the output element will have the same timestamp and be in the same windows as the input element passed to DoFn.processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.ProcessContext)).

If invoked from DoFn.startBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context) or DoFn.finishBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context), this will attempt to use the WindowFn of the input PCollection to determine what windows the element should be in, throwing an exception if the WindowFn attempts to access any information about the input element. The output element will have a timestamp of negative infinity.

outputWithTimestamp

public abstract void outputWithTimestamp(O output,
                       Instant timestamp)

Adds the given element to the main output PCollection, with the given timestamp.

If invoked from DoFn.processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.ProcessContext)), the timestamp must not be older than the input element's timestamp minus DoFn.getAllowedTimestampSkew(). The output element will be in the same windows as the input element.

If invoked from DoFn.startBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context) or DoFn.finishBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context), this will attempt to use the WindowFn of the input PCollection to determine what windows the element should be in, throwing an exception if the WindowFn attempts to access any information about the input element except for the timestamp.

sideOutput

public abstract <T> void sideOutput(TupleTag<T> tag,
                  T output)

Adds the given element to the side output PCollection with the given tag.

The caller of ParDo uses ParDo.withOutputTags(com.google.cloud.dataflow.sdk.values.TupleTag<O>, com.google.cloud.dataflow.sdk.values.TupleTagList) to specify the tags of side outputs that it consumes. Non-consumed side outputs, e.g., outputs for monitoring purposes only, don't necessarily need to be specified.

The output element will have the same timestamp and be in the same windows as the input element passed to DoFn.processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.ProcessContext)).

If invoked from DoFn.startBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context) or DoFn.finishBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context), this will attempt to use the WindowFn of the input PCollection to determine what windows the element should be in, throwing an exception if the WindowFn attempts to access any information about the input element. The output element will have a timestamp of negative infinity.

Throws:

java.lang.IllegalArgumentException - if the number of outputs exceeds the limit of 1,000 outputs per DoFn

See Also:

ParDo.withOutputTags(com.google.cloud.dataflow.sdk.values.TupleTag<O>, com.google.cloud.dataflow.sdk.values.TupleTagList)

sideOutputWithTimestamp

public abstract <T> void sideOutputWithTimestamp(TupleTag<T> tag,
                               T output,
                               Instant timestamp)

Adds the given element to the specified side output PCollection, with the given timestamp.

If invoked from DoFn.processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.ProcessContext)), the timestamp must not be older than the input element's timestamp minus DoFn.getAllowedTimestampSkew(). The output element will be in the same windows as the input element.

If invoked from DoFn.startBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context) or DoFn.finishBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context), this will attempt to use the WindowFn of the input PCollection to determine what windows the element should be in, throwing an exception if the WindowFn attempts to access any information about the input element except for the timestamp.

Throws:

java.lang.IllegalArgumentException - if the number of outputs exceeds the limit of 1,000 outputs per DoFn

See Also:

ParDo.withOutputTags(com.google.cloud.dataflow.sdk.values.TupleTag<O>, com.google.cloud.dataflow.sdk.values.TupleTagList)

createAggregator

public abstract <AI,AA,AO> Aggregator<AI> createAggregator(java.lang.String name,
                                         Combine.CombineFn<? super AI,AA,AO> combiner)

Returns an aggregator with aggregation logic specified by the CombineFn argument. The name provided should be unique across aggregators created within the containing ParDo transform application.

All instances of this DoFn in the containing ParDo transform application should define aggregators consistently, i.e., an aggregator with a given name always specifies the same combiner in all DoFn instances in the containing ParDo transform application.

Throws:

java.lang.IllegalArgumentException - if the given CombineFn is not supported as aggregator's combiner, or if the given name collides with another aggregator or system-provided counter.

createAggregator

public abstract <AI,AO> Aggregator<AI> createAggregator(java.lang.String name,
                                      SerializableFunction<java.lang.Iterable<AI>,AO> combiner)

Returns an aggregator with aggregation logic specified by the SerializableFunction argument. The name provided should be unique across aggregators created within the containing ParDo transform application.

All instances of this DoFn in the containing ParDo transform application should define aggregators consistently, i.e., an aggregator with a given name always specifies the same combiner in all DoFn instances in the containing ParDo transform application.

Throws:

java.lang.IllegalArgumentException - if the given SerializableFunction is not supported as aggregator's combiner, or if the given name collides with another aggregator or system-provided counter.