com.google.cloud.dataflow.sdk.util

Class ReduceFnRunner<K,InputT,OutputT,W extends BoundedWindow>

java.lang.Object

com.google.cloud.dataflow.sdk.util.ReduceFnRunner<K,InputT,OutputT,W>

Type Parameters:

K - The type of key being processed.

InputT - The type of values associated with the key.

OutputT - The output type that will be produced for each key.

W - The type of windows this operates on.

All Implemented Interfaces:

ActiveWindowSet.MergeCallback<W>

public class ReduceFnRunner<K,InputT,OutputT,W extends BoundedWindow>
extends Object
implements ActiveWindowSet.MergeCallback<W>

Manages the execution of a ReduceFn after a GroupByKey.GroupByKeyOnly has partitioned the PCollection by key.

The onTrigger(com.google.cloud.dataflow.sdk.util.ReduceFn<K, InputT, OutputT, W>.Context, boolean, boolean) relies on a TriggerRunner to manage the execution of the triggering logic. The ReduceFnRunners responsibilities are:

• Tracking the windows that are active (have buffered data) as elements arrive and triggers are fired.
• Holding the watermark based on the timestamps of elements in a pane and releasing it when the trigger fires.
• Dropping data that exceeds the maximum allowed lateness.
• Calling the appropriate callbacks on ReduceFn based on trigger execution, timer firings, etc.
• Scheduling garbage collection of state associated with a specific window, and making that happen when the appropriate timer fires.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DROPPED_DUE_TO_CLOSED_WINDOW_COUNTER
static String	DROPPED_DUE_TO_LATENESS_COUNTER

Constructor Summary

Constructors

Constructor and Description
ReduceFnRunner(K key, WindowingStrategy<?,W> windowingStrategy, TimerInternals timerInternals, WindowingInternals<?,KV<K,OutputT>> windowingInternals, Aggregator<Long,Long> droppedDueToClosedWindow, Aggregator<Long,Long> droppedDueToLateness, ReduceFn<K,InputT,OutputT,W> reduceFn)

Method Summary

Modifier and Type	Method and Description
void	onMerge(Collection<W> mergedWindows, W resultWindow, boolean isResultWindowNew) Called when windows merge.
void	onTimer(TimerInternals.TimerData timer) Called when a timer fires.
void	persist() Make sure that all the state built up in this runner has been persisted.
void	processElement(WindowedValue<InputT> value)
static <T> StateContents<T>	stateContentsOf(T value)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DROPPED_DUE_TO_CLOSED_WINDOW_COUNTER

public static final String DROPPED_DUE_TO_CLOSED_WINDOW_COUNTER

See Also:

Constant Field Values

DROPPED_DUE_TO_LATENESS_COUNTER

public static final String DROPPED_DUE_TO_LATENESS_COUNTER

See Also:

Constant Field Values

Constructor Detail

ReduceFnRunner

public ReduceFnRunner(K key,
                      WindowingStrategy<?,W> windowingStrategy,
                      TimerInternals timerInternals,
                      WindowingInternals<?,KV<K,OutputT>> windowingInternals,
                      Aggregator<Long,Long> droppedDueToClosedWindow,
                      Aggregator<Long,Long> droppedDueToLateness,
                      ReduceFn<K,InputT,OutputT,W> reduceFn)

Method Detail

processElement

public void processElement(WindowedValue<InputT> value)

persist

public void persist()

Make sure that all the state built up in this runner has been persisted.

onMerge

public void onMerge(Collection<W> mergedWindows,
                    W resultWindow,
                    boolean isResultWindowNew)

Called when windows merge.

Specified by:

onMerge in interface ActiveWindowSet.MergeCallback<W extends BoundedWindow>

onTimer

public void onTimer(TimerInternals.TimerData timer)

Called when a timer fires.

stateContentsOf

public static <T> StateContents<T> stateContentsOf(T value)