com.google.cloud.dataflow.sdk.runners

Class DirectPipelineRunner.TestCombineDoFn<K,VI,VA,VO>

java.lang.Object

com.google.cloud.dataflow.sdk.transforms.DoFn<KV<K,java.lang.Iterable<VI>>,KV<K,VO>>

com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.TestCombineDoFn<K,VI,VA,VO>

All Implemented Interfaces:

java.io.Serializable

Enclosing class:

DirectPipelineRunner

public static class DirectPipelineRunner.TestCombineDoFn<K,VI,VA,VO>
extends DoFn<KV<K,java.lang.Iterable<VI>>,KV<K,VO>>

The implementation may split the KeyedCombineFn into ADD, MERGE and EXTRACT phases (see CombineValuesFn). In order to emulate this for the DirectPipelineRunner and provide an experience closer to the service, go through heavy seralizability checks for the equivalent of the results of the ADD phase, but after the GroupByKey shuffle, and the MERGE phase. Doing these checks ensure that not only is the accumulator coder serializable, but the accumulator coder can actually serialize the data in question.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.transforms.DoFn

DoFn.Context, DoFn.KeyedState, DoFn.ProcessContext, DoFn.RequiresKeyedState, DoFn.RequiresWindowAccess

Constructor Summary

Constructors

Constructor and Description
DirectPipelineRunner.TestCombineDoFn(Combine.KeyedCombineFn<? super K,? super VI,VA,VO> fn, Coder<VA> accumCoder, boolean testSerializability)

Method Summary

Methods

Modifier and Type	Method and Description
static <K,VA,VI> java.util.List<VA>	addInputsRandomly(Combine.KeyedCombineFn<? super K,? super VI,VA,?> fn, K key, java.lang.Iterable<VI> values, java.util.Random random)
static <K,VI,VA,VO> DirectPipelineRunner.TestCombineDoFn<K,VI,VA,VO>	create(Combine.GroupedValues<K,VI,VO> transform, boolean testSerializability)
<T> T	ensureSerializableByCoder(Coder<T> coder, T value, java.lang.String errorContext)
void	processElement(DoFn.ProcessContext c) Processes an input element.

Methods inherited from class com.google.cloud.dataflow.sdk.transforms.DoFn

finishBundle, getAllowedTimestampSkew, startBundle

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DirectPipelineRunner.TestCombineDoFn

public DirectPipelineRunner.TestCombineDoFn(Combine.KeyedCombineFn<? super K,? super VI,VA,VO> fn,
                                    Coder<VA> accumCoder,
                                    boolean testSerializability)

Method Detail

create

public static <K,VI,VA,VO> DirectPipelineRunner.TestCombineDoFn<K,VI,VA,VO> create(Combine.GroupedValues<K,VI,VO> transform,
                                                                   boolean testSerializability)

processElement

public void processElement(DoFn.ProcessContext c)
                    throws java.lang.Exception

Description copied from class: DoFn

Processes an input element.

Specified by:

processElement in class DoFn<KV<K,java.lang.Iterable<VI>>,KV<K,VO>>

Throws:

java.lang.Exception

addInputsRandomly

public static <K,VA,VI> java.util.List<VA> addInputsRandomly(Combine.KeyedCombineFn<? super K,? super VI,VA,?> fn,
                                             K key,
                                             java.lang.Iterable<VI> values,
                                             java.util.Random random)

ensureSerializableByCoder

public <T> T ensureSerializableByCoder(Coder<T> coder,
                              T value,
                              java.lang.String errorContext)