com.google.cloud.dataflow.sdk.runners

Class DirectPipelineRunner.TestCombineDoFn<K,InputT,AccumT,OutputT>

java.lang.Object

com.google.cloud.dataflow.sdk.transforms.DoFn<KV<K,Iterable<InputT>>,KV<K,OutputT>>

com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.TestCombineDoFn<K,InputT,AccumT,OutputT>

All Implemented Interfaces:

Serializable

Enclosing class:

DirectPipelineRunner

public static class DirectPipelineRunner.TestCombineDoFn<K,InputT,AccumT,OutputT>
extends DoFn<KV<K,Iterable<InputT>>,KV<K,OutputT>>

The implementation may split the Combine.KeyedCombineFn into ADD, MERGE and EXTRACT phases (see com.google.cloud.dataflow.sdk.runners.worker.CombineValuesFn). In order to emulate this for the DirectPipelineRunner and provide an experience closer to the service, go through heavy seralizability checks for the equivalent of the results of the ADD phase, but after the GroupByKey shuffle, and the MERGE phase. Doing these checks ensure that not only is the accumulator coder serializable, but the accumulator coder can actually serialize the data in question.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.transforms.DoFn

DoFn.Context, DoFn.ProcessContext, DoFn.RequiresWindowAccess

Constructor Summary

Constructors

Constructor and Description
TestCombineDoFn(Combine.KeyedCombineFn<? super K,? super InputT,AccumT,OutputT> fn, Coder<AccumT> accumCoder, boolean testSerializability)

Method Summary

Modifier and Type	Method and Description
static <K,AccumT,InputT> List<AccumT>	addInputsRandomly(Combine.KeyedCombineFn<? super K,? super InputT,AccumT,?> fn, K key, Iterable<InputT> values, Random random)
static <K,InputT,AccumT,OutputT> DirectPipelineRunner.TestCombineDoFn<K,InputT,AccumT,OutputT>	create(Combine.GroupedValues<K,InputT,OutputT> transform, PCollection<KV<K,Iterable<InputT>>> input, boolean testSerializability)
<T> T	ensureSerializableByCoder(Coder<T> coder, T value, String errorContext)
void	processElement(DoFn.ProcessContext c) Processes an input element.

Methods inherited from class com.google.cloud.dataflow.sdk.transforms.DoFn

createAggregator, createAggregator, finishBundle, getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, startBundle

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TestCombineDoFn

public TestCombineDoFn(Combine.KeyedCombineFn<? super K,? super InputT,AccumT,OutputT> fn,
                       Coder<AccumT> accumCoder,
                       boolean testSerializability)

Method Detail

create

public static <K,InputT,AccumT,OutputT> DirectPipelineRunner.TestCombineDoFn<K,InputT,AccumT,OutputT> create(Combine.GroupedValues<K,InputT,OutputT> transform,
                                                                                                             PCollection<KV<K,Iterable<InputT>>> input,
                                                                                                             boolean testSerializability)

processElement

public void processElement(DoFn.ProcessContext c)
                    throws Exception

Description copied from class: DoFn

Processes an input element.

Specified by:

processElement in class DoFn<KV<K,Iterable<InputT>>,KV<K,OutputT>>

Throws:

Exception

addInputsRandomly

public static <K,AccumT,InputT> List<AccumT> addInputsRandomly(Combine.KeyedCombineFn<? super K,? super InputT,AccumT,?> fn,
                                                               K key,
                                                               Iterable<InputT> values,
                                                               Random random)

ensureSerializableByCoder

public <T> T ensureSerializableByCoder(Coder<T> coder,
                                       T value,
                                       String errorContext)