com.google.cloud.dataflow.sdk.transforms

Class CombineFns

java.lang.Object

com.google.cloud.dataflow.sdk.transforms.CombineFns

public class CombineFns
extends Object

Static utility methods that create combine function instances.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CombineFns.CoCombineResult A tuple of outputs produced by a composed combine functions.
static class	CombineFns.ComposeCombineFnBuilder A builder class to construct a composed CombineFnBase.GlobalCombineFn.
static class	CombineFns.ComposedCombineFn<DataT> A composed Combine.CombineFn that applies multiple CombineFns.
static class	CombineFns.ComposedCombineFnWithContext<DataT> A composed CombineWithContext.CombineFnWithContext that applies multiple CombineFnWithContexts.
static class	CombineFns.ComposedKeyedCombineFn<DataT,K> A composed Combine.KeyedCombineFn that applies multiple KeyedCombineFns.
static class	CombineFns.ComposedKeyedCombineFnWithContext<DataT,K> A composed CombineWithContext.KeyedCombineFnWithContext that applies multiple KeyedCombineFnWithContexts.
static class	CombineFns.ComposeKeyedCombineFnBuilder A builder class to construct a composed CombineFnBase.PerKeyCombineFn.

Constructor Summary

Constructors

Constructor and Description
CombineFns()

Method Summary

Modifier and Type	Method and Description
static CombineFns.ComposeCombineFnBuilder	compose() Returns a CombineFns.ComposeCombineFnBuilder to construct a composed CombineFnBase.GlobalCombineFn.
static CombineFns.ComposeKeyedCombineFnBuilder	composeKeyed() Returns a CombineFns.ComposeKeyedCombineFnBuilder to construct a composed CombineFnBase.PerKeyCombineFn.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CombineFns

public CombineFns()

Method Detail

composeKeyed

public static CombineFns.ComposeKeyedCombineFnBuilder composeKeyed()

Returns a CombineFns.ComposeKeyedCombineFnBuilder to construct a composed CombineFnBase.PerKeyCombineFn.

The same TupleTag cannot be used in a composition multiple times.

Example:

 PCollection<KV<K, Integer>> latencies = ...;

 TupleTag<Integer> maxLatencyTag = new TupleTag<Integer>();
 TupleTag<Double> meanLatencyTag = new TupleTag<Double>();

 SimpleFunction<Integer, Integer> identityFn =
     new SimpleFunction<Integer, Integer>() {
       public Integer apply(Integer input) {
           return input;
       }};
 PCollection<KV<K, CoCombineResult>> maxAndMean = latencies.apply(
     Combine.perKey(
         CombineFns.composeKeyed()
            .with(identityFn, new MaxIntegerFn(), maxLatencyTag)
            .with(identityFn, new MeanFn<Integer>(), meanLatencyTag)));

 PCollection<T> finalResultCollection = maxAndMean
     .apply(ParDo.of(
         new DoFn<KV<K, CoCombineResult>, T>() {
           public void processElement(ProcessContext c) throws Exception {
             KV<K, CoCombineResult> e = c.element();
             Integer maxLatency = e.getValue().get(maxLatencyTag);
             Double meanLatency = e.getValue().get(meanLatencyTag);
             .... Do Something ....
             c.output(...some T...);
           }
         }));
 

compose

public static CombineFns.ComposeCombineFnBuilder compose()

Returns a CombineFns.ComposeCombineFnBuilder to construct a composed CombineFnBase.GlobalCombineFn.

The same TupleTag cannot be used in a composition multiple times.

Example:

 PCollection<Integer> globalLatencies = ...;

 TupleTag<Integer> maxLatencyTag = new TupleTag<Integer>();
 TupleTag<Double> meanLatencyTag = new TupleTag<Double>();

 SimpleFunction<Integer, Integer> identityFn =
     new SimpleFunction<Integer, Integer>() {
       public Integer apply(Integer input) {
           return input;
       }};
 PCollection<CoCombineResult> maxAndMean = globalLatencies.apply(
     Combine.globally(
         CombineFns.compose()
            .with(identityFn, new MaxIntegerFn(), maxLatencyTag)
            .with(identityFn, new MeanFn<Integer>(), meanLatencyTag)));

 PCollection<T> finalResultCollection = maxAndMean
     .apply(ParDo.of(
         new DoFn<CoCombineResult, T>() {
           public void processElement(ProcessContext c) throws Exception {
             CoCombineResult e = c.element();
             Integer maxLatency = e.get(maxLatencyTag);
             Double meanLatency = e.get(meanLatencyTag);
             .... Do Something ....
             c.output(...some T...);
           }
         }));