com.google.cloud.dataflow.sdk.util.common.worker

Class PartialGroupByKeyOperation

java.lang.Object

com.google.cloud.dataflow.sdk.util.common.worker.Operation

com.google.cloud.dataflow.sdk.util.common.worker.ReceivingOperation

com.google.cloud.dataflow.sdk.util.common.worker.PartialGroupByKeyOperation

All Implemented Interfaces:

Receiver

public class PartialGroupByKeyOperation
extends ReceivingOperation

A partial group-by-key operation.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PartialGroupByKeyOperation.BufferingGroupingTable<K,V> A grouping table that simply buffers all inserted values in a list.
static interface	PartialGroupByKeyOperation.Combiner<K,InputT,AccumT,OutputT> Provides client-specific operations for combining values.
static class	PartialGroupByKeyOperation.CombiningGroupingTable<K,InputT,AccumT> A grouping table that uses the given combiner to combine values in place.
static interface	PartialGroupByKeyOperation.GroupingKeyCreator<K> Provides client-specific operations for grouping keys.
static interface	PartialGroupByKeyOperation.PairInfo Provides client-specific operations for working with elements that are key/value or key/values pairs.
static class	PartialGroupByKeyOperation.SamplingSizeEstimator<T> Implements size estimation by adaptively delegating to an underlying (potentially more expensive) estimator for some elements and returning the average value for others.
static interface	PartialGroupByKeyOperation.SizeEstimator<T> Provides client-specific operations for size estimates.
static class	PartialGroupByKeyOperation.StructuralByteArray A wrapper around a byte[] that uses structural, value-based equality rather than byte[]'s normal object identity.

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Operation

Operation.InitializationState

Field Summary

Fields inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Operation

finishState, initializationState, initializationStateLock, operationName, processState, receivers, startState, stateSampler

Constructor Summary

Constructors

Constructor and Description
PartialGroupByKeyOperation(PartialGroupByKeyOperation.GroupingKeyCreator<?> groupingKeyCreator, PartialGroupByKeyOperation.SizeEstimator<?> keySizeEstimator, PartialGroupByKeyOperation.SizeEstimator<?> valueSizeEstimator, PartialGroupByKeyOperation.Combiner combineFn, PartialGroupByKeyOperation.PairInfo pairInfo, OutputReceiver outputReceiver, String counterPrefix, CounterSet.AddCounterMutator addCounterMutator, StateSampler stateSampler) Invoked by tests.
PartialGroupByKeyOperation(PartialGroupByKeyOperation.GroupingKeyCreator<?> groupingKeyCreator, PartialGroupByKeyOperation.SizeEstimator<?> keySizeEstimator, PartialGroupByKeyOperation.SizeEstimator<?> valueSizeEstimator, PartialGroupByKeyOperation.PairInfo pairInfo, OutputReceiver outputReceiver, String counterPrefix, CounterSet.AddCounterMutator addCounterMutator, StateSampler stateSampler) Invoked by tests.
PartialGroupByKeyOperation(String operationName, PartialGroupByKeyOperation.GroupingKeyCreator<?> groupingKeyCreator, PartialGroupByKeyOperation.SizeEstimator<?> keySizeEstimator, PartialGroupByKeyOperation.SizeEstimator<?> valueSizeEstimator, double sizeEstimatorSampleRate, PartialGroupByKeyOperation.Combiner combineFn, PartialGroupByKeyOperation.PairInfo pairInfo, OutputReceiver[] receivers, String counterPrefix, CounterSet.AddCounterMutator addCounterMutator, StateSampler stateSampler)
PartialGroupByKeyOperation(String operationName, PartialGroupByKeyOperation.GroupingKeyCreator<?> groupingKeyCreator, PartialGroupByKeyOperation.SizeEstimator<?> keySizeEstimator, PartialGroupByKeyOperation.SizeEstimator<?> valueSizeEstimator, PartialGroupByKeyOperation.Combiner combineFn, PartialGroupByKeyOperation.PairInfo pairInfo, OutputReceiver[] receivers, String counterPrefix, CounterSet.AddCounterMutator addCounterMutator, StateSampler stateSampler)
PartialGroupByKeyOperation(String operationName, PartialGroupByKeyOperation.GroupingKeyCreator<?> groupingKeyCreator, PartialGroupByKeyOperation.SizeEstimator<?> keySizeEstimator, PartialGroupByKeyOperation.SizeEstimator<?> valueSizeEstimator, PartialGroupByKeyOperation.PairInfo pairInfo, OutputReceiver[] receivers, String counterPrefix, CounterSet.AddCounterMutator addCounterMutator, StateSampler stateSampler)

Method Summary

Modifier and Type	Method and Description
void	finish() Finishes this Operation's execution.
long	getGroupingTableBytes() Returns the amount of memory the grouping table currently consumes.
void	process(Object elem) Processes the element.
void	setMaxGroupingTableBytes(long maxSize) Sets the maximum amount of memory the grouping table is allowed to consume before it has to be flushed.
boolean	supportsRestart() Returns true if this Operation can be started again after it is finished.

Methods inherited from class com.google.cloud.dataflow.sdk.util.common.worker.ReceivingOperation

attachInput

Methods inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Operation

start

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PartialGroupByKeyOperation

public PartialGroupByKeyOperation(String operationName,
                                  PartialGroupByKeyOperation.GroupingKeyCreator<?> groupingKeyCreator,
                                  PartialGroupByKeyOperation.SizeEstimator<?> keySizeEstimator,
                                  PartialGroupByKeyOperation.SizeEstimator<?> valueSizeEstimator,
                                  PartialGroupByKeyOperation.PairInfo pairInfo,
                                  OutputReceiver[] receivers,
                                  String counterPrefix,
                                  CounterSet.AddCounterMutator addCounterMutator,
                                  StateSampler stateSampler)

PartialGroupByKeyOperation

public PartialGroupByKeyOperation(String operationName,
                                  PartialGroupByKeyOperation.GroupingKeyCreator<?> groupingKeyCreator,
                                  PartialGroupByKeyOperation.SizeEstimator<?> keySizeEstimator,
                                  PartialGroupByKeyOperation.SizeEstimator<?> valueSizeEstimator,
                                  PartialGroupByKeyOperation.Combiner combineFn,
                                  PartialGroupByKeyOperation.PairInfo pairInfo,
                                  OutputReceiver[] receivers,
                                  String counterPrefix,
                                  CounterSet.AddCounterMutator addCounterMutator,
                                  StateSampler stateSampler)

PartialGroupByKeyOperation

public PartialGroupByKeyOperation(String operationName,
                                  PartialGroupByKeyOperation.GroupingKeyCreator<?> groupingKeyCreator,
                                  PartialGroupByKeyOperation.SizeEstimator<?> keySizeEstimator,
                                  PartialGroupByKeyOperation.SizeEstimator<?> valueSizeEstimator,
                                  double sizeEstimatorSampleRate,
                                  PartialGroupByKeyOperation.Combiner combineFn,
                                  PartialGroupByKeyOperation.PairInfo pairInfo,
                                  OutputReceiver[] receivers,
                                  String counterPrefix,
                                  CounterSet.AddCounterMutator addCounterMutator,
                                  StateSampler stateSampler)

PartialGroupByKeyOperation

public PartialGroupByKeyOperation(PartialGroupByKeyOperation.GroupingKeyCreator<?> groupingKeyCreator,
                                  PartialGroupByKeyOperation.SizeEstimator<?> keySizeEstimator,
                                  PartialGroupByKeyOperation.SizeEstimator<?> valueSizeEstimator,
                                  PartialGroupByKeyOperation.PairInfo pairInfo,
                                  OutputReceiver outputReceiver,
                                  String counterPrefix,
                                  CounterSet.AddCounterMutator addCounterMutator,
                                  StateSampler stateSampler)

Invoked by tests.

PartialGroupByKeyOperation

public PartialGroupByKeyOperation(PartialGroupByKeyOperation.GroupingKeyCreator<?> groupingKeyCreator,
                                  PartialGroupByKeyOperation.SizeEstimator<?> keySizeEstimator,
                                  PartialGroupByKeyOperation.SizeEstimator<?> valueSizeEstimator,
                                  PartialGroupByKeyOperation.Combiner combineFn,
                                  PartialGroupByKeyOperation.PairInfo pairInfo,
                                  OutputReceiver outputReceiver,
                                  String counterPrefix,
                                  CounterSet.AddCounterMutator addCounterMutator,
                                  StateSampler stateSampler)

Invoked by tests.

Method Detail

process

public void process(Object elem)
             throws Exception

Description copied from interface: Receiver

Processes the element.

Throws:

Exception

finish

public void finish()
            throws Exception

Description copied from class: Operation

Finishes this Operation's execution. Called after all predecessor producing operations have been finished.

Overrides:

finish in class Operation

Throws:

Exception

supportsRestart

public boolean supportsRestart()

Description copied from class: Operation

Returns true if this Operation can be started again after it is finished.

Overrides:

supportsRestart in class Operation

setMaxGroupingTableBytes

public void setMaxGroupingTableBytes(long maxSize)

Sets the maximum amount of memory the grouping table is allowed to consume before it has to be flushed.

getGroupingTableBytes

public long getGroupingTableBytes()

Returns the amount of memory the grouping table currently consumes.