com.google.cloud.dataflow.sdk.util.common.worker

Class PartialGroupByKeyOperation.SamplingSizeEstimator<T>

java.lang.Object

com.google.cloud.dataflow.sdk.util.common.worker.PartialGroupByKeyOperation.SamplingSizeEstimator<T>

All Implemented Interfaces:

PartialGroupByKeyOperation.SizeEstimator<T>

Enclosing class:

PartialGroupByKeyOperation

public static class PartialGroupByKeyOperation.SamplingSizeEstimator<T>
extends Object
implements PartialGroupByKeyOperation.SizeEstimator<T>

Implements size estimation by adaptively delegating to an underlying (potentially more expensive) estimator for some elements and returning the average value for others.

Field Summary

Fields

Modifier and Type	Field and Description
static double	CONFIDENCE_INTERVAL_SIGMA The degree of confidence required in our expected value predictions before we allow under-sampling.
static double	CONFIDENCE_INTERVAL_SIZE The desired size of our confidence interval (relative to the measured expected value).
static long	DEFAULT_MIN_SAMPLED Default number of elements that must be measured before elements are skipped.

Constructor Summary

Constructors

Constructor and Description
SamplingSizeEstimator(PartialGroupByKeyOperation.SizeEstimator<T> underlying, double minSampleRate, double maxSampleRate)
SamplingSizeEstimator(PartialGroupByKeyOperation.SizeEstimator<T> underlying, double minSampleRate, double maxSampleRate, long minSampled, Random random)

Method Summary

Modifier and Type	Method and Description
long	estimateSize(T element)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONFIDENCE_INTERVAL_SIGMA

public static final double CONFIDENCE_INTERVAL_SIGMA

The degree of confidence required in our expected value predictions before we allow under-sampling.

The value of 3.0 is a confidence interval of about 99.7% for a a high-degree-of-freedom t-distribution.

See Also:

Constant Field Values

CONFIDENCE_INTERVAL_SIZE

public static final double CONFIDENCE_INTERVAL_SIZE

The desired size of our confidence interval (relative to the measured expected value).

The value of 0.25 is plus or minus 25%.

See Also:

Constant Field Values

DEFAULT_MIN_SAMPLED

public static final long DEFAULT_MIN_SAMPLED

Default number of elements that must be measured before elements are skipped.

See Also:

Constant Field Values

Constructor Detail

SamplingSizeEstimator

public SamplingSizeEstimator(PartialGroupByKeyOperation.SizeEstimator<T> underlying,
                             double minSampleRate,
                             double maxSampleRate)

SamplingSizeEstimator

public SamplingSizeEstimator(PartialGroupByKeyOperation.SizeEstimator<T> underlying,
                             double minSampleRate,
                             double maxSampleRate,
                             long minSampled,
                             Random random)

Method Detail

estimateSize

public long estimateSize(T element)
                  throws Exception

Specified by:

estimateSize in interface PartialGroupByKeyOperation.SizeEstimator<T>

Throws:

Exception