com.google.cloud.dataflow.sdk.runners.worker

Class GroupingShuffleRangeTracker

java.lang.Object

com.google.cloud.dataflow.sdk.runners.worker.GroupingShuffleRangeTracker

All Implemented Interfaces:

RangeTracker<ByteArrayShufflePosition>

public class GroupingShuffleRangeTracker
extends Object
implements RangeTracker<ByteArrayShufflePosition>

A RangeTracker for positions used by GroupingShuffleReader (ByteArrayShufflePosition).

These positions roughly correspond to hashes of keys. In case of hash collisions, multiple groups can have the same position. In that case, the first group at a particular position is considered a split point (because it is the first to be returned when reading a position range starting at this position), others are not.

Constructor Summary

Constructors

Constructor and Description
GroupingShuffleRangeTracker(ByteArrayShufflePosition startPosition, ByteArrayShufflePosition stopPosition)

Method Summary

Modifier and Type	Method and Description
double	getFractionConsumed() Returns the approximate fraction of positions in the source that have been consumed by successful RangeTracker.tryReturnRecordAt(boolean, PositionT) calls, or 0.0 if no such calls have happened.
ByteArrayShufflePosition	getLastGroupStart()
ByteArrayShufflePosition	getStartPosition() Returns the starting position of the current range, inclusive.
ByteArrayShufflePosition	getStopPosition() Returns the ending position of the current range, exclusive.
String	toString()
boolean	tryReturnRecordAt(boolean isAtSplitPoint, ByteArrayShufflePosition groupStart) Atomically determines whether a record at the given position can be returned and updates internal state.
boolean	trySplitAtPosition(ByteArrayShufflePosition splitPosition) Atomically splits the current range [RangeTracker.getStartPosition(), RangeTracker.getStopPosition()) into a "primary" part [RangeTracker.getStartPosition(), splitPosition) and a "residual" part [splitPosition, RangeTracker.getStopPosition()), assuming the current last-consumed position is within [RangeTracker.getStartPosition(), splitPosition) (i.e., splitPosition has not been consumed yet).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

GroupingShuffleRangeTracker

public GroupingShuffleRangeTracker(@Nullable
                                   ByteArrayShufflePosition startPosition,
                                   @Nullable
                                   ByteArrayShufflePosition stopPosition)

Method Detail

getStartPosition

public ByteArrayShufflePosition getStartPosition()

Description copied from interface: RangeTracker

Returns the starting position of the current range, inclusive.

Specified by:

getStartPosition in interface RangeTracker<ByteArrayShufflePosition>

getStopPosition

public ByteArrayShufflePosition getStopPosition()

Description copied from interface: RangeTracker

Returns the ending position of the current range, exclusive.

Specified by:

getStopPosition in interface RangeTracker<ByteArrayShufflePosition>

getLastGroupStart

public ByteArrayShufflePosition getLastGroupStart()

tryReturnRecordAt

public boolean tryReturnRecordAt(boolean isAtSplitPoint,
                                 ByteArrayShufflePosition groupStart)

Description copied from interface: RangeTracker

Atomically determines whether a record at the given position can be returned and updates internal state. In particular:

• If isAtSplitPoint is true, and recordStart is outside the current range, returns false;
• Otherwise, updates the last-consumed position to recordStart and returns true.

This method MUST be called on all split point records. It may be called on every record.

Specified by:

tryReturnRecordAt in interface RangeTracker<ByteArrayShufflePosition>

trySplitAtPosition

public boolean trySplitAtPosition(ByteArrayShufflePosition splitPosition)

Description copied from interface: RangeTracker

Atomically splits the current range [RangeTracker.getStartPosition(), RangeTracker.getStopPosition()) into a "primary" part [RangeTracker.getStartPosition(), splitPosition) and a "residual" part [splitPosition, RangeTracker.getStopPosition()), assuming the current last-consumed position is within [RangeTracker.getStartPosition(), splitPosition) (i.e., splitPosition has not been consumed yet).

Updates the current range to be the primary and returns true. This means that all further calls on the current object will interpret their arguments relative to the primary range.

If the split position has already been consumed, or if no RangeTracker.tryReturnRecordAt(boolean, PositionT) call was made yet, returns false. The second condition is to prevent dynamic splitting during reader start-up.

Specified by:

trySplitAtPosition in interface RangeTracker<ByteArrayShufflePosition>

getFractionConsumed

public double getFractionConsumed()

Description copied from interface: RangeTracker

Returns the approximate fraction of positions in the source that have been consumed by successful RangeTracker.tryReturnRecordAt(boolean, PositionT) calls, or 0.0 if no such calls have happened.

Specified by:

getFractionConsumed in interface RangeTracker<ByteArrayShufflePosition>

toString

public String toString()

Overrides:

toString in class Object