com.google.cloud.dataflow.sdk.runners.worker

Class ShuffleSink<T>

java.lang.Object

com.google.cloud.dataflow.sdk.util.common.worker.Sink<WindowedValue<T>>

com.google.cloud.dataflow.sdk.runners.worker.ShuffleSink<T>

Type Parameters:

T - the type of the elements written to the sink

public class ShuffleSink<T>
extends Sink<WindowedValue<T>>

A sink that writes to a shuffle dataset.

Nested Class Summary

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Sink

Sink.SinkWriter<ElemT>

Constructor Summary

Constructors

Constructor and Description
ShuffleSink(PipelineOptions options, byte[] shuffleWriterConfig, com.google.cloud.dataflow.sdk.runners.worker.ShuffleSink.ShuffleKind shuffleKind, Coder<WindowedValue<T>> coder, CounterSet.AddCounterMutator addCounterMutator)

Method Summary

Modifier and Type	Method and Description
static com.google.cloud.dataflow.sdk.runners.worker.ShuffleSink.ShuffleKind	parseShuffleKind(String shuffleKind)
Sink.SinkWriter<WindowedValue<T>>	writer() Returns a Writer that allows writing to this Sink.
Sink.SinkWriter<WindowedValue<T>>	writer(com.google.cloud.dataflow.sdk.runners.worker.ShuffleEntryWriter writer, String datasetId) Returns a SinkWriter that allows writing to this ShuffleSink, using the given ShuffleEntryWriter.

Methods inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Sink

supportsRestart

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ShuffleSink

public ShuffleSink(PipelineOptions options,
                   byte[] shuffleWriterConfig,
                   com.google.cloud.dataflow.sdk.runners.worker.ShuffleSink.ShuffleKind shuffleKind,
                   Coder<WindowedValue<T>> coder,
                   CounterSet.AddCounterMutator addCounterMutator)
            throws Exception

Throws:

Exception

Method Detail

parseShuffleKind

public static com.google.cloud.dataflow.sdk.runners.worker.ShuffleSink.ShuffleKind parseShuffleKind(String shuffleKind)
                                                                                             throws Exception

Throws:

Exception

writer

public Sink.SinkWriter<WindowedValue<T>> writer(com.google.cloud.dataflow.sdk.runners.worker.ShuffleEntryWriter writer,
                                                String datasetId)

Returns a SinkWriter that allows writing to this ShuffleSink, using the given ShuffleEntryWriter. The dataset ID is used to construct names of counters that track per-worker per-dataset bytes written to shuffle.

writer

public Sink.SinkWriter<WindowedValue<T>> writer()
                                         throws IOException

Description copied from class: Sink

Returns a Writer that allows writing to this Sink.

Specified by:

writer in class Sink<WindowedValue<T>>

Throws:

IOException