com.google.cloud.dataflow.sdk.runners.dataflow

Class BasicSerializableSourceFormat

java.lang.Object

com.google.cloud.dataflow.sdk.runners.dataflow.BasicSerializableSourceFormat

All Implemented Interfaces:

SourceFormat

public class BasicSerializableSourceFormat
extends Object
implements SourceFormat

A helper class for supporting sources defined as Source.

Provides a bridge between the high-level Source API and the raw API-level SourceFormat API, by encoding the serialized Source in a parameter of the API Source message.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BasicSerializableSourceFormat.BoundedSourceSplit<T> A DynamicSplitResult specified explicitly by a pair of BoundedSource objects describing the primary and residual sources.

Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.util.common.worker.SourceFormat

SourceFormat.OperationRequest, SourceFormat.OperationResponse, SourceFormat.SourceSpec

Field Summary

Fields

Modifier and Type	Field and Description
static String	TOO_MANY_SOURCE_SPLITS_ERROR

Constructor Summary

Constructors

Constructor and Description
BasicSerializableSourceFormat(PipelineOptions options)

Method Summary

Modifier and Type	Method and Description
static <T> Reader<WindowedValue<T>>	create(PipelineOptions options, CloudObject spec, Coder<WindowedValue<T>> coder, ExecutionContext executionContext) Factory method allowing this class to satisfy the implicit contract of ReaderFactory.
static Source<?>	deserializeFromCloudSource(Map<String,Object> spec)
static <T> void	evaluateReadHelper(Read.Bounded<T> transform, DirectPipelineRunner.EvaluationContext context)
SourceFormat.OperationResponse	performSourceOperation(SourceFormat.OperationRequest request) Executes a protocol-level split SourceOperationRequest for bounded sources by deserializing its source to a BoundedSource, splitting it, and serializing results back.
static com.google.api.services.dataflow.model.Source	serializeToCloudSource(Source<?> source, PipelineOptions options)
static com.google.api.services.dataflow.model.DynamicSourceSplit	toSourceSplit(BasicSerializableSourceFormat.BoundedSourceSplit<?> sourceSplitResult, PipelineOptions options)
static <T> void	translateReadHelper(Source<T> source, PTransform<?,? extends PValue> transform, DataflowPipelineTranslator.TranslationContext context)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TOO_MANY_SOURCE_SPLITS_ERROR

public static final String TOO_MANY_SOURCE_SPLITS_ERROR

See Also:

Constant Field Values

Constructor Detail

BasicSerializableSourceFormat

public BasicSerializableSourceFormat(PipelineOptions options)

Method Detail

toSourceSplit

public static com.google.api.services.dataflow.model.DynamicSourceSplit toSourceSplit(BasicSerializableSourceFormat.BoundedSourceSplit<?> sourceSplitResult,
                                                                                      PipelineOptions options)

performSourceOperation

public SourceFormat.OperationResponse performSourceOperation(SourceFormat.OperationRequest request)
                                                      throws Exception

Executes a protocol-level split SourceOperationRequest for bounded sources by deserializing its source to a BoundedSource, splitting it, and serializing results back.

Specified by:

performSourceOperation in interface SourceFormat

Throws:

Exception

create

public static <T> Reader<WindowedValue<T>> create(PipelineOptions options,
                                                  CloudObject spec,
                                                  Coder<WindowedValue<T>> coder,
                                                  ExecutionContext executionContext)
                                           throws Exception

Factory method allowing this class to satisfy the implicit contract of ReaderFactory.

Throws:

Exception

deserializeFromCloudSource

public static Source<?> deserializeFromCloudSource(Map<String,Object> spec)
                                            throws Exception

Throws:

Exception

serializeToCloudSource

public static com.google.api.services.dataflow.model.Source serializeToCloudSource(Source<?> source,
                                                                                   PipelineOptions options)
                                                                            throws Exception

Throws:

Exception

evaluateReadHelper

public static <T> void evaluateReadHelper(Read.Bounded<T> transform,
                                          DirectPipelineRunner.EvaluationContext context)

translateReadHelper

public static <T> void translateReadHelper(Source<T> source,
                                           PTransform<?,? extends PValue> transform,
                                           DataflowPipelineTranslator.TranslationContext context)