com.google.cloud.dataflow.sdk.io

## Class UnboundedSource<OutputT,CheckpointMarkT extends UnboundedSource.CheckpointMark>

- java.lang.Object
- - com.google.cloud.dataflow.sdk.io.Source<OutputT>
  - - com.google.cloud.dataflow.sdk.io.UnboundedSource<OutputT,CheckpointMarkT>

Type Parameters:
  `OutputT` - Type of records output by this source.
  `CheckpointMarkT` - Type of checkpoint marks used by the readers of this source.

All Implemented Interfaces:
  HasDisplayData, Serializable

    public abstract class UnboundedSource<OutputT,CheckpointMarkT extends UnboundedSource.CheckpointMark>
    extends Source<OutputT>

A `Source` that reads an unbounded amount of input and, because of that, supports some additional operations such as checkpointing, watermarks, and record ids.

- Checkpointing allows sources to not re-read the same data again in the case of failures.
- Watermarks allow for downstream parts of the pipeline to know up to what point in time the data is complete.
- Record ids allow for efficient deduplication of input records; many streaming sources do not guarantee that a given record will only be read a single time.

See `Window` and `Trigger` for more information on timestamps and watermarks.

See Also:
  Serialized Form

### Nested Class Summary

Nested Classes
`static interface` `UnboundedSource.CheckpointMark` A marker representing the progress and state of an `UnboundedSource.UnboundedReader`.
`static class` `UnboundedSource.UnboundedReader<OutputT>` A `Reader` that reads an unbounded amount of input.

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

`Source.Reader<T>`

### Constructor Summary

Constructors
Constructor and Description
`UnboundedSource()`

### Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
`abstract UnboundedSource.UnboundedReader<OutputT>` `createReader(PipelineOptions options, CheckpointMarkT checkpointMark)` Create a new `UnboundedSource.UnboundedReader` to read from this source, resuming from the given checkpoint if present.
`abstract List<? extends UnboundedSource<OutputT,CheckpointMarkT>>` `generateInitialSplits(int desiredNumSplits, PipelineOptions options)` Returns a list of `UnboundedSource` objects representing the instances of this source that should be used when executing the workflow.
`abstract Coder<CheckpointMarkT>` `getCheckpointMarkCoder()` Returns a `Coder` for encoding and decoding the checkpoints for this source, or null if the checkpoints do not need to be durably committed.
`boolean` `requiresDeduping()` Returns whether this source requires explicit deduping.

### Methods inherited from class com.google.cloud.dataflow.sdk.io.Source

`getDefaultOutputCoder, populateDisplayData, validate`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### UnboundedSource

    public UnboundedSource()

### Method Detail

#### generateInitialSplits

    public abstract List<? extends UnboundedSource<OutputT,CheckpointMarkT>> generateInitialSplits(int desiredNumSplits,
                                                                                                   PipelineOptions options)
                                                                                            throws Exception

Returns a list of `UnboundedSource` objects representing the instances of this source that should be used when executing the workflow. Each split should return a separate partition of the input data.

For example, for a source reading from a growing directory of files, each split could correspond to a prefix of file names.

Some sources are not splittable, such as reading from a single TCP stream. In that case, only a single split should be returned.

Some data sources automatically partition their data among readers. For these types of inputs, `n` identical replicas of the top-level source can be returned.

The size of the returned list should be as close to `desiredNumSplits` as possible, but does not have to match exactly. A low number of splits will limit the amount of parallelism in the source.

Throws:
  `Exception`

#### createReader

    public abstract UnboundedSource.UnboundedReader<OutputT> createReader(PipelineOptions options,
                                                                          @Nullable
                                                                          CheckpointMarkT checkpointMark)

Create a new `UnboundedSource.UnboundedReader` to read from this source, resuming from the given checkpoint if present.

#### getCheckpointMarkCoder

    @Nullable
    public abstract Coder<CheckpointMarkT> getCheckpointMarkCoder()

Returns a `Coder` for encoding and decoding the checkpoints for this source, or null if the checkpoints do not need to be durably committed.

#### requiresDeduping

    public boolean requiresDeduping()

Returns whether this source requires explicit deduping.

This is needed if the underlying data source can return the same record multiple times, such a queuing system with a pull-ack model. Sources where the records read are uniquely identified by the persisted state in the CheckpointMark do not need this.