com.google.cloud.dataflow.sdk.io

## Class UnboundedSource.UnboundedReader<OutputT>

- java.lang.Object
- - com.google.cloud.dataflow.sdk.io.Source.Reader<OutputT>
  - - com.google.cloud.dataflow.sdk.io.UnboundedSource.UnboundedReader<OutputT>

All Implemented Interfaces:
  AutoCloseable

Enclosing class:
  UnboundedSource<OutputT,CheckpointMarkT extends UnboundedSource.CheckpointMark>

    @Experimental(value=SOURCE_SINK)
    public abstract static class UnboundedSource.UnboundedReader<OutputT>
    extends Source.Reader<OutputT>

A `Reader` that reads an unbounded amount of input.

A given `UnboundedReader` object will only be accessed by a single thread at once.

### Field Summary

Fields
`static long` `BACKLOG_UNKNOWN` Constant representing an unknown amount of backlog.

### Constructor Summary

Constructors
Constructor and Description
`UnboundedReader()`

### Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
`abstract boolean` `advance()` Advances the reader to the next valid record.
`abstract UnboundedSource.CheckpointMark` `getCheckpointMark()` Returns a `UnboundedSource.CheckpointMark` representing the progress of this `UnboundedReader`.
`byte[]` `getCurrentRecordId()` Returns a unique identifier for the current record.
`abstract UnboundedSource<OutputT,?>` `getCurrentSource()` Returns the `UnboundedSource` that created this reader.
`long` `getSplitBacklogBytes()` Returns the size of the backlog of unread data in the underlying data source represented by this split of this source.
`long` `getTotalBacklogBytes()` Returns the size of the backlog of unread data in the underlying data source represented by all splits of this source.
`abstract Instant` `getWatermark()` Returns a timestamp before or at the timestamps of all future elements read by this reader.
`abstract boolean` `start()` Initializes the reader and advances the reader to the first record.

### Methods inherited from class com.google.cloud.dataflow.sdk.io.Source.Reader

`close, getCurrent, getCurrentTimestamp`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### BACKLOG_UNKNOWN

    public static final long BACKLOG_UNKNOWN

Constant representing an unknown amount of backlog.

See Also:
  Constant Field Values

### Constructor Detail

#### UnboundedReader

    public UnboundedReader()

### Method Detail

#### start

    public abstract boolean start()
                           throws IOException

Initializes the reader and advances the reader to the first record.

This method should be called exactly once. The invocation should occur prior to calling `advance()` or `Source.Reader.getCurrent()`. This method may perform expensive operations that are needed to initialize the reader.

Returns `true` if a record was read, `false` if there is no more input currently available. Future calls to `advance()` may return `true` once more data is available. Regardless of the return value of `start`, `start` will not be called again on the same `UnboundedReader` object; it will only be called again when a new reader object is constructed for the same source, e.g. on recovery.

Specified by:
  `start` in class `Source.Reader<OutputT>`
Returns:
  `true` if a record was read, `false` if there is no more input available.
Throws:
  `IOException`

#### advance

    public abstract boolean advance()
                             throws IOException

Advances the reader to the next valid record.

Returns `true` if a record was read, `false` if there is no more input available. Future calls to `advance()` may return `true` once more data is available.

Specified by:
  `advance` in class `Source.Reader<OutputT>`
Returns:
  `true` if a record was read, `false` if there is no more input available.
Throws:
  `IOException`

#### getCurrentRecordId

    public byte[] getCurrentRecordId()
                              throws NoSuchElementException

Returns a unique identifier for the current record. This should be the same for each instance of the same logical record read from the underlying data source.

It is only necessary to override this if `UnboundedSource.requiresDeduping()` has been overridden to return true.

For example, this could be a hash of the record contents, or a logical ID present in the record. If this is generated as a hash of the record contents, it should be at least 16 bytes (128 bits) to avoid collisions.

This method has the same restrictions on when it can be called as `Source.Reader.getCurrent()` and `Source.Reader.getCurrentTimestamp()`.

Throws:
  `NoSuchElementException` - if the reader is at the beginning of the input and `start()` or `advance()` wasn't called, or if the last `start()` or `advance()` returned `false`.

#### getWatermark

    public abstract Instant getWatermark()

Returns a timestamp before or at the timestamps of all future elements read by this reader.

This can be approximate. If records are read that violate this guarantee, they will be considered late, which will affect how they will be processed. See `Window` for more information on late data and how to handle it.

However, this value should be as late as possible. Downstream windows may not be able to close until this watermark passes their end.

For example, a source may know that the records it reads will be in timestamp order. In this case, the watermark can be the timestamp of the last record read. For a source that does not have natural timestamps, timestamps can be set to the time of reading, in which case the watermark is the current clock time.

See `Window` and `Trigger` for more information on timestamps and watermarks.

May be called after `advance()` or `start()` has returned false, but not before `start()` has been called.

#### getCheckpointMark

    public abstract UnboundedSource.CheckpointMark getCheckpointMark()

Returns a `UnboundedSource.CheckpointMark` representing the progress of this `UnboundedReader`.

The elements read up until this is called will be processed together as a bundle. Once the result of this processing has been durably committed, `UnboundedSource.CheckpointMark.finalizeCheckpoint()` will be called on the `UnboundedSource.CheckpointMark` object.

The returned object should not be modified.

May be called after `advance()` or `start()` has returned false, but not before `start()` has been called.

#### getSplitBacklogBytes

    public long getSplitBacklogBytes()

Returns the size of the backlog of unread data in the underlying data source represented by this split of this source.

One of this or `getTotalBacklogBytes()` should be overridden in order to allow the runner to scale the amount of resources allocated to the pipeline.

#### getTotalBacklogBytes

    public long getTotalBacklogBytes()

Returns the size of the backlog of unread data in the underlying data source represented by all splits of this source.

One of this or `getSplitBacklogBytes()` should be overridden in order to allow the runner to scale the amount of resources allocated to the pipeline.

#### getCurrentSource

    public abstract UnboundedSource<OutputT,?> getCurrentSource()

Returns the `UnboundedSource` that created this reader. This will not change over the life of the reader.

Specified by:
  `getCurrentSource` in class `Source.Reader<OutputT>`