com.google.cloud.dataflow.sdk.io

## Class OffsetBasedSource<T>

- java.lang.Object
- - com.google.cloud.dataflow.sdk.io.Source<T>
  - - com.google.cloud.dataflow.sdk.io.BoundedSource<T>
    - - com.google.cloud.dataflow.sdk.io.OffsetBasedSource<T>

Type Parameters:
  `T` - Type of records represented by the source.

All Implemented Interfaces:
  HasDisplayData, Serializable

Direct Known Subclasses:
  FileBasedSource

    public abstract class OffsetBasedSource<T>
    extends BoundedSource<T>

A `BoundedSource` that uses offsets to define starting and ending positions.

`OffsetBasedSource` is a common base class for all bounded sources where the input can be represented as a single range, and an input can be efficiently processed in parallel by splitting the range into a set of disjoint ranges whose union is the original range. This class should be used for sources that can be cheaply read starting at any given offset. `OffsetBasedSource` stores the range and implements splitting into bundles.

Extend `OffsetBasedSource` to implement your own offset-based custom source. `FileBasedSource`, which is a subclass of this, adds additional functionality useful for custom sources that are based on files. If possible implementors should start from `FileBasedSource` instead of `OffsetBasedSource`.

Consult `RangeTracker` for important semantics common to all sources defined by a range of positions of a certain type, including the semantics of split points (`OffsetBasedSource.OffsetBasedReader.isAtSplitPoint()`).

See Also:
  `BoundedSource`, `FileBasedSource`, `RangeTracker`, Serialized Form

### Nested Class Summary

Nested Classes
`static class` `OffsetBasedSource.OffsetBasedReader<T>` A `Source.Reader` that implements code common to readers of all `OffsetBasedSource`s.

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

`BoundedSource.BoundedReader<T>`

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

`Source.Reader<T>`

### Constructor Summary

Constructors
Constructor and Description
`OffsetBasedSource(long startOffset, long endOffset, long minBundleSize)`

### Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
`boolean` `allowsDynamicSplitting()` Whether this source should allow dynamic splitting of the offset ranges.
`abstract OffsetBasedSource<T>` `createSourceForSubrange(long start, long end)` Returns an `OffsetBasedSource` for a subrange of the current source.
`long` `getBytesPerOffset()` Returns approximately how many bytes of data correspond to a single offset in this source.
`long` `getEndOffset()` Returns the specified ending offset of the source.
`long` `getEstimatedSizeBytes(PipelineOptions options)` An estimate of the total size (in bytes) of the data that would be read from this source.
`abstract long` `getMaxEndOffset(PipelineOptions options)` Returns the actual ending offset of the current source.
`long` `getMinBundleSize()` Returns the minimum bundle size that should be used when splitting the source into sub-sources.
`long` `getStartOffset()` Returns the starting offset of the source.
`void` `populateDisplayData(DisplayData.Builder builder)` Register display data for the given transform or component.
`List<? extends OffsetBasedSource<T>>` `splitIntoBundles(long desiredBundleSizeBytes, PipelineOptions options)` Splits the source into bundles of approximately `desiredBundleSizeBytes`.
`String` `toString()`
`void` `validate()` Checks that this source is valid, before it can be used in a pipeline.

### Methods inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

`createReader, producesSortedKeys`

### Methods inherited from class com.google.cloud.dataflow.sdk.io.Source

`getDefaultOutputCoder`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Constructor Detail

#### OffsetBasedSource

    public OffsetBasedSource(long startOffset,
                             long endOffset,
                             long minBundleSize)

Parameters:
  `startOffset` - starting offset (inclusive) of the source. Must be non-negative.
  `endOffset` - ending offset (exclusive) of the source. Use `Long.MAX_VALUE` to indicate that the entire source after `startOffset` should be read. Must be `> startOffset`.
  `minBundleSize` - minimum bundle size in offset units that should be used when splitting the source into sub-sources. This value may not be respected if the total range of the source is smaller than the specified `minBundleSize`. Must be non-negative.

### Method Detail

#### getStartOffset

    public long getStartOffset()

Returns the starting offset of the source.

#### getEndOffset

    public long getEndOffset()

Returns the specified ending offset of the source. Any returned value greater than or equal to `getMaxEndOffset(PipelineOptions)` should be treated as `getMaxEndOffset(PipelineOptions)`.

#### getMinBundleSize

    public long getMinBundleSize()

Returns the minimum bundle size that should be used when splitting the source into sub-sources. This value may not be respected if the total range of the source is smaller than the specified `minBundleSize`.

#### getEstimatedSizeBytes

    public long getEstimatedSizeBytes(PipelineOptions options)
                               throws Exception

Description copied from class: `BoundedSource`
An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before any decompression or other processing done by the reader.

Specified by:
  `getEstimatedSizeBytes` in class `BoundedSource<T>`
Throws:
  `Exception`

#### splitIntoBundles

    public List<? extends OffsetBasedSource<T>> splitIntoBundles(long desiredBundleSizeBytes,
                                                                 PipelineOptions options)
                                                          throws Exception

Description copied from class: `BoundedSource`
Splits the source into bundles of approximately `desiredBundleSizeBytes`.

Specified by:
  `splitIntoBundles` in class `BoundedSource<T>`
Throws:
  `Exception`

#### validate

    public void validate()

Description copied from class: `Source`
Checks that this source is valid, before it can be used in a pipeline.

It is recommended to use `Preconditions` for implementing this method.

Specified by:
  `validate` in class `Source<T>`

#### toString

    public String toString()

Overrides:
  `toString` in class `Object`

#### getBytesPerOffset

    public long getBytesPerOffset()

Returns approximately how many bytes of data correspond to a single offset in this source. Used for translation between this source's range and methods defined in terms of bytes, such as `getEstimatedSizeBytes(com.google.cloud.dataflow.sdk.options.PipelineOptions)` and `splitIntoBundles(long, com.google.cloud.dataflow.sdk.options.PipelineOptions)`.

Defaults to `1` byte, which is the common case for, e.g., file sources.

#### getMaxEndOffset

    public abstract long getMaxEndOffset(PipelineOptions options)
                                  throws Exception

Returns the actual ending offset of the current source. The value returned by this function will be used to clip the end of the range `[startOffset, endOffset)` such that the range used is `[startOffset, min(endOffset, maxEndOffset))`.

As an example in which `OffsetBasedSource` is used to implement a file source, suppose that this source was constructed with an `endOffset` of `Long.MAX_VALUE` to indicate that a file should be read to the end. Then this function should determine the actual, exact size of the file in bytes and return it.

Throws:
  `Exception`

#### createSourceForSubrange

    public abstract OffsetBasedSource<T> createSourceForSubrange(long start,
                                                                 long end)

Returns an `OffsetBasedSource` for a subrange of the current source. The subrange `[start, end)` must be within the range `[startOffset, endOffset)` of the current source, i.e. `startOffset <= start < end <= endOffset`.

#### allowsDynamicSplitting

    public boolean allowsDynamicSplitting()

Whether this source should allow dynamic splitting of the offset ranges.

True by default. Override this to return false if the source cannot support dynamic splitting correctly. If this returns false, `OffsetBasedSource.OffsetBasedReader.splitAtFraction(double)` will refuse all split requests.

#### populateDisplayData

    public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: `Source`
Register display data for the given transform or component.

`populateDisplayData(DisplayData.Builder)` is invoked by Pipeline runners to collect display data via `DisplayData.from(HasDisplayData)`. Implementations may call `super.populateDisplayData(builder)` in order to register display data in the current namespace, but should otherwise use `subcomponent.populateDisplayData(builder)` to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:
  `populateDisplayData` in interface `HasDisplayData`
Overrides:
  `populateDisplayData` in class `Source<T>`
Parameters:
  `builder` - The builder to populate with display data.
See Also:
  `HasDisplayData`