com.google.cloud.dataflow.sdk.io

## Class BlockBasedSource<T>

- java.lang.Object
- - com.google.cloud.dataflow.sdk.io.Source<T>
  - - com.google.cloud.dataflow.sdk.io.BoundedSource<T>
    - - com.google.cloud.dataflow.sdk.io.OffsetBasedSource<T>
      - - com.google.cloud.dataflow.sdk.io.FileBasedSource<T>
        - - com.google.cloud.dataflow.sdk.io.BlockBasedSource<T>

Type Parameters:
  `T` - The type of records to be read from the source.

All Implemented Interfaces:
  HasDisplayData, Serializable

Direct Known Subclasses:
  AvroSource

    @Experimental(value=SOURCE_SINK)
    public abstract class BlockBasedSource<T>
    extends FileBasedSource<T>

A `BlockBasedSource` is a `FileBasedSource` where a file consists of blocks of records.

`BlockBasedSource` should be derived from when a file format does not support efficient seeking to a record in the file, but can support efficient seeking to a block. Alternatively, records in the file cannot be offset-addressed, but blocks can (it is not possible to say that record {code i} starts at offset `m`, but it is possible to say that block `j` starts at offset `n`).

The records that will be read from a `BlockBasedSource` that corresponds to a subrange of a file `[startOffset, endOffset)` are those records such that the record is contained in a block that starts at offset `i`, where `i >= startOffset` and `i < endOffset`. In other words, a record will be read from the source if its first byte is contained in a block that begins within the range described by the source.

This entails that it is possible to determine the start offsets of all blocks in a file.

Progress reporting for reading from a `BlockBasedSource` is inaccurate. A `BlockBasedSource.BlockBasedReader` reports its current offset as `(offset of current block) + (current block size) * (fraction of block consumed)`. However, only the offset of the current block is required to be accurately reported by subclass implementations. As such, in the worst case, the current offset is only updated at block boundaries.

`BlockBasedSource` supports dynamic splitting. However, because records in a `BlockBasedSource` are not required to have offsets and progress reporting is inaccurate, `BlockBasedReader` only supports splitting at block boundaries. In other words, `BlockBasedSource.BlockBasedReader.atSplitPoint` returns true iff the current record is the first record in a block. See `FileBasedSource.FileBasedReader` for discussion about split points.

See Also:
  Serialized Form

### Nested Class Summary

Nested Classes
`protected static class` `BlockBasedSource.Block<T>` A `Block` represents a block of records that can be read.
`protected static class` `BlockBasedSource.BlockBasedReader<T>` A `Reader` that reads records from a `BlockBasedSource`.

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.FileBasedSource

`FileBasedSource.FileBasedReader<T>, FileBasedSource.Mode`

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.OffsetBasedSource

`OffsetBasedSource.OffsetBasedReader<T>`

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

`BoundedSource.BoundedReader<T>`

### Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

`Source.Reader<T>`

### Constructor Summary

Constructors
Constructor and Description
`BlockBasedSource(String fileOrPatternSpec, long minBundleSize)` Creates a `BlockBasedSource` based on a file name or pattern.
`BlockBasedSource(String fileName, long minBundleSize, long startOffset, long endOffset)` Creates a `BlockBasedSource` for a single file.

### Method Summary

All Methods Instance Methods Abstract Methods
`protected abstract BlockBasedSource<T>` `createForSubrangeOfFile(String fileName, long start, long end)` Creates a `BlockBasedSource` for the specified range in a single file.
`protected abstract BlockBasedSource.BlockBasedReader<T>` `createSingleFileReader(PipelineOptions options)` Creates a `BlockBasedReader`.

### Methods inherited from class com.google.cloud.dataflow.sdk.io.FileBasedSource

`createReader, createSourceForSubrange, expandFilePattern, getEstimatedSizeBytes, getFileOrPatternSpec, getMaxEndOffset, getMode, isSplittable, populateDisplayData, splitIntoBundles, toString, validate`

### Methods inherited from class com.google.cloud.dataflow.sdk.io.OffsetBasedSource

`allowsDynamicSplitting, getBytesPerOffset, getEndOffset, getMinBundleSize, getStartOffset`

### Methods inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

`producesSortedKeys`

### Methods inherited from class com.google.cloud.dataflow.sdk.io.Source

`getDefaultOutputCoder`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Constructor Detail

#### BlockBasedSource

    public BlockBasedSource(String fileOrPatternSpec,
                            long minBundleSize)

Creates a `BlockBasedSource` based on a file name or pattern. Subclasses must call this constructor when creating a `BlockBasedSource` for a file pattern. See `FileBasedSource` for more information.

#### BlockBasedSource

    public BlockBasedSource(String fileName,
                            long minBundleSize,
                            long startOffset,
                            long endOffset)

Creates a `BlockBasedSource` for a single file. Subclasses must call this constructor when implementing `createForSubrangeOfFile(java.lang.String, long, long)`. See documentation in `FileBasedSource`.

### Method Detail

#### createForSubrangeOfFile

    protected abstract BlockBasedSource<T> createForSubrangeOfFile(String fileName,
                                                                   long start,
                                                                   long end)

Creates a `BlockBasedSource` for the specified range in a single file.

Specified by:
  `createForSubrangeOfFile` in class `FileBasedSource<T>`
Parameters:
  `fileName` - file backing the new `FileBasedSource`.
  `start` - starting byte offset of the new `FileBasedSource`.
  `end` - ending byte offset of the new `FileBasedSource`. May be Long.MAX_VALUE, in which case it will be inferred using `FileBasedSource.getMaxEndOffset(com.google.cloud.dataflow.sdk.options.PipelineOptions)`.

#### createSingleFileReader

    protected abstract BlockBasedSource.BlockBasedReader<T> createSingleFileReader(PipelineOptions options)

Creates a `BlockBasedReader`.

Specified by:
  `createSingleFileReader` in class `FileBasedSource<T>`