Class ChunkedFileProcessor

java.lang.Object
cloud.opencode.base.io.file.ChunkedFileProcessor
public final class ChunkedFileProcessor extends Object
Chunked File Processor - Memory-efficient large file processing 大文件分块处理器 - 内存高效的大文件处理

Provides memory-efficient processing of large files using chunked reading, memory mapping, and parallel processing with virtual threads.

提供使用分块读取、内存映射和虚拟线程并行处理的大文件内存高效处理。

Features | 主要功能:

  • Chunked reading with configurable buffer size - 可配置缓冲区大小的分块读取
  • Memory-mapped file processing - 内存映射文件处理
  • Parallel chunk processing with virtual threads - 虚拟线程并行块处理
  • Stream-based chunk iteration - 基于流的块迭代
  • Progress tracking and callbacks - 进度跟踪和回调
  • Resume support for interrupted processing - 中断处理的恢复支持

Usage Examples | 使用示例:

// Simple chunked processing
ChunkedFileProcessor.process(largePath, 1024 * 1024, chunk -> {
    // Process each 1MB chunk
    processData(chunk.data());
});

// With progress tracking
ChunkedFileProcessor.builder()
    .path(largePath)
    .chunkSize(8 * 1024 * 1024)  // 8MB chunks
    .onProgress((processed, total) -> {
        System.out.printf("Progress: %.1f%%\n", 100.0 * processed / total);
    })
    .process(chunk -> saveChunk(chunk));

// Parallel processing
List<Result> results = ChunkedFileProcessor.builder()
    .path(largePath)
    .parallel(true)
    .processAndCollect(chunk -> analyzeChunk(chunk));

// Stream-based iteration
try (Stream<Chunk> chunks = ChunkedFileProcessor.streamChunks(path, chunkSize)) {
    chunks.filter(c -> c.index() < 10)
          .forEach(this::process);
}

Security | 安全性:

  • Thread-safe: Yes, parallel mode uses virtual threads safely - 线程安全: 是，并行模式安全使用虚拟线程
  • Null-safe: No, path and processor must not be null - 空值安全: 否，路径和处理器不可为null

Performance | 性能特性:

  • Time complexity: O(n) where n=file size; parallel mode divides work across virtual threads for I/O-bound speedup - 时间复杂度: O(n)，n 为文件大小；并行模式通过虚拟线程分配工作以加速 I/O 操作
  • Space complexity: O(k) where k=chunk buffer size; memory-efficient as only one chunk (or p chunks in parallel) is loaded at a time - 空间复杂度: O(k)，k 为块缓冲区大小；内存高效，每次只加载一个块（并行时为 p 个块）
Since:
JDK 25, opencode-base-io V1.0.0
Author:
Leon Soo www.LeonSoo.com
See Also:

  • Field Details

    • DEFAULT_CHUNK_SIZE

      public static final int DEFAULT_CHUNK_SIZE
      Default chunk size: 4MB
      See Also:

    • MIN_CHUNK_SIZE

      public static final int MIN_CHUNK_SIZE
      Minimum chunk size: 4KB
      See Also:

    • MAX_MMAP_SIZE

      public static final long MAX_MMAP_SIZE
      Maximum chunk size for memory mapping: 1GB
      See Also:

  • Method Details

    • process

      public static void process(Path path, Consumer<ChunkedFileProcessor.Chunk> processor)
      Processes a file in chunks with the default chunk size. 使用默认块大小分块处理文件。
      Parameters:
      path - the file path - 文件路径
      processor - the chunk processor - 块处理器

    • process

      public static void process(Path path, int chunkSize, Consumer<ChunkedFileProcessor.Chunk> processor)
      Processes a file in chunks. 分块处理文件。
      Parameters:
      path - the file path - 文件路径
      chunkSize - the chunk size in bytes - 块大小（字节）
      processor - the chunk processor - 块处理器

    • streamChunks

      public static Stream<ChunkedFileProcessor.Chunk> streamChunks(Path path, int chunkSize)
      Creates a stream of chunks from a file. 从文件创建块流。
      Parameters:
      path - the file path - 文件路径
      chunkSize - the chunk size in bytes - 块大小（字节）
      Returns:
      a stream of chunks - 块流

    • countChunks

      public static long countChunks(Path path, int chunkSize)
      Counts the number of chunks in a file. 计算文件中的块数。
      Parameters:
      path - the file path - 文件路径
      chunkSize - the chunk size - 块大小
      Returns:
      the number of chunks - 块数

    • builder

      public static ChunkedFileProcessor.Builder builder()
      Creates a new builder. 创建新的构建器。
      Returns:
      the builder - 构建器

    • writeInChunks

      public static void writeInChunks(Path path, byte[] data, int chunkSize, BiConsumer<Long,Long> progressCallback)
      Writes data to a file in chunks. 分块写入数据到文件。
      Parameters:
      path - the file path - 文件路径
      data - the data to write - 要写入的数据
      chunkSize - the chunk size - 块大小
      progressCallback - optional progress callback - 可选的进度回调

    • copyInChunks

      public static void copyInChunks(Path source, Path target, int chunkSize, BiConsumer<Long,Long> progressCallback)
      Copies a file in chunks. 分块复制文件。
      Parameters:
      source - the source path - 源路径
      target - the target path - 目标路径
      chunkSize - the chunk size - 块大小
      progressCallback - optional progress callback - 可选的进度回调