Class TokenizerUtil

java.lang.Object
cloud.opencode.base.string.parse.TokenizerUtil
public final class TokenizerUtil extends Object
Tokenizer Utility - Provides string tokenization methods. 分词器工具 - 提供字符串分词方法。

Features | 主要功能:

  • Default whitespace tokenization - 默认空白分词
  • Custom delimiter tokenization - 自定义分隔符分词

Usage Examples | 使用示例:

List<String> tokens = TokenizerUtil.tokenize("hello world"); // ["hello", "world"]
List<String> custom = TokenizerUtil.tokenize("a;b;c", ";");  // ["a", "b", "c"]

Security | 安全性:

  • Thread-safe: Yes (stateless utility) - 线程安全: 是（无状态工具类）
  • Null-safe: Yes - 空值安全: 是

Performance | 性能特性:

  • Time complexity: O(n) where n = input length - O(n), n为输入长度
  • Space complexity: O(t) where t = number of tokens - O(t), t为词元数
Since:
JDK 25, opencode-base-string V1.0.0
Author:
Leon Soo www.LeonSoo.com
See Also: