Class Tokenizer

java.lang.Object
cloud.opencode.base.expression.parser.Tokenizer
public class Tokenizer extends Object
Expression Tokenizer 表达式词法分析器

Tokenizes expression strings into a list of tokens.

将表达式字符串分解为词法单元列表。

Features | 主要功能:

  • String literals with escape sequences (\n, \t, \\, \', \") - 带转义序列的字符串字面量
  • Number literals with decimal, scientific notation, and type suffixes (L, D, F) - 带小数、科学记数法和类型后缀的数字字面量
  • Keyword recognition: true, false, null, and, or, not, matches, instanceof - 关键字识别
  • Multi-character operators: ==, !=, <=, >=, **, &&, ||, ?., .?[, .![ - 多字符运算符

Usage Examples | 使用示例:

List<Token> tokens = Tokenizer.tokenize("price * 1.1 + tax");
// tokens: [IDENTIFIER("price"), STAR, NUMBER(1.1), PLUS, IDENTIFIER("tax"), EOF]

// Or instance-based
Tokenizer tokenizer = new Tokenizer("a + b");
List<Token> tokens = tokenizer.tokenize();

Security | 安全性:

  • Thread-safe: No, stateful tokenizer instance - 线程安全: 否，有状态的词法分析器实例
  • Null-safe: Yes, null expression treated as empty string - 空值安全: 是，null表达式视为空字符串

Performance | 性能特性:

  • Time complexity: O(n) for tokenize where n is the expression length - 时间复杂度: tokenize 为 O(n)，n为表达式长度
  • Space complexity: O(n) for the token list - 空间复杂度: O(n)，存储词法单元列表
Since:
JDK 25, opencode-base-expression V1.0.0
Author:
Leon Soo www.LeonSoo.com
See Also:

  • Constructor Details

    • Tokenizer

      public Tokenizer(String expression)
      Create tokenizer for expression 为表达式创建词法分析器
      Parameters:
      expression - the expression | 表达式

  • Method Details

    • tokenize

      public List<Token> tokenize()
      Tokenize the expression 对表达式进行词法分析
      Returns:
      the token list | 词法单元列表

    • tokenize

      public static List<Token> tokenize(String expression)
      Create tokenizer and tokenize 创建词法分析器并进行分析
      Parameters:
      expression - the expression | 表达式
      Returns:
      the token list | 词法单元列表