Uses of Interface
cloud.opencode.base.hash.simhash.Tokenizer

Packages that use Tokenizer

Package	Description
cloud.opencode.base.hash.simhash

Uses of Tokenizer in cloud.opencode.base.hash.simhash

Methods in cloud.opencode.base.hash.simhash that return Tokenizer

Modifier and Type	Method	Description
static Tokenizer	Tokenizer.characters()	Creates a character tokenizer 创建字符分词器
static Tokenizer	Tokenizer.cjkCharacters()	Creates a Chinese character tokenizer (single characters for CJK) 创建中文字符分词器（CJK单字符）
static Tokenizer	Tokenizer.combined(Tokenizer... tokenizers)	Combines multiple tokenizers 组合多个分词器
static Tokenizer	Tokenizer.ngram(int n)	Creates an N-gram tokenizer 创建 N-gram 分词器
static Tokenizer	Tokenizer.whitespace()	Creates a whitespace tokenizer 创建空格分词器
static Tokenizer	Tokenizer.words()	Creates a word boundary tokenizer (alphanumeric words) 创建单词边界分词器（字母数字单词）

Methods in cloud.opencode.base.hash.simhash with parameters of type Tokenizer

Modifier and Type	Method	Description
static Tokenizer	Tokenizer.combined(Tokenizer... tokenizers)	Combines multiple tokenizers 组合多个分词器
SimHashBuilder	SimHashBuilder.tokenizer(Tokenizer tokenizer)	Sets a Tokenizer instance 设置Tokenizer实例