Uses of Interface
cloud.opencode.base.hash.simhash.Tokenizer
Packages that use Tokenizer
-
Uses of Tokenizer in cloud.opencode.base.hash.simhash
Methods in cloud.opencode.base.hash.simhash that return TokenizerModifier and TypeMethodDescriptionstatic TokenizerTokenizer.characters()Creates a character tokenizer 创建字符分词器static TokenizerTokenizer.cjkCharacters()Creates a Chinese character tokenizer (single characters for CJK) 创建中文字符分词器(CJK单字符)static TokenizerCombines multiple tokenizers 组合多个分词器static TokenizerTokenizer.ngram(int n) Creates an N-gram tokenizer 创建 N-gram 分词器static TokenizerTokenizer.whitespace()Creates a whitespace tokenizer 创建空格分词器static TokenizerTokenizer.words()Creates a word boundary tokenizer (alphanumeric words) 创建单词边界分词器(字母数字单词)Methods in cloud.opencode.base.hash.simhash with parameters of type Tokenizer