Class ChineseTokenizer

Tokenize Chinese text as individual chinese characters.

The difference between ChineseTokenizer and CJKTokenizer is that they have different token parsing logic.

For example, if the Chinese text "C1C2C3C4" is to be indexed:

  • The tokens returned from ChineseTokenizer are C1, C2, C3, C4.
  • The tokens returned from the CJKTokenizer are C1C2, C2C3, C3C4.

Therefore the index created by CJKTokenizer is much larger.

The problem is that when searching for C1, C1C2, C1C3, C4C2, C1C2C3 ... the ChineseTokenizer works, but the CJKTokenizer will not work.

Inheritance
System.Object
AttributeSource
TokenStream
Tokenizer
ChineseTokenizer
Inherited Members
Lucene.Net.Analysis.Tokenizer.SetReader(System.IO.TextReader)
System.Object.Equals(System.Object, System.Object)
System.Object.ReferenceEquals(System.Object, System.Object)
System.Object.GetType()
System.Object.MemberwiseClone()
Assembly: Lucene.Net.Analysis.Common.dll
Syntax
[Obsolete("(3.1) Use StandardTokenizer instead, which has the same functionality.")]
[Serializable]
public sealed class ChineseTokenizer : Tokenizer, IDisposable

Constructors

Name Description
ChineseTokenizer(AttributeSource.AttributeFactory, TextReader)
ChineseTokenizer(TextReader)

Methods

Name Description
End()
IncrementToken()
Reset()

Extension Methods