Class LowerCaseTokenizer
LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.
Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.
You must specify the required LuceneVersion compatibility when creating LowerCaseTokenizer:
- As of 3.1, CharTokenizer uses an int based API to normalize and detect token characters. See IsTokenChar(Int32) and Normalize(Int32) for details.
Inheritance
Inherited Members
Assembly: Lucene.Net.Analysis.Common.dll
Syntax
[Serializable]
public sealed class LowerCaseTokenizer : LetterTokenizer, IDisposableConstructors
| Name | Description |
|---|---|
| LowerCaseTokenizer(LuceneVersion, AttributeSource.AttributeFactory, TextReader) | Construct a new LowerCaseTokenizer using a given AttributeSource.AttributeFactory. |
| LowerCaseTokenizer(LuceneVersion, TextReader) | Construct a new LowerCaseTokenizer. |
Methods
| Name | Description |
|---|---|
| Normalize(Int32) | Converts char to lower case ToLower(Int32). |
