Class LetterTokenizer
A LetterTokenizer is a tokenizer that divides text at non-letters. That's to say, it defines tokens as maximal strings of adjacent letters, as defined by System.Char.IsLetter(System.Char) predicate.
Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.
You must specify the required LuceneVersion compatibility when creating LetterTokenizer:
- As of 3.1, CharTokenizer uses an System.Int32 based API to normalize and detect token characters. See IsTokenChar(Int32) and Normalize(Int32) for details.
Inherited Members
Lucene.Net.Analysis.Tokenizer.SetReader(System.IO.TextReader)
System.Object.Equals(System.Object, System.Object)
System.Object.ReferenceEquals(System.Object, System.Object)
System.Object.GetType()
System.Object.MemberwiseClone()
Assembly: Lucene.Net.Analysis.Common.dll
Syntax
[Serializable]
public class LetterTokenizer : CharTokenizer, IDisposableConstructors
| Name | Description |
|---|---|
| LetterTokenizer(LuceneVersion, AttributeSource.AttributeFactory, TextReader) | Construct a new LetterTokenizer using a given AttributeSource.AttributeFactory. |
| LetterTokenizer(LuceneVersion, TextReader) | Construct a new LetterTokenizer. |
Methods
| Name | Description |
|---|---|
| IsTokenChar(Int32) | Collects only characters which satisfy IsLetter(Int32). |
