Class WhitespaceTokenizer
A WhitespaceTokenizer is a tokenizer that divides text at whitespace. Adjacent sequences of non-Whitespace characters form tokens.
You must specify the required LuceneVersion compatibility when creating WhitespaceTokenizer:
- As of 3.1, CharTokenizer uses an int based API to normalize and detect token characters. See IsTokenChar(Int32) and Normalize(Int32) for details.
Inherited Members
Lucene.Net.Analysis.Tokenizer.SetReader(System.IO.TextReader)
System.Object.Equals(System.Object, System.Object)
System.Object.ReferenceEquals(System.Object, System.Object)
System.Object.GetType()
System.Object.MemberwiseClone()
Assembly: Lucene.Net.Analysis.Common.dll
Syntax
[Serializable]
public sealed class WhitespaceTokenizer : CharTokenizer, IDisposableConstructors
| Name | Description |
|---|---|
| WhitespaceTokenizer(LuceneVersion, AttributeSource.AttributeFactory, TextReader) | Construct a new WhitespaceTokenizer using a given AttributeSource.AttributeFactory. |
| WhitespaceTokenizer(LuceneVersion, TextReader) |
Methods
| Name | Description |
|---|---|
| IsTokenChar(Int32) | Collects only characters which do not satisfy System.Char.IsWhiteSpace(System.Char). |
