Class Tokenizer

A Tokenizer is a TokenStream whose input is a Reader.

This is an abstract class; subclasses must override IncrementToken()

NOTE: Subclasses overriding IncrementToken() must call Lucene.Net.Util.AttributeSource.ClearAttributes before setting attributes.

System.Object

Lucene.Net.Util.AttributeSource

Tokenizer

TokenStream.IncrementToken()

TokenStream.End()

TokenStream.Reset()

TokenStream.Close()

TokenStream.Dispose()

public abstract class Tokenizer : TokenStream, IDisposable

Name	Description
Tokenizer()	Construct a tokenizer with null input.
Tokenizer(AttributeSource)	Construct a token stream processing the given input using the given AttributeSource.
Tokenizer(AttributeSource, IO.TextReader)	Construct a token stream processing the given input using the given AttributeSource.
Tokenizer(AttributeSource.AttributeFactory)	Construct a tokenizer with null input using the given AttributeFactory.
Tokenizer(AttributeSource.AttributeFactory, IO.TextReader)	Construct a token stream processing the given input using the given AttributeFactory.
Tokenizer(IO.TextReader)	Construct a token stream processing the given input.

Name	Description
input	The text source for this Tokenizer.

Name	Description
CorrectOffset(Int32)	Return the corrected offset. If input is a CharStream subclass this method calls CorrectOffset(Int32), else returns `currentOff`.
Dispose(Boolean)
Reset(IO.TextReader)	Expert: Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.