Class Token

Analyzed token with morphological data from its dictionary.

Inheritance
System.Object
Token
Assembly: Lucene.Net.Analysis.Kuromoji.dll
Syntax
public class Token : object

Constructors

Name Description
Token(Int32, Char[], Int32, Int32, JapaneseTokenizerType, Int32, IDictionary)

Properties

Name Description
Length

length of surfaceForm
Offset

offset into surfaceForm
Position

Get index of this token in input text. Returns position of token.
PositionLength

Gets or Sets the length (in tokens) of this token. For normal tokens this is 1; for compound tokens it's > 1.
SurfaceForm

surfaceForm

Methods

Name Description
GetBaseForm()

base form or null if token is not inflected
GetInflectionForm()

inflection form or null
GetInflectionType()

inflection type or null
GetPartOfSpeech()

part of speech.
GetPronunciation()

pronunciation. null if token doesn't have pronunciation.
GetReading()

reading. null if token doesn't have reading.
GetSurfaceFormString()

surfaceForm as a String
IsKnown()

Returns true if this token is known word.
IsUnknown()

Returns true if this token is unknown word.
IsUser()

Returns true if this token is defined in user dictionary.
ToString()

Extension Methods