Constructor JapaneseTokenizer

JapaneseTokenizer(TextReader, UserDictionary, Boolean, JapaneseTokenizerMode)

Create a new JapaneseTokenizer.

Uses the default AttributeFactory.

Declaration
public JapaneseTokenizer(TextReader input, UserDictionary userDictionary, bool discardPunctuation, JapaneseTokenizerMode mode)
Parameters
Type Name Description
TextReader input

TextReader containing text.
UserDictionary userDictionary

Optional: if non-null, user dictionary.
System.Boolean discardPunctuation

true if punctuation tokens should be dropped from the output.
JapaneseTokenizerMode mode

Tokenization mode.

JapaneseTokenizer(AttributeSource.AttributeFactory, TextReader, UserDictionary, Boolean, JapaneseTokenizerMode)

Create a new JapaneseTokenizer.

Declaration
public JapaneseTokenizer(AttributeSource.AttributeFactory factory, TextReader input, UserDictionary userDictionary, bool discardPunctuation, JapaneseTokenizerMode mode)
Parameters
Type Name Description
AttributeSource.AttributeFactory factory

The AttributeFactory to use.
TextReader input

TextReader containing text.
UserDictionary userDictionary

Optional: if non-null, user dictionary.
System.Boolean discardPunctuation

true if punctuation tokens should be dropped from the output.
JapaneseTokenizerMode mode

Tokenization mode.