Namespace Lucene.Net.Analysis.Cn.Smart.Hhmm

Finds the optimal segmentation of a sentence into Chinese words

@lucene.experimental

SmartChineseAnalyzer internal token

@lucene.experimental

Filters a SegToken by converting full-width latin to half-width, then lowercasing latin. Additionally, all punctuation is converted into COMMON_DELIMITER