Method Filter
Filter(SegToken)
Filter an input SegToken
Full-width latin will be converted to half-width, then all latin will be lowercased. All punctuation is converted into COMMON_DELIMITER
Declaration
public virtual SegToken Filter(SegToken token)Parameters
| Type | Name | Description |
|---|---|---|
| SegToken | token | Input SegToken. |
Returns
| Type | Description |
|---|---|
| SegToken | Normalized SegToken. |
