Constructor Token

Token()

Constructs a Token will null text.

Declaration
public Token()

Token(Int32, Int32)

Constructs a Token with null text and start & end offsets.

Declaration
public Token(int start, int end)
Parameters
Type Name Description
System.Int32 start

start offset in the source text

System.Int32 end

end offset in the source text

Token(Int32, Int32, String)

Constructs a Token with null text and start & end offsets plus the Token type.

Declaration
public Token(int start, int end, string typ)
Parameters
Type Name Description
System.Int32 start

start offset in the source text

System.Int32 end

end offset in the source text

System.String typ

the lexical type of this Token

Token(Int32, Int32, Int32)

Constructs a Token with null text and start & end offsets plus flags. NOTE: flags is EXPERIMENTAL.

Declaration
public Token(int start, int end, int flags)
Parameters
Type Name Description
System.Int32 start

start offset in the source text

System.Int32 end

end offset in the source text

System.Int32 flags

The bits to set for this token

Token(String, Int32, Int32)

Constructs a Token with the given term text, and start & end offsets. The type defaults to "word." NOTE: for better indexing speed you should instead use the char[] termBuffer methods to set the term text.

Declaration
public Token(string text, int start, int end)
Parameters
Type Name Description
System.String text

term text

System.Int32 start

start offset in the source text

System.Int32 end

end offset in the source text

Token(String, Int32, Int32, String)

Constructs a Token with the given text, start and end offsets, & type. NOTE: for better indexing speed you should instead use the char[] termBuffer methods to set the term text.

Declaration
public Token(string text, int start, int end, string typ)
Parameters
Type Name Description
System.String text

term text

System.Int32 start

start offset in the source text

System.Int32 end

end offset in the source text

System.String typ

token type

Token(String, Int32, Int32, Int32)

Constructs a Token with the given text, start and end offsets, & type. NOTE: for better indexing speed you should instead use the char[] termBuffer methods to set the term text.

Declaration
public Token(string text, int start, int end, int flags)
Parameters
Type Name Description
System.String text

term text

System.Int32 start

start offset in the source text

System.Int32 end

end offset in the source text

System.Int32 flags

token type bits

Token(Char[], Int32, Int32, Int32, Int32)

Constructs a Token with the given term buffer (offset & length), start and end offsets

Declaration
public Token(char[] startTermBuffer, int termBufferOffset, int termBufferLength, int start, int end)
Parameters
Type Name Description
System.Char[] startTermBuffer

buffer containing term text

System.Int32 termBufferOffset

the index in the buffer of the first character

System.Int32 termBufferLength

number of valid characters in the buffer

System.Int32 start

start offset in the source text

System.Int32 end

end offset in the source text