Abstract
Tokenizer Class
This class provides methods for tokenizing text into spans of tokens.
Static
The regular expression used for tokenization.
Tokenize a given text into spans of tokens.
The text to be tokenized.
An array of spans representing the tokens in the text.
This method tokenizes the text and returns an array of spans, each containing the start index, end index, and the token itself.
Tokenize a given text into an array of tokens.
An array of tokens extracted from the text.
This method uses a regular expression to split the text into tokens based on the defined regex.
Tokenizer Class
Description
This class provides methods for tokenizing text into spans of tokens.