Type Alias TokenizerSpans

TokenizerSpans: [start: number, end: number, token: string]

Tokenizer Spans

This type represents the spans of tokens in a text, including their start and end indices and the token itself.