Class TMToken

Class representing a token in the TokenManager.

Description

This class implements the TMTokens interface and represents a token with its start, end, text, and current state.

Implements

TMTokens

Index

Constructors

Properties

currentState end start text type

Methods

Constructors

constructor

new TMToken(
    start: number,
    end: number,
    text: string,
    currentState: string,
): TMToken
Constructor for TMToken.
Parameters
- start: number
  The start index of the token.
- end: number
  The end index of the token.
- text: string
  The text of the token.
- currentState: string
  The current state of the token.
Returns TMToken
- Defined in managers/TokenManager.ts:43

Properties

currentState

currentState: string

The current state of the token.

end

end: number

The end index of the token.

start

start: number

The start index of the token.

text

text: string

The text of the token.

type

type: string = 'token'

The type of the token, default is 'token'.

Methods

`Static`fromObject

fromObject(obj: TokenizerSpans): TMToken
Static method to create a TMToken from a TokenizerSpans object.
Parameters
- obj: TokenizerSpans
  The TokenizerSpans object to convert.
Returns TMToken
- A new TMToken instance.
- Defined in managers/TokenManager.ts:55