compiler/stage2_l0/src/tokens.l0

Module tokens

Overview Symbols grouped by source file: compiler/stage2_l0/src/tokens.l0

Module: tokens

Source: compiler/stage2_l0/src/tokens.l0 Language: Dea/L0

Imports / Includes

  • std.vector
  • std.text
  • sys.rt
  • std.linear_map
  • std.unit
  • std.string
  • std.io

Symbols

Enum TokenRecovery

TokenRecovery

Non-recursive logical token payload used by lexer-error wrappers.

Variants:

  • TR_NONE
  • TR_INT(text: string, value: int)
  • TR_BYTE(text: string, value: byte)
  • TR_STRING(text: string, value: string)

Enum TokenType

TokenType

Defines the TokenType enumeration.

Variants:

  • TT_EOF
  • TT_IDENT(text: string)
  • TT_UNDERSCORE
  • TT_INT(text: string, value: int)
  • TT_BYTE(text: string, value: byte)
  • TT_STRING(text: string, value: string)
  • TT_MODULE
  • TT_IMPORT
  • TT_FUNC
  • TT_STRUCT
  • TT_ENUM
  • TT_TYPE
  • TT_EXTERN
  • TT_LET
  • TT_RETURN
  • TT_MATCH
  • TT_CASE
  • TT_IF
  • TT_ELSE
  • TT_WHILE
  • TT_FOR
  • TT_BREAK
  • TT_CONTINUE
  • TT_TRUE
  • TT_FALSE
  • TT_NULL
  • TT_AS
  • TT_NEW
  • TT_DROP
  • TT_WITH
  • TT_CLEANUP
  • TT_LBRACE
  • TT_RBRACE
  • TT_LPAREN
  • TT_RPAREN
  • TT_LBRACKET
  • TT_RBRACKET
  • TT_COMMA
  • TT_SEMI
  • TT_COLON
  • TT_DOUBLE_COLON
  • TT_ARROW_FUNC
  • TT_ARROW_MATCH
  • TT_EQ
  • TT_PLUS
  • TT_MINUS
  • TT_STAR
  • TT_SLASH
  • TT_MODULO
  • TT_LT
  • TT_GT
  • TT_LE
  • TT_GE
  • TT_EQEQ
  • TT_NE
  • TT_ANDAND
  • TT_OROR
  • TT_BANG
  • TT_QUESTION
  • TT_DOT
  • TT_AMP
  • TT_PIPE
  • TT_CARET
  • TT_TILDE
  • TT_LSHIFT
  • TT_RSHIFT
  • TT_FUTURE_EXTENSION
  • TT_LEXER_ERROR(code: string, message: string, line_end: int, column_end: int, recovery: TokenRecovery)

Function token_to_string

func token_to_string(token: Token) -> string

Render a token as human-readable text.

Parameters:

  • token: Token to format.

Returns: Readable token spelling for diagnostics and debugging output.

Function token_recovery_to_string

func token_recovery_to_string(recovery: TokenRecovery) -> string

Render a logical recovery payload.

Parameters:

  • recovery: Recovery payload to format.

Returns: Human-readable recovery token spelling.

Function token_recovery_is_some

func token_recovery_is_some(recovery: TokenRecovery) -> bool

Return whether a lexer-error wrapper has a parser-visible recovery token.

Parameters:

  • recovery: Recovery payload to inspect.

Returns: true if the payload represents a logical token.

Function token_recovered

func token_recovered(wrapper: Token) -> Token

Build the logical token represented by a lexer-error recovery payload.

Parameters:

  • wrapper: Physical lexer-error token.

Returns: Logical recovered token, or the wrapper when there is no recovery.

Function token_logical_predecessor_ends_expression

func token_logical_predecessor_ends_expression(token: Token, previous: bool) -> bool

Return whether the parser-visible predecessor ends an expression.

Lexer-error wrappers with recovery contribute the recovered token context; wrappers without recovery preserve the preceding logical context.

Parameters:

  • token: Newly scanned physical token.
  • previous: Whether the previous parser-visible token ended an expression.

Returns: Expression-ending context for signed-literal lexing.

Function is_reserved_ident_word

func is_reserved_ident_word(word: string) -> bool

Check whether identifier text is reserved as a builtin type name.

Parameters:

  • word: Identifier text to inspect.

Returns: true if word is reserved and cannot be used as a plain identifier.

Function is_reserved_keyword

func is_reserved_keyword(token: Token) -> bool

Check whether a token is reserved as a keyword or future extension.

Parameters:

  • token: Token to inspect.

Returns: true if the token cannot be used as a variable name.

Function ident_to_token

func ident_to_token(ident: string, index: int, line: int, col: int) -> Token

Convert identifier text into the appropriate token at a source location.

Parameters:

  • ident: Identifier text as it appeared in source.
  • index: Zero-based byte offset in the source.
  • line: One-based source line.
  • col: One-based source column.

Returns: Keyword, underscore, future-extension, or identifier token for ident.

Function is_expr_ending_token_type

func is_expr_ending_token_type(token_type: TokenType) -> bool

Check whether a token type can legally terminate an expression.

Parameters:

  • token_type: Token type to inspect.

Returns: true if the token type can appear at the end of an expression.

Function tv_create

func tv_create() -> TokenVector

Create an empty token vector.

Returns: Token vector with an initial capacity.

Function tv_push

func tv_push(self: TokenVector, token: Token)

Append a token to a token vector.

Parameters:

  • self: Token vector to extend.
  • token: Token value to append.

Function tv_get

func tv_get(self: TokenVector, index: int) -> Token

Return a token from a token vector.

Parameters:

  • self: Token vector to inspect.
  • index: Zero-based token index.

Returns: Token at index.

Function tv_len

func tv_len(self: TokenVector) -> int

Return the number of tokens in a token vector.

Parameters:

  • self: Token vector to inspect.

Returns: Token count.

Function token_release_payload

func token_release_payload(token: Token)

Release ARC-managed payload strings held by a token.

Parameters:

  • token: Token whose payload should be released.

Function token_recovery_release_payload

func token_recovery_release_payload(recovery: TokenRecovery)

Release ARC-managed payload strings held by a token recovery payload.

Parameters:

  • recovery: Recovery payload whose strings should be released.

Function tv_free

func tv_free(self: TokenVector)

Free a token vector and release any token payload strings.

Parameters:

  • self: Token vector to free.

Struct Token

Represents the Token structure.

Token Field token_type

token_type: TokenType

Token Field index

index: int

Token Field line

line: int

Token Field column

column: int

Struct TokenVector

Represents the TokenVector structure.

TokenVector Field vec

vec: VectorBase*