l0_lexer.py

Defined in module l0_lexer (l0_lexer.py).

Tokenizes L0 source code.

This lexer scans source text and produces a list of Token objects. It tracks line and column information for diagnostics.

Public functions

auto __init__(self, str source, str filename = "<input>", Optional] diagnostics[List[Diagnostic] = None) -> None: Initialize the lexer.
auto from_source(cls, str source) -> "Lexer": Create a lexer from a source string with default settings.
auto tokenize(self) -> List[Token]: Perform tokenization of the source string.

auto _error(self, str message, int line, int column) -> None: Add an error diagnostic.
auto _lexer_error_token(self, str text, int line, int column, List diagnostics[Diagnostic], Optional recovery[Token] = None) -> Token: Build a deferred lexer-error wrapper token.
auto _wrap_new_diagnostics(self, int diagnostic_start, str text, int line, int column, Token recovery) -> Token: Move newly emitted recoverable diagnostics into a wrapper token.
auto _at_end(self) -> bool: Check if all input has been scanned.
auto _peek(self) -> str: Peek at the current character without advancing.
auto _peek_next(self) -> str: Peek at the next character without advancing.
auto _advance(self) -> str: Advance the current index and return the character.
auto _next_token(self) -> Token: Scan and return the next token.
auto _read_byte_literal(self, int start_col, int start_line) -> str: Scan a byte/character literal.
auto _read_string_literal(self) -> str: Scan a double-quoted string literal.
auto _read_valid_char_escape(self) -> str: Scan a single escape sequence.
auto _read_number(self, str c, int start_col, int start_line, bool is_negative = False) -> str: Scan an integer literal.
auto _skip_ws_and_comments(self) -> None: Skip whitespace and both line and block comments.
auto _collect_invalid_characters(self) -> str: Consume the rest of an invalid-character run and return its text.

Initialize the lexer.

Parameters
source	The source code text.
filename	The source filename for diagnostics.
diagnostics	Optional list to collect diagnostics into.

Create a lexer from a source string with default settings.

Parameters
source	The source code text.
Returns	A new Lexer instance.

Perform tokenization of the source string.

Returns	A list of all tokens, ending with an EOF token.

Consume the rest of an invalid-character run and return its text.

The run stops at printable ASCII or whitespace, so a diagnostic span never crosses a line break.