pub fn scan(text: &str) -> Vec<Token>Expand description
Scan text into a flat sequence of Tokens covering every byte (no
gaps), classifying identifiers/numbers/strings/date-literals/operators by
simple char class. Never fails — unrecognized characters become single-char
Operator tokens so the whole string is always covered.
Every start/end is a char boundary and the tokens tile text exactly
(no gaps, no overlap): highlight_formula slices text by those ranges.