Add more expressions to parser, add EOL and newline tokens

* Lexer recognizes semicolons as EOL tokens, and newline tokens
* Parser can be configured to ignore newlines (which is also used
  internally)
* Newlines are allowed in lists, tuples, and parenthesized expressions
* Add a bunch of tests for the new stuff

Signed-off-by: Alek Ratzloff <alekratz@gmail.com>
This commit is contained in:
2020-05-05 16:38:07 -04:00
parent d9edf21d16
commit a4f289fb53
6 changed files with 274 additions and 22 deletions

View File

@@ -31,6 +31,8 @@ pub enum TokenKind {
Splat,
FSlash,
Bang,
Eol,
Newline,
}
impl Display for TokenKind {
@@ -65,6 +67,8 @@ impl Display for TokenKind {
Splat => "splat (or times)",
FSlash => "fslash (or divide)",
Bang => "not",
Eol => "line end",
Newline => "newline",
};
Display::fmt(s, fmt)
}