Lexer 标记 SKIP 和 NEWLINE 值。如何隐藏它们？

Question

我下面的词法分析器代码在应该隐藏它们的地方同时生成“SKIP”和“NEWLINE”值。有人可以告诉我该怎么做才能隐藏它们吗？

        tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification)
        get_token = re.compile(tok_regex, re.DOTALL).match
        mo = get_token(self.input, self.p)

        self.p = mo.end()

        if self.p >= len(self.input):
            self.c = "EOF"
        else:
            if mo is not None:
                self.c = mo.lastgroup
                self.val = mo.group(self.c)
                if self.c == 'NEWLINE':
                    self.line += 1

    def next_token(self):

        keywords = {'digraph', 'subgraph', 'node', 'edge', 'strict', 'graph'}
        self.tokenize()
        if self.c != 'SKIP' and self.c != "NEWLINE":

            if self.c == 'ID' and self.val in keywords:
                self.c = self.val
        return TokenTypes.Token(self.c, self.val)
        # if self.p != len(self.input):
        #    raise RuntimeError('Unexpected character %r on line %d' %(self.input[self.p], self.line))

def main():
    statements = '''
        digraph G {
            main [shape=box]; 
        }
    '''
    lexer = Lexer(statements)
    t = lexer.next_token()
    i = 0
    while (t.typ != "EOF"):
        i += 1
        print(t)
        t = lexer.next_token()

if __name__ == "__main__":
    sys.exit(main())

我正在尝试通过从上述词法分析器导入标记来为点语法生成抽象语法树。但是不需要的 SKIP 和 NEWLINE 令牌正在制造麻烦。

这是一个示例输入：

digraph G { main [shape=box]; }

产生输出：

NEWLINE digraph SKIP ID SKIP LBRACE NEWLINE ID SKIP LBRACKET ID EQUAL ID RBRACKET END SKIP NEWLINE SKIP RBRACE

但它应该输出：

digraph ID LBRACE ID LBRACKET ID EQUAL ID RBRACKET END RBRACE

Answer 1

如果遇到NEWLINE 或SKIP，我相信您想更改next_token() 方法以解析另一个令牌。示例：

def next_token(self):
    keywords = {'digraph', 'subgraph', 'node', 'edge', 'strict', 'graph'}

    # Parse next token.
    self.tokenize()
    while self.c == "SKIP" or self.c == "NEWLINE":
        # Discard this token and parse another token.
        self.tokenize()

    if self.c == 'ID' and self.val in keywords:
        self.c = self.val

    return TokenTypes.Token(self.c, self.val)
    # if self.p != len(self.input):
    #    raise RuntimeError('Unexpected character %r on line %d' %(self.input[self.p], self.line))