使用 antlr 解析电子邮件

Question

我已经尝试了整整一周来使用 antlr 构建一种允许我解析电子邮件的语法。

我的目标是不是将整个电子邮件详尽地解析为令牌，而是解析为相关部分。

这是我必须处理的文档格式。 // 描述不属于消息的内联 cmets：

Subject : [SUBJECT_MARKER] + lorem ipsum...
// marks a message that needs to be parsed.
// Subject marker can be something like "help needed", "action required"

Body: 

// irrelevant text we can ignore, discard or skip
Hi George,
Hope you had a good weekend. Another fluff sentence ...
// end of irrelevant text


// beginning of the SECTION_TYPE_1. SECTION_TYPE_1 marker is "answers below:" 
[SECTION_TYPE_1]

Meaningful text block that needs capturing, made of many sentences: Lorem ipsum ...

[SENTENCE_MARKER] - Sentences that needs to be captured.
[SENTENCE_MARKER] - Sentences that needs to be captured.
[SENTENCE_MARKER] - Sentences that needs to be captured.

[SECTION_END_MARKER] // this is "\n\n"

// SENTENCE_MARKER can be "a)", "b)" or anything that is in the form "[a-zA-Z]')'"
// one important requirement is that this SENTENCE_MARKER matches only inside a section. Either SECTION_TYPE_1 or SECTION_TYPE_2


// alternatively instead of [SECTION_TYPE_1] we can have [SECTION_TYPE_2].
// if we have SECTION_TYPE_1 then try to parse SECTION_TYPE_1 else try to parse SECTION_TYPE_2.enter code here

[SECTION_TYPE_2] // beginning of the section type 1;

Meaningful text bloc that needs capturing. Many sentences Lorem ipsum ...

[SENTENCE_MARKER] - Sentences that needs to be captured.
[SENTENCE_MARKER] - Sentences that needs to be captured.
[SENTENCE_MARKER] - Sentences that needs to be captured.

[SECTION_END_MARKER] // same as above

我面临的问题如下：

• 我没有找到跳过开头文本的好方法 消息并仅在标记后才开始应用解析规则 被发现。 SECTION_TYPE_1
• 捕获部分开始和句子标记之间的部分内的所有文本
• 在 SECTION_END 标记之后忽略后面的所有文本。

Answer 1

Antlr 是结构化文本的解析器，理想情况下是明确的结构化文本。除非您的源消息具有相对明确定义的特征来可靠地标记感兴趣的消息部分，否则 Antlr 不太可能工作。

更好的方法是使用自然语言处理器 (NLP) 包来识别每个句子或短语的形式和对象，从而识别出感兴趣的内容。 Stanford NLP package 是众所周知的 (Github)。

更新

必要的语法格式如下：

message : subject ( sec1 | sec2 | fluff )* EOF ;

subject : fluff* SUBJECT_MARKER subText EOL ;
subText : ( word | HWS )+ ;

sec1    : ( SECTION_TYPE_1 content )+ SECTION_END_MARKER     ;
sec2    : ( SECTION_TYPE_2 content )+ SECTION_END_MARKER     ;
content : ( word | ws )+ ;

word    : CHAR+ ;
ws      : ( EOL | HWS )+ ;

fluff   : . ;

SUBJECT_MARKER      : 'marker' ;
SECTION_TYPE_1      : 'text1' ;
SECTION_TYPE_2      : 'text2' ;
SENTENCE_MARKER     : [a-zA-Z0-9] ')' ;

EOL                 : '\r'? '\n';
HWS                 : [ \t] ;
CHAR                : . ;

成功将取决于各种标记的明确程度——而且肯定会有歧义。要么修改语法以明确处理歧义，要么推迟到树遍历/分析阶段来解决。