【问题标题】:Bug in double negation of regex character classes?正则表达式字符类的双重否定中的错误?
【发布时间】:2014-02-21 12:10:56
【问题描述】:

更新:下面描述的 Java 11 错误似乎已修复

(可能更早修复了,但我不知道具体是哪个版本。Bug reportnhahtdh's answer 中链接的类似问题建议 Java 9)。


TL;DR(修复前):
为什么[^\\D2][^[^0-9]2][^2[^0-9]] 在 Java 中得到不同的结果?


用于测试的代码。你现在可以跳过它。

String[] regexes = { "[[^0-9]2]", "[\\D2]", "[013-9]", "[^\\D2]", "[^[^0-9]2]", "[^2[^0-9]]" };
String[] tests = { "x", "1", "2", "3", "^", "[", "]" };

System.out.printf("match | %9s , %6s | %6s , %6s , %6s , %10s%n", (Object[]) regexes);
System.out.println("-----------------------------------------------------------------------");
for (String test : tests)
    System.out.printf("%5s | %9b , %6b | %7b , %6b , %10b , %10b %n", test,
            test.matches(regexes[0]), test.matches(regexes[1]),
            test.matches(regexes[2]), test.matches(regexes[3]),
            test.matches(regexes[4]), test.matches(regexes[5]));

假设我需要接受以下字符的正则表达式

  • 不是数字,
  • 2 除外。

所以这样的正则表达式应该代表除0134、...、9 之外的每个字符。我至少可以用两种方式编写它,这将是 所有非数字2 的总和:

  • [[^0-9]2]
  • [\\D2]

这两个正则表达式都按预期工作

match , [[^0-9]2] ,  [\D2]
--------------------------
    x ,      true ,   true
    1 ,     false ,  false
    2 ,      true ,   true
    3 ,     false ,  false
    ^ ,      true ,   true
    [ ,      true ,   true
    ] ,      true ,   true

现在假设我想反转接受的字符。(所以我想接受除 2 之外的所有数字) 我可以创建明确包含所有接受的字符的正则表达式,例如

  • [013-9]

或尝试通过将两个先前描述的正则表达式包装在另一个 [^...] 中来否定它

  • [^\\D2]
  • [^[^0-9]2]
    甚至
  • [^2[^0-9]]

但令我惊讶的是,只有前两个版本按预期工作

match | [[^0-9]2] ,  [\D2] | [013-9] , [^\D2] , [^[^0-9]2] , [^2[^0-9]] 
------+--------------------+------------------------------------------- 
    x |      true ,   true |   false ,  false ,       true ,       true 
    1 |     false ,  false |    true ,   true ,      false ,       true 
    2 |      true ,   true |   false ,  false ,      false ,      false 
    3 |     false ,  false |    true ,   true ,      false ,       true 
    ^ |      true ,   true |   false ,  false ,       true ,       true 
    [ |      true ,   true |   false ,  false ,       true ,       true 
    ] |      true ,   true |   false ,  false ,       true ,       true 

所以我的问题是为什么[^[^0-9]2][^2[^0-9]] 的行为不像[^\D2]?我可以以某种方式更正这些正则表达式,以便能够在其中使用 [^0-9] 吗?

【问题讨论】:

  • 只是为了让你知道[^[^0-9]][0-9]不同
  • @anubhava 谢谢。这很有趣。据我所知,它的行为与[^0-9] 相同。你能解释一下吗?根据我在发布的答案中读到的内容,它应该更像not nothing 的联合(它可能用[^0-9] 表示所有字符。但在这种情况下Everything UNION Anything 仍然是Everything。在这种情况下,它似乎不是使用了联合交集,这有点令人困惑。
  • 抱歉不得不去开会。 IMO 在嵌套的[ and ] 中没有否定。所以[^[^0-9]] 被解释为与^[^0-9] 的UNION 相同,并且实际上是[^0-9] 本身。
  • @anubhava 如果[^[^0-9]]^[^0-9] 的联合,那么^ 应该与[^[^0-9^]] 匹配,但"^".matches("[^[^0-9^]]") 返回false :/(除非我误解了你) .
  • 嗯不确定[^3[^2]] 匹配所有内容,即任何数字或任何非数字。

标签: java regex


【解决方案1】:

如果您从 Oracle 网站下载或使用 OpenJDK,您的 JRE/JDK 随附的 Oracle 实现 Pattern 类的字符类解析代码中会出现一些奇怪的巫术。我没有检查其他 JVM(尤其是 GNU Classpath)实现如何解析问题中的正则表达式。

从现在开始,任何对Pattern 类及其内部工作的引用都严格限制在Oracle 的实现(参考实现)中。

阅读和理解Pattern 类如何解析嵌套否定需要一些时间,如问题所示。但是,我编写了一个程序1Pattern 对象(带有Reflection API)中提取信息以查看编译结果。下面的输出来自于在 Java HotSpot Client VM 版本 1.7.0_51 上运行我的程序。

1:目前,程序是一团糟。当我完成并重构它时,我会用一个链接更新这篇文章。

[^0-9]
Start. Start unanchored match (minLength=1)
CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
  Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
LastNode
Node. Accept match

这没什么好奇怪的。

[^[^0-9]]
Start. Start unanchored match (minLength=1)
CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
  Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
LastNode
Node. Accept match
[^[^[^0-9]]]
Start. Start unanchored match (minLength=1)
CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
  Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
LastNode
Node. Accept match

上面接下来的 2 个案例被编译成与[^0-9] 相同的程序,这是反直觉

[[^0-9]2]
Start. Start unanchored match (minLength=1)
Pattern.union (character class union). Match any character matched by either character classes below:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
    [U+0032]
    2
LastNode
Node. Accept match
[\D2]
Start. Start unanchored match (minLength=1)
Pattern.union (character class union). Match any character matched by either character classes below:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Ctype. Match POSIX character class DIGIT (US-ASCII)
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
    [U+0032]
    2
LastNode
Node. Accept match

如问题所述,上述两种情况都没有什么奇怪的。

[013-9]
Start. Start unanchored match (minLength=1)
Pattern.union (character class union). Match any character matched by either character classes below:
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 2 character(s):
    [U+0030][U+0031]
    01
  Pattern.rangeFor (character range). Match any character within the range from code point U+0033 to code point U+0039 (both ends inclusive)
LastNode
Node. Accept match
[^\D2]
Start. Start unanchored match (minLength=1)
Pattern.setDifference (character class subtraction). Match any character matched by the 1st character class, but NOT the 2nd character class:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
      Ctype. Match POSIX character class DIGIT (US-ASCII)
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
    [U+0032]
    2
LastNode
Node. Accept match

如问题中所述,这两种情况按预期工作。但是,请注意引擎如何对第一个字符类 (\D) 进行补码,并将集合差异应用于由剩余字符组成的字符类。

[^[^0-9]2]
Start. Start unanchored match (minLength=1)
Pattern.setDifference (character class subtraction). Match any character matched by the 1st character class, but NOT the 2nd character class:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
    [U+0032]
    2
LastNode
Node. Accept match
[^[^[^0-9]]2]
Start. Start unanchored match (minLength=1)
Pattern.setDifference (character class subtraction). Match any character matched by the 1st character class, but NOT the 2nd character class:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
    [U+0032]
    2
LastNode
Node. Accept match
[^[^[^[^0-9]]]2]
Start. Start unanchored match (minLength=1)
Pattern.setDifference (character class subtraction). Match any character matched by the 1st character class, but NOT the 2nd character class:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
    [U+0032]
    2
LastNode
Node. Accept match

Keppil 在评论中通过测试确认,上面的输出显示上面的所有 3 个正则表达式都编译到同一个程序中!

[^2[^0-9]]
Start. Start unanchored match (minLength=1)
Pattern.union (character class union). Match any character matched by either character classes below:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
      [U+0032]
      2
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
LastNode
Node. Accept match

我们得到UNION(NOT(2), NOT(0-9)),而不是NOT(UNION(2, NOT(0-9)),也就是0-13-9,相当于NOT(2)

[^2[^[^0-9]]]
Start. Start unanchored match (minLength=1)
Pattern.union (character class union). Match any character matched by either character classes below:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
      [U+0032]
      2
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
LastNode
Node. Accept match

由于相同的错误,正则表达式[^2[^[^0-9]]] 编译为与[^2[^0-9]] 相同的程序。

有一个似乎具有相同性质的未解决错误:JDK-6609854


说明

初步

下面是Pattern类的实现细节,在进一步阅读之前应该知道:

  • Pattern 类将String 编译成一个节点链,每个节点负责一个小而明确的职责,并将工作委托给链中的下一个节点。 Node 类是所有节点的基类。
  • CharProperty 类是所有与字符类相关的Nodes 的基类。
  • BitClass 类是 CharProperty 类的子类,它使用 boolean[] 数组来加速匹配 Latin-1 字符(代码点 add 方法,允许在编译期间添加字符。
  • CharProperty.complementPattern.unionPattern.intersection 是对应于集合操作的方法。他们所做的事情是不言自明的。
  • Pattern.setDifferenceasymmetric set difference

乍一看解析字符类

在看CharProperty clazz(boolean consume)方法的完整代码之前,它是负责解析一个字符类的方法,让我们看一个极其简化的代码版本来理解代码的流程:

private CharProperty clazz(boolean consume) {
    // [Declaration and initialization of local variables - OMITTED]
    BitClass bits = new BitClass();
    int ch = next();
    for (;;) {
        switch (ch) {
            case '^':
                // Negates if first char in a class, otherwise literal
                if (firstInClass) {
                    // [CODE OMITTED]
                    ch = next();
                    continue;
                } else {
                    // ^ not first in class, treat as literal
                    break;
                }
            case '[':
                // [CODE OMITTED]
                ch = peek();
                continue;
            case '&':
                // [CODE OMITTED]
                continue;
            case 0:
                // [CODE OMITTED]
                // Unclosed character class is checked here
                break;
            case ']':
                // [CODE OMITTED]
                // The only return statement in this method
                // is in this case
                break;
            default:
                // [CODE OMITTED]
                break;
        }
        node = range(bits);

        // [CODE OMITTED]
        ch = peek();
    }
}

代码基本上读取输入(输入String 转换为null-terminated int[] 的代码点)直到它到达] 或字符串的结尾(未封闭的字符类)。

代码有点混淆continuebreakswitch 块内混合在一起。不过,只要你意识到continue属于外部for循环,break属于switch块,代码就很容易理解了:

  • continue 结尾的案例将永远不会执行switch 语句之后的代码。
  • break 结尾的情况可以执行switch 语句之后的代码(如果它还没有return)。

通过上面的观察,我们可以看到,每当发现一个字符是非特殊的并且应该包含在字符类中时,我们都会执行switch语句之后的代码,其中node = range(bits); 是第一条语句。

如果检查source code,方法CharProperty range(BitClass bits) 会解析“字符类中的单个字符或字符范围”。该方法要么返回传入的相同BitClass 对象(添加了新字符),要么返回CharProperty 类的新实例。

血淋淋的细节

接下来我们看完整版代码(省略部分解析字符类交集&amp;&amp;):

private CharProperty clazz(boolean consume) {
    CharProperty prev = null;
    CharProperty node = null;
    BitClass bits = new BitClass();
    boolean include = true;
    boolean firstInClass = true;
    int ch = next();
    for (;;) {
        switch (ch) {
            case '^':
                // Negates if first char in a class, otherwise literal
                if (firstInClass) {
                    if (temp[cursor-1] != '[')
                        break;
                    ch = next();
                    include = !include;
                    continue;
                } else {
                    // ^ not first in class, treat as literal
                    break;
                }
            case '[':
                firstInClass = false;
                node = clazz(true);
                if (prev == null)
                    prev = node;
                else
                    prev = union(prev, node);
                ch = peek();
                continue;
            case '&':
                // [CODE OMITTED]
                // There are interesting things (bugs) here,
                // but it is not relevant to the discussion.
                continue;
            case 0:
                firstInClass = false;
                if (cursor >= patternLength)
                    throw error("Unclosed character class");
                break;
            case ']':
                firstInClass = false;

                if (prev != null) {
                    if (consume)
                        next();

                    return prev;
                }
                break;
            default:
                firstInClass = false;
                break;
        }
        node = range(bits);

        if (include) {
            if (prev == null) {
                prev = node;
            } else {
                if (prev != node)
                    prev = union(prev, node);
            }
        } else {
            if (prev == null) {
                prev = node.complement();
            } else {
                if (prev != node)
                    prev = setDifference(prev, node);
            }
        }
        ch = peek();
    }
}

查看switch语句的case '[':中的代码和switch语句之后的代码:

  • node 变量存储解析 unit(独立字符、字符范围、速记字符类、POSIX/Unicode 字符类或嵌套字符类)的结果李>
  • prev 变量存储了到目前为止的编译结果,并且总是在我们编译 node 中的 unit 后立即更新。

由于记录字符类是否取反的局部变量boolean include,从来没有传递给任何方法调用,所以只能在这个方法中单独作用。唯一读取和处理include 的位置是在switch 语句之后。

正在建设中

【讨论】:

  • 这很有趣。感谢您的意见。
  • @Pshemo:我写信给 core-lib-dev,结果发现问题已经被讨论过,并且已经建议了一个补丁since 2011。我确实在the email I wrote to them 中更详细地解释了当前的行为。我应该用这些信息更新这篇文章吗?
  • 这令人印象深刻。我对如何在 Java 中实现正则表达式的了解非常有限。我认为最好将此信息包含在您的答案中(尤其是在CharProperty clazz(boolean consume) 方法中关于case:']' 的部分),以显示正则表达式如何/为什么以它的方式处理我们的特殊情况,以及如何纠正这种情况。
  • @Pshemo:你可以慢慢来(我也会慢慢来,因为我现在有点累)。
  • @user1803551:这里的程序github.com/nhahtdh/pattern-dissector(它落后于我当前的代码,但应该输出类似于上面的内容)。但是,我没有时间完成其余的分析。这将需要 6-8 小时的专注工作(半开玩笑,但半真半假)
【解决方案2】:

根据JavaDoc page 嵌套类生成两个类的联合,这使得使用该表示法创建交集是不可能的:

要创建联合,只需将一个类嵌套在另一个类中,例如 [0-4[6-8]]。这个特殊的联合创建了一个匹配数字 0、1、2、3、4、6、7 和 8 的单个字符类。

要创建交叉点,您必须使用&amp;&amp;

要创建一个仅匹配所有嵌套类共有的字符的单个字符类,请使用 &&,如 [0-9&&[345]]。这个特殊的交集创建了一个字符类,只匹配两个字符类共有的数字:3、4 和 5。

你问题的最后一部分对我来说仍然是个谜。 [^2][^0-9] 的并集确实应该是 [^2],所以 [^2[^0-9]] 的行为符合预期。 [^[^0-9]2] 表现得像 [^0-9] 确实很奇怪。

【讨论】:

  • 感谢您的回答。这也是我一开始的想法。看起来[^2[^0-9]] [^2] 是先创建的,然后正则表达式引擎使用联合将它与[^0-9] 结合起来,所以它不会改变任何东西,因为这两个类的总和是[^2] ([^0-9][^2]) 的子集。困扰我的是为什么[^[^0-9]2] 的行为与[^0-9] 相同,但与[^2] 不同?
  • @Pshemo:稍微更新了答案。我在想javadoc中的所有示例都将嵌套类作为最后一个元素。如果不遵循该约定,行为是否会有点不确定?
  • 这很令人费解。也许如果[] 是外部否定字符类的第一个元素,而不是使用联合交集。德摩根可以在这里责备,但我不知道如何将他与这种情况联系起来。
  • 更让我困惑的是[^[^0-9]2][^[^[^0-9]]2][^[^[^[^0-9]]]2] 都产生了相同的结果。我尝试查看代码,但不太容易理解。
  • @Keppil 是否支持多个嵌套级别?
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2014-11-21
  • 2014-01-15
  • 2016-04-30
  • 2017-04-20
  • 2014-02-12
  • 1970-01-01
相关资源
最近更新 更多