【问题标题】:Unicode character range not being consumed by Regex正则表达式未使用 Unicode 字符范围
【发布时间】:2017-12-02 05:45:46
【问题描述】:

注意

C# Regular Expressions with \Uxxxxxxxx characters in the pattern 已经提出了另一个问题。这个问题的不同之处在于它不是关于如何计算代理对,而是如何在正则表达式中表示高于 0 的 unicode 平面。从我的问题中应该清楚,我已经理解为什么这些代码单元被表示为 2 个字符 - 它们是代理对(这是另一个问题所问的)。我的问题是如何一般地转换它们(因为我无法控制提供给程序的正则表达式的外观),以便它们可以被 .NET 正则表达式引擎使用。

请注意,我现在有办法做到这一点,并想将我的答案添加到我的问题中,但由于现在这被标记为重复,我无法添加我的答案。

我有一些测试数据正在传递给我要移植到 c# 的 Java 库。我以一个特定的问题案例为例。原始字符类是 UTF-32 = \U0001BCA0-\U0001BCA3,.NET 不容易使用它 - 我们得到一个 "Unrecognized escape sequence \U" 错误。

我尝试转换为 UTF-16,并确认 \U0001BCA0\U0001BCA3 的结果符合预期。

UTF-32      | Codepoint   | High Surrogate  | Low Surrogate  | UTF-16
---------------------------------------------------------------------------
0x0001BCA0  | 113824      | 55343           | 56480          | \uD82F\uDCA0
0x0001BCA3  | 113827      | 55343           | 56483          | \uD82F\uDCA3

但是,当我将字符串 "([\uD82F\uDCA0-\uD82F\uDCA3])" 传递给 Regex 类的构造函数时,我得到一个异常 "[x-y] range in reverse order"

虽然很清楚字符是按正确的顺序指定的(它在 Java 中有效),但我反过来尝试并得到相同的错误消息。

我还尝试将 UTF-32 字符从 \U0001BCA0-\U0001BCA3 更改为 \x01BCA0-\x01BCA3,但仍然收到异常 "[x-y] range in reverse order"

那么,如何让 .NET Regex 类成功解析此字符范围?

注意:我尝试更改代码以生成包含所有字符而不是范围的正则表达式字符类,它似乎可以工作,但这将使我的正则表达式成为几十个字符变成几千个字符,这肯定不会对性能产生奇迹。

实际的正则表达式示例

再一次,上面是一个更大的字符串失败的孤立示例。我正在寻找的是一种转换此类正则表达式的通用方法,以便它们可以被 .NET Regex 类解析。

"([\\u0000-\\u0009\\u000B\\u000C\\u000E-\\u001F\\u007F-\\u009F\\u00AD" +
"\\u061C\\u180E\\u200B\\u200E\\u200F\\u2028-\\u202E\\u2060-\\u206F\\uD800-" +
"\\uDFFF\\uFEFF\\uFFF0-\\uFFFB\\U0001BCA0-\\U0001BCA3\\U0001D173-" +
"\\U0001D17A\\U000E0000-\\U000E001F\\U000E0080-\\U000E00FF\\U000E01F0-\\U000E0FFF] " +
"| [\\u000D] | [\\u000A]) ()"

【问题讨论】:

  • 这绝对不是重复的。问题不在于如何计算代理对,而是如何在正则表达式中表示高于 0 的 unicode 平面。
  • @WiktorStribiżew - 请重新打开我的问题,以便我可以添加我的答案。这不是链接问题的重复。

标签: c# .net regex unicode


【解决方案1】:

您假设Regex"\uD82F\uDCA0" 识别为复合字符。事实并非如此,因为 .NET 中字符串的内部表示是 16 位 Unicode。

Unicode 有code points 的概念,这是一个独立于物理表示的抽象概念。根据使用的实际编码,并非所有代码点都可以显示在一个字符中。在 UTF-8 中,这变得非常明显,因为所有高于 127 的代码点都需要两个或更多字符。在 .NET 中,字符是 Unicode,这意味着对于大于 0 的 planes,您需要组合字符。这些仍然被正则表达式引擎识别为单个字符。

长话短说:不要将字符组合视为代码点,将它们视为单个字符。所以在你的情况下,正则表达式是:

using System;
using System.Text.RegularExpressions;

public class Program
{
    public static void Main()
    {
        var regex = new Regex("(\uD82F[\uDCA0-\uDCA3])");
        Console.WriteLine(regex.Match("\uD82F\uDCA2").Success);
    }
}

你可以try out the code here

【讨论】:

  • 对于没有指定相同高代理字符的范围,我需要做什么?同样,这个例子是一个孤立的案例。我的实际字符串具有字符类,其中指定了许多代码点范围。
  • 如果您需要在正则表达式中执行此操作,您必须将您的范围拆分为子范围并使用(range1|range2)。如果您对非正则表达式解决方案持开放态度,您可以使用Encoding.UTF32 将其转换为二进制文件,并在二进制文件中搜索代码点。请注意,对于 Regex 解决方案,每个代码点范围最多需要 3 个子范围。
【解决方案2】:

C# 中的字符串是 UTF-16 编码的。这就是为什么这个正则表达式被视为:

  • 符号'\uD82F'
  • 范围\uDCA0-\uD82F
  • 符号'\uDCA3'

\uDCA0-\uD82F 的范围明显不正确,导致[x-y] range in reverse order 异常。

不幸的是,您的问题没有简单的解决方案,因为它是由 C# 字符串的性质引起的。不能将 UTF-32 符号放入一个 C# 字符中,也不能使用多字符串作为范围边界。

可能的解决方法是使用半正则表达式解决方案:从字符串中提取此类符号并通过纯 C# 代码执行比较。当然它看起来很丑,但我没有看到另一种方法可以在 C# 中使用原始正则表达式来实现这一点。

【讨论】:

  • 谢谢。至少现在我对它发生的原因有了一个合理的解释。然而,代码的全部要点是它使用一组文件驱动的规则来构建一个正则表达式,然后将该正则表达式与生产代码进行比较,以确保它以相同的方式工作。我将不得不考虑如何最好地处理这个问题。
【解决方案3】:

虽然这个问题的其他贡献者提供了一些线索,但我需要一个答案。我的测试是一个规则引擎,它由从文件输入构建的正则表达式驱动,因此不能将逻辑硬编码到 C# 中。

但是,我确实在这里了解到

  1. .NET Regex 类不支持代理对和
  2. 您可以通过使用正则表达式更改来伪造对代理对范围的支持

当然,在我的数据驱动案例中,我无法手动将正则表达式更改为 .NET 可以接受的格式 - 我需要将其自动化。因此,我创建了下面的 Utf32Regex 类,它直接在构造函数中接受 UTF32 字符,并在内部将它们转换为 .NET 可以理解的正则表达式。

例如,它将转换正则表达式

"[abc\\U00011DEF-\\U00013E07]"

"(?:[abc]|\\uD807[\\uDDEF-\\uDFFF]|[\\uD808-\\uD80E][\\uDC00-\\uDFFF]|\\uD80F[\\uDC00-\\uDE07])"

或者

"([\\u0000-\\u0009\\u000B\\u000C\\u000E-\\u001F\\u007F-\\u009F\\u00AD" +
"\\u061C\\u180E\\u200B\\u200E\\u200F\\u2028-\\u202E\\u2060-\\u206F\\uD800-" +
"\\uDFFF\\uFEFF\\uFFF0-\\uFFFB\\U0001BCA0-\\U0001BCA3\\U0001D173-" +
"\\U0001D17A\\U000E0000-\\U000E001F\\U000E0080-\\U000E00FF\\U000E01F0-\\U000E0FFF] " +
"| [\\u000D] | [\\u000A]) ()"

"((?:[\\u0000-\\u0009\\u000B\\u000C\\u000E-\\u001F\\u007F-\\u009F\\u00AD\\u061C\\u180E" + 
"\\u200B\\u200E\\u200F\\u2028-\\u202E\\u2060-\\u206F\\uD800-\\uDFFF\\uFEFF\\uFFF0-\\uFFFB]|" + 
"\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40[\\uDC00-\\uDC1F]|" + 
"\\uDB40[\\uDC80-\\uDCFF]|\\uDB40[\\uDDF0-\\uDFFF]|[\\uDB41-\\uDB42][\\uDC00-\\uDFFF]|" + 
"\\uDB43[\\uDC00-\\uDFFF]) | [\\u000D] | [\\u000A]) ()"

Utf32Regex.cs

using System;
using System.Globalization;
using System.Text;
using System.Text.RegularExpressions;

/// <summary>
/// Patches the <see cref="Regex"/> class so it will automatically convert and interpret
/// UTF32 characters expressed like <c>\U00010000</c> or UTF32 ranges expressed
/// like <c>\U00010000-\U00010001</c>.
/// </summary>
public class Utf32Regex : Regex
{
    private const char MinLowSurrogate = '\uDC00';
    private const char MaxLowSurrogate = '\uDFFF';

    private const char MinHighSurrogate = '\uD800';
    private const char MaxHighSurrogate = '\uDBFF';

    // Match any character class such as [A-z]
    private static readonly Regex characterClass = new Regex(
        "(?<!\\\\)(\\[.*?(?<!\\\\)\\])",
        RegexOptions.Compiled);

    // Match a UTF32 range such as \U000E01F0-\U000E0FFF
    // or an individual character such as \U000E0FFF
    private static readonly Regex utf32Range = new Regex(
        "(?<begin>\\\\U(?:00)?[0-9A-Fa-f]{6})-(?<end>\\\\U(?:00)?[0-9A-Fa-f]{6})|(?<begin>\\\\U(?:00)?[0-9A-Fa-f]{6})",
        RegexOptions.Compiled);

    public Utf32Regex()
        : base()
    {
    }

    public Utf32Regex(string pattern)
        : base(ConvertUTF32Characters(pattern))
    {
    }

    public Utf32Regex(string pattern, RegexOptions options)
        : base(ConvertUTF32Characters(pattern), options)
    {
    }

    public Utf32Regex(string pattern, RegexOptions options, TimeSpan matchTimeout)
        : base(ConvertUTF32Characters(pattern), options, matchTimeout)
    {
    }

    private static string ConvertUTF32Characters(string regexString)
    {
        StringBuilder result = new StringBuilder();
        // Convert any UTF32 character ranges \U00000000-\U00FFFFFF to their
        // equivalent UTF16 characters
        ConvertUTF32CharacterClassesToUTF16Characters(regexString, result);
        // Now find all of the individual characters that were not in ranges and
        // fix those as well.
        ConvertUTF32CharactersToUTF16(result);

        return result.ToString();
    }

    private static void ConvertUTF32CharacterClassesToUTF16Characters(string regexString, StringBuilder result)
    {
        Match match = characterClass.Match(regexString); // Reset
        int lastEnd = 0;
        if (match.Success)
        {
            do
            {
                string characterClass = match.Groups[1].Value;
                string convertedCharacterClass = ConvertUTF32CharacterRangesToUTF16Characters(characterClass);

                result.Append(regexString.Substring(lastEnd, match.Index - lastEnd)); // Remove the match
                result.Append(convertedCharacterClass); // Append replacement 

                lastEnd = match.Index + match.Length;
            } while ((match = match.NextMatch()).Success);
        }
        result.Append(regexString.Substring(lastEnd)); // Append tail
    }

    private static string ConvertUTF32CharacterRangesToUTF16Characters(string characterClass)
    {
        StringBuilder result = new StringBuilder();
        StringBuilder chars = new StringBuilder();

        Match match = utf32Range.Match(characterClass); // Reset
        int lastEnd = 0;
        if (match.Success)
        {
            do
            {
                string utf16Chars;
                string rangeBegin = match.Groups["begin"].Value.Substring(2);

                if (!string.IsNullOrEmpty(match.Groups["end"].Value))
                {
                    string rangeEnd = match.Groups["end"].Value.Substring(2);
                    utf16Chars = UTF32RangeToUTF16Chars(rangeBegin, rangeEnd);
                }
                else
                {
                    utf16Chars = UTF32ToUTF16Chars(rangeBegin);
                }

                result.Append(characterClass.Substring(lastEnd, match.Index - lastEnd)); // Remove the match
                chars.Append(utf16Chars); // Append replacement 

                lastEnd = match.Index + match.Length;
            } while ((match = match.NextMatch()).Success);
        }
        result.Append(characterClass.Substring(lastEnd)); // Append tail of character class

        // Special case - if we have removed all of the contents of the
        // character class, we need to remove the square brackets and the
        // alternation character |
        int emptyCharClass = result.IndexOf("[]");
        if (emptyCharClass >= 0)
        {
            result.Remove(emptyCharClass, 2);
            // Append replacement ranges (exclude beginning |)
            result.Append(chars.ToString(1, chars.Length - 1));
        }
        else
        {
            // Append replacement ranges
            result.Append(chars.ToString());
        }

        if (chars.Length > 0)
        {
            // Wrap both the character class and any UTF16 character alteration into
            // a non-capturing group.
            return "(?:" + result.ToString() + ")";
        }
        return result.ToString();
    }

    private static void ConvertUTF32CharactersToUTF16(StringBuilder result)
    {
        while (true)
        {
            int where = result.IndexOf("\\U00");
            if (where < 0)
            {
                break;
            }
            string cp = UTF32ToUTF16Chars(result.ToString(where + 2, 8));
            result.Replace(where, where + 10, cp);
        }
    }

    private static string UTF32RangeToUTF16Chars(string hexBegin, string hexEnd)
    {
        var result = new StringBuilder();
        int beginCodePoint = int.Parse(hexBegin, NumberStyles.HexNumber);
        int endCodePoint = int.Parse(hexEnd, NumberStyles.HexNumber);

        var beginChars = char.ConvertFromUtf32(beginCodePoint);
        var endChars = char.ConvertFromUtf32(endCodePoint);
        int beginDiff = endChars[0] - beginChars[0];

        if (beginDiff == 0)
        {
            // If the begin character is the same, we can just use the syntax \uD807[\uDDEF-\uDFFF]
            result.Append("|");
            AppendUTF16Character(result, beginChars[0]);
            result.Append('[');
            AppendUTF16Character(result, beginChars[1]);
            result.Append('-');
            AppendUTF16Character(result, endChars[1]);
            result.Append(']');
        }
        else
        {
            // If the begin character is not the same, create 3 ranges
            // 1. The remainder of the first
            // 2. A range of all of the middle characters
            // 3. The beginning of the last

            result.Append("|");
            AppendUTF16Character(result, beginChars[0]);
            result.Append('[');
            AppendUTF16Character(result, beginChars[1]);
            result.Append('-');
            AppendUTF16Character(result, MaxLowSurrogate);
            result.Append(']');

            // We only need a middle range if the ranges are not adjacent
            if (beginDiff > 1)
            {
                result.Append("|");
                // We only need a character class if there are more than 1
                // characters in the middle range
                if (beginDiff > 2)
                {
                    result.Append('[');
                }
                AppendUTF16Character(result, (char)(Math.Min(beginChars[0] + 1, MaxHighSurrogate)));
                if (beginDiff > 2)
                {
                    result.Append('-');
                    AppendUTF16Character(result, (char)(Math.Max(endChars[0] - 1, MinHighSurrogate)));
                    result.Append(']');
                }
                result.Append('[');
                AppendUTF16Character(result, MinLowSurrogate);
                result.Append('-');
                AppendUTF16Character(result, MaxLowSurrogate);
                result.Append(']');
            }

            result.Append("|");
            AppendUTF16Character(result, endChars[0]);
            result.Append('[');
            AppendUTF16Character(result, MinLowSurrogate);
            result.Append('-');
            AppendUTF16Character(result, endChars[1]);
            result.Append(']');
        }
        return result.ToString();
    }

    private static string UTF32ToUTF16Chars(string hex)
    {
        int codePoint = int.Parse(hex, NumberStyles.HexNumber, CultureInfo.InvariantCulture);
        return UTF32ToUTF16Chars(codePoint);
    }

    private static string UTF32ToUTF16Chars(int codePoint)
    {
        StringBuilder result = new StringBuilder();
        UTF32ToUTF16Chars(codePoint, result);
        return result.ToString();
    }

    private static void UTF32ToUTF16Chars(int codePoint, StringBuilder result)
    {
        // Use regex alteration to on the entire range of UTF32 code points
        // to ensure each one is treated as a group.
        result.Append("|");
        AppendUTF16CodePoint(result, codePoint);
    }

    private static void AppendUTF16CodePoint(StringBuilder text, int cp)
    {
        var chars = char.ConvertFromUtf32(cp);
        AppendUTF16Character(text, chars[0]);
        if (chars.Length == 2)
        {
            AppendUTF16Character(text, chars[1]);
        }
    }

    private static void AppendUTF16Character(StringBuilder text, char c)
    {
        text.Append(@"\u");
        text.Append(Convert.ToString(c, 16).ToUpperInvariant());
    }
}

StringBuilderExtensions.cs

public static class StringBuilderExtensions
{
    /// <summary>
    /// Searches for the first index of the specified character. The search for
    /// the character starts at the beginning and moves towards the end.
    /// </summary>
    /// <param name="text">This <see cref="StringBuilder"/>.</param>
    /// <param name="value">The string to find.</param>
    /// <returns>The index of the specified character, or -1 if the character isn't found.</returns>
    public static int IndexOf(this StringBuilder text, string value)
    {
        return IndexOf(text, value, 0);
    }

    /// <summary>
    /// Searches for the index of the specified character. The search for the
    /// character starts at the specified offset and moves towards the end.
    /// </summary>
    /// <param name="text">This <see cref="StringBuilder"/>.</param>
    /// <param name="value">The string to find.</param>
    /// <param name="startIndex">The starting offset.</param>
    /// <returns>The index of the specified character, or -1 if the character isn't found.</returns>
    public static int IndexOf(this StringBuilder text, string value, int startIndex)
    {
        if (text == null)
            throw new ArgumentNullException("text");
        if (value == null)
            throw new ArgumentNullException("value");

        int index;
        int length = value.Length;
        int maxSearchLength = (text.Length - length) + 1;

        for (int i = startIndex; i < maxSearchLength; ++i)
        {
            if (text[i] == value[0])
            {
                index = 1;
                while ((index < length) && (text[i + index] == value[index]))
                    ++index;

                if (index == length)
                    return i;
            }
        }

        return -1;
    }

    /// <summary>
    /// Replaces the specified subsequence in this builder with the specified
    /// string.
    /// </summary>
    /// <param name="text">this builder.</param>
    /// <param name="start">the inclusive begin index.</param>
    /// <param name="end">the exclusive end index.</param>
    /// <param name="str">the replacement string.</param>
    /// <returns>this builder.</returns>
    /// <exception cref="IndexOutOfRangeException">
    /// if <paramref name="start"/> is negative, greater than the current
    /// <see cref="StringBuilder.Length"/> or greater than <paramref name="end"/>.
    /// </exception>
    /// <exception cref="ArgumentNullException">if <paramref name="str"/> is <c>null</c>.</exception>
    public static StringBuilder Replace(this StringBuilder text, int start, int end, string str)
    {
        if (str == null)
        {
            throw new ArgumentNullException(nameof(str));
        }
        if (start >= 0)
        {
            if (end > text.Length)
            {
                end = text.Length;
            }
            if (end > start)
            {
                int stringLength = str.Length;
                int diff = end - start - stringLength;
                if (diff > 0)
                { // replacing with fewer characters
                    text.Remove(start, diff);
                }
                else if (diff < 0)
                {
                    // replacing with more characters...need some room
                    text.Insert(start, new char[-diff]);
                }
                // copy the chars based on the new length
                for (int i = 0; i < stringLength; i++)
                {
                    text[i + start] = str[i];
                }
                return text;
            }
            if (start == end)
            {

                text.Insert(start, str);
                return text;
            }
        }
        throw new IndexOutOfRangeException();
    }
}

请注意,这没有经过很好的测试,可能不是很健壮,但出于测试目的,它应该没问题。

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 2021-05-20
    • 1970-01-01
    • 2014-04-11
    • 2014-02-26
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多