在 Swift 中规范化（组合和分解）utf8 字符串

Question

Unicode 字符串中带有重音符号的字符可以用“短”（组合）和“长”（分解）格式表示。这意味着在 Xcode 中，字符串 a 的长度为 8，而字符串 b 的长度为 10，即使它们看起来相同：

let a:String = "δέκα" // 8 bytes
print(a.data(using:String.Encoding.utf8)!.count)

let b:String = "δέκα" // 10 bytes
print(b.data(using:String.Encoding.utf8)!.count)

我需要“缩小”字符串以确保它们始终采用较短的“组合”格式。这是如何在 Swift 中完成的？

---

脚注：我知道像这样（如下）完全去除重音是可能的。我不想那样做，我只想“组合”角色。

let usPosixLocale = Locale(identifier: "en_US_POSIX")
let out = "δέκα".folding(options: [.caseInsensitive, .diacriticInsensitive], locale: usPosixLocale)

我知道.widthInsensitive 选项，但文档似乎表明它仅适用于亚洲字符。所以具体来说，这 不 用于组合或分解字符：

let out = a.folding(options: [.widthInsensitive], locale: usPosixLocale)

---

更新

这是代码的第二个较长版本，为了清楚起见显示了字节差异。

let a:String = String(bytes:[206, 180, 206, 173, 206, 186, 206, 177], encoding:.utf8)!
print(a, a.data(using:String.Encoding.utf8)!.count)

let b:String = String(bytes:[206, 180, 206, 181, 204, 129, 206, 186, 206, 177], encoding:.utf8)!
print(b, b.data(using:String.Encoding.utf8)!.count)

let usPosixLocale = Locale(identifier: "en_US_POSIX")
let out = b.folding(options: [.widthInsensitive], locale: usPosixLocale)
    print(out.data(using:String.Encoding.utf8)!.count)

Answer 1

precomposedStringWithCanonicalMapping 进行标准化：

let a = "δέκα"
print(a, Data(a.utf8).count) // δέκα 8

let b = "δε\u{0301}κα"
print(b, Data(b.utf8).count) // δέκα 10

let bn = b.precomposedStringWithCanonicalMapping
print(bn, Data(bn.utf8).count) // δέκα 8

“字面”比较表明a 与bn 相同，但与b 不同：

print(b.compare(a, options: .literal) == .orderedSame)  // false
print(bn.compare(a, options: .literal) == .orderedSame) // true

备注： precomposedStringWithCanonicalMapping 生成“Unicode 规范化表格 C”。还有precomposedStringWithCompatibilityMapping，它产生“Unicode 规范化表格 KC”。见

• 1.2 Normalization Forms

Unicode 标准中的精确差异。粗略地说，后者折叠了更多“在许多情况下被不当区分”的差异。例子：

let c = "\u{fb01}" // LATIN SMALL LIGATURE FI
print(c, c.precomposedStringWithCanonicalMapping, c.precomposedStringWithCompatibilityMapping)
// fi fi fi

let d = "2\u{2075}"
print(d, d.precomposedStringWithCanonicalMapping, d.precomposedStringWithCompatibilityMapping)
// 2⁵ 2⁵ 25

let e = "\u{2165}" // ROMAN NUMERAL SIX
print(e, e.precomposedStringWithCanonicalMapping, e.precomposedStringWithCompatibilityMapping)
// Ⅵ Ⅵ VI

Answer 2

感谢@matt指向CFStringNormalize(_:_:)

你可以这样做 -

import Foundation
import CoreFoundation

extension String {
    func normalizedCanonicallyComposed() -> String {
        let mutable = NSMutableString(string: self) as CFMutableString
        CFStringNormalize(mutable, .KC) // OR .C
        return mutable as String
    }
}

用法

let a: String = String(bytes: [206, 180, 206, 173, 206, 186, 206, 177], encoding: .utf8)!
print(a, a.data(using: .utf8)!.count)

let b: String = String(bytes: [206, 180, 206, 181, 204, 129, 206, 186, 206, 177], encoding: .utf8)!
print(b, b.data(using: .utf8)!.count)
        
print("Before - \(b), count: \(b.data(using: .utf8)!.count)")
let c = b.normalizedCanonicallyComposed()
print("After - \(c), count: \(c.data(using: .utf8)!.count)")

输出

δέκα 8
δέκα 10
Before - δέκα, count: 10
After - δέκα, count: 8