【问题标题】:Using python difflib to compare more than two files使用 python difflib 比较两个以上的文件
【发布时间】:2022-10-08 00:22:51
【问题描述】:

我想了解一下例如多台 (3+) 计算机的 ldd 依赖关系列表,通过将它们相互比较并突出显示差异。例如,如果我有一个如下所示的 dict:

my_ldd_outputs = {
  01:"<ldd_output>",
  02:"<ldd_output>", 
  ...
  09:"<ldd_output>",
  10:"<ldd_output>"
}

我希望输出看起来像

<identical line 1>
<identical line 2>
<identical line 3>
<differing line 4> (computer 01 02)
<differing line 4> (computer 04 05 06 07)
<differing line 4> (computer 08 09 10)
<identical line 5>
<identical line 6>
...

我的第一种方法涉及python difflib,我的想法是首先获取一个数据结构,其中来自上述my_ldd_outputs 字典的所有ldd_output 列表(只是与\n 拆分的结果)长度相同,并且任何缺失另一个ldd_output 字符串中存在的行添加了一个字符串。因此,如果两个文件看起来像这样:

ldd_1 = """
<identical line 1>
<identical line 2>
<differing line 3>
<identical line 4>
<extra line 5>
<identical line 6>
"""

ldd_2 = """
<identical line 1>
<identical line 2>
<differing line 3>
<identical line 4>
<identical line 6>
"""

我的目标是将这些文件存储为

ldd_1 = """
<identical line 1>
<identical line 2>
<differing line 3>
<identical line 4>
<extra line 5>
<identical line 6>
"""

ldd_2 = """
<identical line 1>
<identical line 2>
<differing line 3>
<identical line 4>
<None>
<identical line 6>
"""

最终只需遍历转换文件的每一行(现在都具有相同的长度)并比较每一行的差异并忽略任何&lt;None&gt; 条目,以便可以连续打印差异。

我创建了一个函数,它使用 python difflib&lt;None&gt; 字符串填充其他文件中缺少的行。但是,我不确定如何扩展此功能以合并任意数量的差异

def generate_diff(file_1, file_2):
    #differing hashvalues from ldd can be ignored, we only care about version and path
    def remove_hashvalues(input):
        return re.sub("([a-zA-Z0-9_.-]{32}\/|\([a-zA-Z0-9_.-]*\))", "<>", input)
    diff = [line.strip() for line in difflib.ndiff(remove_hashvalues(base).splitlines(keepends=True),remove_hashvalues(file_2).splitlines(keepends=True))]
    list_1 = []
    list_2 = []
    i = 0
    while i<len(diff):
        if diff[i].strip():
            if diff[i][0:2]=="- ":
                lost = []
                gained = []
                while diff[i][0:2]=="- " or diff[i][0:2]=="? ":
                    if diff[i][0:2]=="- ": lost.append(diff[i][1:].strip())
                    i+=1
                while diff[i][0:2]=="+ " or diff[i][0:2]=="? ":
                    if diff[i][0:2]=="+ ": gained.append(diff[i][1:].strip())
                    i+=1
                while len(lost) != len(gained):
                    lost.append("<None>") if len(lost)<len(gained) else gained.insert(0,"<None>")
                list_1+=lost; list_2+=gained
            elif diff[i][0:2]=="+ ":
                list_1.append("<None>"); list_2.append(diff[i][1:].strip())
            if not diff[i][0:2]=="? ":
                list_1.append(diff[i].strip()); list_2.append(diff[i].strip())
        i+=1
    return list_1, list_2

我还发现this tool 允许比较多个文件,但不幸的是它不是为了比较代码而设计的。

编辑:我调整了@AyoubKaanich 的解决方案建议,以创建一个更简化的版本来满足我的需求:

from collections import defaultdict
import re
def transform(input):
    input = re.sub("([a-zA-Z0-9_.-]{32}\/|\([a-zA-Z0-9_.-]*\))", "<>", input) # differing hashvalues can be ignored, we only care about version and path
    return sorted(input.splitlines())
def generate_diff(outputs: dict):
    mapping = defaultdict(set)
    for target, output in outputs.items():
        for line in transform(output):
            mapping[line.strip()].add(target)
    result = []
    current_line = None
    color_index = 0
    for line in sorted(mapping.keys()):
        if len(outputs) == len(mapping[line]):
            if current_line: current_line = None
            result.append((line))
        else:
            if current_line != line.split(" ")[0]:
                current_line = line.split(" ")[0]
                color_index+=1
            result.append((f"\033[3{color_index%6+1}m{line}\033[0m",mapping[line]))
    return result

唯一的缺点是这不适用于字符串在任意部分变化的差异,而不仅仅是开头,这是difflib 擅长检测的。但是,对于ldd 的情况,由于始终首先列出依赖项,因此按字母顺序排序并获取字符串的第一部分是可行的。

【问题讨论】:

    标签: python ssh diff difflib ldd


    【解决方案1】:

    纯 Python 解决方案,没有库或额外的依赖项。

    注意:由于一些假设,此解决方案有效:

    • 行的顺序无关紧要
    • 一行存在或缺失(没有逻辑来检查行之间的相似性)
    
    from collections import defaultdict
    import re
    
    def transform(input):
        # differing hashvalues from ldd can be ignored, we only care about version and path
        input = re.sub("([a-zA-Z0-9_.-]{32}/|([a-zA-Z0-9_.-]*))", "<>", input)
        return sorted(input.splitlines())
    
    def generate_diff(outputs: dict, common_threshold = 0):
        """
            common_threshold: how many outputs need to contain line to consider it common
                and mark outputs that do not have it as missing
        """
        assert(common_threshold <= len(outputs))
    
        mapping = defaultdict(set)
        for target, output in outputs.items():
            for line in transform(output):
                mapping[line].add(target)
        
        for line in sorted(mapping.keys()):
            found = mapping[line]
            if len(outputs) == len(found):
                print('  ' + line)
            elif len(found) >= common_threshold:
                missed_str = ",".join(map(str, set(outputs.keys()) - found))
                print(f'- {line}  ({missed_str})')
            else:
                added_str = ",".join(map(str, found))
                print(f'+ {line}  ({added_str})')
    
    

    样品执行

    
    my_ldd_outputs = {
    'A': """
    linux-vdso.so.1 (0x00007ffde4f09000)
    libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007fe0594f3000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe0592cb000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fe059690000)
    """,
    'B': """
    linux-vdso.so.1 (0x00007fff697b6000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1c54045000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f1c54299000)
    """,
    'C': """
    linux-vdso.so.1 (0x00007fffd61f9000)
    libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007f08a51a3000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f08a4f7b000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f08a5612000)
    """,
    'D': """
    linux-vdso.so.1 (0x00007ffcf9ddd000)
    libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007fa2e381b000)
    libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007fa2e37ef000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa2e35c7000)
    libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007fa2e3530000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa2e3cd7000)
    """,
    'E': """
    linux-vdso.so.1 (0x00007ffc2deab000)
    libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007f31fed91000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f31fed75000)
    libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f31fed49000)
    libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f31fecf5000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f31feacd000)
    libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f31fea34000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f31ff2af000)
    libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f31fe969000)
    libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f31fe93a000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f31fe934000)
    libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f31fe926000)
    libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f31fe91f000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f31fe909000)
    """
    }
    generate_diff(my_ldd_outputs, 2)
    
    

    输出

      /lib64/ld-linux-x86-64.so.2 <>
      libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 <>
    + libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 <>  (E)
    - libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 <>  (B,A)
    + libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 <>  (E)
    + libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 <>  (E)
    + libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 <>  (E)
    + libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 <>  (E)
    + libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 <>  (E)
    - libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 <>  (C,B,A)
    + libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 <>  (E)
    - libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 <>  (C,B,A)
    + libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 <>  (A)
    + libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 <>  (E)
      linux-vdso.so.1 <>
    

    【讨论】:

    • 我在我的帖子中加入了您的解决方案,如果没有人有更好的建议,我会将其标记为已解决
    【解决方案2】:

    来自 Git 的 git 差异。

    git add ldd.tmp1 ldd.tmp2 ldd.tmp3
    git diff -- | grep + > result.tmp
    

    https://git-scm.com/docs/git-diff

    还:

    • 融合
    • Kdiff3

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 2013-03-29
      • 1970-01-01
      • 1970-01-01
      • 2019-06-27
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      相关资源
      最近更新 更多