在特定情况下连接文件列表中的文件答案

【问题标题】：Concatenate files from list of files in specific cases在特定情况下连接文件列表中的文件
【发布时间】：2022-01-08 12:46:00
【问题描述】：

我在一个名为 data.tsv 的文件中有一个文件名列表。每行对应相同的样本 ID，每个 ID 最多可包含 8 个文件。我需要合并以“_1.[variable extension]”和“_2.[variable extension]”结尾的文件。

下面的数据，但是 stackover 流程将制表符转换为空格 - 应该用制表符分隔：

I5_fastq_path_1 I5_fastq_path_2 I5_fastq_path_3 I5_fastq_path_4 I5_fastq_path_5 I5_fastq_path_6 I5_fastq_path_7 I5_fastq_path_8                             
/some/path/to/directory/PD7597b_1_1.fastq.gz    /some/path/to/directory/PD7597b_1_2.fastq.gz    /some/path/to/directory/PD7597b_2_1.fastq.gz    /some/path/to/directory/PD7597b_2_2.fastq.gz    /some/path/to/directory/PD7597b_3_1.fastq.gz    /some/path/to/directory/PD7597b_3_2.fastq.gz    NA  NA                              
/some/path/to/directory/WTCHG_65902_709501_1.fastq.gz   /some/path/to/directory/WTCHG_65902_709501_2.fastq.gz   /some/path/to/directory/WTCHG_68106_709501_1.fastq.gz   /some/path/to/directory/WTCHG_68106_709501_2.fastq.gz   /some/path/to/directory/WTCHG_68107_709501_1.fastq.gz   /some/path/to/directory/WTCHG_68107_709501_2.fastq.gz   /some/path/to/directory/WTCHG_68108_709501_1.fastq.gz   /some/path/to/directory/WTCHG_68108_709501_1.fastq.gz                               
/some/path/to/directory/WTCHG_65902_702501_1.fastq.gz   /some/path/to/directory/WTCHG_65902_702501_2.fastq.gz   /some/path/to/directory/WTCHG_68106_702501_1.fastq.gz   /some/path/to/directory/WTCHG_68106_702501_2.fastq.gz   /some/path/to/directory/WTCHG_68107_702501_1.fastq.gz   /some/path/to/directory/WTCHG_68107_702501_2.fastq.gz   NA  NA                              
/some/path/to/directory/WTCHG_87945_712502_1.fastq.gz   /some/path/to/directory/WTCHG_87945_712502_2.fastq.gz   /some/path/to/directory/WTCHG_88506_712502_1.fastq.gz   /some/path/to/directory/WTCHG_88506_712502_2.fastq.gz   /some/path/to/directory/WTCHG_88507_712502_1.fastq.gz   /some/path/to/directory/WTCHG_88507_712502_2.fastq.gz   NA  NA                              
/some/path/to/directory/IFS003-W_DHE04956-7_HW3LFCCXX_L5_1.fq.gz    /some/path/to/directory/IFS003-W_DHE04956-7_HW3LFCCXX_L5_2.fq.gz    /some/path/to/directory/IFS003-W_DHE04956-7_HW3LFCCXX_L6_1.fq.gz    /some/path/to/directory/IFS003-W_DHE04956-7_HW3LFCCXX_L6_2.fq.gz    NA  NA  NA  NA                              
/some/path/to/directory/WTCHG_65902_710501_1.fastq.gz   /some/path/to/directory/WTCHG_65902_710501_2.fastq.gz   /some/path/to/directory/WTCHG_68106_710501_1.fastq.gz   /some/path/to/directory/WTCHG_68106_710501_2.fastq.gz   /some/path/to/directory/WTCHG_68107_710501_1.fastq.gz   /some/path/to/directory/WTCHG_68107_710501_2.fastq.gz   NA  NA                              
/some/path/to/directory/NG178_S1_R1_001.fastq.gz    /some/path/to/directory/NG178_S1_R2_001.fastq.gz    /some/path/to/directory/NG178_S3_R1_001.fastq.gz    /some/path/to/directory/NG178_S3_R2_001.fastq.gz    NA  NA  NA  NA                              
/some/path/to/directory/NG232_S8_R1_001.fastq.gz    /some/path/to/directory/NG232_S8_R2_001.fastq.gz    /some/path/to/directory/NG232_S2_R1_001.fastq.gz    /some/path/to/directory/NG232_S2_R2_001.fastq.gz    NA  NA  NA  NA                              
/some/path/to/directory/NG367_S19_R1_001.fastq.gz   /some/path/to/directory/NG367_S19_R2_001.fastq.gz   /some/path/to/directory/NG367_S6_R1_001.fastq.gz    /some/path/to/directory/NG367_S6_R2_001.fastq.gz    NA  NA  NA  NA                              
/research/HGIGData/2022_NGS_BACKUP/IBD/PR0006_NDHE04397-A5-A5_HMTN3CCXY_L1_1.fq.gz  /research/HGIGData/2022_NGS_BACKUP/IBD/PR0006_NDHE04397-A5-A5_HMTN3CCXY_L1_2.fq.gz  /research/HGIGData/2022_NGS_BACKUP/IBD/PR0006_NDHE04397-A5-A5_HMTVYCCXY_L3_1.fq.gz  /research/HGIGData/2022_NGS_BACKUP/IBD/PR0006_NDHE04397-A5-A5_HMTVYCCXY_L3_2.fq.gz  /research/HGIGData/2022_NGS_BACKUP/IBD/PR0006_NDHE04397-A5-A5_HMVCKCCXY_L5_1.fq.gz  /research/HGIGData/2022_NGS_BACKUP/IBD/PR0005-A5_HMVCKCCXY_L5_2.fq.gz   NA  NA                              6_NDHE04397-A

如果更简单，您可以在 html 中运行代码，并且在复制到文本编辑器时以制表符分隔。

<table>
    <thead>
        <tr>
            <td>I5_fastq_path_1</td>
            <td>I5_fastq_path_2</td>
            <td>I5_fastq_path_3</td>
            <td>I5_fastq_path_4</td>
            <td>I5_fastq_path_5</td>
            <td>I5_fastq_path_6</td>
            <td>I5_fastq_path_7</td>
            <td>I5_fastq_path_8</td>
        </tr>
    </thead>
    <tr>
        <td>/some/path/to/directory/PD7597b_1_1.fastq.gz</td>
        <td>/some/path/to/directory/PD7597b_1_2.fastq.gz</td>
        <td>/some/path/to/directory/PD7597b_2_1.fastq.gz</td>
        <td>/some/path/to/directory/PD7597b_2_2.fastq.gz</td>
        <td>/some/path/to/directory/PD7597b_3_1.fastq.gz</td>
        <td>/some/path/to/directory/PD7597b_3_2.fastq.gz</td>
        <td>NA</td>
        <td>NA</td>
    </tr>
    <tr>
        <td>/some/path/to/directory/WTCHG_65902_709501_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_65902_709501_2.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68106_709501_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68106_709501_2.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68107_709501_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68107_709501_2.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68108_709501_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68108_709501_1.fastq.gz</td>
    </tr>
    <tr>
        <td>/some/path/to/directory/WTCHG_65902_702501_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_65902_702501_2.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68106_702501_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68106_702501_2.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68107_702501_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68107_702501_2.fastq.gz</td>
        <td>NA</td>
        <td>NA</td>
    </tr>
    <tr>
        <td>/some/path/to/directory/WTCHG_87945_712502_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_87945_712502_2.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_88506_712502_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_88506_712502_2.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_88507_712502_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_88507_712502_2.fastq.gz</td>
        <td>NA</td>
        <td>NA</td>
    </tr>
    <tr>
        <td>/some/path/to/directory/IFS003-W_DHE04956-7_HW3LFCCXX_L5_1.fq.gz</td>
        <td>/some/path/to/directory/IFS003-W_DHE04956-7_HW3LFCCXX_L5_2.fq.gz</td>
        <td>/some/path/to/directory/IFS003-W_DHE04956-7_HW3LFCCXX_L6_1.fq.gz</td>
        <td>/some/path/to/directory/IFS003-W_DHE04956-7_HW3LFCCXX_L6_2.fq.gz</td>
        <td>NA</td>
        <td>NA</td>
        <td>NA</td>
        <td>NA</td>
    </tr>
    <tr>
        <td>/some/path/to/directory/WTCHG_65902_710501_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_65902_710501_2.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68106_710501_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68106_710501_2.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68107_710501_1.fastq.gz</td>
        <td>/some/path/to/directory/WTCHG_68107_710501_2.fastq.gz</td>
        <td>NA</td>
        <td>NA</td>
    </tr>
    <tr>
        <td>/some/path/to/directory/NG178_S1_R1_001.fastq.gz</td>
        <td>/some/path/to/directory/NG178_S1_R2_001.fastq.gz</td>
        <td>/some/path/to/directory/NG178_S3_R1_001.fastq.gz</td>
        <td>/some/path/to/directory/NG178_S3_R2_001.fastq.gz</td>
        <td>NA</td>
        <td>NA</td>
        <td>NA</td>
        <td>NA</td>
    </tr>
    <tr>
        <td>/some/path/to/directory/NG232_S8_R1_001.fastq.gz</td>
        <td>/some/path/to/directory/NG232_S8_R2_001.fastq.gz</td>
        <td>/some/path/to/directory/NG232_S2_R1_001.fastq.gz</td>
        <td>/some/path/to/directory/NG232_S2_R2_001.fastq.gz</td>
        <td>NA</td>
        <td>NA</td>
        <td>NA</td>
        <td>NA</td>
    </tr>
    <tr>
        <td>/some/path/to/directory/NG367_S19_R1_001.fastq.gz</td>
        <td>/some/path/to/directory/NG367_S19_R2_001.fastq.gz</td>
        <td>/some/path/to/directory/NG367_S6_R1_001.fastq.gz</td>
        <td>/some/path/to/directory/NG367_S6_R2_001.fastq.gz</td>
        <td>NA</td>
        <td>NA</td>
        <td>NA</td>
        <td>NA</td>
    </tr>
    <tr>
        <td>/research/HGIGData/2022_NGS_BACKUP/IBD/PR0006_NDHE04397-A5-A5_HMTN3CCXY_L1_1.fq.gz</td>
        <td>/research/HGIGData/2022_NGS_BACKUP/IBD/PR0006_NDHE04397-A5-A5_HMTN3CCXY_L1_2.fq.gz</td>
        <td>/research/HGIGData/2022_NGS_BACKUP/IBD/PR0006_NDHE04397-A5-A5_HMTVYCCXY_L3_1.fq.gz</td>
        <td>/research/HGIGData/2022_NGS_BACKUP/IBD/PR0006_NDHE04397-A5-A5_HMTVYCCXY_L3_2.fq.gz</td>
        <td>/research/HGIGData/2022_NGS_BACKUP/IBD/PR0006_NDHE04397-A5-A5_HMVCKCCXY_L5_1.fq.gz</td>
        <td>/research/HGIGData/2022_NGS_BACKUP/IBD/PR0006_NDHE04397-A5-A5_HMVCKCCXY_L5_2.fq.gz</td>
        <td>NA</td>
        <td>NA</td>
    </tr>
</table>

如果每行有 4 列，则合并第 1 列和第 3 列中的文件，并合并第 2 列和第 4 列中的文件。

如果每行有 6 列，则合并第 1、3、5 列中的文件，并合并第 2、4 和 6 列中的文件。

此模式最多可延续 8 列。请注意，并非所有列都具有相同数量的文件，但它们始终是偶数。

我已经成功地为 4 列做到了这一点，但我想要一种方法来为 6 列和 8 列执行此操作，并且可能更多，而无需对其进行硬编码...我的硬编码尝试如下：

while read -r line; do
    path1=$(echo "$line"| cut -f1)
    path2=$(echo "$line"| cut -f2)
    path3=$(echo "$line"| cut -f3)
    path4=$(echo "$line"| cut -f4)

    #extract sample ID and append 'merged'
    ID=$(echo "$path1" | cut -d "." -f1 | sed 's#.*/##' | sed 's/_1/_merged/')

    #merge files
cat ${path1} ${path3} > /research/merged_fq/${ID}_1.fq.gz
cat ${path2} ${path4} > /research/merged_fq/${ID}_2.fq.gz

#create file with new file paths in
a=$(echo "/research/merged_fq/${ID}_1.fq.gz")
b=$(echo "/research/merged_fq/${ID}_2.fq.gz")
echo "$a" "$b" >>new_files
done < data.tsv

【问题讨论】：

这看起来像 HTML，而不是制表符分隔的数据。你能澄清一下吗？只需复制粘贴文本文件并添加代码格式，以我们可以轻松复制/粘贴minimal reproducible example 的形式提供制表符分隔的数据（尽管可能请注意，Stack Overflow 降价渲染引擎将在渲染时将制表符替换为空格它）。
这与这个问题有关吗？ stackoverflow.com/questions/70629122/…
如果数据确实是制表符分隔的，while IFS=$'\t' read -r path1 path2 path3 path4 path5 path6 path7 path8; do ... 如果缺少某些字段，这些变量将为空。（设计这种数据格式的人可能会收到不愉快的生命威胁。）
对PD7597b_1_1运行sed 's/_1/_merged/'会给你PD7597b_merged_1。这就是你想要的吗？
@triplee 当我复制和粘贴（转换为空格）时，我无法让 tab sep 工作，所以我将它转换为 html，以便在运行时可以以相同的格式查看它。对不起，如果那是错误的。

标签： bash csv unix

【解决方案1】：

修复了所有脚本并在各处添加了 cmets，因此您可以详细了解它的工作原理：

#!/usr/bin/env bash

# Path where to store merged fq files
mergePath='/research/merged_fq'

{
  # Print header for merged files tsv
  printf 'MergedOddFile\tMergedEvenFile\n'

  # Read in dummy variable to skip header row of data.tsv
  read -r _

  # Iterate reading rows of data until End Of File
  while read -r row; do
    # Map row fields to files array
    read -ra files <<<"$row"

    # Initialize arrays for odd and even files
    oddFiles=()
    evenFiles=()

    # Extract sample ID _merged, suffix for odd and even fastq files

    # Remove path to get file name only
    oddBaseName="${files[0]##*/}"

    # Remove everything from after the last '_' and add '_merged'
    oddID="${oddBaseName%_*}_merged"

    # Remove everything until the last '_'
    oddSuffix="${oddBaseName##*_}"

    # Strip the all the .extension like .fq.gz
    oddSuffix="${oddSuffix%%.*}"

    # Same for even
    evenBaseName="${files[1]##*/}"
    evenID="${evenBaseName%_*}_merged"
    evenSuffix="${evenBaseName##*_}"
    evenSuffix="${evenSuffix%%.*}"

    # Iterate the files array indexes
    for i in "${!files[@]}"; do

      # Stop processing files when name is 'NA' or empty
      [[ 'NA' == "${files[i]}" || -z "${files[i]}" ]] && break

      # Add file to corresponding merge array
      if ((i % 2)); then
        evenFiles+=("${files[i]}")
      else
        oddFiles+=("${files[i]}")
      fi
    done

    # Compose merged file names
    oddMerge="${mergePath}/${oddID}_${oddSuffix}.fq.gz"
    evenMerge="${mergePath}/${evenID}_${evenSuffix}.fq.gz"

    # Merge odd and even files separately
    cat "${oddFiles[@]}" > "$oddMerge"
    cat "${evenFiles[@]}" > "$evenMerge"

    # Log merged files
    printf '%s\t%s\n' "$oddMerge" "$evenMerge"
  done
} < data.tsv > merged_files.tsv

【讨论】：

这太不可思议了，谢谢。我将逐行询问事情以了解您的所作所为。
@lecb 检查我的最后一分钟修复以跳过 data.tsv 中的标题行