如何使用for-loop，if-else合并两个文件？

Question

根据以下数据：

data01 =

pos     ids      sample1_value   sample2_value
2969    a:b:c    12:13:15        12:13:15
3222    a:b:c    13:13:16        21:33:41
3416    a:b:c    19:13:18        21:33:41
5207    a:b:c    11:33:41        91:33:41
5238    a:b:c    21:13:45        31:27:63
5398    a:b:c    31:27:63        28:63:41
5403    a:b:c    15:7:125        71:33:41
5426    a:b:c    12:13:25        82:25:14
5434    a:b:c    12:17:15        52:33:52

假设我为每个样本计算了另一个 id (d) 值，但不是在每一行中。

data02 = 

pos     ids      sample1_value    sample2_value
2969    d        21               96
3416    d        52               85
5207    d        63               85
5398    d        27               52
5403    d        63               52
5434    d        81               63

问题：

我想为每个样本的每一行写下这个 d 值。

是否可以使用 for 循环将值写回？

预期的最终输出：

pos     ids       sample1_value    sample2_value
2969    a:b:c:d   12:13:15:21      12:13:15:.
3222    a:b:c:d   13:13:16:.       21:33:41:.
3416    a:b:c:d   19:13:18:52      21:33:41:.
................................
.......................... in the same way as above

我只为 sample01 尝试了以下代码：

data01 = open('data01.txt', 'r')
header01 = data01.readline()
data01 = data01.read().rstrip('n').split('\n')

# similar code for data02

data01_new = open('data01_new.txt', 'w')
data01_new.write(header01 + '\n')


for lines in data01:
    values01 = lines.split('\t')
    pos01 = values01[0]
    ids01 = values01[1]
    sample1_val01 = values01[2]

    for lines in data02:
        values02 = lines.split('\t')
        pos02 = values02[0]
        ids02 = values02[1]
        sample1_val02 = values02[2]

        if pos01 == pos02:
            data01_update = open('data01_new.txt', 'a')
            data01_update.write('\t'.join(pos1, (ids01+':'+ids02), sample1_val01+':'+sample1_val02)

        else:
            data01_update = open('data01_new.txt', 'a')
            data01_update.write('\t'.join(pos1, (ids01+':'+ids02), sample1_val01+':'+'.')

• 我知道嵌套循环使用文件大小的乘积会浪费大量时间。
• 我的 if/else 逻辑用匹配更新行，但后来被不匹配覆盖。

是否可以使用for-loop and if-else 解决此问题？

如果没有，我该如何使用 pandas 解决这个问题？

Answer 1

这里有一种方法，先合并pos上的两个数据，然后加入id，sample1和sample 2，最后只使用想要的列

data = data1.merge(data2, on = 'pos',how = 'outer').fillna('.')

data['ids'] = data['ids_x'] + ':'+ data['ids_y']
data['sample1_value'] = data['sample1_value_x'].astype(str) + ':'+ 
data['sample1_value_y'].astype(str)
data['sample2_value'] = data['sample2_value_x'].astype(str) + ':'+ 
data['sample2_value_y'].astype(str)
data = data[['pos', 'ids', 'sample1_value', 'sample2_value']]


    pos     ids     sample1_value   sample2_value
0   2969    a:b:c:d 12:13:15:21.0   12:13:15:96.0
1   3222    a:b:c:. 13:13:16:.      21:33:41:.
2   3416    a:b:c:d 19:13:18:52.0   21:33:41:85.0
3   5207    a:b:c:d 11:33:41:63.0   91:33:41:85.0
4   5238    a:b:c:. 21:13:45:.      31:27:63:.
5   5398    a:b:c:d 31:27:63:27.0   28:63:41:52.0
6   5403    a:b:c:d 15:7:125:63.0   71:33:41:52.0
7   5426    a:b:c:. 12:13:25:.      82:25:14:.
8   5434    a:b:c:d 12:17:15:81.0   52:33:52:63.0

Answer 2

这是对您当前逻辑的修正。

循环浏览更新文件，一次一行。 对于每一行，在主文件中前进到匹配的pos； 在途中写出不匹配的行。

当您找到匹配项时，更新信息（您已经知道该怎么做）。

data01 = open('data01.txt', 'r')
header01 = data01.readline()

# similar code for data02

data01_new = open('data01_new.txt', 'w')
data01_new.write(header01 + '\n')


line01 = data01.readline()
values01 = data01.readline().split(\t)
pos01 = values01[0]

for line02 in data02:
    # Parse next update line.
    values02 = line02.split('\t')
    ids02 = values02[1]
    sample1_val02 = values02[2]
    pos02 = values02[0]

    # Find the next line of master file to update.
    # Extract the pos until it matches update pos.
    while pos01 < pos02:
        # Write the previous line (not matched or updated).
        data01_new.write(line01)

        values01 = data01.readline().split(\t)
        pos01 = values01[0]

    ids01 = values01[1]
    sample1_val01 = values01[2]

    # At this point, you have pos01 == pos02
    # Update the information as needed;
    #   put the result into line01,
    #   so it gets written on the next "while" iteration.

Answer 3

如果文件按“pos”排序，则可以同时处理一行。

def parse_line(line):
    return line.split()

line1 = f1.readline()
line2 = f2.readline()

while line1:

    pos1, id1, v1, w1 = parse_line(line1)
    pos2, id2, v2, w2 = parse_line(line2)

    if pos2 == pos1:
        out_file.write('{:s}\t{:s}:{:s}\t{:s}:{:s}\t{:s}:{:s}\n'.format(
            pos1, id1, id2, v1, v2, w1, w2))
        line2 = f2.readline()
    else:
        out_file.write('{:s}\t{:s}:{:s}\t{:s}:{:s}\t{:s}:{:s}\n'.format(
            pos1, id1, id2, v1, '.', w1, '.'))

    line1 = f1.readline()

输出：

2969    a:b:c:d 12:13:15:21 12:13:15:96
3222    a:b:c:d 13:13:16:.  21:33:41:.
3416    a:b:c:d 19:13:18:52 21:33:41:85
5207    a:b:c:d 11:33:41:63 91:33:41:85
5238    a:b:c:d 21:13:45:.  31:27:63:.
5398    a:b:c:d 31:27:63:27 28:63:41:52
5403    a:b:c:d 15:7:125:63 71:33:41:52
5426    a:b:c:d 12:13:25:.  82:25:14:.
5434    a:b:c:d 12:17:15:81 52:33:52:63

Answer 4

#Merge two DFs on pos column
df3 = pd.merge(data01,data02,how='left',on='pos',suffixes=['','_y']).fillna('.')

#transfer data to a numpy array
data = df3.iloc[:,1:].values.astype(np.str).reshape(-1,2,3).transpose(1,0,2)

#concatenate relevant columns with ':' as delimeter.
df3.iloc[:,1:4] =np.core.defchararray.add(np.core.defchararray.add(data[0],':'),data[1])

#take the columns required.
df_final = df3[['pos', 'ids', 'sample1_value', 'sample2_value']]

Out[1372]: 
    pos      ids  sample1_value  sample2_value
0  2969  a:b:c:d  12:13:15:21.0  12:13:15:96.0
1  3222  a:b:c:.     13:13:16:.     21:33:41:.
2  3416  a:b:c:d  19:13:18:52.0  21:33:41:85.0
3  5207  a:b:c:d  11:33:41:63.0  91:33:41:85.0
4  5238  a:b:c:.     21:13:45:.     31:27:63:.
5  5398  a:b:c:d  31:27:63:27.0  28:63:41:52.0
6  5403  a:b:c:d  15:7:125:63.0  71:33:41:52.0
7  5426  a:b:c:.     12:13:25:.     82:25:14:.
8  5434  a:b:c:d  12:17:15:81.0  52:33:52:63.0