删除文件中的重复记录 [重复]

Question

可能重复：
How might I remove duplicate lines from a file?

我有一个要删除的包含重复记录的文件。这是我尝试过的

import sys  

for line in sys.stdin:  
    line = line.rstrip()  
    line = line.split()  
    idlist = []   
    if idlist == []:  
        idlist = line[1]  
    else:  
    idlist.append(line[1])  
    print line[0], idlist  

#没用

还有这个

for line in sys.stdin:  
    line = line.rstrip()  
    line = line.split()  
    lines_seen = set()  
    dup = line[1]  
    if dup not in lines_seen:  
        lines_seen = dup  
    else:  
        lines_seen.append(dup)  
    print line[0], lines_seen  
    
sys.stdin.close()

#也不起作用！

这就是输入的样子

BLE 1234
BLE 1223
LLE 3456
ELE 1223
BLE 4444
ELE 5555
BLE 4444

这就是我希望输出的样子

BLE 1234
BLE 1223
LLE 3456
BLE 4444
ELE 5555

谢谢！ 边缘

Answer 1

import fileinput

ss = '''BLE 1234
BLE 1223
LLE 3456
ELE 1223
BLE 4444
ELE 5555
BLE 4444 
'''
with open('klmp.txt','w') as f:
    f.write(ss)





seen = []
for line in fileinput.input('klmp.txt',inplace=1):
    b = line.split()[1]
    if b not in seen:
        seen.append(b)
        print line.strip()

在 SO 中使用单词“fileinput”搜索，我发现：

How to delete all blank lines in the file with the help of python?

Answer 2

elem1_seen = set()                 # first initialize an empty set of seen elem[1]
lines_out = []                     # list of "unique" output lines
for line in sys.stdin:             # iterate over input
    elems = line.rstrip().split()  # split line into two elements
    if elems[1] not in elem1_seen: # if second element not seen before...
        lines_out.append(line)     # append the whole line to output
        elem1_seen.add(elems[1])   # add this second element to seen before set
print lines_out                    # print output

Answer 3

主要问题是您正在更改变量类型，这会造成一些混乱：

import sys  

for line in sys.stdin:  
    line = line.rstrip()   #Line is a string  
    line = line.split()    #Line is a list
    idlist = []            #idlist is a list
    if idlist == []:  
        idlist = line[1]   #id list is a string
    else:  
        idlist.append(line[1])  #and now?
    print line[0], idlist