循环遍历 pandas 列以获取 wmd 相似性答案

【问题标题】：loop over pandas column for wmd similarity循环遍历 pandas 列以获取 wmd 相似性
【发布时间】：2020-12-01 00:30:44
【问题描述】：

我有两个数据框。两者都有两列。我想使用 wmd 为 source_label 列中的每个实体与 target_label 列中的实体找到最接近的匹配但是，最后我想要一个包含所有 4 列与实体相关的 DataFrame。

df1

,source_Label,source_uri
'neuronal ceroid lipofuscinosis 8',"http://purl.obolibrary.org/obo/DOID_0110723"
'autosomal dominant distal hereditary motor neuronopathy',"http://purl.obolibrary.org/obo/DOID_0111198"

df2

,target_label,target_uri
'neuronal ceroid ',"http://purl.obolibrary.org/obo/DOID_0110748"
'autosomal dominanthereditary',"http://purl.obolibrary.org/obo/DOID_0111110"

预期结果

,source_label, target_label, source_uri, target_uri, wmd score
'neuronal ceroid lipofuscinosis 8', 'neuronal ceroid ', "http://purl.obolibrary.org/obo/DOID_0110723", "http://purl.obolibrary.org/obo/DOID_0110748", 0.98
'autosomal dominant distal hereditary motor neuronopathy', 'autosomal dominanthereditary', "http://purl.obolibrary.org/obo/DOID_0111198", "http://purl.obolibrary.org/obo/DOID_0111110", 0.65

数据框太大了，我正在寻找一些更快的方法来迭代两个标签列。到目前为止，我尝试过这个：

list_distances = []
temp = []

def preprocess(sentence):
    return [w for w in sentence.lower().split()]

entity = df1['source_label']
target = df2['target_label']

 for i in tqdm(entity):
    for j in target:
        wmd_distance = model.wmdistance(preprocess(i), preprocess(j))
        temp.append(wmd_distance)
    list_distances.append(min(temp))
# print("list_distances", list_distances)
WMD_Dataframe = pd.DataFrame({'source_label': pd.Series(entity),
                              'target_label': pd.Series(target),
                              'source_uri': df1['source_uri'],
                              'target_uri': df2['target_uri'],
                              'wmd_Score': pd.Series(list_distances)}).sort_values(by=['wmd_Score'])
WMD_Dataframe = WMD_Dataframe.reset_index()

首先，此代码无法正常运行，因为其他两列直接来自 dfs'，并且没有考虑与 uri 的实体关系。当实体数以百万计时，如何使其更快。提前致谢。

【问题讨论】：

标签： python pandas numpy gensim word2vec

【解决方案1】：

快速修复：

closest_neighbour_index_df2 = []


def preprocess(sentence):
    return [w for w in sentence.lower().split()]



 
for i in tqdm(entity):
    temp = []
    for j in target:
        wmd_distance = model.wmdistance(preprocess(i), preprocess(j))
        temp.append(wmd_distance)
    # maybe assert to make sure its always right
    closest_neighbour_index_df2.append(np.argmin(np.array(temp))) 
    # return argmin to return index rather than the value. 
    
# Add the indices from df2 to df1

df1['closest_neighbour'] = closest_neighbour_index_df2 
# add information to respective row from df2 using the closest_neighbour column

【讨论】：