使用 glob/merge 删除 NaN 行、特定 excel 文件中的某些列

Question

我想在加载到 excel 文件中的 for 循环中删除最终文件中的 NaN 行，并删除所有公司、电子邮件、创建的重复列，但最终加载到 excel 文件中。

这是我的 for 循环（以及随后合并到单个 DF），目前：

for f in glob.glob("./gowall-users-export-*.xlsx"):
    df = pd.read_excel(f)
    all_users_sheets_hosts.append(df)
    j = re.search('(\d+)', f)
    df.columns = df.columns.str.replace('.*Hosted Meetings.*', 'Hosted Meetings' + ' ' + j.group(1))

all_users_sheets_hosts = reduce(lambda left,right: pd.merge(left,right,on=['First Name', 'Last Name'], how='outer'), all_users_sheets_hosts)

这是生成的 DF 的前几行：

Company_x   First Name  Last Name   Emails_x    Created_x   Hosted Meetings 03112016    Facilitated Meetings_x  Attended Meetings_x Company_y   Emails_y    ... Created_x   Hosted Meetings 04122016    Facilitated Meetings_x  Attended Meetings_x Company_y   Emails_y    Created_y   Hosted Meetings 04212016    Facilitated Meetings_y  Attended Meetings_y
0   TS  X Y X@Y.com 03/10/2016  0.0 0.0 0.0 TS  X@Y.com ... 03/10/2016  0.0 0.0 2.0 NaN NaN NaN NaN NaN NaN
1   TS  X Y X@Y.com 03/10/2016  0.0 0.0 0.0 TS  X@Y.com ... 01/25/2016  0.0 0.0 0.0 NaN NaN NaN NaN NaN NaN
2   TS  X Y X@Y.com 03/10/2016  0.0 0.0 0.0 TS  X@Y.com ... 04/06/2015  9.0 10.0    17.0    NaN NaN NaN NaN NaN NaN

Answer 1

为防止出现多个 Company、Emails、Created、Facilitated Meetings 和 Attended Meetings 列，请将它们从 right 数据帧中删除。要删除包含所有 NaN 值的行，请使用 result.dropna(how='all', axis=0)：

import pandas as pd
import functools

for f in glob.glob("./gowall-users-export-*.xlsx"):
    df = pd.read_excel(f)
    all_users_sheets_hosts.append(df)
    j = re.search('(\d+)', f)
    df.columns = df.columns.str.replace('.*Hosted Meetings.*', 
                                        'Hosted Meetings' + ' ' + j.group(1))

# Drop rows of all NaNs from the final DataFrame in `all_users_sheets_hosts`
all_users_sheets_hosts[-1] = all_users_sheets_hosts[-1].dropna(how='all', axis=0)

def mergefunc(left, right):
    cols = ['Company', 'Emails', 'Created', 'Facilitated Meetings', 'Attended Meetings']
    right = right.drop(cols, axis=1)
    result = pd.merge(left, right, on=['First Name', 'Last Name'], how='outer')
    return result

all_users_sheets_hosts = functools.reduce(mergefunc, all_users_sheets_hosts)

自从Company 等。人。列将仅存在于left DataFrame 中，这些列不会扩散。但是请注意，如果 left 和 right DataFrame 在这些列中具有不同的值，则只会保留 all_users_sheets_hosts 中第一个 DataFrame 中的值。

---

替代方案，如果 left 和 right DataFrames 对于 Company 等具有相同的值。人。列，那么另一种选择是也对这些列进行简单合并：

def mergefunc(left, right):
    cols = ['First Name', 'Last Name', 'Company', 'Emails', 'Created', 
            'Facilitated Meetings', 'Attended Meetings']
    result = pd.merge(left, right, on=cols, how='outer')
    return result
all_users_sheets_hosts = functools.reduce(mergefunc, all_users_sheets_hosts)