使用熊猫创建数据框而不使用 for 循环的有效方法答案

【问题标题】：Efficient way to create dataframe using pandas with out using for loops使用熊猫创建数据框而不使用 for 循环的有效方法
【发布时间】：2018-07-31 15:50:30
【问题描述】：

我正在尝试从下面提到的字典中创建以下数据框。有没有有效的解决方案？

data_dict = {
    'Total_Amount' : '150.00',
    'LinkAPI' : [{"ConfidenceScore":4},{"ConfidenceScore":9}],
    'RecordID' : 5687,
    'ClientId' : 45,
    'Customer_Number' : ["HDMO70232"],
    'RowNumber' : 0,
    'Invoice_Number' : '',
    'Customer_Name' : 'HD MOTORCYCLES SIS/SVC'
}

数据框中的行数，应等于“LinkAPI”列表中的项目数。上述数据的数据框应如下图所示。

ClientId    Customer_Name   Customer_Number Invoice_Number  LinkAPI RecordID    RowNumber   Total_Amount
0   45  HD MOTORCYCLES SIS/SVC  [HDMO70232]     {'ConfidenceScore': 4}  5687    0   150.00
1   45  HD MOTORCYCLES SIS/SVC  [HDMO70232]     {'ConfidenceScore': 9}  5687    0   150.00

我尝试了两种解决方案来实现这一点。我希望有更好的方法来创建数据框。解决方案1：

items_number = len(data_dict['LinkAPI'])
df_dict = {k : [data_dict[k] for _ in range(items_number)] if k != 'LinkAPI' else data_dict[k]
           for k in data_dict.keys()}
df = pd.DataFrame(df_dict)

解决方案 2：

LinkAPI = data_dict["LinkAPI"]

df_new = pd.DataFrame(columns=list(df))  # list(df) is ['ClientId','Customer_Name', 'Customer_Number', 
                                            # 'Invoice_Number', 'LinkAPI','RecordID', 'RowNumber', 'Total_Amount']
i=0
for conf in LinkAPI:
    df_new.loc[i] = [data_dict["Total_Amount"], conf, data_dict["RecordID"], data_dict["ClientId"], data_dict["Customer_Number"],
                    data_dict["RowNumber"], data_dict["Invoice_Number"], data_dict["Customer_Name"]]
    i+=1

【问题讨论】：

标签： python pandas list-comprehension dictionary-comprehension

【解决方案1】：

使用json_normalize:

from pandas.io.json import json_normalize

cols = ['Total_Amount','RecordID','ClientId','Customer_Number',
        'RowNumber','Invoice_Number','Customer_Name']
df = json_normalize(data, 'LinkAPI', cols)
#data borrowed from HYRY
print (df)
   ConfidenceScore  test Total_Amount Invoice_Number  RowNumber  \
0              4.0   NaN       150.00                         0   
1              9.0   NaN       150.00                         0   
2              8.0   NaN      1500.00                         1   
3             10.0   NaN      1500.00                         1   
4             20.0   NaN      1500.00                         1   
5              NaN   2.0      1500.00                         1   

  Customer_Number  ClientId           Customer_Name  RecordID  
0       HDMO70232        45  HD MOTORCYCLES SIS/SVC      5687  
1       HDMO70232        45  HD MOTORCYCLES SIS/SVC      5687  
2       HDMO70232       415  HD MOTORCYCLES SIS/SVC     56287  
3       HDMO70232       415  HD MOTORCYCLES SIS/SVC     56287  
4       HDMO70232       415  HD MOTORCYCLES SIS/SVC     56287  
5       HDMO70232       415  HD MOTORCYCLES SIS/SVC     56287

【讨论】：

感谢您的回答

【解决方案2】：

我将您的数据更改为字典列表：

data = [
{
    'Total_Amount' : '150.00',
    'LinkAPI' : [{"ConfidenceScore":4},{"ConfidenceScore":9}],
    'RecordID' : 5687,
    'ClientId' : 45,
    'Customer_Number' : ["HDMO70232"],
    'RowNumber' : 0,
    'Invoice_Number' : '',
    'Customer_Name' : 'HD MOTORCYCLES SIS/SVC'
},
{
    'Total_Amount' : '1500.00',
    'LinkAPI' : [{"ConfidenceScore":8},{"ConfidenceScore":10}, {"ConfidenceScore":20}, {"test":2}],
    'RecordID' : 56287,
    'ClientId' : 415,
    'Customer_Number' : ["HDMO70232"],
    'RowNumber' : 1,
    'Invoice_Number' : '',
    'Customer_Name' : 'HD MOTORCYCLES SIS/SVC'
},
]

df = pd.DataFrame(data)

df2 = pd.DataFrame(np.concatenate(df.LinkAPI).tolist(), 
                   index=np.repeat(df.index, df.LinkAPI.str.len().astype(int)))

df.drop("LinkAPI", axis=1).join(df2)

输出：

   ClientId           Customer_Name Customer_Number Invoice_Number  RecordID  RowNumber Total_Amount  ConfidenceScore  test
0        45  HD MOTORCYCLES SIS/SVC     [HDMO70232]                     5687          0       150.00              4.0   NaN
0        45  HD MOTORCYCLES SIS/SVC     [HDMO70232]                     5687          0       150.00              9.0   NaN
1       415  HD MOTORCYCLES SIS/SVC     [HDMO70232]                    56287          1      1500.00              8.0   NaN
1       415  HD MOTORCYCLES SIS/SVC     [HDMO70232]                    56287          1      1500.00             10.0   NaN
1       415  HD MOTORCYCLES SIS/SVC     [HDMO70232]                    56287          1      1500.00             20.0   NaN
1       415  HD MOTORCYCLES SIS/SVC     [HDMO70232]                    56287          1      1500.00              NaN   2.0

【讨论】：

【解决方案3】：

我不知道这是否是一个选项，但如果您可以更改您的字典以使所有条目具有相等长度的列表（例如，只需重复您的 data_dict 中当前的值，您就可以使用 pd.DataFrame(data_dict) . 在您的情况下，字典中的每个条目的长度必须等于 2，因为这是字典中最长的条目（LinkAPI):

import pandas as pd
pd.set_option("display.width", 300)  # You can ignore this

data_dict = {
    'Total_Amount' : '150.00',
    'LinkAPI' : [{"ConfidenceScore":4},{"ConfidenceScore":9}],
    'RecordID' : [5687] * 2,
    'ClientId' : [45] * 2,
    'Customer_Number' : ["HDMO70232"] * 2,
    'RowNumber' : [0] * 2,
    'Invoice_Number' : [''] * 2,
    'Customer_Name' : ['HD MOTORCYCLES SIS/SVC'] * 2
}

df = pd.DataFrame(data_dict)

print df

它为您提供以下数据框：

   ClientId           Customer_Name Customer_Number Invoice_Number                  LinkAPI  RecordID  RowNumber Total_Amount
0        45  HD MOTORCYCLES SIS/SVC       HDMO70232                 {u'ConfidenceScore': 4}      5687          0       150.00
1        45  HD MOTORCYCLES SIS/SVC       HDMO70232                 {u'ConfidenceScore': 9}      5687          0       150.00

编辑：

为了澄清，要将字典读取到数据帧，pandas 要求每个条目（字典中的键将成为数据帧中的一列）的长度相同。否则会抛出ValueError:

ValueError: arrays must all be same length

【讨论】：