【发布时间】：2022-10-30 08:50:40
【问题描述】：
我需要导入和转换 xlsx 文件。它们以宽格式编写，我需要从每一行复制一些单元格信息，并将其与所有其他行的信息配对：
[编辑：更改格式以表示更复杂的要求]
源格式
| ID | Property | Activity1name | Activity1timestamp | Activity2name | Activity2timestamp |
|---|---|---|---|---|---|
| 1 | A | a | 1.1.22 00:00 | b | 2.1.22 10:05 |
| 2 | B | a | 1.1.22 03:00 | b | 5.1.22 20:16 |
目标格式
| ID | Property | Activity | Timestamp |
|---|---|---|---|
| 1 | A | a | 1.1.22 00:00 |
| 1 | A | b | 2.1.22 10:05 |
| 2 | B | a | 1.1.22 03:00 |
| 2 | B | b | 5.1.22 20:16 |
以下代码可以很好地转换数据，但过程非常非常慢：
def transform(data_in):
data = pd.DataFrame(columns=columns)
# Determine number of processes entered in a single row of the original file
steps_per_row = int((data_in.shape[1] - (len(columns) - 2)) / len(process_matching) + 1)
data_in = data_in.to_dict("records") # Convert to dict for speed optimization
for row_dict in tqdm(data_in): # Iterate over each row of the original file
new_row = {}
# Set common columns for each process step
for column in column_matching:
new_row[column] = row_dict[column_matching[column]]
for step in range(0, steps_per_row):
rep = str(step+1) if step > 0 else ""
# Iterate for as many times as there are process steps in one row of the original file and
# set specific columns for each process step, keeping common column values identical for current row
for column in process_matching:
new_row[column] = row_dict[process_matching[column]+rep]
data = data.append(new_row, ignore_index=True) # append dict of new_row to existing data
data.index.name = "SortKey"
data[timestamp].replace(r'.000', '', regex=True, inplace=True) # Remove trailing zeros from timestamp # TODO check if works as intended
data.replace(r'^\s*$', float('NaN'), regex=True, inplace=True) # Replace cells with only spaces with nan
data.dropna(axis=0, how="all", inplace=True) # Remove empty rows
data.dropna(axis=1, how="all", inplace=True) # Remove empty columns
data.dropna(axis=0, subset=[timestamp], inplace=True) # Drop rows with empty Timestamp
data.fillna('', inplace=True) # Replace NaN values with empty cells
return data
显然，遍历每一行甚至每一列都不是如何正确使用 pandas 的，但我看不出这种转换如何被矢量化。
我尝试过使用并行化（modin）并尝试使用 dict 或不使用，但它没有工作/帮助。脚本的其余部分实际上只是打开并保存文件，所以问题就在这里。
对于如何提高速度的任何想法，我将不胜感激！
【问题讨论】：
标签： python pandas optimization transformation