【发布时间】：2020-04-14 20:14:05
【问题描述】：
我在 pandas 中有一个 DataFrame，其中包含有关人员及时位置的信息。它大约有 300+ 百万行。
示例：
import pandas as pd
inp = [{'Name': 'John', 'Year':2018, 'Address':'Beverly hills'}, {'Name': 'John', 'Year':2018, 'Address':'Beverly hills'}, {'Name': 'John', 'Year':2019, 'Address':'Beverly hills'}, {'Name': 'John', 'Year':2019, 'Address':'Orange county'}, {'Name': 'John', 'Year':2019, 'Address':'New York'}, {'Name': 'Steve', 'Year':2018, 'Address':'Canada'}, {'Name': 'Steve', 'Year':2019, 'Address':'Canada'}, {'Name': 'Steve', 'Year':2019, 'Address':'Canada'}, {'Name': 'Steve', 'Year':2020, 'Address':'California'}, {'Name': 'Steve', 'Year':2020, 'Address':'Canada'}, {'Name': 'John', 'Year':2020, 'Address':'Canada'}, {'Name': 'John', 'Year':2021, 'Address':'Canada'}, {'Name': 'John', 'Year':2021, 'Address':'Beverly hills'}, {'Name': 'Steve', 'Year':2021, 'Address':'California'}, {'Name': 'Steve', 'Year':2022, 'Address':'California'}, {'Name': 'Steve', 'Year':2018, 'Address':'NewYork'}, {'Name': 'Steve', 'Year':2018, 'Address':'California'}, {'Name': 'Steve', 'Year':2022, 'Address':'NewYork'}]
df = pd.DataFrame(inp)
print (df)
输出：
Address Name Year
0 Beverly hills John 2018
1 Beverly hills John 2018
2 Beverly hills John 2019
3 Orange county John 2019
4 New York John 2019
5 Canada Steve 2018
6 Canada Steve 2019
7 Canada Steve 2019
8 California Steve 2020
9 Canada Steve 2020
10 Canada John 2020
11 Canada John 2021
12 Beverly hills John 2021
13 California Steve 2021
14 California Steve 2022
15 NewYork Steve 2018
16 California Steve 2018
17 NewYork Steve 2022
我想计算特定年份中地址之间的总变化。或者换句话说，有多少人在 2018 年从“加拿大”搬到“加利福尼亚”。
理想输出：
1) 每年的矩阵如下。示例：2019 年（包括 2018 年至 2019 年）的所有地址变化。
+---------------+---------------+---------------+----------+------------+
| From\ To | Beverly hills | Orange county | New York | California |
+---------------+---------------+---------------+----------+------------+
| Beverly hills | 0 | 1 | 0 | 0 |
+---------------+---------------+---------------+----------+------------+
| Orange county | 0 | 0 | 1 | 0 |
+---------------+---------------+---------------+----------+------------+
| New York | 0 | 2 | 0 | 0 |
+---------------+---------------+---------------+----------+------------+
| California | 0 | 0 | 0 | 0 |
+---------------+---------------+---------------+----------+------------+
2)所有年份的地址更改。
+---------------+---------------+------+------+------+
| Address 1 | Address 2 | 2018 | 2019 | 2020 |
+---------------+---------------+------+------+------+
| Beverly hills | Orange county | 0 | 1 | 0 |
+---------------+---------------+------+------+------+
| New York | Canada | 0 | 0 | 1 |
+---------------+---------------+------+------+------+
| Canada | New York | 1 | 0 | 0 |
+---------------+---------------+------+------+------+
| California | Canada | 0 | 1 | 2 |
+---------------+---------------+------+------+------+
到目前为止我的解决方案： 感谢@QuangHoang，我可以使用以下代码捕获“年份”的变化和“地址”的变化：
groups = df.groupby('Name')
for col in ['Year', 'Address']:
df[f'cng-{col}'] = groups[col].shift().fillna(df[col]).ne(df[col]).astype(int)
groups[col].shift() 在每个名称中将相应的列移动 1。 fillna(df[col] 用原始值填充每个（移位的）组中的第一行，表示没有变化。最后，ne(df[col]) 将移位后的值与原始值进行比较以进行更改。
产量：
+----+---------------+-------+------+----------+-------------+
| ID | Address | Name | Year | cng-Year | cng-Address |
+----+---------------+-------+------+----------+-------------+
| 0 | Beverly hills | John | 2018 | 0 | 0 |
+----+---------------+-------+------+----------+-------------+
| 1 | Beverly hills | John | 2018 | 0 | 0 |
+----+---------------+-------+------+----------+-------------+
| 2 | Beverly hills | John | 2019 | 1 | 0 |
+----+---------------+-------+------+----------+-------------+
| 3 | Orange county | John | 2019 | 0 | 1 |
+----+---------------+-------+------+----------+-------------+
| 4 | New York | John | 2019 | 0 | 1 |
+----+---------------+-------+------+----------+-------------+
| 10 | Canada | John | 2020 | 1 | 1 |
+----+---------------+-------+------+----------+-------------+
| 11 | Canada | John | 2021 | 1 | 0 |
+----+---------------+-------+------+----------+-------------+
| 12 | Beverly hills | John | 2021 | 0 | 1 |
+----+---------------+-------+------+----------+-------------+
| 5 | Canada | Steve | 2018 | 0 | 0 |
+----+---------------+-------+------+----------+-------------+
| 15 | NewYork | Steve | 2018 | 1 | 1 |
+----+---------------+-------+------+----------+-------------+
| 16 | California | Steve | 2018 | 0 | 1 |
+----+---------------+-------+------+----------+-------------+
| 6 | Canada | Steve | 2019 | 1 | 0 |
+----+---------------+-------+------+----------+-------------+
| 7 | Canada | Steve | 2019 | 0 | 0 |
+----+---------------+-------+------+----------+-------------+
| 8 | California | Steve | 2020 | 1 | 1 |
+----+---------------+-------+------+----------+-------------+
| 9 | Canada | Steve | 2020 | 0 | 1 |
+----+---------------+-------+------+----------+-------------+
| 13 | California | Steve | 2021 | 1 | 1 |
+----+---------------+-------+------+----------+-------------+
| 14 | California | Steve | 2022 | 1 | 0 |
+----+---------------+-------+------+----------+-------------+
| 17 | NewYork | Steve | 2022 | 1 | 1 |
+----+---------------+-------+------+----------+-------------+
【问题讨论】：
-
如果我理解正确，这更像是一个 networkx 问题，您需要根据日期顺序中的唯一值来识别位置之间的子/父关系。也因为你有 300mil + 行我不认为这最适合熊猫 - 最好在 SQL 或 Pyspark 中执行此操作
-
我已经阅读了更多关于 pandas 和 networkx 的内容，而且确实问题是用 graph、adjacency-matrix 和 networkx 包更好地处理。出于这个原因，我发布了带有相关标签的问题的修改版本 -> stackoverflow.com/q/61307877/2739921
标签： python pandas dataframe compare rows