优化大数据集计算的算法

Question

我再次发现自己被 pandas 难倒，以及如何最好地执行“矢量运算”。我的代码可以工作，但是遍历所有内容需要很长时间。

代码试图做的是循环遍历shapes.cv并确定哪个shape_pt_sequence是stop_id，然后将stop_lat和stop_lon分配给shape_pt_lat和shape_pt_lon，同时也将shape_pt_sequence 标记为is_stop。

地理信息系统

stop_times.csvLINK

trips.csvLINK

shapes.csvLINK

这是我的代码：

import pandas as pd
from haversine import *

'''
iterate through shapes and match stops along a shape_pt_sequence within
x amount of distance. for shape_pt_sequence that is closest, replace the stop
lat/lon to the shape_pt_lat/shape_pt_lon, and mark is_stop column with 1.
'''

# readability assignments for shapes.csv
shapes = pd.read_csv('csv/shapes.csv')
shapes_index = list(set(shapes['shape_id']))
shapes_index.sort(key=int)
shapes.set_index(['shape_id', 'shape_pt_sequence'], inplace=True)

# readability assignments for trips.csv
trips = pd.read_csv('csv/trips.csv')
trips_index = list(set(trips['trip_id']))
trips.set_index(['trip_id'], inplace=True)

# readability assignments for stops_times.csv
stop_times = pd.read_csv('csv/stop_times.csv')
stop_times.set_index(['trip_id','stop_sequence'], inplace=True)
print(len(stop_times.loc[1423492]))

# readability assginments for stops.csv
stops = pd.read_csv('csv/stops.csv')
stops.set_index(['stop_id'], inplace=True)

# for each trip_id
for i in trips_index:
    print('******NEW TRIP_ID******')
    print(i)
    i = i.astype(int)

    # for each stop_sequence in stop_times
    for x in range(len(stop_times.loc[i])):
        stop_lat = stop_times.loc[i,['stop_lat','stop_lon']].iloc[x,[0,1]][0]
        stop_lon = stop_times.loc[i,['stop_lat','stop_lon']].iloc[x,[0,1]][1]
        stop_coordinate = (stop_lat, stop_lon)
        print(stop_coordinate)

        # shape_id that matches trip_id
        print('**SHAPE_ID**')
        trips_shape_id = trips.loc[i,['shape_id']].iloc[0]
        trips_shape_id = int(trips_shape_id)
        print(trips_shape_id)

        smallest = 0

        for y in range(len(shapes.loc[trips_shape_id])):
            shape_lat = shapes.loc[trips_shape_id].iloc[y,[0,1]][0]
            shape_lon = shapes.loc[trips_shape_id].iloc[y,[0,1]][1]

            shape_coordinate = (shape_lat, shape_lon)

            haversined = haversine_mi(stop_coordinate, shape_coordinate)

            if smallest == 0 or haversined < smallest:
                smallest = haversined
                smallest_shape_pt_indexer = y
            else:
                pass

            print(haversined)
            print('{0:.20f}'.format(smallest))

        print('{0:.20f}'.format(smallest))
        print(smallest_shape_pt_indexer)

        # mark is_stop as 1
        shapes.iloc[smallest_shape_pt_indexer,[2]] = 1

        # replace coordinate value
        shapes.loc[trips_shape_id].iloc[y,[0,1]][0] = stop_lat
        shapes.loc[trips_shape_id].iloc[y,[0,1]][1] = stop_lon

shapes.to_csv('csv/shapes.csv', index=False)

Answer 1

你可以做些什么来优化这段代码是使用一些线程/工作者而不是那些。

我推荐使用Pool of Workes，因为它使用起来非常简单。

在：

for i in trips_index:

你可以使用类似的东西：

from multiprocessing import Pool

pool = Pool(processes=4)
result = pool.apply_async(func, trips_index)

比方法 func 会像：

def func(i):
   #code here

您可以简单地将整个 for 循环放入此方法中。 在这个例子中，它可以与 4 个子进程一起工作，这是一个很好的改进。

Answer 2

需要考虑的一点是，一组行程通常具有相同的停靠点序列和相同的形状数据（行程之间的唯一区别是时间）。因此，为 (stop_id, shape_id) 缓存 find-closest-point-on-shape 操作可能是有意义的。我敢打赌，这会使您的运行时间减少一个数量级。