【问题标题】:Pandas - Count of tasks spanning several X minute timeslotsPandas - 跨越几个 X 分钟时隙的任务计数
【发布时间】:2018-10-16 21:54:48
【问题描述】:

考虑以下数据:

Index   Task        Start                       Finish
0       RandomName  2018-10-15T13:30:00+00:00   2018-10-15T13:41:00+00:00
1       RandomName  2018-10-15T13:40:00+00:00   2018-10-15T13:51:00+00:00
2       RandomName  2018-10-15T13:50:00+00:00   2018-10-15T13:51:00+00:00
3       RandomName  2018-10-15T14:10:00+00:00   2018-10-15T14:11:00+00:00
4       RandomName  2018-10-15T14:20:00+00:00   2018-10-15T14:21:00+00:00
5       RandomName  2018-10-15T14:30:00+00:00   2018-10-15T14:31:00+00:00

我要做的是生成此数据帧的 5 分钟段(一种时隙),并计算这些任务在所述段中发生的次数并尝试将其可视化。由于这些任务有持续时间,我首先必须通过以下方式生成细分:

import pandas as pd
from datetime import datetime, timedelta

def main():

   input_file = "input.csv"    
   df = pd.read_csv(
                input_file
                ,parse_dates=['Start','Finish']
                ,names=['Index', 'Job', 'Start', 'Finish']
                ,index_col='Index'
                ,header=None
                )

    # Find the duration of each task.
    df['Start']  = pd.to_datetime(df['Start'],dayfirst=True, errors='coerce')
    df['Finish'] = pd.to_datetime(df['Finish'],dayfirst=True, errors='coerce')
    df.loc[:,'Duration'] = df['Finish'].dt.minute - df['Start'].dt.minute

    # Define the range and split it into 5 minute segments
    rng_min = df['Start'].min()  # Earliest Date
    rng_max = df['Finish'].max() # Latest Date
    current = rng_min
    while current < rng_max:
         current += timedelta(minutes=5)

if __name__ == "__main__":
     main()

一个任务可以扩展到几个 5 分钟的片段,所以它不是一个简单的计数。从这一点开始,我完全不知道该怎么做,所以任何帮助都将不胜感激!

谢谢!

编辑 - 添加更多信息:

任务无关紧要,因为这里的目标是产生空的(可用的)5 分钟片段

编辑 2 - 添加它的外观:

 Timeslot   Start Time           End Time          Tasks Running
  1         10/15/18 13:30  10/15/18 13:35  1
  2         10/15/18 13:35  10/15/18 13:40  1
  3         10/15/18 13:40  10/15/18 13:45  2
  4         10/15/18 13:45  10/15/18 13:50  3

【问题讨论】:

  • 您的预期输出是什么样的?
  • 添加到帖子中 - 谢谢!
  • 可以按如下方式完成: (a) 创建一个字典,键为开始时间,开始时间+5,开始时间+10分钟,直到你覆盖最后一条记录的开始时间。 (b) 将每个任务的完成时间与每个键进行比较。如果它大于键值,则将其附加为列表值。所以你将拥有 {'start_window1':['T1','T2'], 'start_window2':['T1','T2'] ...} 等等,其中 T1,T2 是任务名称(c)计算与键对应的每个列表的长度,以便得到您需要的答案,

标签: python pandas


【解决方案1】:

您可以在时间序列索引上使用resample 后跟reindex 来做您想做的事情:

重采样允许您更改日期时间索引的频率。在这种情况下,您想要“上采样” - 增加数据中的步骤数 然后,重新索引可以让您用 NA 填补空白


import pandas as pd
from datetime import datetime, timedelta
import math


def main(input_file="untitled.txt", minutes_per_segment=5):

    df = pd.read_csv(input_file
                     ,parse_dates=['Start','Finish']
                     ,names=['Index', 'Task', 'Start', 'Finish']
                     ,index_col='Index'
                     ,header=0
                     )

    # Find the duration of each task.
    df['Start']  = pd.to_datetime(df['Start'], dayfirst=True, errors='coerce')
    df['Finish'] = pd.to_datetime(df['Finish'], dayfirst=True, errors='coerce')

    # Get the number of <segments> minute segments that the task 
    # runs for, rounded up to the next integer value
    df['Segments'] = (df.apply(lambda x: math.ceil((x.Finish - 
                                                x.Start).total_seconds()/60/minutes_per_segment), 
                               axis='columns'))

    # You can skip this step if the values in your Task_Name are unique
    # if not, you need something so you can treat each entry independently
    df['Task_ID'] = df.index.astype(str)
    df['Task_Name'] = df.apply(lambda x: '_'.join([x.Task, x.Task_ID]), axis=1)

    # create a new df so that the start and end times are in separate rows
    df2 = pd.concat([df[['Task_Name','Start', 'Segments']]
                         .rename(columns={'Start':'Time'}), 
                     df[['Task_Name','Finish', 'Segments']]
                         .rename(columns={'Finish':'Time'})])

    df2 = df2.sort_values(by='Task_Name').set_index('Time')
    df2.index = pd.DatetimeIndex(df2.index)

    # group by the task name 
    # resample to create 5-minute blocks 
    # clean up columns
    df3 = (df2.groupby('Task_Name')
              .apply(lambda x: x.resample(rule='{interval}T'.format(interval=minutes_per_segment), 
                                          label='right',
                                          closed='right')
                                .asfreq()
                                .ffill()
                    ) 
              .reset_index(level=1)
              .rename(columns={'level_1':'Time'})
              .reset_index(drop=True)) 

    # reset the index as a datetime Index - needed to do the next reindex step 
    df3.set_index('Time', inplace=True)
    df3.index = pd.DatetimeIndex(df3.index)


    # group by the time and aggregate the data:
    #     count the number of tasks in the time group
    #     (optional) create a list of the task names (you can comment out this line, and the name in the 'reorder' step at the bottom, if you don't need this) 
    # reindex to get all the 5-minute segments in the date range
    df4 = (df3.reset_index()
              .groupby('Time')
              .agg({'Task_Name': {'Tasks_Running': 'count', 
                                  'Task_Names': lambda x: list(x) # you can get rid of this line if you prefer
                                 }
                   })
              .reindex(pd.date_range(start=df3.index.min(), 
                                     end=df3.index.max(), 
                                     freq='{segments}min'.format(segments=minutes_per_segment)))
          )

    # remove the multi-index created in the agg step
    df4.columns = [name[1] for name in df4.columns]
    df4.index.name = 'Start_Time'
    df4.reset_index(inplace=True)

    # Fill in the missing task count (any time periods newly added by the reindex will have 0 tasks)
    df4.Tasks_Running.fillna(0, inplace=True)

    # get the end time from the start time column
    df4['End_Time'] = df4.Start_Time.shift(-1).ffill()

    # reorder the columns for ease of reading
    df4 = df4[['Start_Time','End_Time','Tasks_Running', 'Task_Names']] # comment this out if you commented out the line in the df4 agg 

    df4.index.name = 'Timeslot'
    df4.reset_index(inplace=True)

    return df4

if __name__ == "__main__":
     main()

这给了你:

    Timeslot          Start_Time            End_Time  Tasks_Running                    Task_Names 
0          0 2018-10-15 13:30:00 2018-10-15 13:35:00            1.0                 [RandomName0] 
1          1 2018-10-15 13:35:00 2018-10-15 13:40:00            1.0                 [RandomName0] 
2          2 2018-10-15 13:40:00 2018-10-15 13:45:00            2.0    [RandomName0, RandomName1] 
3          3 2018-10-15 13:45:00 2018-10-15 13:50:00            2.0    [RandomName0, RandomName1] 
4          4 2018-10-15 13:50:00 2018-10-15 13:55:00            2.0    [RandomName1, RandomName2] 
5          5 2018-10-15 13:55:00 2018-10-15 14:00:00            2.0    [RandomName1, RandomName2] 
6          6 2018-10-15 14:00:00 2018-10-15 14:05:00            0.0                           NaN 
7          7 2018-10-15 14:05:00 2018-10-15 14:10:00            0.0                           NaN 
8          8 2018-10-15 14:10:00 2018-10-15 14:15:00            1.0                 [RandomName3] 
9          9 2018-10-15 14:15:00 2018-10-15 14:20:00            1.0                 [RandomName3] 
10        10 2018-10-15 14:20:00 2018-10-15 14:25:00            1.0                 [RandomName4] 
11        11 2018-10-15 14:25:00 2018-10-15 14:30:00            1.0                 [RandomName4] 
12        12 2018-10-15 14:30:00 2018-10-15 14:35:00            1.0                 [RandomName5] 
13        13 2018-10-15 14:35:00 2018-10-15 14:35:00            1.0                 [RandomName5] 

【讨论】:

  • 嗨,凯特莉,谢谢!我尝试了您的代码,但有一个错误:分段持续时间从 5 分钟变为 10 分钟(在您的回复示例中也是如此)。
  • 抱歉 - 您需要使用 resample,然后使用 reindex,来创建所有 5 分钟间隔。我已编辑我的答案以更正它。
  • 谢谢你,Katelie,你不知道你的帮助有多感激!不幸的是,在到达重采样部分时我仍然遇到错误: ValueError: cannot reindex from a duplicate axis [Line 40] which is ` .apply(lambda x: x.resample(rule='{interval}T'.format(interval= minutes_per_segment), label='right', closed='right')` 。你能帮帮我吗?是因为 task_name 不是唯一的吗?
  • hmm...您是否有任何行的开始时间与结束时间相同(尤其是以分钟为单位)?它适用于非唯一任务名称和直接重复行的组合,但不适用于结束时间 = 开始时间。如果您需要考虑亚秒级时间,您可能需要更改 timedelta 计算中的total_seconds 位。
  • 如果您愿意,可以再试一次样本数据?我意识到文件读取具有“作业”而不是示例中的“任务”,并且当我编辑时,Task_Name 创建丢失了。我还将 'header=None' 更改为 'header=0',以说明示例数据中的列标签(否则它不会为我读取),但这在您的实际数据中可能会有所不同。我的猜测是列名可能很奇怪?该值错误可能是索引或列问题。
【解决方案2】:

Groupby 是一种有用的数据分割方法。使用 date_range 函数将分段时间分配给列,频率为 5 分钟。展开此列以使用 itertuples() 创建一个新的数据帧,它遍历数据帧的每一行。从这里您可以对数据运行 groupby 函数,或根据需要更改它。

    df['Start'] = pd.to_datetime(df['Start'])
    df['Finish'] = pd.to_datetime(df['Finish'])
    df['Segments'] = df.index.map(lambda x: pd.date_range(start=df['Start'][x], end=df['Finish'][x], freq='5Min'))
    df1 = pd.DataFrame([(d, t.Task) for t in df.itertuples() for d in t.Segments])
    df1 = df1.rename(columns={0:'Time', 1:'Task'})
    grouped = df1.groupby(['Time'])
    for time, group in grouped:
        print(group)

【讨论】:

    【解决方案3】:

    您可以尝试类似的方法:

    #Copying your original dataframe into clipboard buffer
    df = pd.read_clipboard(index_col='Index')
    
    df[['Start', 'Finish']] = df[['Start','Finish']].apply(pd.to_datetime)
    
    df_out = df.apply(lambda x: pd.Series(pd.date_range(x.Start, x.Finish, freq='5T')), axis=1)\
      .stack()\
      .value_counts(bins=pd.date_range(df.Start.min(), df.Finish.max(), freq='5T'))\
      .sort_index()
    
    df_out.index = pd.MultiIndex.from_tuples(df_out.index.to_tuples())
    
    df_out = df_out.rename_axis(['Start', 'Finish']).rename('Task Running').reset_index()
    print(df_out)
    
    df_out.plot('Start','Task Running')
    

    输出(注意:包含时间间隔的开始或结束的模糊性,即应该将 13:35 的值包含在时间间隔的结尾或下一个时间间隔的开始):

                               Start              Finish  Task Running
    0  2018-10-15 13:29:59.999999999 2018-10-15 13:35:00             2
    1  2018-10-15 13:35:00.000000000 2018-10-15 13:40:00             2
    2  2018-10-15 13:40:00.000000000 2018-10-15 13:45:00             1
    3  2018-10-15 13:45:00.000000000 2018-10-15 13:50:00             2
    4  2018-10-15 13:50:00.000000000 2018-10-15 13:55:00             0
    5  2018-10-15 13:55:00.000000000 2018-10-15 14:00:00             0
    6  2018-10-15 14:00:00.000000000 2018-10-15 14:05:00             0
    7  2018-10-15 14:05:00.000000000 2018-10-15 14:10:00             1
    8  2018-10-15 14:10:00.000000000 2018-10-15 14:15:00             0
    9  2018-10-15 14:15:00.000000000 2018-10-15 14:20:00             1
    10 2018-10-15 14:20:00.000000000 2018-10-15 14:25:00             0
    11 2018-10-15 14:25:00.000000000 2018-10-15 14:30:00             1
    

    可视化输出:

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 2019-08-03
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2012-11-14
      • 2017-10-20
      • 2011-09-17
      相关资源
      最近更新 更多