在python中基于时间序列数据中的连续值计算活动答案

【问题标题】：Counting activities based on consecutive values in time series data in python在python中基于时间序列数据中的连续值计算活动
【发布时间】：2020-07-10 11:07:26
【问题描述】：

在时间序列数据框中df：

        id      timestamp                data 
27581   27822   2020-01-02 07:53:05.173  19.5    
27582   27823   2020-01-02 07:53:05.273  20.0    
27647   27888   2020-01-02 10:01:46.380  20.5    
27648   27889   2020-01-02 10:01:46.480  21.0    
27649   27890   2020-01-02 10:01:48.463  21.5    
27650   27891   2020-01-02 10:01:48.563  22.0    
27711   27952   2020-01-02 10:32:19.897  21.5    
27712   27953   2020-01-02 10:32:19.997  21.0
27861   28102   2020-01-02 11:34:41.940  21.5    
...

我想统计并记录data 在31 到35 之间连续超过 20 秒的活动。但是每次数据绝对值变化0.5时记录数据，不考虑时间，所以时间戳是不定期记录的。

希望我可以在这样的表格中获取所有活动：

id    start_time               end_time                 
0     2020-01-02 07:53:05.173  2020-01-02 07:54:04.563      
1     2020-01-02 08:53:05.273  2020-01-02 07:53:05.273      
2     2020-01-02 10:01:46.380  2020-01-02 10:01:59.380       
3     2020-01-02 10:21:46.480  2020-01-02 10:22:10.480       
...

其中start_time 和end_time 是31-35 范围内第一个和最后一个数据的时间戳。

我试过了：

def data_interval(df):
    range_df = []
    df1=df[df['data']>0]
    df1=df1.drop_duplicates(['timestamp'])
    df1=df1.sort_values(by=['timestamp'])

    status = 0       # Let 0 represent data out of desired range, and 1 data in desired range.

    for i in range(len(df1)):     

      if status == 0:
        if ((df1.loc[i, 'data'] >= 31) and
            (df1.loc[i, 'data'] <= 35)):

          data_in_range = df1.loc[i, 'data']
          start_time = df1.loc[i, 'timestamp']

        continue

      if status == 1:

        if ((df1.loc[i, 'data'] < 31) or
            (df1.loc[i, 'data'] > 35)):

          end_time = df1.loc[i, 'timestamp']

          range_df.append([data_in_range, 
                           start_time, 
                           end_time]) 

          status = 0 

        continue 


    final_df=pd.DataFrame(range_df,columns=['Data','Start_time', 'End_time'])

    return final_df

final_df = data_interval(df)
final_df

返回错误

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2896             try:
-> 2897                 return self._engine.get_loc(key)
   2898             except KeyError:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 0

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
8 frames
/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2897                 return self._engine.get_loc(key)
   2898             except KeyError:
-> 2899                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2900         indexer = self.get_indexer([key], method=method, tolerance=tolerance)
   2901         if indexer.ndim > 1 or indexer.size > 1:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 0

如果我们可以将这些活动绘制成图表来验证细节，那就更好了。

【问题讨论】：

这当然是可能的，但必须记住，SO 不是免费的编码和设计网站。请尝试一下，并在遇到特定编程问题时提出问题。
堆栈跟踪不完整？我没有看到你端的函数调用
@MadPhysicist 我已经复制并粘贴了我所看到的整个 Traceback。我想我用final_df = data_interval(df) 调用了该函数，但我不确定为什么它不在 Traceback 中。
可能在循环前添加 df1.reset_index()
@trigonom 谢谢。我在for i in range(len(df1)): 行前面添加了您的代码，并且输出是相同的...

标签： python pandas algorithm numpy plot

【解决方案1】：

您可以为此使用traces。要从数据框中获取时间序列：

import traces

data = traces.TimeSeries(data=df[['timestamp', 'data']].itertuples(index=False))

或直接制作一个 TimeSeries，例如像这样：

data = traces.TimeSeries(data=[
    (datetime(2016, 9, 27, 23, 0, 0), 30),
    (datetime(2016, 9, 27, 23, 0, 5), 33),
    (datetime(2016, 9, 27, 23, 0, 19), 33),
    (datetime(2016, 9, 27, 23, 0, 15), 34),
    (datetime(2016, 9, 27, 23, 0, 31), 25),
    (datetime(2016, 9, 27, 23, 0, 45), 30),
    (datetime(2016, 9, 27, 23, 0, 45), 38),
    (datetime(2016, 9, 27, 23, 1, 45), 33),
    (datetime(2016, 9, 27, 23, 1, 55), 24),
    (datetime(2016, 9, 27, 23, 2, 15), 24),
])

要获取 TimeSeries 在范围内的时间间隔：

# create a new TimeSeries of whether the original is between 31 and 35
inrange = traces.TimeSeries(default=False)
for t, v in data:
    inrange[t] = (31 < v < 35)

# remove repeated measurements
inrange.compact()

# make a list of intervals that are between 31 and 35 for more than 20 seconds
intervals_in_range = []
for t0, t1, value in inrange.iterperiods(value=True):
    if (t1 - t0).total_seconds() > 20:
        intervals_in_range.append((t0, t1))

# [(datetime.datetime(2016, 9, 27, 23, 0, 5), datetime.datetime(2016, 9, 27, 23, 0, 31))]

并制作一个情节：

import matplotlib.dates as mdates
fig, ax = data.plot()
ax.fill_between([data.first_key(), data.last_key()], 31, 35, color="#eee")
ax.xaxis.set_major_formatter(mdates.DateFormatter('%H:%M:%S'))

【讨论】：