【问题标题】:Pandas: rolling mean by time interval熊猫:按时间间隔滚动平均值
【发布时间】:2013-03-24 04:21:55
【问题描述】:

我有一堆投票数据;我想计算一个 Pandas 滚动平均值,以根据三天的窗口估算每一天。根据this questionrolling_* 函数根据指定数量的值计算窗口,而不是特定的日期时间范围。

如何实现此功能?

示例输入数据:

polls_subset.tail(20)
Out[185]: 
            favorable  unfavorable  other

enddate                                  
2012-10-25       0.48         0.49   0.03
2012-10-25       0.51         0.48   0.02
2012-10-27       0.51         0.47   0.02
2012-10-26       0.56         0.40   0.04
2012-10-28       0.48         0.49   0.04
2012-10-28       0.46         0.46   0.09
2012-10-28       0.48         0.49   0.03
2012-10-28       0.49         0.48   0.03
2012-10-30       0.53         0.45   0.02
2012-11-01       0.49         0.49   0.03
2012-11-01       0.47         0.47   0.05
2012-11-01       0.51         0.45   0.04
2012-11-03       0.49         0.45   0.06
2012-11-04       0.53         0.39   0.00
2012-11-04       0.47         0.44   0.08
2012-11-04       0.49         0.48   0.03
2012-11-04       0.52         0.46   0.01
2012-11-04       0.50         0.47   0.03
2012-11-05       0.51         0.46   0.02
2012-11-07       0.51         0.41   0.00

每个日期的输出只有一行。

【问题讨论】:

标签: python pandas time-series rolling-computation


【解决方案1】:

同时,添加了时间窗口功能。看到这个link

In [1]: df = DataFrame({'B': range(5)})

In [2]: df.index = [Timestamp('20130101 09:00:00'),
   ...:             Timestamp('20130101 09:00:02'),
   ...:             Timestamp('20130101 09:00:03'),
   ...:             Timestamp('20130101 09:00:05'),
   ...:             Timestamp('20130101 09:00:06')]

In [3]: df
Out[3]: 
                     B
2013-01-01 09:00:00  0
2013-01-01 09:00:02  1
2013-01-01 09:00:03  2
2013-01-01 09:00:05  3
2013-01-01 09:00:06  4

In [4]: df.rolling(2, min_periods=1).sum()
Out[4]: 
                       B
2013-01-01 09:00:00  0.0
2013-01-01 09:00:02  1.0
2013-01-01 09:00:03  3.0
2013-01-01 09:00:05  5.0
2013-01-01 09:00:06  7.0

In [5]: df.rolling('2s', min_periods=1).sum()
Out[5]: 
                       B
2013-01-01 09:00:00  0.0
2013-01-01 09:00:02  1.0
2013-01-01 09:00:03  3.0
2013-01-01 09:00:05  3.0
2013-01-01 09:00:06  7.0

【讨论】:

  • 这应该是最佳答案。
  • rolling 可以采用的偏移量(如'2s')参数的文档在这里:pandas.pydata.org/pandas-docs/stable/user_guide/…
  • 如果数据框中有多个列怎么办;我们如何指定特定的列?
  • @Brain_overflowed 设置为索引
  • min_period 使用这种方法似乎不可靠。对于 min_periods > 1,由于时间戳精度/可变采样率,您可能会在不期望的地方获得 NaN
【解决方案2】:

这样的事情怎么样:

首先将数据帧重新采样为一维间隔。这取所有重复日期的平均值。使用fill_method 选项填写缺失的日期值。接下来,将重新采样的帧传递给pd.rolling_mean,窗口为 3,min_periods=1:

pd.rolling_mean(df.resample("1D", fill_method="ffill"), window=3, min_periods=1)

            favorable  unfavorable     other
enddate
2012-10-25   0.495000     0.485000  0.025000
2012-10-26   0.527500     0.442500  0.032500
2012-10-27   0.521667     0.451667  0.028333
2012-10-28   0.515833     0.450000  0.035833
2012-10-29   0.488333     0.476667  0.038333
2012-10-30   0.495000     0.470000  0.038333
2012-10-31   0.512500     0.460000  0.029167
2012-11-01   0.516667     0.456667  0.026667
2012-11-02   0.503333     0.463333  0.033333
2012-11-03   0.490000     0.463333  0.046667
2012-11-04   0.494000     0.456000  0.043333
2012-11-05   0.500667     0.452667  0.036667
2012-11-06   0.507333     0.456000  0.023333
2012-11-07   0.510000     0.443333  0.013333

更新:正如 Ben 在 cmets 中指出的那样,with pandas 0.18.0 the syntax has changed。使用新语法,这将是:

df.resample("1d").sum().fillna(0).rolling(window=3, min_periods=1).mean()

【讨论】:

  • 抱歉,Pandas newb,ffill 究竟使用什么规则来提供缺失值?
  • 有几个填充选项。 ffill 代表前向填充并简单地传播最新的非缺失值。类似地,bfill 用于反向填充,以相反的顺序执行相同的操作。
  • 也许我在这里弄错了,但是您是否忽略了同一天的多个读数(当采用滚动平均值时,您会期望两个读数比一个读数更重......)
  • 很好的答案。请注意,在 pandas 0.18.0 中,syntax changed。新语法为:df.resample("1D").ffill(limit=0).rolling(window=3, min_periods=1).mean()
  • 在我使用的 pandas 0.18.1 版中复制原始答案的结果:df.resample("1d").mean().rolling(window=3, min_periods=1).mean()
【解决方案3】:

我只是有同样的问题,但数据点的间距不规则。重采样在这里并不是一个真正的选择。所以我创建了自己的函数。也许它对其他人也有用:

from pandas import Series, DataFrame
import pandas as pd
from datetime import datetime, timedelta
import numpy as np

def rolling_mean(data, window, min_periods=1, center=False):
    ''' Function that computes a rolling mean

    Parameters
    ----------
    data : DataFrame or Series
           If a DataFrame is passed, the rolling_mean is computed for all columns.
    window : int or string
             If int is passed, window is the number of observations used for calculating 
             the statistic, as defined by the function pd.rolling_mean()
             If a string is passed, it must be a frequency string, e.g. '90S'. This is
             internally converted into a DateOffset object, representing the window size.
    min_periods : int
                  Minimum number of observations in window required to have a value.

    Returns
    -------
    Series or DataFrame, if more than one column    
    '''
    def f(x):
        '''Function to apply that actually computes the rolling mean'''
        if center == False:
            dslice = col[x-pd.datetools.to_offset(window).delta+timedelta(0,0,1):x]
                # adding a microsecond because when slicing with labels start and endpoint
                # are inclusive
        else:
            dslice = col[x-pd.datetools.to_offset(window).delta/2+timedelta(0,0,1):
                         x+pd.datetools.to_offset(window).delta/2]
        if dslice.size < min_periods:
            return np.nan
        else:
            return dslice.mean()

    data = DataFrame(data.copy())
    dfout = DataFrame()
    if isinstance(window, int):
        dfout = pd.rolling_mean(data, window, min_periods=min_periods, center=center)
    elif isinstance(window, basestring):
        idx = Series(data.index.to_pydatetime(), index=data.index)
        for colname, col in data.iterkv():
            result = idx.apply(f)
            result.name = colname
            dfout = dfout.join(result, how='outer')
    if dfout.columns.size == 1:
        dfout = dfout.ix[:,0]
    return dfout


# Example
idx = [datetime(2011, 2, 7, 0, 0),
       datetime(2011, 2, 7, 0, 1),
       datetime(2011, 2, 7, 0, 1, 30),
       datetime(2011, 2, 7, 0, 2),
       datetime(2011, 2, 7, 0, 4),
       datetime(2011, 2, 7, 0, 5),
       datetime(2011, 2, 7, 0, 5, 10),
       datetime(2011, 2, 7, 0, 6),
       datetime(2011, 2, 7, 0, 8),
       datetime(2011, 2, 7, 0, 9)]
idx = pd.Index(idx)
vals = np.arange(len(idx)).astype(float)
s = Series(vals, index=idx)
rm = rolling_mean(s, window='2min')

【讨论】:

  • 你能包括相关的进口吗?
  • 您能否提供一个示例输入数据帧,如果计算时间间隔滑动窗口,该数据帧将起作用,谢谢
  • 在原帖中添加了一个例子。
  • 同样可以现在使用s.rolling('2min', min_periods=1).mean()完成
【解决方案4】:

user2689410 的代码正是我所需要的。提供我的版本(归功于 user2689410),由于一次计算 DataFrame 中整行的平均值,速度更快。

希望我的后缀约定是可读的:_s:字符串,_i:int,_b:bool,_ser:系列和_df:DataFrame。如果您发现多个后缀,则 type 可以是两者。

import pandas as pd
from datetime import datetime, timedelta
import numpy as np

def time_offset_rolling_mean_df_ser(data_df_ser, window_i_s, min_periods_i=1, center_b=False):
    """ Function that computes a rolling mean

    Credit goes to user2689410 at http://stackoverflow.com/questions/15771472/pandas-rolling-mean-by-time-interval

    Parameters
    ----------
    data_df_ser : DataFrame or Series
         If a DataFrame is passed, the time_offset_rolling_mean_df_ser is computed for all columns.
    window_i_s : int or string
         If int is passed, window_i_s is the number of observations used for calculating
         the statistic, as defined by the function pd.time_offset_rolling_mean_df_ser()
         If a string is passed, it must be a frequency string, e.g. '90S'. This is
         internally converted into a DateOffset object, representing the window_i_s size.
    min_periods_i : int
         Minimum number of observations in window_i_s required to have a value.

    Returns
    -------
    Series or DataFrame, if more than one column

    >>> idx = [
    ...     datetime(2011, 2, 7, 0, 0),
    ...     datetime(2011, 2, 7, 0, 1),
    ...     datetime(2011, 2, 7, 0, 1, 30),
    ...     datetime(2011, 2, 7, 0, 2),
    ...     datetime(2011, 2, 7, 0, 4),
    ...     datetime(2011, 2, 7, 0, 5),
    ...     datetime(2011, 2, 7, 0, 5, 10),
    ...     datetime(2011, 2, 7, 0, 6),
    ...     datetime(2011, 2, 7, 0, 8),
    ...     datetime(2011, 2, 7, 0, 9)]
    >>> idx = pd.Index(idx)
    >>> vals = np.arange(len(idx)).astype(float)
    >>> ser = pd.Series(vals, index=idx)
    >>> df = pd.DataFrame({'s1':ser, 's2':ser+1})
    >>> time_offset_rolling_mean_df_ser(df, window_i_s='2min')
                          s1   s2
    2011-02-07 00:00:00  0.0  1.0
    2011-02-07 00:01:00  0.5  1.5
    2011-02-07 00:01:30  1.0  2.0
    2011-02-07 00:02:00  2.0  3.0
    2011-02-07 00:04:00  4.0  5.0
    2011-02-07 00:05:00  4.5  5.5
    2011-02-07 00:05:10  5.0  6.0
    2011-02-07 00:06:00  6.0  7.0
    2011-02-07 00:08:00  8.0  9.0
    2011-02-07 00:09:00  8.5  9.5
    """

    def calculate_mean_at_ts(ts):
        """Function (closure) to apply that actually computes the rolling mean"""
        if center_b == False:
            dslice_df_ser = data_df_ser[
                ts-pd.datetools.to_offset(window_i_s).delta+timedelta(0,0,1):
                ts
            ]
            # adding a microsecond because when slicing with labels start and endpoint
            # are inclusive
        else:
            dslice_df_ser = data_df_ser[
                ts-pd.datetools.to_offset(window_i_s).delta/2+timedelta(0,0,1):
                ts+pd.datetools.to_offset(window_i_s).delta/2
            ]
        if  (isinstance(dslice_df_ser, pd.DataFrame) and dslice_df_ser.shape[0] < min_periods_i) or \
            (isinstance(dslice_df_ser, pd.Series) and dslice_df_ser.size < min_periods_i):
            return dslice_df_ser.mean()*np.nan   # keeps number format and whether Series or DataFrame
        else:
            return dslice_df_ser.mean()

    if isinstance(window_i_s, int):
        mean_df_ser = pd.rolling_mean(data_df_ser, window=window_i_s, min_periods=min_periods_i, center=center_b)
    elif isinstance(window_i_s, basestring):
        idx_ser = pd.Series(data_df_ser.index.to_pydatetime(), index=data_df_ser.index)
        mean_df_ser = idx_ser.apply(calculate_mean_at_ts)

    return mean_df_ser

【讨论】:

    【解决方案5】:

    这个例子似乎需要一个加权平均值,正如@andyhayden 的评论中所建议的那样。例如,10/25 有两个民意调查,10/26 和 10/27 各有一个。如果您只是重新采样然后取平均值,那么与 10/25 的投票相比,这实际上为 10/26 和 10/27 的投票提供了两倍的权重。

    要对每个民意调查赋予同等​​权重,而不是赋予每个同等权重,您可以执行以下操作。

    >>> wt = df.resample('D',limit=5).count()
    
                favorable  unfavorable  other
    enddate                                  
    2012-10-25          2            2      2
    2012-10-26          1            1      1
    2012-10-27          1            1      1
    
    >>> df2 = df.resample('D').mean()
    
                favorable  unfavorable  other
    enddate                                  
    2012-10-25      0.495        0.485  0.025
    2012-10-26      0.560        0.400  0.040
    2012-10-27      0.510        0.470  0.020
    

    这为您提供了基于民意调查的平均值而不是基于天的平均值的原始成分。与以前一样,民意调查在 10/25 进行平均,但 10/25 的权重也会被存储,并且是 10/26 或 10/27 权重的两倍,以反映在 10/25 进行了两次民意调查。

    >>> df3 = df2 * wt
    >>> df3 = df3.rolling(3,min_periods=1).sum()
    >>> wt3 = wt.rolling(3,min_periods=1).sum()
    
    >>> df3 = df3 / wt3  
    
                favorable  unfavorable     other
    enddate                                     
    2012-10-25   0.495000     0.485000  0.025000
    2012-10-26   0.516667     0.456667  0.030000
    2012-10-27   0.515000     0.460000  0.027500
    2012-10-28   0.496667     0.465000  0.041667
    2012-10-29   0.484000     0.478000  0.042000
    2012-10-30   0.488000     0.474000  0.042000
    2012-10-31   0.530000     0.450000  0.020000
    2012-11-01   0.500000     0.465000  0.035000
    2012-11-02   0.490000     0.470000  0.040000
    2012-11-03   0.490000     0.465000  0.045000
    2012-11-04   0.500000     0.448333  0.035000
    2012-11-05   0.501429     0.450000  0.032857
    2012-11-06   0.503333     0.450000  0.028333
    2012-11-07   0.510000     0.435000  0.010000
    

    请注意,10/27 的滚动平均值现在是 0.51500(民意调查加权)而不是 52.1667(日加权)。

    另请注意,自 0.18.0 版起,resamplerolling 的 API 发生了变化。

    rolling (what's new in pandas 0.18.0)

    resample (what's new in pandas 0.18.0)

    【讨论】:

      【解决方案6】:

      当我尝试使用 window='1M' 时,我发现 user2689410 代码中断,因为营业月的增量引发了此错误:

      AttributeError: 'MonthEnd' object has no attribute 'delta'
      

      我添加了直接传递相对时间增量的选项,因此您可以为用户定义的时间段执行类似的操作。

      感谢您的指点,这是我的尝试-希望它有用。

      def rolling_mean(data, window, min_periods=1, center=False):
      """ Function that computes a rolling mean
      Reference:
          http://stackoverflow.com/questions/15771472/pandas-rolling-mean-by-time-interval
      
      Parameters
      ----------
      data : DataFrame or Series
             If a DataFrame is passed, the rolling_mean is computed for all columns.
      window : int, string, Timedelta or Relativedelta
               int - number of observations used for calculating the statistic,
                     as defined by the function pd.rolling_mean()
               string - must be a frequency string, e.g. '90S'. This is
                        internally converted into a DateOffset object, and then
                        Timedelta representing the window size.
               Timedelta / Relativedelta - Can directly pass a timedeltas.
      min_periods : int
                    Minimum number of observations in window required to have a value.
      center : bool
               Point around which to 'center' the slicing.
      
      Returns
      -------
      Series or DataFrame, if more than one column
      """
      def f(x, time_increment):
          """Function to apply that actually computes the rolling mean
          :param x:
          :return:
          """
          if not center:
              # adding a microsecond because when slicing with labels start
              # and endpoint are inclusive
              start_date = x - time_increment + timedelta(0, 0, 1)
              end_date = x
          else:
              start_date = x - time_increment/2 + timedelta(0, 0, 1)
              end_date = x + time_increment/2
          # Select the date index from the
          dslice = col[start_date:end_date]
      
          if dslice.size < min_periods:
              return np.nan
          else:
              return dslice.mean()
      
      data = DataFrame(data.copy())
      dfout = DataFrame()
      if isinstance(window, int):
          dfout = pd.rolling_mean(data, window, min_periods=min_periods, center=center)
      
      elif isinstance(window, basestring):
          time_delta = pd.datetools.to_offset(window).delta
          idx = Series(data.index.to_pydatetime(), index=data.index)
          for colname, col in data.iteritems():
              result = idx.apply(lambda x: f(x, time_delta))
              result.name = colname
              dfout = dfout.join(result, how='outer')
      
      elif isinstance(window, (timedelta, relativedelta)):
          time_delta = window
          idx = Series(data.index.to_pydatetime(), index=data.index)
          for colname, col in data.iteritems():
              result = idx.apply(lambda x: f(x, time_delta))
              result.name = colname
              dfout = dfout.join(result, how='outer')
      
      if dfout.columns.size == 1:
          dfout = dfout.ix[:, 0]
      return dfout
      

      还有一个用 3 天时间窗口计算平均值的例子:

      from pandas import Series, DataFrame
      import pandas as pd
      from datetime import datetime, timedelta
      import numpy as np
      from dateutil.relativedelta import relativedelta
      
      idx = [datetime(2011, 2, 7, 0, 0),
                 datetime(2011, 2, 7, 0, 1),
                 datetime(2011, 2, 8, 0, 1, 30),
                 datetime(2011, 2, 9, 0, 2),
                 datetime(2011, 2, 10, 0, 4),
                 datetime(2011, 2, 11, 0, 5),
                 datetime(2011, 2, 12, 0, 5, 10),
                 datetime(2011, 2, 12, 0, 6),
                 datetime(2011, 2, 13, 0, 8),
                 datetime(2011, 2, 14, 0, 9)]
      idx = pd.Index(idx)
      vals = np.arange(len(idx)).astype(float)
      s = Series(vals, index=idx)
      # Now try by passing the 3 days as a relative time delta directly.
      rm = rolling_mean(s, window=relativedelta(days=3))
      >>> rm
      Out[2]: 
      2011-02-07 00:00:00    0.0
      2011-02-07 00:01:00    0.5
      2011-02-08 00:01:30    1.0
      2011-02-09 00:02:00    1.5
      2011-02-10 00:04:00    3.0
      2011-02-11 00:05:00    4.0
      2011-02-12 00:05:10    5.0
      2011-02-12 00:06:00    5.5
      2011-02-13 00:08:00    6.5
      2011-02-14 00:09:00    7.5
      Name: 0, dtype: float64
      

      【讨论】:

        【解决方案7】:

        为了保持基本,我使用了一个循环和类似的东西来帮助你开始(我的索引是日期时间):

        import pandas as pd
        import datetime as dt
        
        #populate your dataframe: "df"
        #...
        
        df[df.index<(df.index[0]+dt.timedelta(hours=1))] #gives you a slice. you can then take .sum() .mean(), whatever
        

        然后您可以在该切片上运行函数。您可以看到如何添加一个迭代器以使窗口的开始不是数据帧索引中的第一个值,然后滚动窗口(例如,您也可以使用 > 规则作为开始)。

        请注意,对于超大数据或非常小的增量,这可能效率较低,因为您的切片可能会变得更加费力(对我来说,对于数十万行数据和几列来说已经足够好了,尽管对于几个星期的每小时窗口来说)

        【讨论】:

          【解决方案8】:

          检查您的索引是否真的是datetime,而不是str 可能会有所帮助:

          data.index = pd.to_datetime(data['Index']).values
          

          【讨论】:

            【解决方案9】:

            可视化滚动平均值,看看它是否有意义。我不明白为什么在请求滚动平均值时使用 sum。

              df=pd.read_csv('poll.csv',parse_dates=['enddate'],dtype={'favorable':np.float,'unfavorable':np.float,'other':np.float})
            
              df.set_index('enddate')
              df=df.fillna(0)
            
             fig, axs = plt.subplots(figsize=(5,10))
             df.plot(x='enddate', ax=axs)
             plt.show()
            
            
             df.rolling(window=3,min_periods=3).mean().plot()
             plt.show()
             print("The larger the window coefficient the smoother the line will appear")
             print('The min_periods is the minimum number of observations in the window required to have a value')
            
             df.rolling(window=6,min_periods=3).mean().plot()
             plt.show()
            

            【讨论】:

              猜你喜欢
              • 2020-07-30
              • 1970-01-01
              • 2016-09-10
              • 2017-08-03
              • 1970-01-01
              • 2015-01-12
              • 1970-01-01
              相关资源
              最近更新 更多