How to filter minute data using day and symbol?答案

【问题标题】：How to filter minute data using day and symbol?How to filter minute data using day and symbol?
【发布时间】：2022-12-27 15:12:28
【问题描述】：

Assuming I have a dataframe of minute OHLC:

                       timestamp    open    high     low  ...      vwap  symbol  volume_10_day        date
0      2022-09-22 08:00:00+00:00  3.8400  3.9700  3.8400  ...  3.898279     APE           None  2022-12-22
1      2022-09-22 08:05:00+00:00  3.9100  3.9600  3.9000  ...  3.913727     APE           None  2022-12-22
2      2022-09-22 08:10:00+00:00  3.9300  3.9500  3.9000  ...  3.927569     APE           None  2022-12-22
3      2022-09-22 08:15:00+00:00  3.9300  3.9500  3.9200  ...  3.922965     APE           None  2022-12-22
4      2022-09-22 08:20:00+00:00  3.9500  3.9800  3.9500  ...  3.965291     APE           None  2022-12-22
...                          ...     ...     ...     ...  ...       ...     ...            ...         ...
21288  2022-12-24 00:35:00+00:00  2.2400  2.2400  2.2200  ...  2.227360    XPON           None  2022-12-23
21289  2022-12-24 00:40:00+00:00  2.2488  2.2488  2.2488  ...  2.248800    XPON           None  2022-12-23
21290  2022-12-24 00:45:00+00:00  2.2500  2.2500  2.2200  ...  2.227422    XPON           None  2022-12-23
21291  2022-12-24 00:50:00+00:00  2.2500  2.2500  2.2200  ...  2.229057    XPON           None  2022-12-23
21292  2022-12-24 00:55:00+00:00  2.2395  2.2395  2.1700  ...  2.202498    XPON           None  2022-12-23

[21293 rows x 11 columns]

And a DF of daily data:

   level_0  index        date symbol  ...  change_1_day  change_10_day  volume_10_day  volume_1_day
0    22177  22177  2022-12-20   ICCM  ...    177.599829           None           None       30005.0
1    30404  30404  2022-12-22    APE  ...     75.182482           None           None        2224.0
2    46210  46210  2022-12-21   SINT  ...     57.161981           None           None      857345.0
3    47737  47737  2022-12-23   XPON  ...    139.185751           None           None         284.0

How can I use the second dataframe to filter the first, so that I only get back a DF of minute data which contain the symbol and days of the second DF?

【问题讨论】：

Please post a reproducible example to get a better answer: merge both df (how="left") and then drop_na. You get the date with df["timestamp"].dt.date

标签： python pandas numpy

【解决方案1】：

As an option, you can create columns with dates in each dataframe.

dfd['day'] = dfd['timestamp'].dt.date
dfm['day'] = dfm['timestamp'].dt.date

Make grouping in the minute chart by the 'day' column. Create an empty dataframe and fill in where there are dates and symbol matches.

import pandas as pd

dfd['timestamp'] =  pd.to_datetime(dfd['timestamp'], errors='raise')#day
dfm['timestamp'] =  pd.to_datetime(dfm['timestamp'], errors='raise')#minute
dfd['day'] = dfd['timestamp'].dt.date
dfm['day'] = dfm['timestamp'].dt.date


df = pd.DataFrame()

def my_func(x):
    global df
    ind = x.index[0]
    aaa = dfd[(dfd['day'] == x['day'][ind]) & (dfd['symbol'] == x['symbol'][ind])]
    if len(aaa) > 0:#match found add these rows
        df = pd.concat([df, x])

dfm.groupby('day').apply(my_func)

print(df)

Output

                  timestamp     open     high       vwap symbol         day
0 2020-11-05 09:00:00+00:00  11.2000  11.2000  11.199735    AAL  2020-11-05
1 2020-11-05 09:03:00+00:00  11.0900  11.0900  11.090000    AAL  2020-11-05
2 2020-11-05 09:07:00+00:00  11.1600  11.1800  11.167725    AAL  2020-11-05
3 2020-11-05 09:08:00+00:00  11.1800  11.1800  11.180000    AAL  2020-11-05
4 2020-11-05 09:09:00+00:00  11.1800  11.1800  11.180000    AAL  2020-11-05
5 2022-10-24 20:33:00+00:00  94.6000  94.6000  94.601900   SPOT  2022-10-24
6 2022-10-24 20:41:00+00:00  94.6600  94.6600  94.648500   SPOT  2022-10-24
7 2022-10-24 20:49:00+00:00  94.0000  94.0000  93.959055   SPOT  2022-10-24
8 2022-10-24 22:45:00+00:00  94.5000  94.5000  94.500000   SPOT  2022-10-24
9 2022-10-24 23:59:00+00:00  94.3999  94.3999  94.399162   SPOT  2022-10-24

【讨论】：