【问题标题】:How to fix dates in the generated csv file?如何修复生成的 csv 文件中的日期?
【发布时间】:2019-10-29 18:12:20
【问题描述】:

Im writing a program that you put the log and it generates a csv about the exchange of virtual desktops, but Im 有一个大问题,因为它生成 csv 时会获取所有虚拟桌面 1,然后是所有 2、3、4、5 和 6。

我希望它在一天之内,我该怎么做?

我的代码现在是这样的,如何获取一天中的事件而不用数字分隔?

filepath = 'dgnet.log'
with open(filepath) as fp:
    line = fp.readline()
    cnt = 1
    Fls = []
    while line:
        if "Taskbarbuttons auf Desktop" in line.strip():
            loc = []
            loc.append(line.strip().split("> ---")[0])
            loc.append(line.strip().split("> ---")[1])
            Fls.append(loc)
        line = fp.readline()
        cnt += 1
df = pd.DataFrame(Fls, columns=['Time', 'Name'])
gk = df.groupby('Name')
finallist = []
for name, group in gk:
    g = group
    g['Time2'] = g['Time'].shift(1)
    for i in range(len(g)):
        loc1 = []
        if i % 2 == 0:
            x = np.nan
        else:
            x = g.iloc[i, 2]
        loc1.append(x)
        loc1.append(g.iloc[i, 0])
        loc1.append(g.iloc[i, 1])
        finallist.append(loc1)
df1 = pd.DataFrame(finallist, columns=['StartTime', 'EndTime', 'Name'])
df1.fillna(0, inplace=True)
df2 = df1[df1['StartTime'] != 0]
df2['date'] = ""

d1 = 1
d = datetime.datetime.now()
l = len(df2)
Lxs = []
for i in range(len(df2)):
    if i == 0:
        df2.iloc[i, 3] = (datetime.datetime.now()).date()
    if i != 0:
        s = pd.to_datetime(df2.iloc[i, 0]).time()
        e = pd.to_datetime(df2.iloc[i, 1]).time()
        pe = pd.to_datetime(df2.iloc[i - 1, 1]).time()
        if e > s:
            df2.iloc[i, 3] = (datetime.datetime.now() + datetime.timedelta(days=d1)).date()
        if pe > s:
            d1 = d1 + 1
            df2.iloc[i, 3] = (datetime.datetime.now() + datetime.timedelta(days=d1)).date()
        #             print(i,datetime.datetime.now() + datetime.timedelta(days=d1))
        elif e < s:
            df2.iloc[i, 3] = np.NaN
            Lxs.append([df2.iloc[i, 0], "23:59:00", df2.iloc[i, 2],
                        (datetime.datetime.now() + datetime.timedelta(days=d1)).date()])
            d1 = d1 + 1
            Lxs.append(["00:00:01", df2.iloc[i, 1], df2.iloc[i, 2],
                        (datetime.datetime.now() + datetime.timedelta(days=d1)).date()])
            l = l + 1

Lxsdf = pd.DataFrame(Lxs, columns=['StartTime', 'EndTime', 'Name', 'date'])
x = df2.append(Lxsdf)
x.fillna(0, inplace=True)
x1 = x[x['date'] != 0]
x1.index = x1['date']
x1.sort_index(inplace=True)
x1['EndTime'] = pd.to_datetime(x1['EndTime'])
x1['StartTime'] = pd.to_datetime(x1['StartTime'])
x1['s1'] = x1['EndTime'] - x1['StartTime']
x1.to_csv('file3.csv')

我希望是这样的:

taskbarbuttons auf desktop 1 = 50 seconds
taskbarbuttons auf desktop 2 = 1 minutes

目前是:

09:42:58    23:07:00     Taskbarbuttons auf Desktop 1
12:59:04    13:42:08     Taskbarbuttons auf Desktop 1
03:59:13    04:44:39     Taskbarbuttons auf Desktop 1
13:24:59    13:25:26     Taskbarbuttons auf Desktop 1
19:24:15    13:03:02     Taskbarbuttons auf Desktop 1
13:12:29    13:33:43     Taskbarbuttons auf Desktop 1
13:35:43    07:45:14     Taskbarbuttons auf Desktop 1
07:53:27    10:51:30     Taskbarbuttons auf Desktop 1

然后是所有 2、3、4、5 和 6。

【问题讨论】:

  • 这个问题我已经读了两遍了,还是不知道你在问什么。
  • 而不是line = p.readline()(两次)和while line - 所以你有3行代码 - 你可以使用单行代码:for line in p:。从文件中读取行是一种非常流行的方法。
  • 我不明白你们中的一些计算。您应该使用变量的名称,这意味着什么 - 即。 start_time 代替 send_time 代替 e
  • 可能大部分计算都可以在没有for i in range(len(df2)) 的情况下完成,但使用像df['StartTime'] = pd.to_datetime(df['StartTime']).time() 这样的一行
  • 我会支持@furas 到目前为止所说的一切,并钦佩他们的耐心。你能包括一些示例数据吗?一旦我可以实际运行该程序,我就可以查看它。

标签: python pandas csv dataframe


【解决方案1】:

我制作的大多数元素都没有iterrows()。我只需要在午夜过后将行分成两行。

为了测试,我使用io.StringIO() 来模拟带有数据的文件。

我尝试添加 cmets 来描述代码 导入日期时间 将熊猫导入为 pd 将 numpy 导入为 np

# read full file
#text = open(filename).read()

text = '''09:43:04> --- Taskbarbuttons auf Desktop 2

12:37:20> --- Taskbarbuttons auf Desktop 6

23:07:00> --- Taskbarbuttons auf Desktop 1

23:07:07> --- Taskbarbuttons auf Desktop 2

23:07:09> --- Taskbarbuttons auf Desktop 3

12:59:04> --- Taskbarbuttons auf Desktop 1

13:41:53> --- Taskbarbuttons auf Desktop 5

13:47:09> --- Taskbarbuttons auf Desktop 3'''


# --- read file ---

#filepath = 'dgnet.log'
#with open(filepath) as fp:

import io
with io.StringIO(text) as fp: # simulate file with data
    data = []
    for line in fp:
        line = line.strip()
        if "Taskbarbuttons auf Desktop" in line:
            row = line.strip().split("> ---")
            data.append(row)

# --- calculations ---

# create DataFrame

df = pd.DataFrame(data, columns=['StartTime', 'Name'])

# convert string to datetime. I don't convert to time

df['StartTime'] = pd.to_datetime(df['StartTime'])

# create second column to calculate difference 

df['EndTime'] = df['StartTime'].shift(-1)

# fill last cell with current datetime (without microsecond)

iloc_endtime = df.columns.get_loc('EndTime')
df.iloc[-1, iloc_endtime] = datetime.datetime.now().replace(microsecond=0)

print('--- df at start ---')
# I use columns names because sometimes I get columns in different order
print(df[['Name', 'StartTime', 'EndTime']])

# iterate rows to split at midnight and add day number I also add day to `StartTime` and a `EndTime` 

new_df = pd.DataFrame()
day = 1

for index, row in df.iterrows():
    #print(row)

    if row['EndTime'] < row['StartTime']:
        # split into two rows on midnight
        row2 = row.copy()

        # first row ends at midnight-1second
        row['EndTime']    = row['EndTime'].replace(hour=23, minute=59, second=59)

        # second row starts at midnight
        row2['StartTime'] = row['StartTime'].replace(hour=0, minute=0, second=0)

        # add day to first row
        dt = pd.Timedelta(day,'D')
        row['StartTime'] += dt
        row['EndTime']   += dt
        row['Day'] = day
        new_df = new_df.append(row)

        # add day+1 to second row
        day += 1

        dt = pd.Timedelta(day,'D')
        row2['StartTime'] += dt
        row2['EndTime']   += dt
        row2['Day'] = day
        new_df = new_df.append(row2)
    else:
        # add only day
        dt = pd.Timedelta(day,'D')
        row['StartTime'] += dt
        row['EndTime']   += dt
        row['Day'] = day
        new_df = new_df.append(row)

df = new_df 

print('--- splited rows and added days ---')        
print(df[['Name', 'StartTime', 'EndTime', 'Day']])

# calculate END - START

df['DiffTime'] = df['EndTime'] - df['StartTime']

print('--- diff time ---')
print(df[['Name', 'StartTime', 'EndTime', 'Day', 'DiffTime']])

# group by `Name` or `Name` and `Day` to sum DiffTimes

#total_time = df.groupby(['Name'])['DiffTime'].sum()  #.dt.seconds
total_time = df.groupby(['Name', 'Day'])['DiffTime'].sum()  #.dt.seconds

print('--- total time ---')
print(total_time)

# reset index to convert index to column 
final_df = total_time.reset_index()

print('--- final df ---')
print(final_df)

最终的 DataFrame 看起来像

                            Name  Day DiffTime
0   Taskbarbuttons auf Desktop 1  1.0 00:00:07
1   Taskbarbuttons auf Desktop 1  2.0 00:42:49
2   Taskbarbuttons auf Desktop 2  1.0 02:54:18
3   Taskbarbuttons auf Desktop 3  1.0 00:52:50
4   Taskbarbuttons auf Desktop 3  2.0 22:13:29
5   Taskbarbuttons auf Desktop 5  2.0 00:05:16
6   Taskbarbuttons auf Desktop 6  1.0 10:29:40

可以按日期和名称排序

final_df = final_df.sort_values(['Day', 'Name']) )

获取数据帧

                            Name  Day DiffTime
0   Taskbarbuttons auf Desktop 1  1.0 00:00:07
2   Taskbarbuttons auf Desktop 2  1.0 02:54:18
3   Taskbarbuttons auf Desktop 3  1.0 00:52:50
6   Taskbarbuttons auf Desktop 6  1.0 10:29:40
1   Taskbarbuttons auf Desktop 1  2.0 00:42:49
4   Taskbarbuttons auf Desktop 3  2.0 22:23:32
5   Taskbarbuttons auf Desktop 5  2.0 00:05:16

可用于显示预期结果。

例如

for index, row in final_df.iterrows():
    day = int(row['Day'])
    number = row['Name'].split(' ')[-1]
    time = str(row['DiffTime']).split(' ')[-1]
    print('day {} desktop {} was used for {}'.format(day, number, time))

给予

day 1 desktop 1 was used for 00:00:07
day 1 desktop 2 was used for 02:54:18
day 1 desktop 3 was used for 00:52:50
day 1 desktop 6 was used for 10:29:40
day 2 desktop 1 was used for 00:42:49
day 2 desktop 3 was used for 22:38:24
day 2 desktop 5 was used for 00:05:16

【讨论】:

  • 使用iterrows 可能是个糟糕的主意! pandas docs 状态 iterrowsnot 跨行保留 dtypes”,比 itertuples 慢,并且“你应该永远不要修改你的东西迭代。”小心!
  • @AlexanderCécile 我知道 pandas 有方法可以在没有循环的情况下制作东西,并且对于大多数修改,我从原始代码中减少了循环。我唯一不知道如何在没有循环的情况下使用一行来创建两个新行并同时增加 day - 我需要计算天数,因为原始 DataFrame 有没有日期的时间。所以我使用了 iterrows 并创建了新的数据框。也许你知道如何摆脱这个循环。
  • 我将看看如何摆脱循环,但我的意思是,如果您使用显式迭代,那么使用 itertuples 会好得多!出于好奇,我将进行更改,并运行一些基准测试。
猜你喜欢
  • 1970-01-01
  • 2017-12-26
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2017-05-08
  • 1970-01-01
  • 2015-03-11
  • 1970-01-01
相关资源
最近更新 更多