读取 pandas 中的非结构化数据答案

【问题标题】：Read unstructured data in pandas读取 pandas 中的非结构化数据
【发布时间】：2020-08-08 13:58:21
【问题描述】：

我在一个文本文件中有以下非结构化数据，这是来自 Discord 的消息日志数据。

[06-Nov-19 03:36 PM] Dyno#0000

{Embed}
Server
**Message deleted in #reddit-feed**
Author: ? | Message ID: 171111183099756545

[12-Nov-19 01:35 PM] Dyno#0000

{Embed}
Member Left
@Unknown User
ID: 171111183099756545

[16-Nov-19 11:25 PM] Dyno#0000

{Embed}
Member Joined
@User
ID: 171111183099756545

基本上我的目标是解析数据并提取所有加入和留言，然后绘制服务器中成员的增长。有些消息是不相关的，每个消息块的行长也不同。

Date        Member-change
4/24/2020   2
4/25/2020   -1
4/26/2020   3

我尝试在循环中解析数据，但由于数据是非结构化的并且具有不同长度的行，我对如何设置它感到困惑。有没有办法忽略所有没有“成员加入”和“成员离开”的块？

【问题讨论】：

这似乎是您需要构建的东西，而不是使用内置方法。像with open(file) as f: data=f.readlines(); for line in f: if line.startswith('{')... 这样的东西。看看你到目前为止所做的尝试会很有帮助
您不能真正“忽略”文本，您必须在其中搜索您要查找的字符串。您能否发布一小部分用于解析和收集数据的代码示例。

标签： python pandas dataframe jupyter-notebook data-science

【解决方案1】：

它是结构化文本，只是不符合您的预期。如果文本以一致的格式编写，则文件可以被结构化，即使我们通常认为结构化文本是基于字段的。

字段由基于日期的标题分隔，后跟 {embed} 关键字，然后是您感兴趣的命令。

#! /usr/bin/env python
# -*- coding: utf-8 -*-

import re
from itertools import count

# Get rid of the newlines for convenience
message = message_log.replace("\n", " ")

# Use a regular expression to split the log file into records
rx = r"(\[\d{2}-\w{3}-\d{2})"
replaced = re.split(rx, message)

# re.split will leave a blank entry as the first entry
replaced.pop(0)

# Each record will be a separate entry in a list 
# Unfortunately the date component gets put in a different section of the list
# from the record is refers to and needs to be merged back together
merge_list = list()

for x, y in zip(count(step=2), replaced):
    try:
        merge_list.append(replaced[x] + replaced[x+1])
    except:
        continue

# Now a nice clean record list exists, it is possible to get the user count
n = 0
for z in merge_list:
    # Split the record into date and context
    log_date = re.split("(\d{2}-\w{3}-\d{2})", z)
    # Work out whether the count should be incremented or decremented
    if "{Embed} Member Joined" in z:
        n = n + 1
    elif "{Embed} Member Left" in z:
        n = n - 1
    else:
        continue
    # log_date[1] is needed to get the date from the record
    print(log_date[1] + " " + str(n))

【讨论】：