如何将第 4 列的制表符分隔文本文件解析为 json 并删除某些键？答案

【问题标题】：How to parse tab-delimited text file with 4th column as json and remove certain keys?如何将第 4 列的制表符分隔文本文件解析为 json 并删除某些键？
【发布时间】：2020-03-06 17:42:01
【问题描述】：

我有一个 26 Gb 的文本文件，行格式如下

/type/edition /books/OL10000135M 4 2010-04-24T17:54:01.503315 {"publishers": ["Bernan Press"], "physical_format": "Hardcover", "subtitle": "9th November - 3rd December, 1992", "key": "/books/OL10000135M", "title": "Parliamentary Debates, House of Lords, Bound Volumes, 1992-93", "identifiers": {"goodreads": ["6850240"]}, "isbn_13": ["9780107805401"], "languages": [{"key": "/languages/eng"}], "number_of_pages": 64, "isbn_10": ["0107805405"], "publish_date": "December 1993", "last_modified": {"type": "/type/datetime", "value": "2010-04-24T17:54:01.503315"}, "authors": [{"key": "/authors/OL2645777A"}], "latest_revision": 4, "works": [{"key": "/works/OL7925046W"}], "type": {"key": "/type/edition"}, "subjects": ["Government - Comparative", "Politics / Current Events"], "revision": 4}

我试图只获取最后一列是 json 并且从那个 Json 我只是试图保存“title”、“isbn 13”、“isbn 10”

我只能用这段代码保存最后一列

csv.field_size_limit(sys.maxsize)
# File names: to read in from and read out to
input_file = '../inputFile/ol_dump_editions_2019-10-31.txt'
output_file = '../outputFile/output.txt'

## ==================== ##
##  Using module 'csv'  ##
## ==================== ##
with open(input_file) as to_read:
    with open(output_file, "w") as tmp_file:
        reader = csv.reader(to_read, delimiter = "\t")
        writer = csv.writer(tmp_file)

        desired_column = [4]        # text column

        for row in reader:     # read one row at a time
            myColumn = list(row[i] for i in desired_column)   # build the output row (process)
            writer.writerow(myColumn) # write it

但这不会返回正确的 json 对象，而是返回旁边带有双引号的所有内容。另外，我将如何从 json 中提取某些值作为新的 json

编辑：

"{""publishers"": [""Bernan Press""], ""physical_format"": ""Hardcover"", ""subtitle"": ""9th November - 3rd December, 1992"", ""key"": ""/books/OL10000135M"", ""title"": ""Parliamentary Debates, House of Lords, Bound Volumes, 1992-93"", ""identifiers"": {""goodreads"": [""6850240""]}, ""isbn_13"": [""9780107805401""], ""languages"": [{""key"": ""/languages/eng""}], ""number_of_pages"": 64, ""isbn_10"": [""0107805405""], ""publish_date"": ""December 1993"", ""last_modified"": {""type"": ""/type/datetime"", ""value"": ""2010-04-24T17:54:01.503315""}, ""authors"": [{""key"": ""/authors/OL2645777A""}], ""latest_revision"": 4, ""works"": [{""key"": ""/works/OL7925046W""}], ""type"": {""key"": ""/type/edition""}, ""subjects"": [""Government - Comparative"", ""Politics / Current Events""], ""revision"": 4}"

编辑 2：

所以我试图读取这个文件，它是一个制表符分隔的文件，包含以下列：

type - 记录类型（/type/edition、/type/work 等） key - 记录的唯一键。（/books/OL1M 等）修订 - 记录的修订号 last_modified - 最后修改的时间戳 JSON - JSON 格式的完整记录

我正在尝试读取 JSON 文件，而我只是试图从该 Json 中获取“title”、“isbn 13”、“isbn 10”作为 json 并将其作为一行保存到文件中

所以每一行都应该看起来像原来的，但只有那些键和值

【问题讨论】：

请分享您的代码返回的内容。
抱歉我更新了

标签： python json csv parsing text

【解决方案1】：

这是一种直接的方法。您需要重复此操作，并在读取文件的每一行时逐行提取所需的数据（在 Python 中处理文本文件读取的默认方式）。

import json

line = '/type/edition   /books/OL10000135M  4   2010-04-24T17:54:01.503315  {"publishers": ["Bernan Press"], "physical_format": "Hardcover", "subtitle": "9th November - 3rd December, 1992", "key": "/books/OL10000135M", "title": "Parliamentary Debates, House of Lords, Bound Volumes, 1992-93", "identifiers": {"goodreads": ["6850240"]}, "isbn_13": ["9780107805401"], "languages": [{"key": "/languages/eng"}], "number_of_pages": 64, "isbn_10": ["0107805405"], "publish_date": "December 1993", "last_modified": {"type": "/type/datetime", "value": "2010-04-24T17:54:01.503315"}, "authors": [{"key": "/authors/OL2645777A"}], "latest_revision": 4, "works": [{"key": "/works/OL7925046W"}], "type": {"key": "/type/edition"}, "subjects": ["Government - Comparative", "Politics / Current Events"], "revision": 4}'

csv_cols = line.split('\t')
json_data = json.loads(csv_cols[4])
#print(json.dumps(json_data, indent=4))

desired = {key: json_data[key] for key in ("title", "isbn_13", "isbn_10")}
result = json.dumps(desired, indent=4)
print(result)

采样线的输出：

{
    "title": "Parliamentary Debates, House of Lords, Bound Volumes, 1992-93",
    "isbn_13": [
        "9780107805401"
    ],
    "isbn_10": [
        "0107805405"
    ]
}

【讨论】：

谢谢！但我试图用键和值来保留它的 json
是的，但问题出在编写它的时候。它没有正确写入文件{,"""",p,u,b,l,i,s,h,e,r,s,"""",:, ,[,"""",B,e,r,n,a,n, ,P,r,e,s,s,"""",],",", ,"""",p,h,y,s,i,c,a,l,_,f,o,r,m,a,t,"""",:, ,"""",H,a,r,d,c,o,v,e,r,"""",",", ,"""",s,u,b,t,i,t,l,e,"""",:, ,"""",9,t,h, ,N,o,v,e,m,b,e,r, ,-, ,3,r,d, ,D,e,c,e,m,b,e,r,",", ,1,9,9,2,"""",",",t,a,r,y, ,D,e,b,a,t,e,s,",", ,H,o,u,s,e, ,o,f, ,L,o,r,d,s,",", ,B,o,u,n,d, ,V,o,l,u,m,e,s,",", ,1,9,9,2,-,9,3,"""",",", ,"""",i,d,e,n,t,i,f,i,e,d,a,t,e,"""",:, ,"""",D,e,c,e,m,b,e,r, }
那么你一定没有正确地做到这一点。要将结果写入文件，请使用类似：outfile.write(result + '\n').

【解决方案2】：

因此，鉴于您当前的代码返回以下内容：

result = '{""publishers"": [""Bernan Press""], ""physical_format"": ""Hardcover"", ""subtitle"": ""9th November - 3rd December, 1992"", ""key"": ""/books/OL10000135M"", ""title"": ""Parliamentary Debates, House of Lords, Bound Volumes, 1992-93"", ""identifiers"": {""goodreads"": [""6850240""]}, ""isbn_13"": [""9780107805401""], ""languages"": [{""key"": ""/languages/eng""}], ""number_of_pages"": 64, ""isbn_10"": [""0107805405""], ""publish_date"": ""December 1993"", ""last_modified"": {""type"": ""/type/datetime"", ""value"": ""2010-04-24T17:54:01.503315""}, ""authors"": [{""key"": ""/authors/OL2645777A""}], ""latest_revision"": 4, ""works"": [{""key"": ""/works/OL7925046W""}], ""type"": {""key"": ""/type/edition""}, ""subjects"": [""Government - Comparative"", ""Politics / Current Events""], ""revision"": 4}'

看起来你需要做的是：首先 - 用常规双引号替换那些双双引号，否则事情无法解析：

res = result.replace('""','"')

现在res 可以转换为 JSON 对象：

import json
my_json = json.loads(res)

my_json 现在看起来像这样：

{'authors': [{'key': '/authors/OL2645777A'}],
 'identifiers': {'goodreads': ['6850240']},
 'isbn_10': ['0107805405'],
 'isbn_13': ['9780107805401'],
 'key': '/books/OL10000135M',
 'languages': [{'key': '/languages/eng'}],
 'last_modified': {'type': '/type/datetime',
  'value': '2010-04-24T17:54:01.503315'},
 'latest_revision': 4,
 'number_of_pages': 64,
 'physical_format': 'Hardcover',
 'publish_date': 'December 1993',
 'publishers': ['Bernan Press'],
 'revision': 4,
 'subjects': ['Government - Comparative', 'Politics / Current Events'],
 'subtitle': '9th November - 3rd December, 1992',
 'title': 'Parliamentary Debates, House of Lords, Bound Volumes, 1992-93',
 'type': {'key': '/type/edition'},
 'works': [{'key': '/works/OL7925046W'}]}

您可以方便地从此对象中获取所需的任何字段：

my_json['title']
# 'Parliamentary Debates, House of Lords, Bound Volumes, 1992-93'
my_json['isbn_10'][0]
# '0107805405'

【讨论】：

我是否需要像下面的答案所说的那样使用 pandas 进行解析，或者我的方式是将所有内容都放入 ram 中？因为我认为它一次只能读取和写入一行
你当然也可以使用 Pandas，很棒的工具。我总是更喜欢使用 csv 和 json 之类的包进行这种复杂的解析操作，并通过显式循环和条件获得我想要的灵活性。 Pandas 函数可以通过稍微更神奇的函数让您获得相同的结果，这可能有一个学习曲线。
并且执行myColumn = list(row[i] for i in desired_column) # build the output row (process) res = myColumn.replace('""','"') writer.writerow(res) # write it 会返回错误AttributeError: 'list' object has no attribute 'replace'
我们一步一步来。 myColumn.replace('""','"') 会产生错误吗？
是第 21 行，即该行给出错误 AttributeError: 'list' object has no attribute 'replace'

【解决方案3】：

特别是因为您的示例太大，我建议使用专用库，例如 pandas，它有一个 read_csv 方法，甚至是 dask，它支持内存不足操作。

这两个系统都会自动为您解析出报价，而 dask 会直接从磁盘“分段”解析，因此您无需尝试将 26GB 加载到 RAM 中。

在这两个库中，您可以像这样访问所需的列：

data = read_csv(PATH)
data["ColumnName"]

然后您可以使用 json.loads() (import json) 或使用 pandas/dask json 实现来解析这些行。如果您可以提供更多关于您期望的细节，我可以帮助您起草更具体的代码示例。

祝你好运！

【讨论】：

谢谢！我添加了第二个编辑，可以更好地解释问题，我对这个数据世界很陌生，任何帮助都将不胜感激
是否可以在 pandas 中使用 chunks 来代替我熟悉 pandas 的 dask

【解决方案4】：

我将您的数据保存到一个文件中，看看我是否可以只读取行，如果可行，请告诉我：

lines = zzread.split('\n')  
temp=[] 
for to_read in lines: 
    if len(to_read) == 0:  
        break  
    new_to_read = '{' + to_read.split('{',1)[1] 
    temp.append(json.loads(new_to_read)) 
for row in temp: 
      print(row['isbn_13'])

如果可行，这应该为您创建一个 json：

lines = zzread.split('\n')  
temp=[] 
for to_read in lines: 
    if len(to_read) == 0:  
        break  
    new_to_read = '{' + to_read.split('{',1)[1] 
    temp.append(json.loads(new_to_read)) 
new_json=[]
for row in temp: 
    new_json.append({'title': row['title'], 'isbn_13': row['isbn_13'], 'isbn_10': row['isbn_10']})

【讨论】：