抓取脚本提供的 JSON 并转换为数据框

Question

我正在尝试从该职位发布（脚本 json 到 df）中获取详细信息，但无法取得进展。

import requests 
from bs4 import BeautifulSoup 
import pandas as pd
from pandas.io.json import json_normalize
import time
import re
import json
from pandas.compat import StringIO


URLS=['https://www.iimjobs.com/j/specialist-manager-operations-student-services-elearning-education-management-organization-4-6-yrs-774235.html?ref=cl','https://www.iimjobs.com/j/specialist-manager-operations-student-services-elearning-education-management-organization-4-6-yrs-774235.html?ref=cl']


i=0
for URL in URLS:
#     time.sleep(5)   
    r = requests.get(URL) 
    soup = BeautifulSoup(r.content, 'html5lib') 
    # print(soup.prettify()) 
    table=soup.find("script" , type='application/ld+json').text
    data = json.loads(json.dumps(table))
    if i == 0:
        df = pd.read_json(data)  
    if i != 0:
        dfnew=pd.read_json(data)
        df=df.append(dfnew)     
    i=i+1
df.to_csv('jobs.csv', index=False)
print(df)

有人可以帮我解决这个问题吗？详细错误如下：

ValueError                                Traceback (most recent call last)
<ipython-input-9-5f9199aa28fd> in <module>
     30     data = json.loads(json.dumps(table))
     31     if i == 0:
---> 32         df = pd.read_json(data)
     33     if i != 0:
     34         print(URL)

C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\json\json.py in read_json(path_or_buf, orient, typ, dtype, convert_axes, convert_dates, keep_default_dates, numpy, precise_float, date_unit, encoding, lines, chunksize, compression)
    425         return json_reader
    426 
--> 427     result = json_reader.read()
    428     if should_close:
    429         try:

C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\json\json.py in read(self)
    535             )
    536         else:
--> 537             obj = self._get_object_parser(self.data)
    538         self.close()
    539         return obj

C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\json\json.py in _get_object_parser(self, json)
    554         obj = None
    555         if typ == 'frame':
--> 556             obj = FrameParser(json, **kwargs).parse()
    557 
    558         if typ == 'series' or obj is None:

C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\json\json.py in parse(self)
    650 
    651         else:
--> 652             self._parse_no_numpy()
    653 
    654         if self.obj is None:

C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\json\json.py in _parse_no_numpy(self)
    869         if orient == "columns":
    870             self.obj = DataFrame(
--> 871                 loads(json, precise_float=self.precise_float), dtype=None)
    872         elif orient == "split":
    873             decoded = {str(k): v for k, v in compat.iteritems(

ValueError: Unexpected character found when decoding object value

我实际上是在尝试抓取职位详细信息，例如职位、描述、技能、jobLocation 等。但是为这个特定 URL 提供的 JSON 似乎失败了，我还没有弄清楚这一点

Answer 1

问题是脚本 json 中的某些值包含 html 的双引号（即：class=""）。因此，它将其视为新字符串的结束和开始，并且没有逗号或有效的键：值。

因此，如果您处理好它，它应该可以工作：

import requests 
from bs4 import BeautifulSoup 
import pandas as pd
from pandas.io.json import json_normalize
import time
import re
import json
from pandas.compat import StringIO


URLS=['https://www.iimjobs.com/j/specialist-manager-operations-student-services-elearning-education-management-organization-4-6-yrs-774235.html?ref=cl','https://www.iimjobs.com/j/specialist-manager-operations-student-services-elearning-education-management-organization-4-6-yrs-774235.html?ref=cl']


i=0
for URL in URLS:
#     time.sleep(5)   
    r = requests.get(URL) 
    soup = BeautifulSoup(r.content, 'html5lib') 
    # print(soup.prettify()) 
    table=soup.find("script" , type='application/ld+json').text
    data = json.loads(json.dumps(table))
    if i == 0:
        try:
            df = pd.read_json(data)  
        except:
            data = data.replace('=""', '=')
            df = pd.read_json(data)  
    if i != 0:
        try:
            dfnew=pd.read_json(data)
            df=df.append(dfnew) 

        except:
            data = data.replace('=""', '=')
            dfnew=pd.read_json(data)
            df=df.append(dfnew) 

    i=i+1
df.to_csv('jobs.csv', index=False)
print(df)