网页抓取错误：“NoneType”对象没有属性“文本”答案

【问题标题】：Webscraping Error: 'NoneType' object has no attribute 'text'网页抓取错误：“NoneType”对象没有属性“文本”
【发布时间】：2020-12-09 19:38:18
【问题描述】：

您好，我正在尝试从网站上抓取数据，它运行了很长一段时间，现在我收到以下错误消息：“'NoneType' object has no attribute 'text'”和它不再抓取任何数据。他们一定是在网站上改变了一些东西，但我不知道是什么。

行中的错误发生：“data_page = soup_page.find('script', text=r_page).text”

这是我的代码：

from bs4 import BeautifulSoup as bs
import urllib.request
from urllib.request import urlopen
import requests
import time
import re
from datetime import datetime
import pandas as pd
import json

for seite in range(1):
    
    print("Loop " + str(seite) + " startet.")
    df = pd.DataFrame()
    l=[]

    try:
        
        page = ("https://www.immobilienscout24.de/Suche/radius/neubauwohnung-kaufen?centerofsearchaddress=Krefeld;47799;Grenzstra%C3%9Fe;;;Bockum&geocoordinates=51.33798;6.58608;1.0&enteredFrom=result_list")
        print(page)

        res_page = requests.get(page)
        soup_page = bs(res_page.content, 'lxml')
        r_page = re.compile(r'resultListModel:(.*)')
        data_page = soup_page.find('script', text=r_page).text
        script_page = r_page.findall(data_page)[0].rstrip(',')
        results_page = json.loads(script_page)

        for item in results_page['searchResponseModel']['resultlist.resultlist']['resultlistEntries'][0]['resultlistEntry']:
            l.append(item['@id'])
            if 'similarObjects' in item:
                for i in item['similarObjects'][0]['similarObject']:
                    if isinstance(i,dict):
                        l.append(i['@id'])
                    elif i == '@id':
                        l.append(item['similarObjects'][0]['similarObject'][i])
        l = list(set(l))

【问题讨论】：

r_page 行是问题所在，它没有找到任何东西。打印soup_page 并确保“resultListModel”在那里可能是一个好主意，因为它可能不是。这应该会提示您为什么会失败。

标签： python web-scraping beautifulsoup nonetype

【解决方案1】：

如果您不指定User-Agent 和Accept-Language HTTP 标头，服务器将返回验证码页面：

import json
import requests
import pandas as pd
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0',
    'Accept-Language': 'en-US,en;q=0.5'
}

for seite in range(1):    
    print("Loop " + str(seite) + " startet.")
    df = pd.DataFrame()
    l=[]

    page = ("https://www.immobilienscout24.de/Suche/radius/neubauwohnung-kaufen?centerofsearchaddress=Krefeld;47799;Grenzstra%C3%9Fe;;;Bockum&geocoordinates=51.33798;6.58608;1.0&enteredFrom=result_list")
    res_page = requests.get(page, headers=headers)
    soup_page = BeautifulSoup(res_page.content, 'lxml')
    r_page = re.compile(r'resultListModel:(.*)')
    data_page = soup_page.find('script', text=r_page).string
    script_page = r_page.findall(data_page)[0].rstrip(',')
    results_page = json.loads(script_page)

    for item in results_page['searchResponseModel']['resultlist.resultlist']['resultlistEntries']:
        item = item['resultlistEntry']
        l.append(item['@id'])
        if 'similarObjects' in item:
            for i in item['similarObjects'][0]['similarObject']:
                if isinstance(i,dict):
                    l.append(i['@id'])
                elif i == '@id':
                    l.append(item['similarObjects'][0]['similarObject'][i])
    l = list(set(l))

print(l)

打印：

['119256589', '119215242', '119254488', '119256425', '119254296', '119256175', '119240835']

【讨论】：