Findall 到 div 标签使用美丽的汤产生空白返回

Question

<div class="columns small-5 medium-4 cell header">Ref No.</div>
<div class="columns small-7 medium-8 cell">110B60329</div>                                                          

网址是https://www.saa.gov.uk/search/?SEARCHED=1&ST=&SEARCH_TERM=city+of+edinburgh%2C+BOSWALL+PARKWAY%2C+EDINBURGH&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&DRILL_SEARCH_TERM=BOSWALL+PARKWAY%2C+EDINBURGH&DD_TOWN=EDINBURGH&DD_STREET=BOSWALL+PARKWAY&UARN=110B60329&PPRN=000000000001745&ASSESSOR_IDX=10&DISPLAY_MODE=FULL#results

我想运行一个循环并返回“110B60329”。我运行了漂亮的汤并完成了 find_all(div)，然后我根据它们的类将 2 个不同的标签定义为 head 和 data。然后我通过'head'标签运行迭代，希望它会返回我定义为数据的div标签中的信息。

Python 返回一个空白（cmd 提示符重新打印了 filepth）。

有没有人知道我可以如何解决这个问题。我的完整代码是.....谢谢

import requests
from bs4 import BeautifulSoup as soup
import csv


url = 'https://www.saa.gov.uk/search/?SEARCHED=1&ST=&SEARCH_TERM=city+of+edinburgh%2C+BOSWALL+PARKWAY%2C+EDINBURGH&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&DRILL_SEARCH_TERM=BOSWALL+PARKWAY%2C+EDINBURGH&DD_TOWN=EDINBURGH&DD_STREET=BOSWALL+PARKWAY&UARN=110B60329&PPRN=000000000001745&ASSESSOR_IDX=10&DISPLAY_MODE=FULL#results'

baseurl = 'https://www.saa.gov.uk'

session = requests.session()

response = session.get(url)

# content of search page in soup 
html= soup(response.content,"lxml")   
properties_col = html.find_all('div')



for col in properties_col:
    ref = 'n/a'
    des = 'n/a'

    head = col.find_all("div",{"class": "columns small-5 medium-4 cell header"}) 

    data = col.find_all("div",{"class":"columns small-7 medium-8 cell"})

    for i,elem in enumerate(head):
    #for i in range(elems):
        if head [i].text == "Ref No.":
            ref = data[i].text
            print ref             

Answer 1

您可以通过两种方式做到这一点。

1) 如果您确定您正在抓取的网站不会更改其内容，您可以找到该类的所有 div 并通过提供索引来获取内容。

2) 查找所有左侧 div（标题），如果其中一个与您想要的匹配，则让下一个兄弟获取文本。

例子：

import requests
from bs4 import BeautifulSoup as soup

url = 'https://www.saa.gov.uk/search/?SEARCHED=1&ST=&SEARCH_TERM=city+of+edinburgh%2C+BOSWALL+PARKWAY%2C+EDINBURGH&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&DRILL_SEARCH_TERM=BOSWALL+PARKWAY%2C+EDINBURGH&DD_TOWN=EDINBURGH&DD_STREET=BOSWALL+PARKWAY&UARN=110B60329&PPRN=000000000001745&ASSESSOR_IDX=10&DISPLAY_MODE=FULL#results'

baseurl = 'https://www.saa.gov.uk'

session = requests.session()

response = session.get(url)

# content of search page in soup 
html = soup(response.content,"lxml")

#Method 1
LeftBlockData = html.find_all("div", class_="columns small-7 medium-8 cell")
Reference = LeftBlockData[0].get_text().strip()
Description = LeftBlockData[2].get_text().strip()
print(Reference)
print(Description)

#Method 2
for column in html.find_all("div", class_="columns small-5 medium-4 cell header"):
    RightColumn = column.next_sibling.next_sibling.get_text().strip()
    if "Ref No." in column.get_text().strip():
        print (RightColumn)
    if "Description" in column.get_text().strip():
        print (RightColumn)

打印件将输出（按顺序）：

110B60329

商店

110B60329

商店

您的问题是您正在尝试将具有大量制表符的节点文本与非空格字符串匹配。

例如，您的head [i].textvariable 包含 Ref No.，因此如果您将它与Ref No. 进行比较，它会给出错误的结果。剥离它会解决。

Answer 2

import requests
from bs4 import BeautifulSoup

r = requests.get("https://www.saa.gov.uk/search/?SEARCHED=1&ST=&SEARCH_TERM=city+of+edinburgh%2C+BOSWALL+PARKWAY%2C+EDINBURGH&ASSESSOR_ID=&SEARCH_TABLE=valuation_roll_cpsplit&DISPLAY_COUNT=10&TYPE_FLAG=CP&ORDER_BY=PROPERTY_ADDRESS&H_ORDER_BY=SET+DESC&DRILL_SEARCH_TERM=BOSWALL+PARKWAY%2C+EDINBURGH&DD_TOWN=EDINBURGH&DD_STREET=BOSWALL+PARKWAY&UARN=110B60329&PPRN=000000000001745&ASSESSOR_IDX=10&DISPLAY_MODE=FULL#results")
soup = BeautifulSoup(r.text, 'lxml')
for row in soup.find_all(class_='table-row'):

    print(row.get_text(strip=True, separator='|').split('|'))

出来：

['Ref No.', '110B60329']
['Office', 'LOTHIAN VJB']
['Description', 'STORE']
['Property Address', '29 BOSWALL PARKWAY', 'EDINBURGH', 'EH5 2BR']
['Proprietor', 'SCOTTISH MIDLAND CO-OP SOCIETY LTD.']
['Tenant', 'PROPRIETOR']
['Occupier']
['Net Annual Value', '£1,750']
['Marker']
['Rateable Value', '£1,750']
['Effective Date', '01-APR-10']
['Other Appeal', 'NO']
['Reval Appeal', 'NO']

get_text() 是一个非常强大的工具，你可以去掉空格并在文本中放置分隔符。

您可以使用此方法获取干净的数据并对其进行过滤。