Beautiful Soup 找到所有找到没有类的某些 div答案

【问题标题】：Beautiful Soup find all to find certain div with no classBeautiful Soup 找到所有找到没有类的某些 div
【发布时间】：2020-05-13 16:25:02
【问题描述】：

我正在尝试从本页表格中的每个条目中提取 url、日期和超链接文本：https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/default.aspx

以前我只会使用

的代码

r = requests.get("https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/default.aspx")

all = soup.find_all("div", class_ = "the class for this div")

然后我将遍历下面的“全部”列表（用于 url）。

for item in all:
    print(item.find("a")["href"])

问题是这里需要的特定 div 没有类，所以我无法获得该表值的列表。我知道我可以将此类保留为无，但此页面中还有其他不适用的 div。我尝试使用下面的元素，但这会返回单个项目列表。

all = soup.select("#ctl00_ctl45_g_76f28544_b3c4_43f4_b435_13e7b563f7f1 > div:nth-child(2)")

我知道我可以在 soup.select() 上应用字符串操作，但我觉得这是不好的做法。

在您更有经验的情况下，解决此问题的最佳方法是什么？

提前致谢！

【问题讨论】：

您应该使用soup.find_all('div', attrs={'class': None}) 不带任何class 属性的div 的外观。但是，我在您的页面中没有找到您正在寻找的<href>？
嗨，还有许多其他没有类的 div，所以我认为这不会起作用:( href 的路径是：//*[@id="ctl00_ctl45_g_76f28544_b3c4_43f4_b435_13e7b563f7f1"]/div[2]/ div[1]/table/tbody/tr[1]/td[2]/h3/a 我试过你的代码，它给了我一个 UnicodeEncodeError: 'charmap' codec can't encode character '\u200b' in position 20497 : 字符映射到 :(
它有效。输入 soup.find_all('div', attrs={'class': None}) print(len(all_div)) 时返回 60 div 没有类
是的，你是对的，我需要将 .encode("utf-8") 添加到我的打印语句中。我将更新我的 for 循环以跳过空结果。谢谢！！！！

标签： python python-3.x beautifulsoup python-requests

【解决方案1】：

使用以下 css 选择器。

from bs4 import BeautifulSoup
r = requests.get("https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/default.aspx")
soup=BeautifulSoup(r.text,'html.parser')
for item in soup.select("a[href^='https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages']"):
    print(item['href'])

或

from bs4 import BeautifulSoup
r = requests.get("https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/default.aspx")
soup=BeautifulSoup(r.text,'html.parser')
for item in soup.select('div.searchfilter-userfilterbox+div a'):
    print(item['href'])

输出：

https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Pure-Tassie---Organic-Pure-Apple-juice-ranges.aspx
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Berg-Deli-Prosciutto-Sliced-100g.aspx
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Kimchi-Company-Kimchi-600g-and-300g.aspx
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Pure-Tassie---Apple-and-Blackcurrant-Juice-1.5L.aspx
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Green-Co.-Enoki-Mushrooms-200g and 300g.aspx
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Black-Swan-crafted-basil-dip-with-cashews-and-parmesan-200g.aspx
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Castard-Soboro-Bread-130g.aspx
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Shelby's-chocolate-chip-cookies.aspx
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/The-Loose-Leaf-Lettuce-Company.aspx
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Supreme-Salads-mesculin-salad-mixes.aspx

或者你可以使用下面的正则表达式。

import re
from bs4 import BeautifulSoup
r = requests.get("https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/default.aspx")
soup=BeautifulSoup(r.text,'html.parser')
for item in soup.find_all("a",href=re.compile('https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages')):
    print(item['href'])

如果你不想要正则表达式，那么试试这个

from bs4 import BeautifulSoup
r = requests.get("https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/default.aspx")
soup=BeautifulSoup(r.text,'html.parser')
for item in soup.find('div',class_='searchfilter-userfilterbox').find_next('div').find_all('a'):
    print(item['href'])

更新：

从 bs4 导入 BeautifulSoup

r = requests.get("https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/default.aspx")
soup=BeautifulSoup(r.text,'html.parser')
for item in soup.find('div',class_='searchfilter-userfilterbox').find_next('div').find_all('a'):
    print(item['href'])
    print(item.find_previous('div').text)
    print(item.find_next('td').text)
    print(item.find_next('td').find_next('td').text)

输出：

https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Pure-Tassie---Organic-Pure-Apple-juice-ranges.aspx
9/05/2020
Juice Isle Pty Ltd
Due to microbial (mycotoxin - Patulin) contamination
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Berg-Deli-Prosciutto-Sliced-100g.aspx
8/05/2020
ALDI
This recall is due to an incorrect back label resulting in an undeclared milk allergen
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Kimchi-Company-Kimchi-600g-and-300g.aspx
28/04/2020
The Kimchi Company Pty Ltd
Due to Potential microbial contamination due to under-allocation of salt.
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Pure-Tassie---Apple-and-Blackcurrant-Juice-1.5L.aspx
24/04/2020
Juicy Isle Pty Ltd
Due to microbial (mycotoxin patulin) contamination
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Green-Co.-Enoki-Mushrooms-200g and 300g.aspx
14/04/2020
Choi's Mushrooms
The recall is due to microbial (Listeria monocytogenes) contamination
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Black-Swan-crafted-basil-dip-with-cashews-and-parmesan-200g.aspx
3/04/2020
Monde Nissin Australia Pty Ltd (Black Swan Foods)
The recall is due to the presence of an undeclared allergen (peanuts).
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Castard-Soboro-Bread-130g.aspx
1/04/2020
Sejong Global Trading Pty Ltd
Due to  the presence of an undeclared allergen (peanut).
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Shelby's-chocolate-chip-cookies.aspx
27/03/2020
Shelby's Healthy Hedonism/Rousche Group Pty Ltd
The recall is due to the presence of undeclared allergens (hazelnut and cashew)
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/The-Loose-Leaf-Lettuce-Company.aspx
20/03/2020
The Loose Leaf Lettuce Company 
The recall is due to microbial (Salmonella) contamination. 
https://www.foodstandards.gov.au/industry/foodrecalls/recalls/Pages/Supreme-Salads-mesculin-salad-mixes.aspx
20/03/2020
Supreme Salads
Due to microbial (Salmonella) contamination.

【讨论】：

嘿，对不起，但我的问题不是特定于 url，我还需要其他属性，即日期和描述。但是，我相信我将能够重新调整您的代码来做到这一点。谢谢
@TobiasFunke ：检查更新的答案与日期和描述。
谢谢，这是最好的解决方案。我不知道 find_next 或 find_previous 函数。再次感谢！