Python 3 BeautifulSoup 获取 URL（href 或 baseURL）如果 div 类标题文本包含“英语”

Question

<div class="gallery" data-tags="19 16 40193 41706 40476 7921 815 425 900 362 229 154 146 13 65 129 766 25 9 51931 188">
    <a href="/g/987654/" class="cover" style="padding:0 0 142.79999999999998% 0">
    <img is="lazyload-image" class="" width="250" height="357" data-src="https://abc.cloud.xyz/galleries/123456/thumb.jpg" alt="" src="https://abc.cloud.xyz/galleries/123456/thumb.jpg">
    <div class="caption">[User] Text ABCDEFGH [English] </div>
    </a>
</div>

该程序不会将 URL/href 保存到 txt 文件中。我认为它找不到 href

如果带有类标题的 div 元素包含 Word English，则应将元素类封面的 href (/g/987654/) 保存在 txt 文件中。

from bs4 import BeautifulSoup
import requests

url = "https://google.com"

response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

base_urls = []
for div in soup.find_all("div", {"class": "caption"}):
    if "English" in div.text:
        a_tag = div.find_previous_sibling("a")
        if a_tag:
            base_urls.append(a_tag["baseURL"])

with open("base_urls.txt", "w") as f:
    for base_url in base_urls:
        f.write(base_url + "\n")

**到目前为止我试过什么 ** 此代码有效，但它将所有 href 保存到 txt 文件中......

from bs4 import BeautifulSoup
import requests

url = "https://google.com"

page = requests.get(url)
soup = BeautifulSoup(page.content, "html.parser")

links = soup.find_all("a")

hrefs = [link["href"] for link in links]

with open("links_test1.txt", "w") as file:
    for href in hrefs:
        file.write(href + "\n")

    from bs4 import BeautifulSoup
    import requests
    
   
    lurl = ["https://web.com/page1","https://web.com/page2","https://web.com/page3"]
    
    
    for url in lurl:
    response = requests.get(url)
    soup = BeautifulSoup(response.text, "html.parser")
 
    base_urls = []
for div in soup.find_all("div", {"class": "caption"}):
    if "English" in div.text:
        a_tag = div.find_previous("a")
        if a_tag:
            base_urls.append(a_tag["href"])
with open("base_urls2.txt", "w") as f:
    for base_url in base_urls:
        f.write(base_url + "\n")

Answer 1

查看 HTML sn-p，您应该使用 .find_previous 而不是 .find_previous_sibling。另外，使用a_tag['href']，而不是a_tag['baseURL']：

from bs4 import BeautifulSoup


html_doc = """
<div class="gallery" data-tags="19 16 40193 41706 40476 7921 815 425 900 362 229 154 146 13 65 129 766 25 9 51931 188">
    <a href="/g/987654/" class="cover" style="padding:0 0 142.79999999999998% 0">
    <img is="lazyload-image" class="" width="250" height="357" data-src="https://abc.cloud.xyz/galleries/123456/thumb.jpg" alt="" src="https://abc.cloud.xyz/galleries/123456/thumb.jpg">
    <div class="caption">[User] Text ABCDEFGH [English] </div>
    </a>
</div>"""

soup = BeautifulSoup(html_doc, "html.parser")


base_urls = []
for div in soup.find_all("div", {"class": "caption"}):
    if "English" in div.text:
        a_tag = div.find_previous("a")
        if a_tag:
            base_urls.append(a_tag["href"])

print(base_urls)

印刷：

['/g/987654/']