抓取 flex 标签答案

【问题标题】：Scraping flex tag抓取 flex 标签
【发布时间】：2023-02-03 09:43:51
【问题描述】：

我想用 BeautifulSoup 抓取网站的段落，但网页中有弹性框，所以程序找不到所选标签。

def content_article(url, file_output):
    """scrape content  web page in a file and the plain code

       url: address of web page of international federation of canoe

       file_output: file name created + plain file name

       return two files: file with HTML code and file with only text information
       """
    response = requests.get(url)
    data= response.content
    soup = bs(data, features="html.parser")
    plain_soup = soup.encode("UTF-8")
    section = soup.find("div", {"class" : "container"})
    print (section)
    paragraphes = section.find_all("p")
    result=""
    for paragraphe in paragraphes:
        print ("paragraphe")
        print(paragraphe)
        result = result + paragraphe.text + "\n"
        print("result")
        print (result)
    url_file = file_output + ".txt"
    file = open(url_file, 'w', encoding="utf_8")
    file.write("infos provenant de" + url + "\n")
    file.write(result)
    file.close()
    url_plain_file = file_output + "_plain.txt"
    plain_file = open(url_plain_file, 'w')
    plain_file.write(str(plain_soup))
    plain_file.close()
    print("the file " + file_output + " has been created")

示例网址：https://www.fifa.com/about-fifa/president/news/gianni-infantino-congratulates-shaikh-salman-on-re-election-as-afc-president

程序找不到标签“container”，因为它在 flex 标签中。

我尝试使用 Selenium，但找不到“已激活”的弹性框。

【问题讨论】：

标签： python web-scraping beautifulsoup

【解决方案1】：

尝试这个。我将不同的子任务分解成单独的功能，然后制作一个结合的功能（即主要）。它会产生您想要的输出，即 2 个文本文件：一个带有包含段落的 html 元素（即 p 标签），另一个带有 p 标签。

from bs4 import BeautifulSoup
from selenium import webdriver
from time import sleep

def get_page_source(url):
    try:
        driver = webdriver.Chrome()
        driver.get(url)
        sleep(3)
        return driver.page_source
    finally: driver.quit()

def store_elements(outpath, p_tags):
    print(p_tags)
    with open(outpath, mode='w') as file:
        file.writelines(p_tags)

def store_texts(outpath, texts):
    with open(outpath, mode='w') as file:
        file.writelines(texts)

def get_elements(page_source, tag_name, attr):
    soup = BeautifulSoup(page_source, 'html.parser')
    return soup.find_all(tag_name, attr)

def get_text_from_elements(elements):
    return [element.text for element in elements]

def main(html_path, text_path):
    pg_source = get_page_source(url)
    p_tags = get_elements(pg_source, 'p', {'class':'p-large ff-text-grey-slate'})
    texts = get_text_from_elements(p_tags)
    store_elements(html_path, p_tags)
    store_texts(text_path, texts)

if __name__ == '__main__':
    # enter 2 paths, one for the html and other for the paragraphs (i.e. texts)
    main()

【讨论】：