如果属性是随机的，我该如何抓取网站？答案

【问题标题】：How can I scrape a website if the attributes are randomized?如果属性是随机的，我该如何抓取网站？
【发布时间】：2023-01-01 23:46:14
【问题描述】：

我正在尝试抓取这个网站：https://novel5s.com/bye-my-irresistible-love-by-goreous-novel5-online-2138/148981.html。

问题是文本保存在属性中并在每次重新加载时随机化。有人可以帮我用 python 废弃这个页面吗？

双击底部的段落，你会发现一切都不是文本。感谢您提供的任何帮助。

我的代码：

from bs4 import BeautifulSoup
from selenium import webdriver
import chromedriver_autoinstaller
from selenium import webdriver
from selenium.webdriver import Keys, ActionChains
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC



chromedriver_autoinstaller.install()


chrome_options = Options()
chrome_options.add_experimental_option( "prefs",{'profile.managed_default_content_settings.javascript': 2})
driver = webdriver.Chrome(options=chrome_options)
driver.maximize_window()

driver.get("https://novel5s.com/bye-my-irresistible-love-by-goreous-novel5-online-2138/148981.html")
time.sleep(2)
text = driver.find_element(By.CSS_SELECTOR,".content-book")
for i in text.find_elements(By.CSS_SELECTOR,"*"):
    attrs=[]
    soup = BeautifulSoup(i.get_attribute("outerHTML"),"html.parser")
    try:
        # print(soup.find("p").text)
        print(soup.find("p").attrs.values())
    except:
        pass
print("null")

我正在尝试获取段落标签不同属性内的完整段落。我知道如何获取标签的所有属性，但问题是有多个属性，其中一些只是为了防止抓取。我想要完整的段落。

编辑：您只需要获取 ::before 和 ::after 属性。

【问题讨论】：

标签： python selenium selenium-webdriver

【解决方案1】：

如果你检查页面，你会看到这本书的文本实际上在类下面：

class="content-book my-4"

所以，瞄准那个班级。

现在你不能只使用：

 soup.find_all(class_="content-book my-4")

因为那会给我们不必要的<script>标签：

<div class="content-book my-4"><p> <strong>Chapter 2 Sick Feeling</strong></p><p> Scarlett’s POV:</p><p> “Anything else?” I asked in disbelief.</p><p> “We have to get up early to see Rita tomorrow,” Charles replied coldly.</p><p> “Okay.”</p><p> I was confused. I could not help but wonder if he returned just to make a point.</p><p> “I’ll sleep here tonight,” he added.</p><p> I came to my senses the instant I heard what he had said. I wanted to ask him if it was really okay for

因此，改为使用 CSS 选择器：

for element in soup.select(".content-book.my-4 p"):
    print(element.text)

这将在content-book my-4 类下选择一个<p>。（这是针对第 2 章的，但它仍然适用于第 1 章）。

import requests
from bs4 import BeautifulSoup


URL = "https://novel5s.com/bye-my-irresistible-love-by-goreous-novel5-online-2138/148982.html"

soup = BeautifulSoup(requests.get(URL).content, "html.parser")

for element in soup.select(".content-book.my-4 p"):
    print(element.text)

输出：

 Chapter 2 Sick Feeling
 Scarlett’s POV:
 “Anything else?” I asked in disbelief.
 “We have to get up early to see Rita tomorrow,” Charles replied coldly.
 “Okay.”
 I was confused. I could not help but wonder if he returned just to make a point.
 “I’ll sleep here tonight,” he added.
 I came to my senses the instant I heard what he had said. I wanted to ask him if it was really okay for him to stay here, but I decided to swallow my words instead.
 “I’m afraid you’ll oversleep because of the jet lag,” he 
...

【讨论】：

检查所有打印的句子，最后几句不完整。等等，我正在编辑代码检查一下。

【解决方案2】：

隐藏文本的顺序似乎是在网页 html 中的 style 元素中编码的，就在包含所有段落的 div 元素下方（见屏幕截图）。

此 style 元素中的代码似乎对应于您在解析时遇到问题的段落元素中的 class 和随机标记。

我的建议是解析这个 style 元素，以正确的顺序提取类和标签，然后从段落元素中解析它们以获得完整的段落。

它仍然需要一些解析和解码，但我希望这会有所帮助！

截图：The element that presumably encodes the text order contained in randomized tags

【讨论】：