我如何刮掉这个元素？答案

【问题标题】：How do I scrape this element?我如何刮掉这个元素？
【发布时间】：2021-03-19 21:26:44
【问题描述】：

我要抓取title=的字符串

我不知道该怎么做。

网址： https://www.google.com/search?tbs=sbi:AMhZZivZpaYCxEKMmd6Bb44wJvnFRn3qBjm55UYqn0gicZ_1JhmTnv84KkKpzLAK35nU8uDqlWr9Kx2I6kTIeVf_1lNlHzPHdnhnLptL3lYjaU2q1gL4ZaKpdTtBvp3jhqqIiEXjsNtBNTN6TdjZ_1rRnzK1eGghX3jkQQiFESYQDwkZHPuXaN7vZ50J5ZZGazHJ8ds_13Hylm51grQwenq506Z1gpaXnHGlte_1ME_1Y0SFltK-FftywdeG_1xDPi4jGiO81ImgKdshrYd9YIpHxyj1fNKaEuOvsH2pZ264QnP3O8fS3V22HA87OgPCY4ubh0N-jJH40b19RAYez5tdPNR2HttPigfixkLcg

有人可以帮帮我吗？

from bs4 import BeautifulSoup
from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://www.google.com/search?tbs=sbi:AMhZZivZpaYCxEKMmd6Bb44wJvnFRn3qBjm55UYqn0gicZ_1JhmTnv84KkKpzLAK35nU8uDqlWr9Kx2I6kTIeVf_1lNlHzPHdnhnLptL3lYjaU2q1gL4ZaKpdTtBvp3jhqqIiEXjsNtBNTN6TdjZ_1rRnzK1eGghX3jkQQiFESYQDwkZHPuXaN7vZ50J5ZZGazHJ8ds_13Hylm51grQwenq506Z1gpaXnHGlte_1ME_1Y0SFltK-FftywdeG_1xDPi4jGiO81ImgKdshrYd9YIpHxyj1fNKaEuOvsH2pZ264QnP3O8fS3V22HA87OgPCY4ubh0N-jJH40b19RAYez5tdPNR2HttPigfixkLcg')
soup = BeautifulSoup(r.text, 'html.parser')
for link in soup.find_all('g-img',class_="GMCzAd BA0A6c"):
    print(link.get('title'))

【问题讨论】：

标签： python beautifulsoup

【解决方案1】：

您需要Selenium，因为内容是动态加载的。

pip install selenium
apt-get update 
apt install chromium-chromedriver

然后你可以得到html 和Selenium 并将它传递给BeautifulSoup：

from selenium import webdriver
from bs4 import BeautifulSoup
import re

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',options=chrome_options)

driver.get("https://www.google.com/search?tbs=sbi:AMhZZivZpaYCxEKMmd6Bb44wJvnFRn3qBjm55UYqn0gicZ_1JhmTnv84KkKpzLAK35nU8uDqlWr9Kx2I6kTIeVf_1lNlHzPHdnhnLptL3lYjaU2q1gL4ZaKpdTtBvp3jhqqIiEXjsNtBNTN6TdjZ_1rRnzK1eGghX3jkQQiFESYQDwkZHPuXaN7vZ50J5ZZGazHJ8ds_13Hylm51grQwenq506Z1gpaXnHGlte_1ME_1Y0SFltK-FftywdeG_1xDPi4jGiO81ImgKdshrYd9YIpHxyj1fNKaEuOvsH2pZ264QnP3O8fS3V22HA87OgPCY4ubh0N-jJH40b19RAYez5tdPNR2HttPigfixkLcg")

soup = BeautifulSoup(driver.page_source)
all_images = [i['title'] for i in soup.find_all('img', attrs={'id': re.compile('^dimg')})] #get the title of all img tags that have an id that starts with dimg

那么all_images[0] 将包含：

https://www.pngfind.com/mpng/TRxwbhw_cartoon-tadpole-hd-png-download/

PS。我不知道你的目标是什么，但可能有easier 解决方案

【讨论】：