Web Scraping - transfermarkt 最有价值的玩家答案

【问题标题】：Web Scraping - transfermarkt most valuable playersWeb Scraping - transfermarkt 最有价值的玩家
【发布时间】：2019-05-03 14:57:29
【问题描述】：

我是网络抓取的新手。

我在这段代码中找不到我的错：

import requests
import csv
from bs4 import BeautifulSoup
url = "https://www.transfermarkt.co.uk/spieler- 
statistik/wertvollstespieler/marktwertetop"
response=requests.get(url)
html_icerigi=response.content
soup=BeautifulSoup(html_icerigi,"html.parser")
footballer = soup.find_all("a",{"class":"spielprofil_tooltip tooltipstered"})
footballer_list=[]
for footballer in footballer_list:
   footballer=footballer.text
    footballer=footballer.strip()
    footballer=footballer.replace("\n","")
    footballer_list.append(["Futbolcu:{}".format(footballer)])
print(footballer_list)

【问题讨论】：

我相信您正在尝试获取 html 动态页面。您可能需要使用 Selenium
实际上我没有收到错误消息，但列表“footballer_list”中没有对象
哦，我明白了。谢谢

标签： python beautifulsoup python-requests

【解决方案1】：

安装Selenium 然后以这种方式访问它。否则，您的代码似乎可以工作

import bs4 
from selenium import webdriver 

browser = webdriver.Chrome()
browser.get('https://www.transfermarkt.co.uk/spieler-statistik/wertvollstespieler/marktwertetop')

html_icerigi = browser.page_source

soup = bs4.BeautifulSoup(html_icerigi,"html.parser")

footballer = soup.find_all("a",{"class":"spielprofil_tooltip tooltipstered"})
footballer_list=[]

for footballer in footballer_list:
    footballer=footballer.text
    footballer=footballer.strip()
    footballer=footballer.replace("\n","")
    footballer_list.append(["Futbolcu:{}".format(footballer)])
print(footballer)

browser.close()

输出：

[<a class="spielprofil_tooltip tooltipstered" href="/kylian-mbappe/profil/spieler/342229" id="342229">Kylian Mbappé</a>, <a class="spielprofil_tooltip tooltipstered" href="/neymar/profil/spieler/68290" id="68290">Neymar</a>, <a class="spielprofil_tooltip tooltipstered" href="/lionel-messi/profil/spieler/28003" id="28003">Lionel Messi</a>, <a class="spielprofil_tooltip tooltipstered" href="/mohamed-salah/profil/spieler/148455" id="148455">Mohamed Salah</a>, <a...

【讨论】：

【解决方案2】：

除了selenium，你还可以使用requests_html来渲染页面。尽管您在问为什么没有获得任何收益，但您的for-loop 是错误的。这意味着你最终会得到空的footballer_list，即使你已经运行了 JavaScript 并获得了完整的 html 代码。

import requests_html
from bs4 import BeautifulSoup

url = "https://www.transfermarkt.co.uk/spieler-statistik/wertvollstespieler/marktwertetop"
with requests_html.HTMLSession() as s:
    resp = s.get(url)
    resp.html.render()
    page = resp.html.raw_html


soup = BeautifulSoup(page,"html.parser")
footballer_all = soup.find_all("a",{"class":"spielprofil_tooltip tooltipstered"})

footballer_list = []

for footballer in footballer_all:
    footballer = footballer.text
    footballer = footballer.strip()
    footballer = footballer.replace("\n","")
    footballer_list.append(["Futbolcu:{}".format(footballer)])

print(footballer_list)

【讨论】：

【解决方案3】：

BeautifulSoup 可以解决问题

有反爬虫，需要设置请求用户代理
附加tooltipstered 类您可以动态删除它。
使用response.text代替转义字符串response.content。

您正在迭代空列表而不是 a 元素列表

footballer_list=[]
for footballer in footballer_list:

不需要的多行变量重写，可能是错误的列表树，你的意思是想要追加dict而不是
```
[['Futbolcu:Kylian Mbappé'], ......, ['Futbolcu:Marlon Freitas']]
```

固定代码：

import requests
import csv
from bs4 import BeautifulSoup

url = "https://www.transfermarkt.co.uk/spieler-statistik/wertvollstespieler/marktwertetop"
heads = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36'}
response = requests.get(url, headers=heads)
html_icerigi = response.text
soup = BeautifulSoup(html_icerigi, "html.parser")
footballers = soup.find_all("a",{"class":"spielprofil_tooltip"})
footballer_list = []
for footballer in footballers:
    footballer_list.append({"Futbolcu" : footballer.text.strip()})

print(footballer_list)
print(footballer_list[5]["Futbolcu"])

结果：

[
 {'Futbolcu': 'Kylian Mbappé'}, 
 ......., 
 {'Futbolcu': 'Marlon Freitas'}
]

【讨论】：