将抓取与 CssSelector 和 BeautifulSoup 相结合

Question

我正在使用 Python 在 Tripadvisor 中搜索巴黎的景点。我想将一部分抓取与 CssSelector 和一部分与 BeautifulSoup 结合起来。

这是带有 CssSelector 的部分：

wd.get("https://www.tripadvisor.it/Attractions-g187147-Activities-a_allAttractions.true-Paris_Ile_de_France.html") 
import pprint 
detail_attractions = [] 
for attraction in list_attractions:
    url = attraction.find_elements_by_css_selector("a")[0].get_attribute("href")
    title = ""
 
    if(len(attraction.find_elements_by_css_selector("._35oYbnyI")) > 0):
      title = attraction.find_elements_by_css_selector("._35oYbnyI")[0].text 
    
    detail_attractions.append({'url': url,
                            'title': title})

这是 BeautifulSoup 的部分：

import requests
from bs4 import BeautifulSoup
import re

header = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"}

resp = requests.get('https://www.tripadvisor.it/Attractions-g187147-Activities-c42-Paris_Ile_de_France.html', headers=header)
if resp.status_code == 200:
    soup = BeautifulSoup(resp.text, 'lxml')
    cards = soup.find_all('div', {'data-automation': 'cardWrapper'})
    for card in cards:
        rating = card.find('svg', {'class': 'zWXXYhVR'})
        match = re.match('Punteggio ([0-9,]+)', rating.attrs['aria-label'])[1]
        print(float(match.replace(',', '.')))

我想在 detail_attractions.append 中添加属性“rating”，并将每个标题与其评分联系起来。

Answer 1

说真的，我不明白在这里使用selenium 的原因。顺便说一句，在这种情况下根本没有实现 selenium。

因此，您不需要大量使用selenium，然后执行另一个请求来解析评级。

以下是处理您的目标的单个调用。

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
from pprint import pp

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0'
}


def main(url):
    r = requests.get(url, headers=headers)
    soup = BeautifulSoup(r.text, 'lxml')
    goal = [(urljoin(url, x.select_one('a')['href']),
             x.select_one('span[name=title]').text.split(' ', 1)[-1],
             x.select_one('.zWXXYhVR')['aria-label'].split()[1])
            for x in soup.select(
        '._3Y-YU9SE[data-automation$=SingleFlexCardSection]')]
    pp(goal)


main('https://www.tripadvisor.it/Attractions-g187147-Activities-a_allAttractions.true-Paris_Ile_de_France.html')

输出：

[('https://www.tripadvisor.it/Attraction_Review-g187147-d188150-Reviews-Musee_d_Orsay-Paris_Ile_de_France.html',
  "Musee d'Orsay",
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d188679-Reviews-Cathedrale_Notre_Dame_de_Paris-Paris_Ile_de_France.html',
  'Cattedrale di Notre Dame',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d190202-Reviews-Sainte_Chapelle-Paris_Ile_de_France.html',
  'Sainte-Chapelle',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d190204-Reviews-Palais_Garnier-Paris_Ile_de_France.html',
  'Opera Garnier',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d188151-Reviews-Eiffel_Tower-Paris_Ile_de_France.html',
  'Torre Eiffel',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d265635-Reviews-Musee_de_l_Orangerie-Paris_Ile_de_France.html',
  "Musee de l'Orangerie",
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d188709-Reviews-Arc_de_Triomphe-Paris_Ile_de_France.html',
  'Arco di Trionfo',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d188757-Reviews-Louvre_Museum-Paris_Ile_de_France.html',
  'Museo del Louvre',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d189284-Reviews-Montmartre-Paris_Ile_de_France.html',
  'Montmartre',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d189683-Reviews-Seine_River-Paris_Ile_de_France.html',
  'La Senna',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d189687-Reviews-Luxembourg_Gardens-Paris_Ile_de_France.html',
  'Giardini del Lussemburgo',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d190685-Reviews-Basilique_du_Sacre_Coeur_de_Montmartre-Paris_Ile_de_France.html',
  'Basilica di Montmartre (Sacre-Coeur)',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d188149-Reviews-Musee_Rodin-Paris_Ile_de_France.html',
  'Musee Rodin',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d292257-Reviews-Le_Marais-Paris_Ile_de_France.html',
  'Le Marais',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d189280-Reviews-Pantheon-Paris_Ile_de_France.html',
  'Pantheon',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d188485-Reviews-Musee_Marmottan_Monet-Paris_Ile_de_France.html',
  'Musee Marmottan',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d191240-Reviews-Saint_Germain_des_Pres_Quarter-Paris_Ile_de_France.html',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d189249-Reviews-Parc_des_Buttes_Chaumont-Paris_Ile_de_France.html',
  'Parc des Buttes Chaumont',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d13986330-Reviews-Atelier_des_Lumieres-Paris_Ile_de_France.html',
  'Atelier des Lumières',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d265614-Reviews-Pont_Alexandre_III-Paris_Ile_de_France.html',
  'Ponte Alessandro III',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d189193-Reviews-Galeries_Lafayette_Paris_Haussmann-Paris_Ile_de_France.html',
  'Galeries Lafayette',
  '4,0'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d314450-Reviews-Observatoire_Panoramique_de_la_Tour_Montparnasse-Paris_Ile_de_France.html',
  'Tour Montparnasse',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d10128174-Reviews-Musee_du_Parfum_Fragonard-Paris_Ile_de_France.html',
  'Musee du Parfum - Fragonard',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d188467-Reviews-Place_des_Vosges-Paris_Ile_de_France.html',
  'Place des Vosges',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d7149635-Reviews-Fondation_Louis_Vuitton-Paris_Ile_de_France.html',
  'Fondation Louis Vuitton',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d188698-Reviews-Pere_Lachaise_Cemetery-Paris_Ile_de_France.html',
  'Il cimitero di Père-Lachaise',
  '4,5'),
 ('https://www.tripadvisor.it/Attraction_Review-g187147-d2507208-Reviews-Musee_des_Arts_Forains-Paris_Ile_de_France.html',
  'Musee des Arts Forains',
  '4,5')]