如何获取 BeautifulSoup 的所有链接

Question

我想要这个链接的酒店的所有链接：https://www.french.hostelworld.com/s?q=Paris,%20Ile-de-France,%20France&country=France&city=Paris&type=city&id=14&from=2021-04-30&to=2021-05-03&guests=2&page=1

类似的东西，列表：['https://www.french.hostelworld.com/pwa/hosteldetails.php/R-sidence-Internationale-de-Paris/Paris/294403?from=2021-04- 30&to=2021-05-03&guests=2', 'https://www.french.hostelworld.com/pwa/hosteldetails.php/Le-Village-Montmartre-by-Hiphophostels/Paris/606?from=2021-04- 30&to=2021-05-03&guests=2'...]

这是我的脚本：

import numpy as np


from time import sleep
from random import randint
import requests
from requests import get
from bs4 import BeautifulSoup
import pandas as pd
import re



url = 'https://www.french.hostelworld.com/s?q=Paris,%20Ile-de-France,%20France&country=France&city=Paris&type=city&id=14&from=2021-04-30&to=2021-05-03&guests=2&page=1'

links1 = []

results = requests.get(url)


soup = BeautifulSoup(results.text, "html.parser")

links1 = [a['href']  for a in soup.find("div", {"class": "page-inner"}).find_all('a', href=True)]


print(links1)

我得到了这个：

(base) C:\Users\evanalonso\PYTHON\webscraping script>python hostelworld.py
['/']

出了点问题，但我想不通，有什么想法吗？

Answer 1

您正在尝试从 div 元素内部检索所有链接：

links1 = [a['href']  for a in soup.find("div", {"class": "page-inner"}).find_all('a', href=True)]

查看该元素的内容。如果我们运行以下 代码：

div = soup.find("div", {"class": "page-inner"})
print(div)

我们看到了：

<div class="page-inner">
  <header data-v-0225089c="">
    <a class="logo nuxt-link-active" data-v-0225089c="" href="/"
    title="Hostelworld">
      <img alt="Hostelworld" data-v-0225089c=""
      src="/_nuxt/img/789e4da.svg" />
    </a>
    <div class="header-icons" data-v-0225089c="">
      <div class="user-authentication item" data-v-0225089c="">
        <button aria-label="Se connecter/cr&#195;&#169;er un compte"
        class="header-option" data-v-0225089c="" id="header-login"
        title="Se connecter/cr&#195;&#169;er un compte">
          <i aria-describedby="icon-core-user-fill-aria-description"
          aria-hidden="true"
          aria-labelledby="icon-core-user-fill-aria-label"
          class="core-icon icon-core-user-fill" data-v-0225089c="">
            <!-- -->
            <!-- -->
          </i>
          <span class="sr-only" data-v-0225089c="">Se
          connecter/cr&#195;&#169;er un compte</span>
        </button>
        <!-- -->
        <!-- -->
      </div>
      <div class="select-list item" data-v-0225089c=""
      data-v-5ec8cd6c="" id="header-language-picker"
      role="listbox">
        <div class="select-list-slot-wrapper" data-v-5ec8cd6c=""
        tabindex="0">
          <div class="icon-menu-item" data-v-0225089c=""
          data-v-5ec8cd6c="" role="option"
          title="Fran&#195;&#167;ais">Fran&#195;&#167;ais</div>
        </div>
        <!-- -->
      </div>
      <div class="select-list item" data-v-0225089c=""
      data-v-5ec8cd6c="" id="header-currency-picker"
      role="listbox">
        <div class="select-list-slot-wrapper" data-v-5ec8cd6c=""
        tabindex="0">
          <div class="icon-menu-item" data-v-0225089c=""
          data-v-5ec8cd6c="" role="option" title="USD">USD</div>
        </div>
        <!-- -->
      </div>
      <button aria-label="Menu"
      class="icon-core-menu-fill header-option item"
      data-v-0225089c="" title="Menu">
        <span class="sr-only" data-v-0225089c="">Menu</span>
      </button>
    </div>
    <!-- -->
    <!-- -->
  </header>
</div>

请注意，除了这个之外，其中没有 <a> 元素：

<a class="logo nuxt-link-active" data-v-0225089c="" href="/"
title="Hostelworld">
  <img alt="Hostelworld" data-v-0225089c=""
  src="/_nuxt/img/789e4da.svg" />
</a>

这意味着您的代码工作正常。问题是， 您在浏览器中看到的内容是在加载后通过 Javascript 加载的 初始 HTML 内容。如果您使用开发人员，您可以看到这一点 浏览器中的工具来查看由 加载页面。对于您给出的示例，我们看到一个请求 JSON 文档如下所示：

GET https://api.m.hostelworld.com/2.2/cities/14/properties/?currency=USD&application=web&user-id=ae71369c-e75b-45e5-8ffa-97223f9daf22&date-start=2021-04-30&num-nights=3&guests=2&per-page=1000&show-rooms=1&property-num-images=30

如果您请求该 URL，您将获得一个 JSON 文档，其中可能包含所有 你想要的信息：

>>> import requests
>>> url='https://api.m.hostelworld.com/2.2/cities/14/properties/?currency=USD&application=web&user-id=ae71369c-e75b-45e5-8ffa-97223f9daf22&date-start=2021-04-30&num-nights=3&guests=2&per-page=1000&show-rooms=1&property-num-images=30'
>>> res=requests.get(url)
>>> print('\n'.join(property['name'] for property in res.json()['properties']))
Enjoy Hostel
Peace & Love Hostel
Generator Paris
Le Village Montmartre by Hiphophostels
Aloha Eiffel Tower by Hiphophostels
MEININGER Paris Porte de Vincennes
Smart Place Paris Gare du Nord by Hiphophostels
.
.
.