跟随第二个请求的链接 - 网络爬虫

Question

请多多包涵。我对 Python 还是很陌生——但玩得很开心。我正在尝试编写一个网络爬虫，该爬虫通过旅游网站的结果进行爬网。我已设法从主页中提取所有相关链接。现在我希望 Python 跟踪每个链接并从每个页面收集信息。但我被困住了。希望你能给我一个提示。

这是我的代码：

  import requests
  from bs4 import BeautifulSoup
  import urllib, collections

  Spider =1

  def trade_spider(max_pages):
  RegionIDArray = {737: "London"}
  for reg in RegionIDArray:
  page = -1
  r = requests.get("https://www.viatorcom.de/London/d" +str(reg) +"&page=" + str(page) , verify = False)
  soup = BeautifulSoup(r.content, "lxml")

  g_data = soup.find_all("h2", {"class": "mtm mbn card-title"})
  for item in g_data:
      Deeplink = item.find_all("a")
      for t in set(t.get("href") for t in Deeplink):
          Deeplink_final = t
          print(Deeplink_final) #The output shows all the links that I would like to follow and gather information from.          
  trade_spider(1)

输出：

    /de/7132/London-attractions/Stonehenge/d737-a113
    /de/7132/London-attractions/Tower-of-London/d737-a93
    /de/7132/London-attractions/London-Eye/d737-a1400
    /de/7132/London-attractions/Thames-River/d737-a1410

输出显示了我想关注并从中收集信息的所有链接。

我的代码的下一步：

   import requests
   from bs4 import BeautifulSoup
   import urllib, collections

   Spider =1

   def trade_spider(max_pages):
   RegionIDArray = {737: "London"}
   for reg in RegionIDArray:
   page = -1
   r = requests.get("https://www.viatorcom.de/London/d" +str(reg) +"&page=" + str(page) , verify = False)
   soup = BeautifulSoup(r.content, "lxml")

   g_data = soup.find_all("h2", {"class": "mtm mbn card-title"})
   for item in g_data:
       Deeplink = item.find_all("a")
       for t in set(t.get("href") for t in Deeplink):
           Deeplink_final = t   
   trade_spider(1)

   def trade_spider2(max_pages):

   r = requests.get("https://www.viatorcom.de" + Deeplink_final,  verify = False)
   soup = BeautifulSoup(r.content, "lxml")
   print(soup)

   trade_spider2(9)

我想将最初抓取的输出附加到我的第二个请求中。但这不起作用。希望你能给我一个提示。

Answer 1

这应该会有所帮助。

import requests
from bs4 import BeautifulSoup
import urllib, collections

Spider =1

def trade_spider2(Deeplink_final):
   r = requests.get("https://www.viatorcom.de" + Deeplink_final,  verify = False)
   soup = BeautifulSoup(r.content, "lxml")
   print(soup)

def trade_spider(max_pages):
   RegionIDArray = {737: "London"}
   for reg in RegionIDArray:
       page = -1
       r = requests.get("https://www.viatorcom.de/London/d" +str(reg) +"&page=" + str(page) , verify = False)
       soup = BeautifulSoup(r.content, "lxml")

       g_data = soup.find_all("h2", {"class": "mtm mbn card-title"})
       for item in g_data:
           Deeplink = item.find_all("a")
           for Deeplink_final in set(t.get("href") for t in Deeplink):
               trade_spider2(Deeplink_final)

trade_spider(1)