每次打开新选项卡时如何捕获 URL 的响应答案

【问题标题】：How do I capture the response of URL every-time new tab opens每次打开新选项卡时如何捕获 URL 的响应
【发布时间】：2020-05-04 00:11:43
【问题描述】：

我正在尝试开发一个网络抓取项目，其中我正在抓取一个名为 Startup India 的网站，您可以使用该网站与初创公司建立联系。在这里，我根据我选择的一些过滤器单击，然后单击每个启动，当我单击每个启动时，我必须进入该启动并抓取它。但我无法抓取数据，因为我无法捕获在初创公司 India 抓取个人资料的响应。

import scrapy
from selenium import webdriver
import os
import logging
class ProductSpider(scrapy.Spider):
    name = "product_spider"
    allowed_domains = ['https://www.startupindia.gov.in/']
    start_urls = ['https://www.startupindia.gov.in/content/sih/en/search.html?industries=sih:industry/advertising&states=sih:location/india/andhra-pradesh&stages=Prototype&roles=Startup&page=0']

    def __init__(self):
        cwd = os.getcwd()
        self.driver = webdriver.Chrome("C:/Users/RAJ/PycharmProjects/WebCrawler/WebCrawler/WebCrawler/spiders/chromedriver.exe")


    def parse(self, response):
        self.driver.get(response.url)


        next = self.driver.find_elements_by_xpath("//*[@id='persona-results']//a[@class='img-wrap']")


        logging.info(next)

        for i in next:
            try:
                logging.info(i.click())
                logging.info(response.url)

                # get the data and write it to scrapy items
            except:
                print("Yolo")

代码将不胜感激

【问题讨论】：

在 Selenium 中有 current_url 方法可以获取当前窗口的 url。
@dermouser123 我试过不行。
你传递给解析函数的参数是什么？
@Yun 我已经通过了自我和回应。
您要查找的数据已通过此 API 调用加载：api.startupindia.gov.in/sih/api/noauth/search/profiles 我会向您的 start_url 引用的此 api 发出请求以获取数据。

标签： python-3.x selenium web-scraping scrapy

【解决方案1】：

我已经设置了scrapy项目并运行scrapy crawl product_spider，它会在点击一个元素后给出新标签页的URL。

import scrapy
from selenium import webdriver
import os
import logging
from selenium.webdriver.chrome.options import Options as ChromeOptions

CHROME_DRIVER_UBUNTU_PATH = "your chrome driver path"


class ProductSpider(scrapy.Spider):
    name = "product_spider"
    allowed_domains = ['https://www.startupindia.gov.in/']
    start_urls = [
        'https://www.startupindia.gov.in/content/sih/en/search.html?industries=sih:industry/advertising&states=sih:location/india/andhra-pradesh&stages=Prototype&roles=Startup&page=0']

    def __init__(self):
        cwd = os.getcwd()
        opts = ChromeOptions()
        opts.add_argument("--headless")  # for headless browser it's not necessary

        self.driver = webdriver.Chrome(executable_path=CHROME_DRIVER_UBUNTU_PATH, chrome_options=opts)

    def parse(self, response):
        self.driver.get(response.url)

        next = self.driver.find_elements_by_xpath("//*[@id='persona-results']//a[@class='img-wrap']")

        for i in next:
            try:
                i.click()  # click on image in page
                # move to new tab open
                self.driver.switch_to.window(self.driver.window_handles[next.index(i) + 1])
                logging.info(self.driver.current_url)

                self.driver.switch_to.window(self.driver.window_handles[0])
                # get the data and write it to scrapy items
            except Exception as e:
                print(e)

【讨论】：