Scrapy 中的 CSS 选择器

【问题标题】：CSS selector in ScrapyScrapy 中的 CSS 选择器
【发布时间】：2022-01-22 00:39:57
【问题描述】： 
import scrapy

class CliffTop(scrapy.Spider):
    name = 'ctop'
    
    start_urls = [
        'https://www.net-a-porter.com/en-in/shop/clothing/tops'
    ]

    def parse(self, response):
        all_div_title= response.css('a::attr(href) div.ProductItem24')
        mname = all_div_title.css('span.ProductItem24__designer::text').extract()
        yield {
            'name': mname
        }

现在当我运行蜘蛛时，我没有得到任何输出。谁能告诉我 CSS 选择器有什么问题？ 优先级：提取名称

【问题讨论】：

    标签： python web-scraping scrapy


    【解决方案1】：

    数据由 javascript 从 xhr 完全动态加载，scrapy 无法呈现 javascript。因此，要获取数据，您需要一个自动化工具，例如带有 scrapy 的 re/json/selenium。这里我用的是json。

    脚本：

    import scrapy

class CliffTop(scrapy.Spider):
    name = 'ctop'
    #start_urls = ['https://www.net-a-porter.com/en-in/shop/clothing/tops']

    def start_requests(self):
        headers= {
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
            "x-ibm-client-id": "95f14cbd-793e-46ec-9f76-6fac2fbb6683",
            "accept-language": "en-US,en;q=0.9,bn;q=0.8,es;q=0.7,ar;q=0.6"
            }
        yield scrapy.Request(
            url='https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60',
            callback=self.parse,
            method= 'GET',
            headers=headers
            
            )
    def parse(self, response):
        titles= response.json()['products']
        for title in titles:
            mname = title['nameEN']
            yield {'name': mname}

    输出：

    {'name': 'Faye cropped shirred stretch-cotton top'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Cyprus cropped cutout shirred linen halterneck top'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Flocked cotton-jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Printed cotton-jersey hoodie'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Cotton-jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Checked twill-trimmed cotton-jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Plum ruffled broderie anglaise organic cotton-poplin blouse'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Grow Up Bee Happy embroidered printed cotton-jersey sweatshirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Cropped flocked cotton-jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Asymmetric cotton-jersey and ruffled taffeta T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Checked cotton-jersey hoodie'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Printed cotton-jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Kaye ribbed organic cotton-jersey turtleneck top'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Ines printed cotton-poplin peplum blouse'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Estelle cropped open-back floral-print linen top'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Ribbed silk turtleneck top'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': '+ Ed Curtis embroidered cotton-jersey T-Shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Cropped flocked organic cotton-jersey hoodie'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Printed cotton-jersey hoodie'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Oversized printed cotton-jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Ribbed stretch organic cotton-jersey tank'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Ribbed modal and cotton-blend jersey tank'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Ethel ruffled embroidered organic cotton-voile blouse'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Asymmetric cotton-jersey and ruffled taffeta T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Zip-detailed cotton-jersey sweatshirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Evie cutout gingham recycled seersucker top'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Ottica cropped printed linen halterneck top'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Printed cotton-jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Cotton-jersey hoodie'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Ruffled broderie anglaise cotton-poplin and jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Button-embellished striped organic cotton-blend jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Calipo cropped shirred linen halterneck top'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Amanda lace-trimmed satin blouse'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Embroidered cotton-jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Amie ruffled floral-print organic cotton-poplin blouse'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'The Daily ribbed stretch-cotton tank'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Ribbed organic cotton-jersey top'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Henley Pima cotton-jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Biggins matte-satin camisole'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Asymmetric embroidered cotton-jersey sweatshirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Oversized ruffled printed organic cotton-poplin blouse'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Patrizia pintucked broderie anglaise-trimmed cotton-poplin top'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Dallas embroidered cotton and TENCEL-blend blouse'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Brixton stretch-jersey camisole'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Savona paisley-print silk crepe de chine blouse'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Striped cotton-poplin tunic'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Valensole ribbed-knit bralette'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Jersey hoodie'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Cropped flocked cotton-jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Appliquéd organic cotton-jersey sweatshirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Cassia cropped scalloped floral-print linen top'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Neza open-back satin camisole'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': '+ NET SUSTAIN Met cutout ribbed organic cotton-blend jersey top'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Embroidered cotton-jersey T-shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Colorado thong bodysuit'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>
{'name': 'Delora floral-print linen shirt'}
2021-12-20 19:43:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.net-a-porter.com/api/nap/search/resources/store/nap_in/productview/byCategory?attrs=true&category=%2Fclothing%2Ftops&locale=en_US&pageNumber=2&pageSize=60>

    .. 以此类推

    【讨论】：

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2020-04-23
      • 2021-07-13
      • 2018-11-16
      • 1970-01-01
      • 2022-01-21
      • 1970-01-01
      相关资源
      最近更新 更多
      热门标签
      Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode