如何循环遍历csv文件scrapy中的起始网址答案

【问题标题】：How to loop through start urls in csv file scrapy如何循环遍历csv文件scrapy中的起始网址
【发布时间】：2020-07-29 21:32:56
【问题描述】：

所以基本上它在我第一次运行蜘蛛时出于某种原因工作，但之后它只抓取一个 URL。

-我的程序正在从列表中抓取我想要抓取的部分。

-将部分列表也转换为文件中的 URL。

-运行并抓取我想要的数据并输入到 csv 文件中。

问题：仅从一个 URL 获取输出不知道从这里去哪里我检查了其他资源并尝试发出 start_request。结果还是一样。

所以基本上我怎样才能让它使用所有的 start_urls 并遍历它们中的每一个而不只是最后一个？

这是蜘蛛：

import csv
import xlrd
import scrapy

wb = xlrd.open_workbook(r'C:\Users\Jatencio\PycharmProjects\testy\test.xlsx')
ws = wb.sheet_by_index(0)
mylist = ws.col_values(0)
print(mylist)

li = []
for el in mylist:
    baseparts = el[:5]
    url1 = 'https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=' + baseparts + '&pageSize=500&pkeyword=' + baseparts
    li.append(url1)
final = list(set(li))


file = open('templist.csv','w+',newline='')
with file:
    write = csv.writer(file, delimiter =',')
    write.writerows(x.split(',') for x in final)

class DigikeSpider(scrapy.Spider):
    name = 'digike'
    allowed_domains = ['digikey.com']
    custom_settings = {
        "USER_AGENT": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36"

    }

    with open('templist.csv') as file:
        start_urls = [line.strip() for line in file]

    def parse(self, response):
        data = {}
        parts1 = []
        # parts=response.css('Table#productTable.productTable')
        for p in response.css('tbody#lnkPart > tr'):

            if p.css('td.tr-mfgPartNumber span::text').get() not in mylist:
                continue

            else:
                parts1 = p.css('td.tr-mfgPartNumber span::text').get()

            if p.css('td.tr-minQty.ptable-param span.desktop::text').get():
                quantity = p.css('td.tr-minQty.ptable-param span.desktop::text').get()
                quantity = quantity.strip()
                cleaned_quantity = int(quantity.replace(',', ''))
            else:
                quantity = 'No quantity'

            if p.css('td.tr-unitPrice.ptable-param center::text').get() == 'Active':
                p.css('td.tr-mfgPartNumber span::text').remove()

            else:
                pass

            if p.css('td.tr-unitPrice.ptable-param center::text').get() == 'Obsolete':
                p.css('td.tr-mfgPartNumber span::text').remove()

            else:
                pass

            if p.css('td.tr-unitPrice.ptable-param center::text').get() == 'Discontinued at Digi-Key':
                p.css('td.tr-mfgPartNumber span::text').remove()

            else:
                pass

            if p.css('td.tr-unitPrice.ptable-param span::text').get():
                price = p.css('td.tr-unitPrice.ptable-param span::text').get()
                cleaned_price = price.strip()
            else:
                price = 'No Price'

            if p.css('td.tr-qtyAvailable.ptable-param span.desktop::text').get():
                stock = p.css('td.tr-qtyAvailable.ptable-param span.desktop::text').get()
                cleaned_stock = stock.strip()

            else:
                pass
            if p.css('#part-status ::text').get():
                status = p.css('#part-status ::text').get()
                cleaned_status = status.strip()

            else:
                pass

            yield {
                'Part': parts1,
                'Quantity': cleaned_quantity,
                'Price': cleaned_price,
                'Stock': cleaned_stock,
                'Status': cleaned_status,

            }

输出

2020-07-30 10:12:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=IS62L&pageSize=500&pkeyword=IS62L> (referer:
 None)
2020-07-30 10:12:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY622&pageSize=500&pkeyword=CY622> (referer:
 None)
2020-07-30 10:12:11 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY622&pageSize=500&pkeyword=CY622
> (referer: None)
Traceback (most recent call last):
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\utils\defer.py", line 120, in iter_errback
    yield next(it)
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\utils\python.py", line 346, in __next__
    return next(self.data)
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\utils\python.py", line 346, in __next__
    return next(self.data)
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\core\spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output
    for x in result:
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\core\spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 340, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\core\spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\core\spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\core\spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "C:\Users\Jatencio\PycharmProjects\testy\testdigi\testdigi\spiders\digike.py", line 93, in parse
    'Quantity': cleaned_quantity,
UnboundLocalError: local variable 'cleaned_quantity' referenced before assignment
2020-07-30 10:12:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=IS62C&pageSize=500&pkeyword=IS62C> (referer:
 None)
2020-07-30 10:12:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=IS62W&pageSize=500&pkeyword=IS62W> (referer:
 None)
2020-07-30 10:12:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY621&pageSize=500&pkeyword=CY621> (referer:
 None)
2020-07-30 10:12:17 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY621&pageSize=500&pkeyword=CY621>
{'Part': 'CY62128ELL-45SXIT', 'Quantity': 1000, 'Price': '$2.29429', 'Stock': '1,000 - Immediate', 'Status': 'Active'}
2020-07-30 10:12:17 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY621&pageSize=500&pkeyword=CY621>
{'Part': 'CY62157EV30LL-45ZSXIT', 'Quantity': 1000, 'Price': '$6.44254', 'Stock': '2,000 - Immediate', 'Status': 'Active'}
2020-07-30 10:12:17 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY621&pageSize=500&pkeyword=CY621>

2020-07-30 10:12:17 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY621&pageSize=500&pkeyword=CY621
> (referer: None)
Traceback (most recent call last):
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\parsel\selector.py", line 368, in remove
    parent = self.root.getparent()
AttributeError: 'str' object has no attribute 'getparent'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\utils\defer.py", line 120, in iter_errback
    yield next(it)
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\utils\python.py", line 346, in __next__
    return next(self.data)
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\utils\python.py", line 346, in __next__
    return next(self.data)
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\core\spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output
    for x in result:
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\core\spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 340, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\core\spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\core\spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\core\spidermw.py", line 64, in _evaluate_iterable
    for r in iterable:
  File "C:\Users\Jatencio\PycharmProjects\testy\testdigi\testdigi\spiders\digike.py", line 55, in parse
    p.css('td.tr-mfgPartNumber span::text').remove()
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\parsel\selector.py", line 164, in remove
    x.remove()
  File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\parsel\selector.py", line 371, in remove
    raise CannotRemoveElementWithoutRoot(
parsel.selector.CannotRemoveElementWithoutRoot: The node you're trying to remove has no root, are you trying to remove a pseudo-element? Try to use 'li' as a selector instead of 'li::text' or '//li' instead of '//li/text()', for exampl
e.
2020-07-30 10:12:17 [scrapy.core.engine] INFO: Closing spider (finished)
2020-07-30 10:12:17 [scrapy.extensions.feedexport] INFO: Stored csv feed (40 items) in: DigiKeyPartsList.csv

print(start_urls): ['https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k =IS62C&pageSize=500&pkeyword=IS62C', 'https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?F V=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=IS62L&pageSize=500&pkeyword=IS62L', 'https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV =-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=IS62W&pageSize=500&pke yword=IS62W', 'https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY621&pageSize=500&pkeyword =CY621', 'https://www.digikey.com/products/en/integrated-circuits-ics /memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY622&pageSize=500&pkeyword=CY622']

新输出

2020-07-30 12:51:31 [scrapy.utils.log] INFO: Scrapy 2.2.1 started (bot: testdigi)
2020-07-30 12:51:31 [scrapy.utils.log] INFO: Versions: lxml 4.5.2.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)], pyO
penSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020), cryptography 3.0, Platform Windows-10-10.0.17134-SP0
2020-07-30 12:51:31 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-07-30 12:51:31 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'testdigi',
 'NEWSPIDER_MODULE': 'testdigi.spiders',
 'SPIDER_MODULES': ['testdigi.spiders'],
 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
               '(KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'}
2020-07-30 12:51:31 [scrapy.extensions.telnet] INFO: Telnet Password: 4abf97dccc166f2d
2020-07-30 12:51:31 [py.warnings] WARNING: c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\scrapy\extensions\feedexport.py:210: ScrapyDeprecationWarning: The `FEED_URI` and `FEED_FORMAT` settings have been deprecated in
favor of the `FEEDS` setting. Please see the `FEEDS` setting docs for more details
  exporter = cls(crawler)

2020-07-30 12:51:31 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2020-07-30 12:51:32 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-07-30 12:51:32 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-07-30 12:51:32 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-07-30 12:51:32 [scrapy.core.engine] INFO: Spider opened
2020-07-30 12:51:32 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-07-30 12:51:32 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-07-30 12:51:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=IS62L&pageSize=500&pkeyword=IS62L> (referer:
 None)
2020-07-30 12:51:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=IS62C&pageSize=500&pkeyword=IS62C> (referer:
 None)
2020-07-30 12:51:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY622&pageSize=500&pkeyword=CY622> (referer:
 None)
2020-07-30 12:51:33 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY622&pageSize=500&pkeyword=CY622>
{'Part': 'CY62256NLL-55ZXIT', 'Quantity': 'No quantity', 'Price': '$1.11989', 'Stock': '0', 'Status': 'Obsolete'}
2020-07-30 12:51:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=IS62W&pageSize=500&pkeyword=IS62W> (referer:
 None)
2020-07-30 12:51:33 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY622&pageSize=500&pkeyword=CY622>
{'Part': 'CY62256VNLL-70ZXIT', 'Quantity': 'No quantity', 'Price': 'No Price', 'Stock': '0', 'Status': 'Obsolete'}
2020-07-30 12:51:33 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY622&pageSize=500&pkeyword=CY622>
{'Part': 'CY62256NLL-55SNXIT', 'Quantity': 'No quantity', 'Price': 'No Price', 'Stock': '0', 'Status': 'Obsolete'}
2020-07-30 12:51:33 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY622&pageSize=500&pkeyword=CY622>
{'Part': 'CY62256VNLL-70SNXIT', 'Quantity': 'No quantity', 'Price': 'No Price', 'Stock': '0', 'Status': 'Obsolete'}
2020-07-30 12:51:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikey.com/products/en/integrated-circuits-ics/memory/774?FV=-8%7C774%2C7%7C1&quantity=0&ColumnSort=0&page=1&k=CY621&pageSize=500&pkeyword=CY621> (referer:
 None)
{'Part': 'CY62148EV30LL-45ZSXIT', 'Quantity': 1000, 'Price': 'No Price', 'Stock': '0', 'Status': 'Active'}
2020-07-30 12:51:34 [scrapy.core.engine] INFO: Closing spider (finished)
2020-07-30 12:51:34 [scrapy.extensions.feedexport] INFO: Stored csv feed (46 items) in: DigiKeyPartsList.csv
2020-07-30 12:51:34 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 2145,
 'downloader/request_count': 5,
 'downloader/request_method_count/GET': 5,
 'downloader/response_bytes': 289446,
 'downloader/response_count': 5,
 'downloader/response_status_count/200': 5,
 'elapsed_time_seconds': 2.311786,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 7, 30, 16, 51, 34, 681758),
 'item_scraped_count': 46,
 'log_count/DEBUG': 51,
 'log_count/INFO': 11,
 'log_count/WARNING': 1,
 'response_received_count': 5,
 'scheduler/dequeued': 5,
 'scheduler/dequeued/memory': 5,
 'scheduler/enqueued': 5,
 'scheduler/enqueued/memory': 5,
 'start_time': datetime.datetime(2020, 7, 30, 16, 51, 32, 369972)}
2020-07-30 12:51:34 [scrapy.core.engine] INFO: Spider closed (finished)

(venv) C:\Users\Jatencio\PycharmProjects\testy\testdigi\testdigi>

【问题讨论】：

我在我的输出中没有运行的每个 url 上都得到一个（引用：无）。
start_urls 中存储的值是多少你能查到吗？（打印、记录或使用调试器）
@renatodvc 我把它加到上面了^

标签： python web-scraping scrapy

【解决方案1】：

现在，通过执行日志，我可以告诉你蜘蛛有两个问题，似乎都与start_urls 无关。

第一个例外：

File "C:\Users\Jatencio\PycharmProjects\testy\testdigi\testdigi\spiders\digike.py", line 93, in parse
    'Quantity': cleaned_quantity,
UnboundLocalError: local variable 'cleaned_quantity' referenced before assignment

您在定义之前引用了cleaned_quantity。问题出在这里：

        if p.css('td.tr-minQty.ptable-param span.desktop::text').get():
            quantity = p.css('td.tr-minQty.ptable-param span.desktop::text').get()
            quantity = quantity.strip()
            cleaned_quantity = int(quantity.replace(',', ''))
        else:
            quantity = 'No quantity'

如果您的 if 语句解析为 false，则从不定义 clean_quantity，并且在您尝试组装项目时会引发错误：

        yield {
            'Part': parts1,
            'Quantity': cleaned_quantity,
            'Price': cleaned_price,
            'Stock': cleaned_stock,
            'Status': cleaned_status,
        }

这仅在几次迭代中发生，而不是全部。

第二个例外：

 File "C:\Users\Jatencio\PycharmProjects\testy\testdigi\testdigi\spiders\digike.py", line 55, in parse
    p.css('td.tr-mfgPartNumber span::text').remove()
[...]
 File "c:\users\jatencio\pycharmprojects\testy\venv\lib\site-packages\parsel\selector.py", line 371, in remove
    raise CannotRemoveElementWithoutRoot(
parsel.selector.CannotRemoveElementWithoutRoot: The node you're trying to remove has no root, are you trying to remove a pseudo-element? Try to use 'li' as a selector instead of 'li::text' or '//li' instead of '//li/text()', for example.

这里的问题是你在解析器调用伪元素中使用.remove()方法，你只能用来从HTML树中删除实际元素，所以我相信这应该可以解决问题：

改变这个：

p.css('td.tr-mfgPartNumber span::text').remove()

到这里：

p.css('td.tr-mfgPartNumber span').remove()

这适用于您使用remove 方法的所有行。

如果这解决了您的问题，请告诉我。

【讨论】：

谢谢，这帮了很大的忙，但现在出现了一个问题，它只为两个 url 提取数据，而不是全部。我收到此错误消息：ScrapyDeprecationWarning：FEED_URI 和 FEED_FORMAT 设置已被弃用，取而代之的是 FEEDS 设置。有关更多详细信息，请参阅FEEDS 设置文档 exporter = cls(crawler)
你是如何执行爬虫的？您是否检查了settings.py 中的这些设置？可以添加整个执行日志吗？
更新了我使用scrapy crawl digike执行的输出
根据您的日志，您的start_urls 中的所有5 个页面均已成功爬取。其中一些页面没有抓取任何内容，这意味着数据不存在或选择器不适合查找您想要的数据。您的执行过程中没有更多错误。
谢谢，我会再看看我的选择器。