如何使用 xpath 从 html 表中检索项目列表？答案

【问题标题】：How to retrieve item list from html table with xpath?如何使用 xpath 从 html 表中检索项目列表？
【发布时间】：2019-01-06 10:59:28
【问题描述】：

我正在尝试将表信息提取到 python 3.7 中的字典中。

表格中的 html 如下所示：

            <dl class="rlxr-specs__block-list">
                <dt class="rlxr-specs__block-list--name">heading</dt>
                <dd class="rlxr-specs__definition-content">
                    <div class="rlxr-specs__definition-title">Key1</div>
                    <span class="rlxr-specs__definition-desc">bla</span>
                </dd>
                <dd class="rlxr-specs__definition-content">
                    <div class="rlxr-specs__definition-title">Key2</div>
                    <span class="rlxr-specs__definition-desc">blub</span>
                </dd>

我的最佳猜测是：

items{}
for row in response.xpath('//dd[@class="rlxr-specs__definition-content"]'):
    items[row.xpath('./div/text()').extract_first()] = items[row.xpath('./span/text()').extract_first()]

我收到一个 Keyerror，其中包含来自页面另一部分的密钥。所以 xpath 选择器中的某些东西一定是错误的。

更多信息：

>>> for row in response.xpath('//dd[@class="rlxr-specs__definition-content"]'):
...     print(row.xpath('./div/text()'))
... 
[<Selector xpath='./div/text()' data='Gehäuse'>]
[<Selector xpath='./div/text()' data='Aufbau des Oyster Gehäuses'>]
[<Selector xpath='./div/text()' data='Durchmesser'>]
[<Selector xpath='./div/text()' data='Material'>]
[<Selector xpath='./div/text()' data='Lünette'>]
[<Selector xpath='./div/text()' data='Aufzugskrone'>]
[<Selector xpath='./div/text()' data='Uhrglas'>]
[<Selector xpath='./div/text()' data='Wasserdichtheit'>]
[<Selector xpath='./div/text()' data='Manufakturwerk'>]
[<Selector xpath='./div/text()' data='Kaliber'>]
[<Selector xpath='./div/text()' data='Ganggenauigkeit'>]
[<Selector xpath='./div/text()' data='Funktionen'>]
[<Selector xpath='./div/text()' data='Oszillator'>]
[<Selector xpath='./div/text()' data='Aufzug'>]
[<Selector xpath='./div/text()' data='Gangreserve'>]
[<Selector xpath='./div/text()' data='Armband'>]
[<Selector xpath='./div/text()' data='Material'>]
[<Selector xpath='./div/text()' data='Schließe'>]
[<Selector xpath='./div/text()' data='Zifferblatt'>]
[<Selector xpath='./div/text()' data='Edelsteinfassung'>]
[]
>>> for row in response.xpath('//dd[@class="rlxr-specs__definition-content"]'):
...     print(row.xpath('./span/text()'))
... 
[<Selector xpath='./span/text()' data='Oyster, 28 mm, Edelstahl Oystersteel und'>]
[<Selector xpath='./span/text()' data='Monoblock-Mittelteil, verschraubter Gehä'>]
[<Selector xpath='./span/text()' data='28 mm'>]
[<Selector xpath='./span/text()' data='Rolesor Everose (Kombination aus Edelsta'>]
[<Selector xpath='./span/text()' data='Diamantlünette'>]
[<Selector xpath='./span/text()' data='Verschraubbare Twinlock-Aufzugskrone mit'>]
[<Selector xpath='./span/text()' data='Kratzfestes Saphirglas, Zykloplupe\xa0zur\xa0V'>]
[<Selector xpath='./span/text()' data='Bis 100 Meter Tiefe wasserdicht'>]
[<Selector xpath='./span/text()' data='Mechanisches Perpetual-Uhrwerk, Selbstau'>]
[<Selector xpath='./span/text()' data='2236, Rolex Manufakturwerk'>]
[<Selector xpath='./span/text()' data='-2/+2 Sekunden pro Tag, gemessen nach de'>]
[<Selector xpath='./span/text()' data='Stunden-, Minuten- und Sekundenzeiger im'>]
[]
[<Selector xpath='./span/text()' data='Selbstaufzugsmechanismus, in beide Richt'>]
[<Selector xpath='./span/text()' data='Circa 55 Stunden'>]
[<Selector xpath='./span/text()' data='Jubilé, fünfreihig'>]
[<Selector xpath='./span/text()' data='Rolesor Everose (Kombination aus Edelsta'>]
[<Selector xpath='./span/text()' data='Verdeckte Crownclasp-Faltschließe'>]
[<Selector xpath='./span/text()' data='Helles Perlmuttzifferblatt mit Diamanten'>]
[<Selector xpath='./span/text()' data='Diamanten, Fassung 18 Karat Gold'>]
[<Selector xpath='./span/text()' data='Chronometer der Superlative  (COSC + Rol'>]
>>>

如何将表格拉到字典中？

【问题讨论】：

替换为items[row.xpath('./div/text()')[0]] = row.xpath('./span/text()')[0]
我确实得到了 IndexError: list index out of range
" IndexError"：一个 xpath(... 返回一个epyt 列表。 Edit 你的问题并显示print(row.xpath('./div/text()')) 和print(row.xpath('./span/text()')) 的first 输出
你是对的，似乎有空元素。编辑问题

标签： python xpath web-scraping scrapy

【解决方案1】：

尝试检查是否有标题和描述值，如果没有值 - 设置默认值：

items{}
for row in response.xpath('//dd[@class="rlxr-specs__definition-content"]'):
    title = row.xpath('./div/text()').extract_first() or "No title"
    description = row.xpath('./span/text()').extract_first() or "No description"
    items[title] = description

【讨论】：