使用 BeautifulSoup/CSS-Selectors 解析同级行

Question

我想要抓取以下 HTML 结构：

<table class="showList">
    <tbody>
        <tr class="showListHeader">
            <th class="artistCol">Artists</th>
            <th class="venueCol">Venue</th>
            <th class="locationCol">Location</th>
        </tr>

        <tr class="dateRow">
            <td>11/26/14</td>
        </tr>

        <tr>
            <td class="artistCol">Artist1</td>
            <td class="venueCol">Venue1</td>
            <td class="locationCol">Location1</td>
        </tr>

        <tr>
            <td class="artistCol">Artist2</td>
            <td class="venueCol">Venue2</td>
            <td class="locationCol">Location2</td>
        </tr>

        <tr>
            <td class="artistCol">Artist3</td>
            <td class="venueCol">Venue3</td>
            <td class="locationCol">Location3</td>
        </tr>

        <tr class="dateRow">
            <td>11/28/14</td>
        </tr>

        <tr>
            <td class="artistCol">Artist4</td>
            <td class="venueCol">Venue4</td>
            <td class="locationCol">Location4</td>
        </tr>

        <tr>
            <td class="artistCol">Artist5</td>
            <td class="venueCol">Venue5</td>
            <td class="locationCol">Location5</td>
        </tr>
    </tbody>
</table>

看起来有点像这样：

+----------+--------+-----------+
|  Artist  | Venue  | Location  |
+----------+--------+-----------+
| 11/26/14 |        |           |
| Artist1  | Venue1 | Location1 |
| Artist2  | Venue2 | Location2 |
| Artist3  | Venue3 | Location3 |
| 11/28/14 |        |           |
| Artist4  | Venue4 | Location4 |
| Artist5  | Venue5 | Location5 |
+----------+--------+-----------+

我想得到以下结果：

[
    {date: 11/26/14, artist: Artist1, venue: Venue1, location: location1},
    {date: 11/26/14, artist: Artist2, venue: Venue2, location: location2},
    {date: 11/26/14, artist: Artist3, venue: Venue3, location: location3},
    {date: 11/28/14, artist: Artist4, venue: Venue4, location: location4},
    {date: 11/28/14, artist: Artist5, venue: Venue5, location: location5}
]

让我失望的是这些条目没有嵌套在日期中，它们是兄弟姐妹。 我知道如何使用 jQuery 来做到这一点，但对使用 CSS 选择器或 Scrapy 支持的 BeautifulSoup 知之甚少。任何有关我将如何解决此问题的帮助将不胜感激，如果我可以提供更多信息，请询问

Answer 1

这个想法是用class="dateRow" 遍历所有tr 标签，为每个tr 获取以下tr 兄弟姐妹；一旦遇到具有dateRow 类的tr 元素，就打破循环：

for tr in response.xpath('//tr[@class="dateRow"]'):
    date = tr.xpath('td/text()').extract()[0]

    for row in tr.xpath('following-sibling::tr'):
        # break the loop once we hit a tr with a class attribute defined
        if row.xpath('@class'):
            break

        artist = row.xpath('td[@class="artistCol"]/text()').extract()[0]
        venue = row.xpath('td[@class="venueCol"]/text()').extract()[0]
        location = row.xpath('td[@class="locationCol"]/text()').extract()[0]

        print {'date': date, 'artist': artist, 'venue': venue, 'location': location}

来自scrapy shell的演示：

$ scrapy shell index.html
>>> for tr in response.xpath('//tr[@class="dateRow"]'):
...     date = tr.xpath('td/text()').extract()[0]
...     for row in tr.xpath('following-sibling::tr'):
...         if row.xpath('@class'):
...             break
...         artist = row.xpath('td[@class="artistCol"]/text()').extract()[0]
...         venue = row.xpath('td[@class="venueCol"]/text()').extract()[0]
...         location = row.xpath('td[@class="locationCol"]/text()').extract()[0]
...         print {'date': date, 'artist': artist, 'venue': venue, 'location': location}
... 
{'date': u'11/26/14', 'venue': u'Venue1', 'location': u'Location1', 'artist': u'Artist1'}
{'date': u'11/26/14', 'venue': u'Venue2', 'location': u'Location2', 'artist': u'Artist2'}
{'date': u'11/26/14', 'venue': u'Venue3', 'location': u'Location3', 'artist': u'Artist3'}
{'date': u'11/28/14', 'venue': u'Venue4', 'location': u'Location4', 'artist': u'Artist4'}
{'date': u'11/28/14', 'venue': u'Venue5', 'location': u'Location5', 'artist': u'Artist5'}

---

UPD（使用您提供的链接）：

for tr in response.xpath('//table[@class="showList"]//tr[@class="dateRow"]'):
    date = tr.xpath('td/a/text()').extract()[0]

    for row in tr.xpath('following-sibling::tr[@class = "dateRow" or @class = " "]'):
        if row.xpath('@id'):
            break

        artist = row.xpath('td[@class="artistCol"]/a/text()').extract()[0]
        venue = row.xpath('td[@class="venueCol"]/a/text()').extract()[0]
        location = row.xpath('td[@class="locationCol"]/a/text()').extract()[0]

        print {'date': date, 'artist': artist, 'venue': venue, 'location': location}

演示：

>>> for tr in response.xpath('//table[@class="showList"]//tr[@class="dateRow"]'):
...     date = tr.xpath('td/a/text()').extract()[0]
...     for row in tr.xpath('following-sibling::tr[@class = "dateRow" or @class = " "]'):
...         if row.xpath('@id'):
...             break
...         artist = row.xpath('td[@class="artistCol"]/a/text()').extract()[0]
...         venue = row.xpath('td[@class="venueCol"]/a/text()').extract()[0]
...         location = row.xpath('td[@class="locationCol"]/a/text()').extract()[0]
...         print {'date': date, 'artist': artist, 'venue': venue, 'location': location}
... 
{'date': u'11/26/14', 'venue': u'The Stone Pony', 'location': u'Asbury Park', 'artist': u'River City Extension'}
{'date': u'11/26/14', 'venue': u'The Wonder Bar', 'location': u'Asbury Park', 'artist': u'Jessica Paris'}
{'date': u'11/26/14', 'venue': u'Third Base Pub', 'location': u'Branchville', 'artist': u'Fish House Road'}
{'date': u'11/26/14', 'venue': u'Bliss Lounge', 'location': u'Clifton', 'artist': u'Erick Morillo'}
{'date': u'11/26/14', 'venue': u'The Claddagh Lounge', 'location': u'Highlands', 'artist': u'Turtle Soup'}
{'date': u'11/26/14', 'venue': u'The Brighton Bar', 'location': u'Long Branch', 'artist': u'Hot Blood'}
...
{'date': u'01/24/15', 'venue': u'Bergen Performing Arts Center', 'location': u'Englewood', 'artist': u'Kashmir'}
{'date': u'01/24/15', 'venue': u"Hansil's Bar & Grill ", 'location': u'Oakland', 'artist': u"Hott Mott's Rhythm & Blues Band"}
{'date': u'01/24/15', 'venue': u'Studio Luloo', 'location': u'Oaklyn', 'artist': u'The Escape'}
{'date': u'01/24/15', 'venue': u'Union County Performing Arts Center', 'location': u'Rahway', 'artist': u'Milton'}
{'date': u'01/24/15', 'venue': u'Starland Ballroom', 'location': u'Sayreville', 'artist': u'Marilyn Manson'}
{'date': u'01/24/15', 'venue': u'Library II', 'location': u'Voorhees', 'artist': u'Hotlanta'}