Xpath：选择正文中的所有文本，除非在页眉、导航或页脚子部分中

Question

我正在尝试使用 Scrapy 从各种网站中抓取内容，并且真的只想要主要内容文本。这意味着避免使用导航文本、页眉文本和页脚文本。我注意到的一件事是很多时候页脚或页眉会在正文中。因此，我需要找到一种方法从 body 的子项中获取所有文本，同时忽略导航、页脚和页眉子项的可能性。

下面似乎很好地抓住了一切：

sel = Selector(response=response)
items = TextItem()
items['text']    = ' '.join(sel.xpath("//body//text()").extract())

但返回一切。

我一直在尝试找到像下面的 Xpath 那样删除导航的方法，但没有取得多大成功。

' '.join(sel.xpath("//body//text()[not(descendant-or-self::nav) and not(descendant-or-self::header) and not(descendant-or-self::footer)]").extract())

有没有办法只用 Xpath 来做到这一点？还是我需要使用更复杂的逻辑和工具，比如 bs4？

感谢您的帮助！

Answer 1

我建议您先从 body 节点中删除不需要的节点（例如使用 lxml）：

from lxml import html
...
    bad_tags = ['nav', 'header', 'footer']
    t = html.fromstring(your_main_body_node)
    for bad_tag in bad_tags:
        for bad_node in t.xpath(f'//{bad_tag}'):
            bad_node.getparent().remove(bad_node)

Answer 2

您可以在 XPath 中使用足够的谓词 []（使用 functions、axes、...）过滤文本节点。

例如，假设您要获取此article 的文本（不包括标题、标题、推广链接（'Get your unlimited Newsweek trial'）中的文本。

你可以这样写：

//div[@class="article-body v_text paywall"]//text()[parent::p or parent::em or parent::a[@target]]

我们从非常具体的元素中获取文本。

对于这个article，您可以在Scrapy 中使用以下代码（将提取标题和标题）：

''.join(response.xpath('//div[@class="story-body__inner"]//text()[parent::p or parent::a]').getall())

输出：

'The US has said it will hold off an a threatened hike in tariffs on $7.5bn (£5.75bn) worth of European and UK goods that it imposed as punishment for subsidies for plane-maker Airbus.The move comes as the two sides wrestle to put to an end their 16-year trade battle over state aid for Airbus and American rival Boeing.The US last year raised border taxes on more than 100 items, including jumpers, single-malt whiskies and cheese.It has said the EU has not done enough."The EU and member states have not taken the actions necessary to come into compliance with WTO decisions," America\'s top trade official, Robert Lighthizer, said on Wednesday.  "The United States, however, is committed to obtaining a long-term resolution to this dispute. The European Union cautiously welcomed the US decision not to increase the amount of goods subject to tariffs."The Commission acknowledges the decision of the US not to exacerbate the ongoing aircraft dispute by increasing tariffs on European products," an EU spokesperson said.Airbus last month said it would alter some deals responsible for the dispute, saying the changes, including increasing its interest rates on loans with France and Spain, eliminated "any justification" for the US border taxes.The move prompted EU officials to call for an end to "unjustified" tariffs. Many American businesses have also protested the duties, which raise prices for American buyers.On Wednesday Airbus spokesman Clay McConnell said in a statement the company "profoundly regrets that, despite Europe\'s recent actions to achieve full compliance, USTR [US Trade Representative] has decided to maintain tariffs on Airbus aircraft - especially at a time when aviation and other sectors are going through an unprecedented crisis."The US announced tariffs on $7.5bn worth of goods last year after the World Trade Organization ruled that state aid provided to Airbus to launch its A380 and A350 jets was illegal and authorised American retaliation. In February, the US raised the rates being charged on aircraft from 10% to 15%, leaving the 25% duty on other items unchanged.This summer, American officials again threatened to raise tariff rates or make new items subject to the import tax.The items threatened with new duties included salmon fillets, gin and olives.The US is required by law to review the tariffs periodically. On Wednesday it announced minor tweaks to the list, for example, removing sweet biscuits made in the UK and adding jams from France and Germany.Trade lawyer Jamieson Greer, former chief of staff to US Trade Representative Robert Lighthizer, told the BBC: "The reality is that this can all be solved if Airbus took some action to provide restitution".The European Union, which brought its own case challenging American subsidies for Boeing, has threatened to hit the US with tariffs of its own. It is waiting for the World Trade Organization to decide how big such a punishment might be. The US in May said it had eliminated the benefits in dispute. That WTO ruling is expected later this year. "In the absence of a settlement, the EU will be ready to fully avail itself of its own sanction rights," Trade Commissioner Phil Hogan said last month.The issue has also complicated trade talks between the US and the UK.UK Trade Secretary Liz Truss raised the matter in talks with Mr Lighthizer this month, as the two sides held a third round of negotiations.The secretary "was clear that the UK considers these tariffs to be unacceptable and continued to push for their immediate removal," the Department for International Trade said.'

仔细查看HTML 代码，找到其中的模式并相应地编写您的XPath 表达式以选择您需要的文本节点。使用 ChroPath 插件或浏览器中的开发工具（F12 键）来测试您的 XPath 表达式。

开始使用 XPath：https://www.w3schools.com/xml/xpath_intro.asp