Python：从 XPath 获取元素值

Question

我对 xpaths 和网络抓取真的很陌生，所以如果这是一个相对较小的问题，我很抱歉。我正在尝试抓取多个网站，以确保更新数据库中的数据。我能够获取部分字符串的 xPath，但不确定如何使用 xPath 获取完整值。

代码：

def xpath_soup(element):
    components = []
    child = element if element.name else element.parent
    for parent in child.parents:

        previous = itertools.islice(parent.children, 0,parent.contents.index(child))
        xpath_tag = child.name
        xpath_index = sum(1 for i in previous if i.name == xpath_tag) + 1
        components.append(xpath_tag if xpath_index == 1 else '%s[%d]' % (xpath_tag, xpath_index))
        child = parent
    components.reverse()
    return '/%s' % '/'.join(components)



page = requests.get("https://www.gaumard.com/obstetricmr")
html = str(BeautifulSoup(page.content, 'html.parser'))
soup = BeautifulSoup(html, 'lxml')
elem = soup.find(string=re.compile('xt-generation mixed reality training solution for VICTORIA® S2200 designed to help learners bridge the gap between theory and practice'))
xPathValue = xpath_soup(elem)
print(xPathValue)

我正在尝试使用 xPathValue 获取元素的完整值。

预期结果将是完整版 xt-generation mixed reality training solution for VICTORIA® S2200 designed to help learners bridge the gap between theory and practice

存在

Obstetric MR™ is a next-generation mixed reality training solution for VICTORIA® S2200 designed to help learners bridge the gap between theory and practice faster than ever before. Using the latest technology in holographic visualization, Obstetric MR brings digital learning content into the physical simulation exercise, allowing participants to link knowledge and skill through an entirely new hands-on training experience. The future of labor and delivery simulation is here.

这个全部价值将来自利用xPathValue。

Answer 1

以下是使用XPath 获取全文的方法。

import requests
from lxml import html

page = requests.get("https://www.gaumard.com/obstetricmr").text
text = html.fromstring(page).xpath('//*[@style="margin: 0 auto;"][2]/div/text()')
print(text[0].strip())

输出：

Obstetric MR™ is a next-generation mixed reality training solution for VICTORIA® S2200 designed to help learners bridge the gap between theory and practice faster than ever before. Using the latest technology in holographic visualization, Obstetric MR brings digital learning content into the physical simulation exercise, allowing participants to link knowledge and skill through an entirely new hands-on training experience. The future of labor and delivery simulation is here.

Answer 2

一个特定的 XPath 不会有太大帮助，因为如前所述，网页可能会有所不同。 用于搜索文本节点并获取包含该字符串的节点的数组或列表的通用 XPath 可以帮助进行一些后期处理。

在 Firefox 控制台上试用：

nodes = $x('//*[contains(text(),"next-generation mixed reality")]', window.document, "nodes");
<- Array [ div ]

nodes[0].textContent;
<- "Obstetric MR™ is a next-generation...(redacted)"

这个 XPath 可以在其他页面上工作
'//*[contains(text(),"next-generation mixed reality")]'
前提是它们包含next-generation mixed reality 字符串。

同样使用 python：

import requests
from lxml import html
url = 'https://www.gaumard.com/obstetricmr'
response = requests.get(url)
html_doc = response.content
xpath0 = '//*[contains(text(),"next-generation mixed reality")]'
result_arr = html.fromstring(html_doc).xpath(xpath0)
result_arr[0].text

输出：

'Obstetric MR™ is a next-generation mixed...'