Xpath 从网页中提取文章文本

Question

我将对该网站上的文章进行网络爬网。

这是我到目前为止所做的：

# HR Version
# the entire crawling process

openfile = open("data/HR.csv", "rb")
r = csv.reader(openfile)
HR_data = []

for i in r:
    url = i[0]
    print url  # to know the status of web crawling
    r = requests.get(url)
    data = html.fromstring(r.text)
    #Inspect line with text
    #//*[@id="article-details"]
    #<section class="entry-content clearfix" itemprop="articleBody"></section>
    texts = data.xpath("//*[@id="article-details"]/p/text()") 
    raw = ''.join(str(i.encode("utf-8")) for i in texts)
    finaldata = raw.replace('\r','').replace('\n','').replace('\r','').replace('\t','')    
    HR_data.append([finaldata])

openfile.close()

有问题的命令如下

texts = data.xpath("//*[@id="article-details"]/p/text()")

它来自这个特定的网页：http://hrmagazine.co.uk/article-details/internal-entrepreneurship-can-boost-your-business

在 Firefox 上使用 Inspect Element，我发现“文本”在以下部分中，包含在以下部分中：

<article id="article-details">
#One <h2> element, followed by multiple <p> elements.
</article>

从文章中只提取段落文本的正确 XPath 是什么？

Answer 1

您几乎编写了正确的 XPath。你需要在h2上替换p

texts = data.xpath("//*[@id="article-details"]/h2/text()")