R-XML 将节点拉入矩阵/DF 以解决缺失节点

Question

我对使用 R 相当陌生，对使用 XML 包和 xpath 也很陌生。我需要从一个看起来像这样的 xml 文件中提取四个元素（除了我已经修剪掉了很多其他 xmlnodes 来简化它）：

<?xml version="1.0" encoding="utf-8"?>
<iati-activities version="1.03" generated-datetime="2015-07-07T16:49:09+00:00">
  <iati-activity last-updated-datetime="2014-08-11T14:36:59+00:00" xml:lang="en" default-currency="EUR">
<iati-identifier>NL-KVK-41160054-100530</iati-identifier>
<title>Improvement of basic health care</title>
<reporting-org ref="NL-KVK-41160054" type="21">Stichting Cordaid</reporting-org>
<participating-org role="Accountable" ref="NL-KVK-41160054" type="21">Cordaid</participating-org>
<participating-org role="Funding" ref="EU" type="15">EU</participating-org>
<participating-org role="Funding" type="21">Cordaid Memisa</participating-org>
<participating-org role="Funding" ref="NL-1" type="10">Dutch Ministry of Foreign Affairs</participating-org>
<participating-org role="Implementing" type="21">CORDAID RCA</participating-org>
<recipient-country percentage="100" code="CF">CENTRAL AFRICAN REPUBLIC</recipient-country>
<budget type="1">
  <period-start iso-date="2010-01-01"></period-start>
  <period-end iso-date="2013-02-28"></period-end>
</budget>
  </iati-activity>
  <iati-activity last-updated-datetime="2013-07-19T14:12:14+00:00" xml:lang="en" default-currency="EUR">
<iati-identifier>NL-KVK-41160054-100625</iati-identifier>
<title>Pigs for Pencils</title>
<reporting-org ref="NL-KVK-41160054" type="21">Stichting Cordaid</reporting-org>
<participating-org role="Funding" ref="NL-1" type="10">Dutch Ministry of Foreign Affairs</participating-org>
<participating-org role="Funding" type="60">Stichting Kapatiran</participating-org>
<participating-org role="Implementing" type="22">PREDA Foundation Inc.</participating-org>
<participating-org role="Accountable" ref="NL-KVK-41160054" type="21">Cordaid</participating-org>
<budget type="2">
  <period-start iso-date="2010-04-20"></period-start>
  <period-end iso-date="2012-10-02"></period-end>
  <value value-date="2010-04-20">12500</value>
</budget>
   </iati-activity>
  <iati-activity last-updated-datetime="2015-04-08T03:01:58+00:00" xml:lang="en" default-currency="EUR">
    <iati-identifier>NL-KVK-41160054-100815</iati-identifier>
<title>Job and housing opportunities for women </title>
<reporting-org ref="NL-KVK-41160054" type="21">Stichting Cordaid</reporting-org>
<participating-org role="Funding" ref="NL-1" type="10">Dutch Ministry of Foreign Affairs</participating-org>
<participating-org role="Implementing" type="22">WISE</participating-org>
<participating-org role="Accountable" ref="NL-KVK-41160054" type="21">Cordaid</participating-org>
<budget type="2">
  <period-start iso-date="2010-10-01"></period-start>
  <period-end iso-date="2011-12-31"></period-end>
  <value value-date="2010-10-01">227000</value>
</budget>
  </iati-activity>
</iati-activities>

这也是我在 StackOverflow 上的第一个问题，所以如果我做得不对（并且 xml 没有完全对齐），请道歉。 我需要的元素，以及我分配给它们的元素是：

UniqueID <- "//iati-activity/iati-identifier"

GrantTitle <- "//iati-activity/title"

GrantAmount <- "//iati-activity/budget/value"

Recipient <- "//iati-activity/participatingorg[@role='Implementing']"

到目前为止（经过多次试验和磨难）我已经想出了这段代码，它通过当前节点（x），拉出 4 个变量，并将它们绑定成一行，然后使用 xpathApply 循环通过 iati-活动节点调用函数并将结果行绑定在一起。

当所有四个元素都存在于每个活动中时，此代码有效。但是，请注意 xml 示例中没有预算/价值节点。这是因为我删除它是为了解决缺少节点的问题，对于我需要的几乎所有元素，它在完整文件中经常出现。

还要注意我的 xpath 表达式末尾的 [1] - 我已经包含这些是因为还有多个标题、所有类型的多个参与组织等。

鉴于某些元素的倍数而其他元素不存在，因此不可能简单地将所有相同的元素拉入一个向量并将其弹出到数据框中。因此需要循环通过每个活动来拉动元素。我的代码目前无法解决缺少的元素（第一个 iati 活动中缺少的预算/值），因为 cbinding（和 rbinding）忽略了空向量。

xmltestNA = xmlInternalTreeParse("XMLtoDF_TestNA.xml", useInternalNodes=TRUE)
bodyToDF <- function(x){
  UniqueID <- xpathSApply(x, "./iati-identifier", xmlValue)
  GrantTitle <- xpathSApply(x, "./title[1]", xmlValue)
  GrantAmount <- xpathSApply(x, "./budget/value[1]", xmlValue)
  Recipient <- xpathSApply(x, "./participating-org[@role='Implementing'][1]", xmlValue)
  cbind(UniqueID=UniqueID, GrantTitle=GrantTitle, GrantAmount=GrantAmount, Recipient=Recipient)
  }
res <-xpathApply(xmltestNA, '//iati-activity', fun=bodyToDF)
IatiNA <-do.call(rbind, res)
IatiNA

如何保留空值/缺失节点，以便将其转换为如下所示的矩阵或数据框：

    UniqueID    GrantTitle  GrantAmount Recipient
1   NL-KVK-41160054-100530  Improvement of basic health care    NA  CORDAID RCA
2   NL-KVK-41160054-100625  Pigs for Pencils    12500   PREDA Foundation Inc.
3   NL-KVK-41160054-100815  Job and housing opportunities for women     227000  WISE

因为我还是新手，代码越简单越好。提前致谢！

Answer 1

如果您的 xpath 查询返回的结果太多或太少，我认为使用节点会更容易

doc <- xmlParse( '<your xml here>')
nodes<- getNodeSet(doc, "//iati-activity")

#Compare
xpathSApply(doc, "//budget/value", xmlValue)
xpathSApply(doc, "//participating-org[@role='Funding']", xmlValue)

sapply(nodes, function(x) xpathSApply(x, "./budget/value", xmlValue))
sapply(nodes, function(x) xpathSApply(x, "./participating-org[@role='Funding']", xmlValue))

添加一个函数来处理丢失或多个节点，然后创建data.frame

xpath2 <-function(x, path, fun = xmlValue, ...){
   y <- xpathSApply(x, path, fun, ...)
   ifelse(length(y) == 0, NA,
    ifelse(length(y) > 1, paste(unlist(y), collapse=", "), y))
}

GrantAmount <- sapply(nodes, xpath2, "./budget/value")
UniqueID    <- sapply(nodes, xpath2, "./iati-identifier")
GrantTitle  <- sapply(nodes, xpath2, "./title")
Recipient   <-  sapply(nodes, xpath2, "./participating-org[@role='Implementing']")
## updated xpath2 so xmlGetAttr will also work
Funding_ref  <- sapply(nodes, xpath2, "./participating-org[@role='Funding']", xmlGetAttr, "ref")
Budget_start <- sapply(nodes, xpath2, ".//period-start", xmlGetAttr, "iso-date")

data.frame(UniqueID, GrantTitle, GrantAmount, Recipient)
                UniqueID                               GrantTitle GrantAmount             Recipient
1 NL-KVK-41160054-100530         Improvement of basic health care        <NA>           CORDAID RCA
2 NL-KVK-41160054-100625                         Pigs for Pencils       12500 PREDA Foundation Inc.
3 NL-KVK-41160054-100815 Job and housing opportunities for women       227000                  WISE

Answer 2

考虑使用 XSLT 转换文件。作为信息，XSLT 是一种声明性专用编程语言，用于以各种结构转换、样式化或重新格式化 XML 文件。遗憾的是，这种在网络时代被遗忘的语言对于日常需求仍然非常有用。

特别是对你来说，你需要一个更简单的结构来让 R 轻松导入。需要注意的是 R 没有专门的、可靠的 XSLT 库。因此，您需要找到一个外部 XSLT 处理器。幸运的是，有几个 R 可以利用的开源解决方案。

XSLT 样式表

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="xml" indent="yes" omit-xml-declaration="no" />

<xsl:template match="iati-activities">
<GrantData>

    <xsl:for-each select="iati-activity">
    <iati-activity>
        <UniqueID><xsl:value-of select="iati-identifier"/></UniqueID>
        <GrantTitle><xsl:value-of select="title"/></GrantTitle>
        <GrantAmount><xsl:value-of select="budget/value"/></GrantAmount>
        <Recipient><xsl:value-of select="participating-org[@role='Implementing']"/></Recipient>
    </iati-activity>
    </xsl:for-each>

</GrantData>

</xsl:template>

</xsl:stylesheet>

Python （使用它的 lxml 库）

#!/usr/bin/python
import lxml.etree as ET

dom = ET.parse('C:\Path\To\RawXMLFile.xml')
xslt = ET.parse('C:\Path\To\RawXSLTFile.xsl')
transform = ET.XSLT(xslt)
newdom = transform(dom)

tree_out = ET.tostring(newdom, encoding='UTF-8', pretty_print=True,  xml_declaration=True)
print(tree_out)

xmlfile = open('C:\Path\To\OutputXMLFile.xml','ab')
xmlfile.write(tree_out)
xmlfile.close()

PHP（使用其 xsl 扩展）

这是为那些认为 PHP 不是通用语言而仅限于网络的人准备的：

<?php

// Load the XML source and XSLT file
$cd = dirname(__FILE__);

$xml = new DOMDocument;
$xml->load('C:\Path\To\RawXMLFile.xml');
$xsl = new DOMDocument;
$xsl->load('C:\Path\To\RawXSLTFile.xsl');    

// Configure the transformer
$proc = new XSLTProcessor;
$proc->importStyleSheet($xsl); // attach the xsl rules    

// Transform XML source
$newXml = $proc->transformToXML($xml);

// Save output to file
$xmlfile = 'C:\Path\To\OutputXMLFile.xml.xml';
file_put_contents($xmlfile, $newXml);  

?>

这两种方法都会为您的 R 数据框需求生成具有更简单结构的 XML（注意 GrantAmount 的空节点）。

<?xml version="1.0"?>

<GrantData>
  <iati-activity>
    <UniqueID>NL-KVK-41160054-100530</UniqueID>
    <GrantTitle>Improvement of basic health care</GrantTitle>
    <GrantAmount/>
    <Recipient>CORDAID RCA</Recipient>
  </iati-activity>
  <iati-activity>
    <UniqueID>NL-KVK-41160054-100625</UniqueID>
    <GrantTitle>Pigs for Pencils</GrantTitle>
    <GrantAmount>12500</GrantAmount>
    <Recipient>PREDA Foundation Inc.</Recipient>
  </iati-activity>
  <iati-activity>
    <UniqueID>NL-KVK-41160054-100815</UniqueID>
    <GrantTitle>Job and housing opportunities for women </GrantTitle>
    <GrantAmount>227000</GrantAmount>
    <Recipient>WISE</Recipient>
  </iati-activity>
</GrantData>

生成更简单的 XML 后，只需导入 R：

library(XML)

# LOADING TRANSFORMED XML INTO R DATA FRAME
doc<-xmlParse("C:\\Path\\To\\OutputXMLFile.xml")
xmldf <- xmlToDataFrame(nodes = getNodeSet(doc, "//iati-activity"))

VBA（使用 MSXML 对象）

最后，有一个可以在 MS Excel 或 MS Access 中运行的 VBA 解决方案。幸运的是，对于最后一个解决方案，您可以使用 R 的 RDCOMClient 包（仅 PC 解决方案）将 VBA 与 R 连接：

library(XML)
library(RDCOMClient)

xmlfile = COMCreate("MSXML2.DOMDocument")
xslfile = COMCreate("MSXML2.DOMDocument")
newxmlfile = COMCreate("MSXML2.DOMDocument")

xmlstr = "C:\\Path\\To\\RawXMLFile.xml"
xslstr = "C:\\Path\\To\\RawXSLTFile.xsl"
newxmlstr = "C:\\Path\\To\\OutputXMLFile.xml"

# LOADING XML & XSLT FILES
xmlfile.async = FALSE
xmlfile$Load(xmlstr)

xslfile.async = FALSE
xslfile$Load(xslstr)

# TRANSFORMING XML FILE USING XLST INTO NEW FILE
xmlfile$transformNodeToObject(xslfile, newxmlfile)
newxmlfile$Save(newxmlstr)

# LOADING TRANSFORMED XML INTO R DATA FRAME
doc<-xmlParse("C:\\Path\\To\\OutputXMLFile.xml")
xmldf <- xmlToDataFrame(nodes = getNodeSet(doc, "//iati-activity"))
View(xmldf)

xmlfile <- NULL
xslfile <- NULL
newxmlfile <- NULL

缺失节点为 NaN 的数据框结果：

    UniqueID                GrantTitle                          GrantAmount    Recipient
1   NL-KVK-41160054-100530  Improvement of basic health care                   CORDAID RCA
2   NL-KVK-41160054-100625  Pigs for Pencils                           12500   PREDA Foundation Inc.
3   NL-KVK-41160054-100815  Job and housing opportunities for women   227000   WISE