(Python) DoD xccdf xml 文件的 XML 到 CSV 转换

Question

我开始编写一个 python 脚本来获取来自 DISA SCAP 检查工具 (SCC) 的 XML 输出并将其转换为 csv，以便我可以轻松地将其摄取到 Splunk 进行存储/分析。以下是 XML 的摘录。

<cdf:Benchmark id="xccdf_mil.disa.stig_benchmark_Mozilla_Firefox_Windows" style="SCAP_1.2" resolved="1" xsi:schemaLocation="http://checklists.nist.gov/xccdf/1.2 http://scap.nist.gov/schema/xccdf/1.2/xccdf_1.2.xsd http://cpe.mitre.org/dictionary/2.0 http://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd http://cpe.mitre.org/language/2.0 http://scap.nist.gov/schema/cpe/2.3/cpe-language_2.3.xsd" xmlns:cdf="http://checklists.nist.gov/xccdf/1.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:dc="http://purl.org/dc/elements/1.1/">
      <cdf:status date="2020-01-24">accepted</cdf:status>
      <cdf:title>Mozilla Firefox Security Technical Implementation Guide</cdf:title>
      <cdf:description>The Mozilla Firefox Security Technical Implementation Guide (STIG) is published as a tool to improve the security of Department of Defense (DoD) information systems. Comments or proposed revisions to this document should be sent via e-mail to the following address: disa.stig_spt@mail.mil</cdf:description>
      <cdf:notice id="terms-of-use"></cdf:notice>
      <cdf:reference href="https://cyber.mil">
            <dc:publisher>DISA</dc:publisher>
            <dc:source>STIG.DOD.MIL</dc:source>
      </cdf:reference>
      <cdf:plain-text id="release-info">Release: 1.5 Benchmark Date: 24 Jan 2020</cdf:plain-text>
      <cdf:platform idref="cpe:/a:mozilla:firefox:::~~~windows~~"></cdf:platform>
      <cdf:version update="http://iase.disa.mil/stigs">001.005</cdf:version>
      <cdf:metadata>
            <dc:creator>DISA</dc:creator>
            <dc:publisher>DISA</dc:publisher>
            <dc:contributor>DISA</dc:contributor>
            <dc:source>STIG.DOD.MIL</dc:source>
      </cdf:metadata>
      <cdf:Profile id="xccdf_mil.disa.stig_profile_MAC-1_Classified">
            <cdf:title>I - Mission Critical Classified</cdf:title>
            <cdf:description>&lt;ProfileDescription&gt;&lt;/ProfileDescription&gt;</cdf:description>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15768" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15771" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15772" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15774" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15775" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15776" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15778" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15779" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15983" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15985" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15986" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-19742" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-19743" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-19744" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-64891" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-79053" selected="true"></cdf:select>
      </cdf:Profile>
      <cdf:Profile id="xccdf_mil.disa.stig_profile_MAC-1_Public">
            <cdf:title>I - Mission Critical Public</cdf:title>
            <cdf:description>&lt;ProfileDescription&gt;&lt;/ProfileDescription&gt;</cdf:description>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15768" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15771" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15772" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15774" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15775" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15776" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15778" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15779" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15983" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15985" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15986" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-19742" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-19743" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-19744" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-64891" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-79053" selected="true"></cdf:select>
      <cdf:Group id="xccdf_mil.disa.stig_group_V-15768" Id="xccdf_mil.disa.stig_group_V-15768">
            <cdf:title>FireFox Preferences – Verification</cdf:title>
            <cdf:description>&lt;GroupDescription&gt;&lt;/GroupDescription&gt;</cdf:description>
            <cdf:Rule id="xccdf_mil.disa.stig_rule_SV-16707r1_rule" weight="10.0" Id="xccdf_mil.disa.stig_rule_SV-16707r1_rule" severity="medium">
                  <cdf:version update="http://iase.disa.mil/stigs">DTBF050</cdf:version>
                  <cdf:title>FireFox is configured to ask which certificate to present to a web site when a certificate is required.</cdf:title>
                  <cdf:description>&lt;VulnDiscussion&gt;When a web site asks for a certificate for user authentication, Firefox must be configured to have the user choose which certificate to present. Websites within DOD require user authentication for access which increases security for DoD information. Access will be denied to the user if certificate management is not configured.&lt;/VulnDiscussion&gt;&lt;FalsePositives&gt;&lt;/FalsePositives&gt;&lt;FalseNegatives&gt;&lt;/FalseNegatives&gt;&lt;Documentable&gt;false&lt;/Documentable&gt;&lt;Mitigations&gt;&lt;/Mitigations&gt;&lt;SeverityOverrideGuidance&gt;&lt;/SeverityOverrideGuidance&gt;&lt;PotentialImpacts&gt;&lt;/PotentialImpacts&gt;&lt;ThirdPartyTools&gt;&lt;/ThirdPartyTools&gt;&lt;MitigationControl&gt;&lt;/MitigationControl&gt;&lt;Responsibility&gt;System Administrator&lt;/Responsibility&gt;&lt;IAControls&gt;&lt;/IAControls&gt;</cdf:description>
                  <cdf:reference>
                        <dc:publisher>DISA</dc:publisher>
                        <dc:identifier>205</dc:identifier>
                        <dc:type>DPMS Target</dc:type>
                  </cdf:reference>
                  <cdf:ident system="http://iase.disa.mil/cci">CCI-001274</cdf:ident>
                  <cdf:fixtext fixref="F-15985r1_fix">Set the value of "security.default_personal_cert" to "Ask Every Time".  Use the Mozilla.cfg file to lock the preference so users cannot change it.</cdf:fixtext>
                  <cdf:fix id="F-15985r1_fix"></cdf:fix>
                  <cdf:check system="http://oval.mitre.org/XMLSchema/oval-definitions-5">
                        <cdf:check-content-ref name="oval:mil.disa.stig.mozilla.firefox.windows:def:20" href="U_Mozilla_Firefox_Windows_V1R5_STIG_SCAP_1-2_Benchmark-oval.xml"></cdf:check-content-ref>
                  </cdf:check>
            </cdf:Rule>
      </cdf:Group>
<cdf:TestResult id="xccdf_mil.disa.stig_testresult_scap_mil.disa.stig_comp_U_Mozilla_Firefox_Windows_V1R5_STIG_SCAP_1-2_Benchmark-xccdf.xml---xccdf_mil.disa.stig_profile_MAC-1_Classified-1" start-time="2020-04-28T16:34:31" version="001.005" end-time="2020-04-28T16:34:31" test-system="cpe:/a:spawar:scc:5.3">
            <cdf:benchmark id="xccdf_mil.disa.stig_benchmark_Mozilla_Firefox_Windows" href="#scap_mil.disa.stig_comp_U_Mozilla_Firefox_Windows_V1R5_STIG_SCAP_1-2_Benchmark-xccdf.xml"></cdf:benchmark>
            <cdf:organization>NIWC Atlantic</cdf:organization>
            <cdf:profile idref="xccdf_mil.disa.stig_profile_MAC-1_Classified"></cdf:profile>
            <cdf:target-id-ref name="SCC_DESKTOP-LAISPM9" href="" system="http://scap.nist.gov/schema/asset-identification/1.1"></cdf:target-id-ref>
            <cdf:platform idref="cpe:/a:mozilla:firefox:::~~~windows~~"></cdf:platform>
            <cdf:rule-result idref="xccdf_mil.disa.stig_rule_SV-16707r1_rule" weight="10.0" version="DTBF050" severity="medium" time="2020-04-28T16:34:31">
                  <cdf:result>fail</cdf:result>
                  <cdf:ident system="http://iase.disa.mil/cci">CCI-001274</cdf:ident>
                  <cdf:fix id="F-15985r1_fix"></cdf:fix>
                  <cdf:check system="http://oval.mitre.org/XMLSchema/oval-definitions-5">
                        <cdf:check-content-ref name="oval:mil.disa.stig.mozilla.firefox.windows:def:20" href="#scap_mil.disa.stig_comp_U_Mozilla_Firefox_Windows_V1R5_STIG_SCAP_1-2_Benchmark-oval.xml"></cdf:check-content-ref>
                  </cdf:check>
            </cdf:rule-result>
            <cdf:rule-result idref="xccdf_mil.disa.stig_rule_SV-16710r3_rule" weight="10.0" version="DTBF105" severity="medium" time="2020-04-28T16:34:31">
                  <cdf:result>fail</cdf:result>
                  <cdf:ident system="http://iase.disa.mil/cci">CCI-000381</cdf:ident>
                  <cdf:fix id="F-15988r3_fix"></cdf:fix>
                  <cdf:check system="http://oval.mitre.org/XMLSchema/oval-definitions-5">
                        <cdf:check-content-ref name="oval:mil.disa.stig.mozilla.firefox.windows:def:40" href="#scap_mil.disa.stig_comp_U_Mozilla_Firefox_Windows_V1R5_STIG_SCAP_1-2_Benchmark-oval.xml"></cdf:check-content-ref>
                  </cdf:check>
            </cdf:rule-result>
</cdf:TestResult>
</cdf:Benchmark>

这是我到目前为止所得到的。它只写我在“xccdf_head”中指定的值。还有更多值要提取，但我想我想在构建之前获取格式。

import csv
import xml.etree.ElementTree as ET

# file to read
tree = ET.parse("./test.xml")
root = tree.getroot()

# create file for writing
xccdf_out = open('./test.csv', 'w')
csvwriter = csv.writer(xccdf_out)
xccdf_head = ['ID']
csvwriter.writerow(xccdf_head)

for group in root.findall('cdf:Benchmark'):
        row = []
        group_id = group.find('cdf:Group').find('cdf:title').text
        row.append(group_id)
        csvwriter.writerow(row)
xccdf_out.close()

我正在寻找可以从两个部分（cdf:Group 和 cdf:Result）中捕获以下内容的输出

'cdf:groupid', cdf:ruleid','cdf:ruleseverity','cdf:title','cdf:description','cdf:result'

...理想情况下，它看起来像这样：

xccdf_mil.disa.stig_group_V-79053, xccdf_mil.disa.stig_rule_SV-93759r3_rule, medium, Background submission of information to Mozilla must be disabled, VulnDiscussion>There should be no background submission(...),fail

Answer 1

另一种方法。

from simplified_scrapy import SimplifiedDoc,utils
html = utils.getFileContent('test.xml') # Get xml data from file
doc = SimplifiedDoc(html)

rows = [['cdf:groupid', 'cdf:ruleid','cdf:ruleseverity','cdf:title','cdf:description','cdf:result']]
groups = doc.selects('cdf:Group') # Get all groups
for group in groups:
  rule = group.select('cdf:Rule') # Get the rule
  # Get result by rule id
  result = doc.select('cdf:TestResult').getElement('cdf:rule-result',attr='idref',value=rule.id).select('cdf:result>text()')
  rows.append([group.id,rule.id,rule.severity,rule.select('cdf:title').text,rule.select('cdf:description').text,result])

utils.save2csv('./test.csv',rows) # Save to csv file

这里有更多示例：https://github.com/yiyedata/simplified-scrapy-demo/tree/master/doc_examples