使用 ElementTree 和 BeautifulSoup 解析文件：有没有办法按标签级别数解析文件？

Question

我有thisxml文件，我基本上想把所有的信息都记录到字典里。

我写了这段代码：

import requests
import xml.etree.ElementTree as ET
import urllib2
import glob
import pprint
from bs4 import BeautifulSoup

#get the XML file
#response = requests.get('https://www.drugbank.ca/drugs/DB01048.xml')
#with open('output.txt', 'w') as input:
#          input.write(response.content)


#set up lists etc
set_of_files = glob.glob('output*txt')
val = lambda x: "{http://www.drugbank.ca}" + str(x)
key_list = ['drugbank-id','name','description','cas-number','unii','average-mass','monoisotopic-mass','state','indication','pharmacodynamics','mechanism-of-action','toxicity','metabolism','absorption','half-life','protein-binding','route-of-elimination','volume-of-distribution','fda-label','msds']
key_dict = {}
method1 = ['groups','synonyms','patent']
method3_inputs = [('dosages','dosage'),('salts','salt'),('products','product'),('mixtures','mixture'),('packagers','packager'),('categories','category'),('atc-codes','atc-code'),('pdb-entries','pdb-entry'),('food-interactions','food-interaction'),('drug-interactions','drug-interaction'),('properties','property'),('external-identifiers','external-identifier'),('external-links','external-link'),('reactions','reaction')]
list_to_run_thru = ['description','direct-parent','kingdom','superclass','class','subclass']
alternative_parents = []
http_add = '{http://www.drugbank.ca}'
substituents = []
ap_sub = lambda x:'{http://www.drugbank.ca}'+ x

def method2(list2_name,list3_name,list4_name):
     temp_list = []
     for i in subnode:
          if i.tag == list2_name:
               for a in i:
                    if a.tag == list3_name:
                         for u in a:
                              if u.tag == list4_name:
                                   temp_list.append(u.text)
     return temp_list

def method3(list1_name):
     list_of_tuples = []
     for i in subnode:
          if i.tag == list1_name:
               temp_list = []
               for a in i:
                    temp_list.append(a.text)
                    list_of_tuples.append(temp_list)
     return list_of_tuples

new_method3 = []
for i in method3_inputs:
     new_k = http_add + i[0]
     new_v = http_add + i[1]
     new_method3.append((new_k,new_v))

for each_file in set_of_files:
     tree = ET.parse(each_file)
     root = tree.getroot()
     for i in key_list:
          for child in root.getchildren():
               if i not in key_dict:
                    key_dict[i] = [child.find(val(i)).text.encode('utf-8')]
               else:
                    key_dict[i].append(child.find(val(i)).text.encode('utf-8'))

     for node in root:
          for subnode in node:
               if subnode.tag == '{http://www.drugbank.ca}general-references':
                    if 'pubmed-id' not in key_dict:
                         key_dict['pubmed-id'] = method2('{http://www.drugbank.ca}articles','{http://www.drugbank.ca}article','{http://www.drugbank.ca}pubmed-id')
                    else:
                         key_dict['pubmed-id'].append(method2('{http://www.drugbank.ca}articles','{http://www.drugbank.ca}article','{http://www.drugbank.ca}pubmed-id'))

               for a,b in zip(method3_inputs,new_method3):
                    if subnode.tag == b[0]:
                         if a[0] not in key_dict:
                              key_dict[a[0]] = method3(b[1])
                         else:
                              key_dict[method3_inputs].append(method3(b[1]))

               if subnode.tag == '{http://www.drugbank.ca}classification': 
                    for each_item in list_to_run_thru:
                         for i in subnode:
                               if i.tag == ap_sub(each_item):
                                   if i.tag == '{http://www.drugbank.ca}alternative-parent':
                                         alternative_parents.append(i.text)  
                                   if i.tag == '{http://www.drugbank.ca}substituent':
                                         substituents.append(i.text)  

               for i in method1:
                    if subnode.tag == '{http://www.drugbank.ca}' + i:
                         for n in subnode:
                              if i not in key_dict:
                                   key_dict[i] = [n.text]
                              elif i in key_dict:
                                   key_dict[i].append(n.text)

               if subnode.tag == '{http://www.drugbank.ca}pathways':
                    if 'pathways'not in key_dict:
                         key_dict['pathways'] = method2('{http://www.drugbank.ca}pathways','{http://www.drugbank.ca}pathway','{http://www.drugbank.ca}drug')
                    else:
                         key_dict['pathways'].append(method2('{http://www.drugbank.ca}pathways','{http://www.drugbank.ca}pathway','{http://www.drugbank.ca}drug'))


key_dict['alternative_parents'] = alternative_parents
key_dict['substituent'] = substituents


html = requests.get('https://www.drugbank.ca/drugs/DB01048').text
soup = BeautifulSoup(html, 'html.parser')
div_targets = soup.find('div', class_='bond-list-container targets')
targets = div_targets.find_all('div', class_='bond card')

for target in targets:
    k = []
    v = []
    for property in target.find_all('dt'):
        k.append(property.get_text())
    for property in target.find_all('dd'):
        v.append(property.get_text())
    key_dict[target.find('strong').get_text()] = dict(zip(k, v))

print key_dict.keys()

代码在描述良好的 XML 文件上运行。但是有一些问题我想看看是否有人可以改进：

1. 您可以清楚地看到我必须对其进行很多硬编码。例如，有些标签是一层深的（例如drugbank-id），有些是两层深的（例如groups -> group），有些是三层深的（例如products, product,name）等等。

你可以看到我已经为两层和三层深度编写了一个函数（这些函数称为method3和method2），然后我在属于每个函数的列表中进行了硬编码。

我想知道谁能向我展示比这更好/更清洁/更高效的代码。我的想法是我有一个 ID 列表（例如描述、产品），并且该功能以某种方式理解“这是一个 3 层标签（例如产品 -> 产品 -> 名称），所以我不必费力所有 3 层标签、所有 2 层标签等中的代码。例如有一个 if 语句“如果标签是 3 层深...执行此功能..如果标签是 2 层深.. ..这样做'。

1. 免责声明：我知道此方法同时使用 BeautifulSoup 和 ET 进行解析，因为我被困在某个部分并获得了帮助 here。不幸的是，我需要掌握一个才能继续前进，而且我发现 HTML 版本比 XML 更令人困惑，所以这就是脚本从 XML 和 HTML 角度解析文件的原因。

   李>

2. 有没有人有任何关于如何使这个脚本更干净的通用 cmets？这个想法是获取每个“顶级”标签（即紧接在“药物类型”标签下方的所有内容），然后基本上将所有这些信息下拉到字典/列表/任何内容中，然后每当我想搜索一些东西，我可以在字典中搜索“顶级”单词，它有我可以阅读的那个标签的子数据（并且级别组织得当，例如，如果标签下面只有一个级别，只是一个字符串很好，但是如果标签下面有很多标签，可能会返回字典/列表/元组/更合适的东西）。

编辑： 根据下面的帮助，我下载了 drugbank.xsd 并运行了这个命令：

pyxbgen -m DB01048 -u drugbank.xsd --no-validate-changes

然后是这个脚本：

from __future__ import print_function
import DB01048

xml = open('DB01048.xml').read()
d_01048 = DB01048.CreateFromDocument(xml)

#print(d_01048.drug[0].state)
#print(d_01048.drug[0].name)
#print(d_01048.drug[0].general_references.articles.article[0].pubmed_id)

和错误：

Traceback (most recent call last):
  File "parse_drug_db.py", line 5, in <module>
    d_01048 = DB01048.CreateFromDocument(xml)
  File "/home/drugs/DB01048.py", line 65, in CreateFromDocument
    saxer.parse(io.BytesIO(xmld))
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/saxer.py", line 388, in endElementNS
    binding_object = this_state.endBindingElement()
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/saxer.py", line 245, in endBindingElement
    return self.__bindingInstance._postDOMValidate()
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/basis.py", line 2652, in _postDOMValidate
    self._validateAttributes()
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/basis.py", line 2246, in _validateAttributes
    au.validate(self)
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/content.py", line 251, in validate
    raise pyxb.MissingAttributeError(type(ctd_instance), self.__name, ctd_instance)
pyxb.exceptions_.MissingAttributeError: Instance of <class 'DB01048.drugbank_type'> lacks required attribute exported-on

Answer 1

更 Python 的方法可能是使用 PyXB 包生成一个从 xml 文件解组的 Python 对象。

• 安装 PyXB 包

pip install PyXB

• 从下载的 xsd 创建 python 模块。将创建 DB01048.py

pyxbgen -m DB01048 -u drugbank-plus.xsd --no-validate-changes

• 将生成的模块用作

  from __future__ import print_function
  import DB01048
  
  xml = open('DB01048.xml').read()
  d_01048 = DB01048.CreateFromDocument(xml)
  
  print(d_01048.drug[0].state)
  print(d_01048.drug[0].name)
  print(d_01048.drug[0].general_references.articles.article[0].pubmed_id)
  

solid Abacavir 17356469

如果出现与某些exported-on 属性相关的 MissingAttribute 错误，请在 DB01048.py 上的第一个 Class 之前添加以下行

pyxb.RequireValidWhenParsing(True) pyxb.RequireValidWhenParsing(False)