使用 BeautifulSoup4 和 Python 从不一致的 HTML 页面中提取数据

Question

我正在尝试从此webpage 中提取数据，但由于页面的 HTML 格式不一致，我遇到了一些问题。我有一个 OGAP ID 列表，我想为我遍历的每个 OGAP ID 提取基因名称和任何文献信息 (PMID #)。感谢此处的其他问题和 BeautifulSoup 文档，我能够始终如一地获得每个 ID 的基因名称，但我在文献部分遇到了麻烦。下面是几个突出显示不一致之处的搜索词。

有效的 HTML 示例

搜索词：OG00131

<tr>
  <td colspan="4" bgcolor="#FBFFCC" class="STYLE28">Literature describing O-GlcNAcylation:
    <br>&nbsp;&nbsp;PMID: 
    <a href="http://www.ncbi.nlm.nih.gov/pubmed/20068230">20068230</a>
    [CAD, ETD MS/MS]; <br>
    <br>
  </td>
</tr>

不起作用的 HTML 示例

搜索词：OG00020

<td align="top" bgcolor="#FBFFCC">
  <div class="STYLE28">Literature describing O-GlcNAcylation: </div>
  <div class="STYLE28">
    <div class="STYLE28">PMID:
      <a href="http://www.ncbi.nlm.nih.gov/pubmed/16408927?dopt=Citation">16408927</a>
      [Azide-tag, nano-HPLC/tandem MS] 
    </div>
    <br>
    Site has not yet been determined. Use 
    <a href="parser2.cgi?ACLY_HUMAN" target="_blank">OGlcNAcScan</a>
    to predict the O-GlcNAc site. </div>
</td>

这是我目前的代码

import urllib2
from bs4 import BeautifulSoup

#define list of genes
   
#initialize variables
gene_list = []
literature = []
# Test list
gene_listID = ["OG00894", "OG00980", "OG00769", "OG00834","OG00852", "OG00131","OG00020"]


for i in range(len(gene_listID)):
    print gene_listID[i]
    # Specifies URL, uses the "%" to sub in different ogapIDs based on a list provided
    dbOGAP = "https://wangj27.u.hpc.mssm.edu/cgi-bin/DB_tb.cgi?textfield=%s&select=Any" % gene_listID[i]
    # Opens the URL as a page
    page = urllib2.urlopen(dbOGAP)
    # Reads the page and parses it through "lxml" format
    soup = BeautifulSoup(page, "lxml")
    
    gene_name = soup.find("td", text="Gene Name").find_next_sibling("td").text
    print gene_name[1:]
    gene_list.append(gene_name[1:])
    
    # PubMed IDs are located near the <td> tag with the term "Data and Source"
    pmid = soup.find("span", text="Data and Source")

    # Based on inspection of the website, need to move up to the parent <td> tag
    pmid_p = pmid.parent

    # Then we move to the next <td> tag, denoted as sibling (since they share parent <tr> (Table row) tag)
    pmid_s = pmid_p.next_sibling
    #for child in pmid_s.descendants:
     #   print child
    # Now we search down the tree to find the next table data (<td>) tag
    pmid_c = pmid_s.find("td")
    temp_lit = []
    # Next we print the text of the data
    #print pmid_c.text
    if "No literature is available" in pmid_c.text:
        temp_lit.append("No literature is available")
        print "Not available"
    else:
    # and then print out a list of urls for each pubmed ID we have
        print "The following is available"
        for link in pmid_c.find_all('a'):
            # the <a> tag includes more than just the link address.
            # for each <a> tag found, print the address (href attribute) and extra bits
            # link.string provides the string that appears to be hyperlinked.
            # In this case, it is the pubmedID
            print link.string
            temp_lit.append("PMID: " + link.string + "  URL: " + link.get('href'))
    literature.append(temp_lit)
    print "\n"

因此，似乎元素就是将代码抛出循环的原因。有没有办法搜索带有文本“PMID”的任何元素并返回其后的文本（如果有 PMID 编号，则返回 url）？如果没有，我是否只想检查每个孩子，寻找我感兴趣的文本？

我使用的是 Python 2.7.10

Answer 1

import requests
from bs4 import BeautifulSoup
import re
gene_listID = ["OG00894", "OG00980", "OG00769", "OG00834","OG00852", "OG00131","OG00020"]
urls = ('https://wangj27.u.hpc.mssm.edu/cgi-bin/DB_tb.cgi?textfield={}&select=Any'.format(i) for i in gene_listID)

for url in urls: 
    r = requests.get(url)
    soup = BeautifulSoup(r.text, 'lxml')
    regex = re.compile(r'http://www.ncbi.nlm.nih.gov/pubmed/\d+')

    a_tag = soup.find('a', href=regex)
    has_pmid = 'PMID' in a_tag.previous_element

    if has_pmid :
        print(a_tag.text, a_tag.next_sibling, a_tag.get("href"))
    else:
        print("Not available")

出来：

18984734  [GalNAz-Biotin tagging, CAD MS/MS];  http://www.ncbi.nlm.nih.gov/pubmed/18984734
20068230  [CAD, ETD MS/MS];  http://www.ncbi.nlm.nih.gov/pubmed/20068230
20068230  [CAD, ETD MS/MS];  http://www.ncbi.nlm.nih.gov/pubmed/20068230
Not available
16408927  [Azide-tag, nano-HPLC/tandem MS];   http://www.ncbi.nlm.nih.gov/pubmed/16408927
Not available
16408927 [Azide-tag, nano-HPLC/tandem MS]  http://www.ncbi.nlm.nih.gov/pubmed/16408927?dopt=Citation

找到第一个匹配目标 url 的标签，它以数字结尾，然后检查它的前一个元素中是否有 'PMID'。 这个网页太不一致了，我尝试了很多次，希望这会有所帮助