带有无效 html 文档的 beautifulsoup

Question

我正在尝试解析文档http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/ecofin/5923en8.htm。 我想提取Commission: 之前的所有内容。

（我需要 Beautifulsoup，因为第二步是提取国家和人名）

如果我这样做：

import urllib
import re
from bs4 import BeautifulSoup
url="http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/ecofin/5923en8.htm"
soup=BeautifulSoup(urllib.urlopen(url))
print soup.find_all(text=re.compile("Commission"))

我得到的唯一结果是：

[u'The Governments of the Member States and the European Commission were represented as follows:']

这是单词的第一次出现，但不是我要查找的行。我认为这是因为该文件无效，但不确定。如果我查看源代码：

<B><U><P>Commission</B></U>:</P>

但如果我打印soup，我可以看到文本，标签重新排序：

<u><b>Commission</b></u>

我怎样才能得到这个元素"Commission:"？

我正在使用 python 2.7 和 Beautifoulsoup 4.3.2。

---

编辑：已解决！

按照alecxe的建议，我换了一行：

soup=BeautifulSoup(urllib.urlopen(url))

与

BeautifulSoup(urllib.urlopen(url), 'html.parser')

它现在可以工作了:)。 谢谢大家。

---

编辑：类似问题

我用相同的解决方案发现了类似的问题：

Beautiful Soup 4 find_all don't find links that Beautiful Soup 3 finds

Beautiful Soup findAll doen't find them all

Answer 1

如果您希望标签之前的所有内容都带有“佣金：”值。你可以在没有 beatifulsoup 的情况下做到这一点......只需将其视为字符串变量并搜索正确的关键字并删除字符串的其余部分。

但是当我运行你的代码时，我得到以下信息：

[u'The Governments of the Member States and the European Commission were represe
nted as follows:', u'Commission', u'The Council held an orientation debate on ke
y economic policy issues with a view to giving guidance to the Commission on the
 questions Ministers wish to be addressed in the broad economic policy guideline
s 1998/99 for which the Commission will present its recommandation later in the
Spring. It was noted that the forthcoming guidelines are of particular importanc
e given the start of stage 3 of EMU.', u'The debate was based on an assessment o
f the economic situation and outlook in the Community carried out by the Commiss
ion and the Economic Policy and Monetary Committees.', u"The Council held an ori
entation debate on the Commission's Communication setting out a possible Communi
ty framework allowing Member States to experiment with reduced VAT rates for lab
our-intensive services in order to boost employment in small businesses without
distorting international competition. ", u'This Communication was tabled by the
Commission as a follow-up to the Employment European Council of last November in
 Luxembourg, which concluded that, in order to make the taxation system more emp
loyment-friendly, "Member States will examine, without obligation, the advisabil
ity of reducing the rate of VAT on labour-intensive services not exposed to cros
s-border competition".', u"In conclusion, the Council invited Coreper to examine
 the technical questions arising from today's debate and to report back to it wi
th a view to deciding on a possible request to the Commission to submit a propos
al in this area. ", u"This technical examination should be carried out, taking i
nto account the criteria indicated in the Commission's Communication for a reduc
ed VAT rate, on the following questions :", u'An initial trial period running un
til the year 2002 should identify the best method for allocating FISIM. At the e
nd of this period, the Commission will assess the results of the trial period an
d decide, by means of a comitology procedure, on the final methodology to be app
lied. However, a unanimous decision by the Council would be needed in order to u
se the new methodology in budgetary calculations on other Community policies and
 notably concerning "own resources".']

Answer 2

遍历p元素并在找到以Commission开头的文本时停止：

import urllib
from bs4 import BeautifulSoup

url="http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/ecofin/5923en8.htm"
soup=BeautifulSoup(urllib.urlopen(url))

for item in soup.find_all('p'):
    if item.text.startswith('Commission'):
        break
    else:
        print item.text

它打印Commission之前的所有内容：

The Governments of the Member States and the European Commission were represented as follows:
Belgium:
...
Ms Helen LIDDELL            Economic Secretary to the Treasury
* * *