提取 CSS 类中的文本

Question

尝试将数据从网页提取到表格。例如

Block Number XXX
Building Name YYY
Street Name zzz
Pin Code 123456789

我正在尝试使用此代码以表格形式获取公司的所有详细信息...

html_doc='https://s3.amazonaws.com/todel162/test.html'

from urllib.request import urlopen
from bs4 import BeautifulSoup

soup = BeautifulSoup(urlopen(html_doc), 'html.parser')

mydivs = soup.findAll("div", {"class": "row"})

for i in mydivs:
    for x in i.findAll("div", {"class": "col-md-3 col-sm-3"}):
        print (x)
        print ('new x')

但我不确定如何提取特定 CSS 类的 HTML 标记中的文本。

Answer 1

您需要的所有项目都在<div class="col-md-3 col-sm-3"> 标记内。并且，除了General Information 之外的所有项目都遵循特定的格式：第一个div 标签是标签，第二个是相应的文本。因此，您可以简单地找到它们，然后将它们交替打印为标签及其文本。

因此，除了 一般信息 之外的所有内容都可以在 <div id="fldFirm"> 标记中找到。由于 General Information 不遵循上述格式，因此您必须先单独抓取它。您可以使用limit=2 获取前两个标签。

import requests
from bs4 import BeautifulSoup

r = requests.get('https://s3.amazonaws.com/todel162/test.html')
soup = BeautifulSoup(r.text, 'lxml')

gen_info = [x.text.strip() for x in soup.find('div', id='divInfoType').find_all('div', class_='col-md-3 col-sm-3', limit=2)]
print('{}: {}'.format(gen_info[0], gen_info[1]))

all_info = [x.text.strip() for x in soup.find('div', id='fldFirm').find_all('div', class_='col-md-3 col-sm-3')]

for i in range(0, len(all_info), 2):
    print('{}: {}'.format(all_info[i], all_info[i+1]))

输出：

Information Type: Other Than Individual
Name: Bellissimo Developers Thane Private Limited (Lodha Developers Pvt Ltd)
Organization Type: Company
Description For Other Type Organization: NA
Do you have any Past Experience ?: No
Block Number: 412, Floor 4 17G
Building Name: Vardhaman Chamber
Street Name: Cawasji Patel Road
Locality: Fort
Land mark: Horniman Circle
State/UT: MAHARASHTRA
Division: Konkan
District: Mumbai City
Taluka: Mumbai City
Village: Mumbai City
Pin Code: 400001
Office Number: 02261334263
Website URL: www.lodhagroup.com

编辑：

要创建数据框，您可以使用：

gen_info = [x.text.strip() for x in soup.find('div', id='divInfoType').find_all('div', class_='col-md-3 col-sm-3', limit=2)]
all_info = gen_info + [x.text.strip() for x in soup.find('div', id='fldFirm').find_all('div', class_='col-md-3 col-sm-3')]
df = pd.DataFrame({'A': all_info[0::2], 'B': all_info[1::2]})

Answer 2

您只想要地址信息还是整个列表？这是整个清单的代码。

html_doc='https://s3.amazonaws.com/todel162/test.html'

from urllib.request import urlopen
from bs4 import BeautifulSoup
import re

soup = BeautifulSoup(urlopen(html_doc), 'html.parser')

mydivs = soup.findAll("div", {"class": "form-group"})

for x in mydivs:
    print (re.sub( '\s+', ' ', x.text ).strip())
    print ('#############')

输出：

Information Type Other Than Individual
#############
Name Bellissimo Developers Thane Private Limited (Lodha Developers Pvt Ltd)
#############
Organization Type Company
#############
Do you have any Past Experience ? No
#############
Block Number 412, Floor 4 17G Building Name Vardhaman Chamber
#############
Street Name Cawasji Patel Road Locality Fort
#############
Land mark Horniman Circle State/UT MAHARASHTRA
#############
Division Konkan District Mumbai City
#############
Taluka Mumbai City Village Mumbai City
#############
Pin Code 400001
#############
Office Number 02261334263
#############
Website URL www.lodhagroup.com
#############

如果你想把它放在一个表格中（非常粗略的例子）：

...
mydivs = soup.findAll("div", {"class": "form-group"})
print ("<table>")
for x in mydivs:
    label = x.find('label')
    if label is not None:
        print ("<tr><td>" + label.text + "</td><td> ")
        label.extract()
    else:
        print ("<tr><td></td><td> ")
    print (re.sub( '\s+', ' ', x.text ).strip())
    print ('</td></tr>')
print ("</table>")

输出：

<table>
<tr><td>Information Type</td><td> 
Other Than Individual
</td></tr>
<tr><td>Name</td><td> 
Bellissimo Developers Thane Private Limited (Lodha Developers Pvt Ltd)
</td></tr>
<tr><td>Organization Type</td><td> 
Company
</td></tr>
<tr><td></td><td> 
Do you have any Past Experience ? No
</td></tr>
<tr><td>Block Number</td><td> 
412, Floor 4 17G Building Name Vardhaman Chamber
</td></tr>
<tr><td>Street Name</td><td> 
Cawasji Patel Road Locality Fort
</td></tr>
<tr><td>Land mark</td><td> 
Horniman Circle State/UT MAHARASHTRA
</td></tr>
<tr><td>Division</td><td> 
Konkan District Mumbai City
</td></tr>
<tr><td>Taluka</td><td> 
Mumbai City Village Mumbai City
</td></tr>
<tr><td>Pin Code</td><td> 
400001
</td></tr>
<tr><td>Office Number</td><td> 
02261334263
</td></tr>
<tr><td>Website URL</td><td> 
www.lodhagroup.com
</td></tr>
</table>