从网站解析的数据最终是一个空数据（bs4、python、lxml）

Question

大家好 Stackoverflow，

我很难从网站解析信息 使用 Beautifulsoup 和 lxml。

我尝试从“https://www1.nyc.gov/events/events-filter.html#page-1”网站带入地址数据。

据我搜索，

它说我需要 1.通过网页的“检查”找到信息的具体类别。 2.需要写g_data = soup.find_all("div", {"class": "event-data-detail"})之类的代码

所以我写了如下代码。

import requests
from bs4 import BeautifulSoup

url = "https://www1.nyc.gov/events/events-filter.html#page-1"
r=requests.get("https://www1.nyc.gov/events/events-filter.html#page-1")

soup = BeautifulSoup(r.content)


links = soup.find_all("a")

g_data = soup.find_all("div", {"class": "event-data-detail"})

print(g_data)

它会显示错误消息

警告（来自警告模块）：文件 “C:/Users/jotna/Desktop/Portfolio/1.py”，第 7 行 soup = BeautifulSoup(r.content) UserWarning: 没有明确指定解析器，所以我使用了最好的 HTML 解析器 这个系统（“lxml”）。这通常不是问题，但如果你运行 这段代码在另一个系统上，或者在不同的虚拟环境中，它 可能使用不同的解析器并且行为不同。

导致此警告的代码位于文件的第 7 行 C:/Users/jotna/Desktop/Portfolio/1.py。为了摆脱这个警告， 将附加参数 'features="lxml"' 传递给 BeautifulSoup 构造函数。

所以我将代码固定如下。 （导致stackoverflow中的帖子建议在最后添加lxml代码）

import lxml
import requests
from bs4 import BeautifulSoup

url = "https://www1.nyc.gov/events/events-filter.html#page-1"
r=requests.get("https://www1.nyc.gov/events/events-filter.html#page-1")

soup = BeautifulSoup(r.content)


links = soup.find_all("a")

for link in links:
   if "http" in link.get("href"):
       print ("<a href='%s'>%s</a>" %(link.get("href"), link.text))

g_data = soup.find_all("div", {"span class": "address"})

print(g_data)

但是它只显示空括号 []

如何才能真正从网站上带地址数据？

为了您的信息，我也上传了网页源的屏幕截图。 enter image description here

Answer 1

使用他们的 json api 代替 bs4，见下面的代码。

import requests
count = 0
for i in range(185):
    count+=1
    link = 'https://www1.nyc.gov/calendar/api/json/search.htm?&sort=DATE&pageNumber='+str(count)
    req = requests.get(link)
    for i in req.json()['items']:
        address = (i['address'])
        print 'Address:', address

输出

Address: Mulberry Street, Little Italy, New York
Address: Various locations Citywide
Address:  SECOND AVENUE between EAST   32 STREET and EAST   33 STREET  Manhattan
Address:  FIRST AVENUE between EAST   92 STREET and EAST   93 STREET  Manhattan
Address:  CARROLL STREET between SMITH STREET and COURT STREET  Brooklyn
Address:  BROADWAY between WEST  114 STREET and WEST  116 STREET  Manhattan
Address:  CORTELYOU ROAD between RUGBY ROAD and ARGYLE ROAD  Brooklyn
Address:  QUEENS BOULEVARD between 70 AVENUE and 69 ROAD  Queens
Address:  79 STREET between NORTHERN BOULEVARD and 34 AVENUE  Queens
Address:  PRINCE STREET between MOTT STREET and MULBERRY STREET  Manhattan
Address:  BUSHWICK AVENUE between NOLL STREET and ARION PLACE  Brooklyn
Address: Alley Pond Park Adventure Center
Address: Atlantic Avenue between 4th Avenue and Hicks Street
Address: Alexander von Humboldt statue - Central Park West and 77th Street
Address:  SEVENTH AVENUE between WEST  110 STREET and WEST  111 STREET  Manhattan
Address: Wave Hill House - West 249th Street and Independence Avenue
Address: Broadway between Liberty Street and Rector Street
Address: Anibal Aviles Playground
Address: Myrtle Avenue between Fresh Pond Road and Wyckoff Avenue