使用 Beautiful Soup 在 Python 中解析网站

Question

鉴于此url（具有给定的“包裹 ID”），任何人都可以弄清楚如何从“详细信息”部分中提取“地址”和“所有者名称 1”吗？

我尝试了下面的代码，但是“soup”不包含“ADDRESS”和“Owner Name 1”信息，然后我会从“soup”中解析出来。

import requests
from bs4 import BeautifulSoup

# website
sitemap = 'https://www.mapsonline.net/chathamma/?ts_cama_id=12E-23G-A15'
# content of website
sitemap_content = requests.get(sitemap).content
# parsing website
soup = BeautifulSoup(sitemap_content, 'html.parser')
print(soup)

感谢您的帮助！！最佳答案获得虚拟高五（当然还有绿色检查）！

Answer 1

数据是通过 Ajax 从外部 URL 加载的。您可以使用此示例如何使用requests/json 模块：

import re
import json
import requests


url = 'https://www.mapsonline.net/chathamma/?ts_cama_id=12E-23G-A15#x=-7789868.005581,5113564.393067,-7788741.45516,5113891.937676'
api_url = 'https://www.mapsonline.net/peopleforms/mo4/mo4_server.php?request=search_triple&sid=undefined&search_field_0=&search_field_1={search_field}&addr_street=&addr_num=&exact=yes&st_id={search_id}&results=limited'
api_url2 = 'https://www.mapsonline.net/peopleforms/mo4/mo4_server.php?request=identify_multi&results=long&sitelayers={sitelayers}&layers={layers}&id={record_id}&site_id={site_id}'

html_data = requests.get(url).text
data = json.loads(re.search(r'MO4\.site_tools = (.*);', html_data).group(1))

for s in data:
    if s['name'] == 'search_triple':
        break

for s2 in data:
    if s2['name'] == 'quicksearch_abutters_search':
        break

search_field = re.search(r'ts_cama_id=(.*?)#', url).group(1)
data = requests.get(api_url.format(search_field=search_field, search_id=s['id'])).json()
# uncomment this to print all data:
# print(json.dumps(data, indent=4))

print(data['results'][0]['results'][0]['values'])

sitelayers = s2['sitetooloptions']['linked_parcel_layer']
site_id = s2['site_id']
layers = s2['sitetooloptions']['abutters_layer']
record_id = data['results'][0]['results'][0]['record_id']
data = requests.get(api_url2.format(sitelayers=sitelayers, layers=layers, record_id=record_id, site_id=site_id)).json()
for h, v in zip(data['results'][0]['headers'], data['results'][0]['results'][0]['values']):
    print('{:<30} {}'.format(h, v))

打印：

['12E-23G-A15', '103 ABSEGAMI RUN', 'MISSY ABBEY LLC']
Parcel ID                      12E-23G-A15
LOC_ID                         F_1073396_2713532
Key                            1131
GISid                          F_1073396_2713532
Map                            12E
Block                          23G
Parcel                         A15
Extension                      0
PropertyType                   R
LocNo                          103
LOCAlpha                       __EMPTY__VALUE__
LOCST                          ABSEGAMI RUN
Owner Name 1                   MISSY ABBEY LLC
Owner Name 2                   __EMPTY__VALUE__
Mailing Addr 1                 PO BOX 1348
Mailing Addr 2                 __EMPTY__VALUE__
Mailing City                   W CHATHAM
Mailing State                  MA
Mailing Zip                    02669-1348
Mailing Country                __EMPTY__VALUE__
StateClass                     1010
StateClassDesc                 SINGLE FAMILY
PrimaryPCTBldg                 100.00
PrimaryPCTLand                 100.00
Deed                           01-Oct-2012 00:00:00
Book                           N/A
Page                           N/A
Certificate                    198342
DeedDateLast                   01-Oct-2012 00:00:00
SaleTypeLast                   R
SalePriceLast                  1200000.0000
BookLast                       N/A
PageLast                       N/A
CertificateLast                198342
DeedDatePrev                   29-Jun-2012 00:00:00
SaleTypePrev                   A
SalePricePrev                  1.0000
BookPrev                       N/A
PagePrev                       N/A
CertificatePrev                197529
BillingSF                      21307
Note                           BORDERING COMMERCIAL
ZoneDesc                       R40
lr_address                     __EMPTY__VALUE__
lr_bldg                        __EMPTY__VALUE__
lr_sales                       __EMPTY__VALUE__
lr_land                        __EMPTY__VALUE__
lr_detach                      __EMPTY__VALUE__
TotalTaxableValue_money        $1,355,500.00
TotalApprValue_money           $1,355,500.00
TotalLandValue_money           $800,100.00
TotalAllBldgsValue_money       $555,400.00
TotalDetachedValue_money       $0.00
PYTotalTaxableValue_money      $1,310,400.00
PYTotalApprValue_money         $1,310,400.00
PYTotalLandValue_money         $748,000.00
PYTotalAllBldgsValue_money     $562,400.00
PYTotalDetachedValue_money     $0.00
ADDRESS                        103 ABSEGAMI RUN
ch_Key                         1131
BldgNumber                     1
_12_baths_data                 1
_12_baths_desc                 __EMPTY__VALUE__
pct_air_cond_data              __EMPTY__VALUE__
pct_air_cond_desc              __EMPTY__VALUE__
baths_data                     A
baths_desc                     AVERAGE
bedrooms_data                  3
bedrooms_desc                  __EMPTY__VALUE__
cdnapp_data                    __EMPTY__VALUE__
cdnapp_desc                    __EMPTY__VALUE__
complex_data                   __EMPTY__VALUE__
complex_desc                   __EMPTY__VALUE__
condo_model_data               __EMPTY__VALUE__
condo_model_desc               __EMPTY__VALUE__
elect_data                     A
elect_desc                     AVERAGE
ext_cover_data                 __EMPTY__VALUE__
ext_cover_desc                 __EMPTY__VALUE__
exterior_data                  A
exterior_desc                  AVERAGE
fixtures_data                  14
fixtures_desc                  __EMPTY__VALUE__
floor_cover_data               16
floor_cover_desc               HARDWD/OTHER
floor_level_data               __EMPTY__VALUE__
floor_level_desc               __EMPTY__VALUE__
foundation_data                4
foundation_desc                FLR & WALL
fuel_source_data               1
fuel_source_desc               OIL
full_baths_data                3
full_baths_desc                __EMPTY__VALUE__
heat_data                      A
heat_desc                      AVERAGE
pct_heated_data                __EMPTY__VALUE__
pct_heated_desc                __EMPTY__VALUE__
heatingcool_data               11
heatingcool_desc               HT-WT/COOL AIR
hvac_data                      __EMPTY__VALUE__
hvac_desc                      __EMPTY__VALUE__
interior_data                  G
interior_desc                  GOOD
int_finish_data                __EMPTY__VALUE__
int_finish_desc                __EMPTY__VALUE__
kitchen_data                   G
kitchen_desc                   GOOD
neighborhood_data              __EMPTY__VALUE__
neighborhood_desc              __EMPTY__VALUE__
roof_cover_data                1
roof_cover_desc                ASPH/CMP SHIN
roof_shape_data                1
roof_shape_desc                GABLE
rooms_data                     0
rooms_desc                     __EMPTY__VALUE__
pct_sprinkler_data             __EMPTY__VALUE__
pct_sprinkler_desc             __EMPTY__VALUE__
stories_data                   1.75
stories_desc                   __EMPTY__VALUE__
unit_model_data                __EMPTY__VALUE__
unit_model_desc                __EMPTY__VALUE__
units_data                     1
units_desc                     __EMPTY__VALUE__
viewloc_data                   __EMPTY__VALUE__
viewloc_desc                   __EMPTY__VALUE__
YearBuilt                      1983
WallHeight                     __EMPTY__VALUE__
Quality                        +
Style                          CAPE
Grantor                        103 ABSEGAMI RUN REALTY TRUST
Grantee                        MISSY ABBEY LLC
BookPageLast                   N/A / N/A
landsf                         21307
Acres                          0.49