在没有 API 的情况下使用 Python 和 bs4 抓取 Google 地图

Question

我正在尝试使用 python 和 BeautifulSoup 从谷歌地图获取数据。例如城市的药店。我将获得位置数据（经纬度）、药房名称（即 MDC 药房）、药房分数（3.2）、rewiews 数（10）、带邮政编码的地址和药房电话号码。 我已经尝试过 python 和 BeautifulSoup 但我被卡住了，因为我不知道如何提取数据。类方法不起作用。当我对结果进行美化和打印时，我已经看到了所有数据。那么如何为熊猫数据框清理它们呢？我需要更多代码来获取干净的数据并将它们添加到列表或 df 中。也是classobject变成noobject类型。这是我的代码：

import requests
from bs4 import BeautifulSoup
r=requests.get("https://www.google.com.tr/maps/search/eczane/@37.4809437,36.7749346,57378m/data=!3m1!1e3")
soup= BeautifulSoup(r.content,"lxml")
a=soup.prettify()
l=soup.find("div",{"class":"mapsConsumerUiSubviewSectionGm2Placeresultcontainer__result-container mapsConsumerUiSubviewSectionGm2Placeresultcontainer__one-action mapsConsumerUiSubviewSectionGm2Placeresultcontainer__wide-margin"})
print(a)

Printresult.jpg

我有这个结果，我需要从这里（上图）提取数据。

我想要这样的结果（如下）。谢谢...

wanted resul(it is just a sample)

Answer 1

可能还值得研究像 SerpApi 这样的第三方解决方案。这是一个免费试用的付费 API。

示例 python 代码（其他库中也有）：

from serpapi import GoogleSearch

params = {
  "api_key": "secret_api_key",
  "engine": "google_maps",
  "q": "eczane",
  "google_domain": "google.com",
  "hl": "en",
  "ll": "@37.5393407,36.707705,11z",
  "type": "search"
}

search = GoogleSearch(params)
results = search.get_dict()

JSON 输出示例：

"local_results": [
  {
    "position": 1,
    "title": "Ocak Eczanesi",
    "place_id": "ChIJcRipbonnLRUR4DG-UuCnB2I",
    "data_id": "0x152de7896ea91871:0x6207a7e052be31e0",
    "data_cid": "7063799122456621536",
    "reviews_link": "https://serpapi.com/search.json?data_id=0x152de7896ea91871%3A0x6207a7e052be31e0&engine=google_maps_reviews&hl=en",
    "photos_link": "https://serpapi.com/search.json?data_id=0x152de7896ea91871%3A0x6207a7e052be31e0&engine=google_maps_photos&hl=en",
    "gps_coordinates": {
      "latitude": 37.5775156,
      "longitude": 36.957789399999996
    },
    "place_id_search": "https://serpapi.com/search.json?data=%214m5%213m4%211s0x152de7896ea91871%3A0x6207a7e052be31e0%218m2%213d37.5775156%214d36.957789399999996&engine=google_maps&google_domain=google.com&hl=en&type=place",
    "rating": 3.5,
    "reviews": 8,
    "type": "Drug store",
    "address": "Kanuni Mh. Milcan Cd. Pk:46100 Merkez, 46100 Dulkadiroğlu/Kahramanmaraş, Turkey",
    "open_state": "Closes soon ⋅ 6PM ⋅ Opens 8:30AM Fri",
    "hours": "Closing soon: 6:00 PM",
    "phone": "+90 344 231 68 00",
    "website": "https://kahramanmaras.bel.tr/nobetci-eczaneler",
    "thumbnail": "https://lh5.googleusercontent.com/p/AF1QipN5CQRdoKc_BdCgSDiEdi0nEkk1X_VUy1PP4wN3=w93-h92-k-no"
  },
  {
    "position": 2,
    "title": "Nobetci eczane",
    "place_id": "ChIJP4eh2WndLRURD6IcnOov0dA",
    "data_id": "0x152ddd69d9a1873f:0xd0d12fea9c1ca20f",
    "data_cid": "15046860514709512719",
    "reviews_link": "https://serpapi.com/search.json?data_id=0x152ddd69d9a1873f%3A0xd0d12fea9c1ca20f&engine=google_maps_reviews&hl=en",
    "photos_link": "https://serpapi.com/search.json?data_id=0x152ddd69d9a1873f%3A0xd0d12fea9c1ca20f&engine=google_maps_photos&hl=en",
    "gps_coordinates": {
      "latitude": 37.591462,
      "longitude": 36.8847051
    },
    "place_id_search": "https://serpapi.com/search.json?data=%214m5%213m4%211s0x152ddd69d9a1873f%3A0xd0d12fea9c1ca20f%218m2%213d37.591462%214d36.8847051&engine=google_maps&google_domain=google.com&hl=en&type=place",
    "rating": 3.3,
    "reviews": 12,
    "type": "Pharmacy",
    "address": "Mimar Sinan, 48007. Sk. No:19, 46050 Kahramanmaraş Merkez/Kahramanmaraş, Turkey",
    "open_state": "Open now",
    "thumbnail": "https://lh5.googleusercontent.com/p/AF1QipNznf-hC_y9KdijwUMqdO9YIcn7rbN8ZQpdIHK5=w163-h92-k-no"
  },
  ...
]

查看documentation了解更多详情。

免责声明：我在 SerpApi 工作。

Answer 2

你不需要硒。你甚至不需要 BeautifulSoup（事实上，它根本没有帮助）。这是获取页面、隔离初始化数据 JSON、对其进行解码并打印生成的 Python 结构的代码。

您需要打印出结构，然后开始数数以找到您想要的数据，但都在这里。

import requests
import json
from pprint import pprint
r=requests.get("https://www.google.com.tr/maps/search/eczane/@37.4809437,36.7749346,57378m/data=!3m1!1e3")

txt = r.text

find1 = "window.APP_INITIALIZATION_STATE="
find2 = ";window.APP"

i1 = txt.find(find1)
i2 = txt.find(find2, i1+1 )
js = txt[i1+len(find1):i2]
data = json.loads(js)
pprint(data)