【问题标题】:How to crawl events from calendar with Python and Beautiful Soup如何使用 Python 和 Beautiful Soup 从日历中抓取事件
【发布时间】:2022-01-21 22:57:42
【问题描述】:

我想从本网站的日历中提取所有活动的链接,然后抓取每个活动的信息。

这是网址: https://thalheim.ch/index.php/aktuell/veranstaltungen

这是我写的代码:

from bs4 import BeautifulSoup
import requests

def get_website_news_links_thalheimCh():
        
    url = 'https://thalheim.ch/index.php/aktuell/veranstaltungen'
    
    response = requests.get(url, allow_redirects=True)
    print("Response for", url, response)

    soup = BeautifulSoup(response.content, 'html.parser')

    all_links = soup.select('table tbody tr div.fc-daygrid-day-events a')
    print(all_links)

result = get_website_news_links_thalheimCh()

对于all_links 变量,我总是得到 []。 我想我做错了什么。 \我查看了在“网络”选项卡上找到的链接,但找不到任何可以帮助我的链接。

【问题讨论】:

  • 注意 首先,总是看看你的汤——这就是真相。内容总是与开发工具中的视图略有不同。 内容是动态提供的,因此可以使用 api 或 selenium 来获取结果。

标签: python web-scraping beautifulsoup


【解决方案1】:

所有数据都来自POST 请求。

您可以通过以下方式获得:

import requests
from bs4 import BeautifulSoup

query_url = "https://thalheim.ch/index.php?option=com_dpcalendar&view=events&format=raw&limit=0&Itemid=504&ids=g-9&date-start=2021-11-29T00:00:00&date-end=2022-01-10T00:00:00"
page = requests.post(query_url).json()

for item in page["data"]:
    location = item["location"][0]["location"] if item["location"] else "N/A"
    desc = BeautifulSoup(item["description"], "lxml")
    print(f"Event: {item['title']} | Location: {location}")
    print(desc.getText(strip=True, separator=" "))
    print("-" * 120)

输出:

Event: Gemeindeversammlung | Location: Thalheim, Aargau 5112
09.12.2021 19:30 - 21:30 [Veranstaltungskalender Thalheim an der Thur] Gemeindeversammlung
------------------------------------------------------------------------------------------------------------------------
Event: Schulsilvester | Location: N/A
17.12.2021 [Veranstaltungskalender Thalheim an der Thur] Schulsilvester
------------------------------------------------------------------------------------------------------------------------
Event: Weihnachtsferien Schule | Location: N/A
20.12.2021 - 31.12.2021 [Veranstaltungskalender Thalheim an der Thur] Weihnachtsferien Schule
------------------------------------------------------------------------------------------------------------------------
Event: TV Thalheim - Abendunterhaltung 2022 (Frühvorstellung) | Location: 
29.12.2021 [Veranstaltungskalender Thalheim an der Thur] TV Thalheim - Abendunterhaltung 2022 (Frühvorstellung)
------------------------------------------------------------------------------------------------------------------------
Event: TV Thalheim - Abendunterhaltung 2022 | Location: 
31.12.2021 [Veranstaltungskalender Thalheim an der Thur] TV Thalheim - Abendunterhaltung 2022
------------------------------------------------------------------------------------------------------------------------
Event: TV Thalheim - Abendunterhaltung 2022 | Location: 
01.01.2022 [Veranstaltungskalender Thalheim an der Thur] TV Thalheim - Abendunterhaltung 2022
------------------------------------------------------------------------------------------------------------------------

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 2013-01-09
    • 2014-01-10
    • 2020-04-22
    • 2017-08-14
    • 1970-01-01
    • 2022-01-07
    • 1970-01-01
    相关资源
    最近更新 更多