如何将日期添加到仅针对多个实例显示一次的数据框中？答案

【问题标题】：How can I add dates to dataframe which only displays once for multiple instances?如何将日期添加到仅针对多个实例显示一次的数据框中？
【发布时间】：2021-12-31 15:19:09
【问题描述】：

我正在尝试抓取 website 以返回频道上正在发生的事件。由于日期的存储方式，我很难将日期添加到事件中 - 一次用于 dateSeparator 类下的当天所有事件。

我想我必须计算每个事件之间的所有事件，然后将日期应用于这些事件，问题是我不知道如何实现这一点。有人可以帮帮我吗？

我的代码

import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

class STG:
    url = "https://sport-tv-guide.live/"
    channel = (f"{url}/tv-guide-live/movistar-deportes")

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(f"{STG.channel}")
html = driver.page_source
soup  = BeautifulSoup(html, features="lxml")

teams = [x.text.strip() for x in soup.findAll('div',{'class':'col-inline'})]
times = [x.text.strip() for x in soup.findAll('div',{'class':'main time col-sm-2 hidden-xs'})]

df = pd.DataFrame(list(zip(times, teams)), columns = ['sport', 'teams'])

df[["sport", "time"]] = df.sport.str.split("\n", expand=True)
df[["teams", "comp"]] = df.teams.str.split("\n", expand=True)
df[["ht", "at"]] = df.teams.str.split(" - ", expand=True)

print(df)

driver.quit()

HTML 示例 - full link

<div class="listData">  
                    <div class="ajax-load text-center" style="display: none;"><p><img data-src="https://img.sport-tv-guide.live/images/loading.gif" height="40px" class="lazyload"></p></div>  
                <div class="dateSeparator">
                             <span class="date"> 1 Jan</span>
                             <div class="dateCenter">Tomorrow</div>
                           </div>
 <a class="article" href="/event/live-tennis-atp-cup-teams-serbia-norway/797348?id=3640101073016">
        <div class="row">
           <div class="main col-xs-2 hidden-sm hidden-md hidden-lg pl-0  text-center">
             <div class="font-size-10"><img data-src="https://img.sport-tv-guide.live/images/t8.png" class="w-40 lazyloaded" title="Tennis" src="https://img.sport-tv-guide.live/images/t8.png"></div>
             06:30           </div>
           <div class="main time col-sm-2 hidden-xs">
             <div class="typeName col-wrap">Tennis</div>
             <b>06:30</b>
             
           </div>
           <div class="col-xs-10">
              <div class="clearfix">
                  <img id="img-inline" class="pr-5 hidden-xs hidden-xxs lazyloaded" data-src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png" height="40" width="40" title="ATP Cup Teams" src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png">                  <div class="col-inline">
                    <div class="text-nowrap">Serbia - Norway</div>
                    <div class="text-nowrap"><b>ATP Cup Teams</b></div>
                  </div>
              </div>
           </div>
           
           <div class="col-xs-12 col-wrap text-right" id="ev3640101073016"></div>
        </div>
</a>   

 <a class="article" href="/event/live-tennis-atp-cup-teams-serbia-norway/797348?id=3640101093016">
        <div class="row">
           <div class="main col-xs-2 hidden-sm hidden-md hidden-lg pl-0  text-center">
             <div class="font-size-10"><img data-src="https://img.sport-tv-guide.live/images/t8.png" class="w-40 lazyloaded" title="Tennis" src="https://img.sport-tv-guide.live/images/t8.png"></div>
             08:30           </div>
           <div class="main time col-sm-2 hidden-xs">
             <div class="typeName col-wrap">Tennis</div>
             <b>08:30</b>
             
           </div>
           <div class="col-xs-10">
              <div class="clearfix">
                  <img id="img-inline" class="pr-5 hidden-xs hidden-xxs lazyloaded" data-src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png" height="40" width="40" title="ATP Cup Teams" src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png">                  <div class="col-inline">
                    <div class="text-nowrap">Serbia - Norway</div>
                    <div class="text-nowrap"><b>ATP Cup Teams</b></div>
                  </div>
              </div>
           </div>
           
           <div class="col-xs-12 col-wrap text-right" id="ev3640101093016"></div>
        </div>
</a>   

 <a class="article" href="/event/live-tennis-atp-cup-teams-serbia-norway/797348?id=3640101113016">
        <div class="row">
           <div class="main col-xs-2 hidden-sm hidden-md hidden-lg pl-0  text-center">
             <div class="font-size-10"><img data-src="https://img.sport-tv-guide.live/images/t8.png" class="w-40 lazyloaded" title="Tennis" src="https://img.sport-tv-guide.live/images/t8.png"></div>
             10:30           </div>
           <div class="main time col-sm-2 hidden-xs">
             <div class="typeName col-wrap">Tennis</div>
             <b>10:30</b>
             
           </div>
           <div class="col-xs-10">
              <div class="clearfix">
                  <img id="img-inline" class="pr-5 hidden-xs hidden-xxs lazyloaded" data-src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png" height="40" width="40" title="ATP Cup Teams" src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png">                  <div class="col-inline">
                    <div class="text-nowrap">Serbia - Norway</div>
                    <div class="text-nowrap"><b>ATP Cup Teams</b></div>
                  </div>
              </div>
           </div>
           
           <div class="col-xs-12 col-wrap text-right" id="ev3640101113016"></div>
        </div>
</a>   

 <a class="article" href="/event/live-tennis-atp-cup-teams-russia-france/797350?id=3640101000016">
        <div class="row">
           <div class="main col-xs-2 hidden-sm hidden-md hidden-lg pl-0  text-center">
             <div class="font-size-10"><img data-src="https://img.sport-tv-guide.live/images/t8.png" class="w-40 lazyloaded" title="Tennis" src="https://img.sport-tv-guide.live/images/t8.png"></div>
             23:00           </div>
           <div class="main time col-sm-2 hidden-xs">
             <div class="typeName col-wrap">Tennis</div>
             <b>23:00</b>
             
           </div>
           <div class="col-xs-10">
              <div class="clearfix">
                  <img id="img-inline" class="pr-5 hidden-xs hidden-xxs lazyloaded" data-src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png" height="40" width="40" title="ATP Cup Teams" src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png">                  <div class="col-inline">
                    <div class="text-nowrap">Russia - France</div>
                    <div class="text-nowrap"><b>ATP Cup Teams</b></div>
                  </div>
              </div>
           </div>
           
           <div class="col-xs-12 col-wrap text-right" id="ev3640101000016"></div>
        </div>
</a>   
<div class="dateSeparator">
                             <span class="date"> 2 Jan</span>
                             <div class="dateCenter">Sunday</div>
                           </div>
 <a class="article" href="/event/live-tennis-atp-cup-teams-russia-france/797350?id=3640101020016">
        <div class="row">
           <div class="main col-xs-2 hidden-sm hidden-md hidden-lg pl-0  text-center">
             <div class="font-size-10"><img data-src="https://img.sport-tv-guide.live/images/t8.png" class="w-40 lazyloaded" title="Tennis" src="https://img.sport-tv-guide.live/images/t8.png"></div>
             01:00           </div>
           <div class="main time col-sm-2 hidden-xs">
             <div class="typeName col-wrap">Tennis</div>
             <b>01:00</b>
             
           </div>
           <div class="col-xs-10">
              <div class="clearfix">
                  <img id="img-inline" class="pr-5 hidden-xs hidden-xxs lazyloaded" data-src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png" height="40" width="40" title="ATP Cup Teams" src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png">                  <div class="col-inline">
                    <div class="text-nowrap">Russia - France</div>
                    <div class="text-nowrap"><b>ATP Cup Teams</b></div>
                  </div>
              </div>
           </div>
           
           <div class="col-xs-12 col-wrap text-right" id="ev3640101020016"></div>
        </div>
</a>   

 <a class="article" href="/event/live-tennis-atp-cup-teams-russia-france/797350?id=3640101041516">
        <div class="row">
           <div class="main col-xs-2 hidden-sm hidden-md hidden-lg pl-0  text-center">
             <div class="font-size-10"><img data-src="https://img.sport-tv-guide.live/images/t8.png" class="w-40 lazyloaded" title="Tennis" src="https://img.sport-tv-guide.live/images/t8.png"></div>
             03:15           </div>
           <div class="main time col-sm-2 hidden-xs">
             <div class="typeName col-wrap">Tennis</div>
             <b>03:15</b>
             
           </div>
           <div class="col-xs-10">
              <div class="clearfix">
                  <img id="img-inline" class="pr-5 hidden-xs hidden-xxs lazyloaded" data-src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png" height="40" width="40" title="ATP Cup Teams" src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png">                  <div class="col-inline">
                    <div class="text-nowrap">Russia - France</div>
                    <div class="text-nowrap"><b>ATP Cup Teams</b></div>
                  </div>
              </div>
           </div>
           
           <div class="col-xs-12 col-wrap text-right" id="ev3640101041516"></div>
        </div>
</a>   

 <a class="article" href="/event/live-tennis-atp-cup-teams-germany-greatbritain/797351?id=3640102073016">
        <div class="row">
           <div class="main col-xs-2 hidden-sm hidden-md hidden-lg pl-0  text-center">
             <div class="font-size-10"><img data-src="https://img.sport-tv-guide.live/images/t8.png" class="w-40 lazyloaded" title="Tennis" src="https://img.sport-tv-guide.live/images/t8.png"></div>
             06:30           </div>
           <div class="main time col-sm-2 hidden-xs">
             <div class="typeName col-wrap">Tennis</div>
             <b>06:30</b>
             
           </div>
           <div class="col-xs-10">
              <div class="clearfix">
                  <img id="img-inline" class="pr-5 hidden-xs hidden-xxs lazyloaded" data-src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png" height="40" width="40" title="ATP Cup Teams" src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png">                  <div class="col-inline">
                    <div class="text-nowrap">Germany - Great Britain</div>
                    <div class="text-nowrap"><b>ATP Cup Teams</b></div>
                  </div>
              </div>
           </div>
           
           <div class="col-xs-12 col-wrap text-right" id="ev3640102073016"></div>
        </div>
</a>   

 <a class="article" href="/event/live-tennis-atp-cup-teams-germany-greatbritain/797351?id=3640102093016">
        <div class="row">
           <div class="main col-xs-2 hidden-sm hidden-md hidden-lg pl-0  text-center">
             <div class="font-size-10"><img data-src="https://img.sport-tv-guide.live/images/t8.png" class="w-40 lazyloaded" title="Tennis" src="https://img.sport-tv-guide.live/images/t8.png"></div>
             08:30           </div>
           <div class="main time col-sm-2 hidden-xs">
             <div class="typeName col-wrap">Tennis</div>
             <b>08:30</b>
             
           </div>
           <div class="col-xs-10">
              <div class="clearfix">
                  <img id="img-inline" class="pr-5 hidden-xs hidden-xxs lazyloaded" data-src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png" height="40" width="40" title="ATP Cup Teams" src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png">                  <div class="col-inline">
                    <div class="text-nowrap">Germany - Great Britain</div>
                    <div class="text-nowrap"><b>ATP Cup Teams</b></div>
                  </div>
              </div>
           </div>
           
           <div class="col-xs-12 col-wrap text-right" id="ev3640102093016"></div>
        </div>
</a>   

 <a class="article" href="/event/live-tennis-atp-cup-teams-germany-greatbritain/797351?id=3640102113016">
        <div class="row">
           <div class="main col-xs-2 hidden-sm hidden-md hidden-lg pl-0  text-center">
             <div class="font-size-10"><img data-src="https://img.sport-tv-guide.live/images/t8.png" class="w-40 lazyloaded" title="Tennis" src="https://img.sport-tv-guide.live/images/t8.png"></div>
             10:30           </div>
           <div class="main time col-sm-2 hidden-xs">
             <div class="typeName col-wrap">Tennis</div>
             <b>10:30</b>
             
           </div>
           <div class="col-xs-10">
              <div class="clearfix">
                  <img id="img-inline" class="pr-5 hidden-xs hidden-xxs ls-is-cached lazyloaded" data-src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png" height="40" width="40" title="ATP Cup Teams" src="https://img.sport-tv-guide.live/images/leaguem-atp-cup-teams-2765.png">                  <div class="col-inline">
                    <div class="text-nowrap">Germany - Great Britain</div>
                    <div class="text-nowrap"><b>ATP Cup Teams</b></div>
                  </div>
              </div>
           </div>
           
           <div class="col-xs-12 col-wrap text-right" id="ev3640102113016"></div>
        </div>
</a>   

 <a class="article" href="/event/live-basketball-acb?uid=3640102175516">
        <div class="row">
           <div class="main col-xs-2 hidden-sm hidden-md hidden-lg pl-0  text-center">
             <div class="font-size-10"><img data-src="https://img.sport-tv-guide.live/images/t5.png" class="w-40 ls-is-cached lazyloaded" title="Basketball" src="https://img.sport-tv-guide.live/images/t5.png"></div>
             16:55           </div>
           <div class="main time col-sm-2 hidden-xs">
             <div class="typeName col-wrap">Basketball</div>
             <b>16:55</b>
             
           </div>
           <div class="col-xs-10">
              <div class="clearfix">
                  <img id="img-inline" class="pr-5 hidden-xs hidden-xxs lazyloaded" data-src="https://img.sport-tv-guide.live/images/leaguem-acb-30.png" height="40" width="40" title="ACB" src="https://img.sport-tv-guide.live/images/leaguem-acb-30.png">                  <div class="col-inline">
                    <div class="text-nowrap">Liga Endesa (T21/22): Unicaja - Casademont Zaragoza</div>
                    <div class="text-nowrap"></div>
                  </div>
              </div>
           </div>
           
           <div class="col-xs-12 col-wrap text-right" id="ev3640102175516"></div>
        </div>
</a>   

 <a class="article" href="/event/live-basketball-nba-torontoraptors-newyorkknicks/789250?id=3640102213016">
        <div class="row">
           <div class="main col-xs-2 hidden-sm hidden-md hidden-lg pl-0  text-center">
             <div class="font-size-10"><img data-src="https://img.sport-tv-guide.live/images/t5.png" class="w-40 ls-is-cached lazyloaded" title="Basketball" src="https://img.sport-tv-guide.live/images/t5.png"></div>
             20:30           </div>
           <div class="main time col-sm-2 hidden-xs">
             <div class="typeName col-wrap">Basketball</div>
             <b>20:30</b>
             
           </div>
           <div class="col-xs-10">
              <div class="clearfix">
                  <img id="img-inline" class="pr-5 hidden-xs hidden-xxs lazyloaded" data-src="https://img.sport-tv-guide.live/images/leaguem-nba-26.png" height="40" width="40" title="NBA" src="https://img.sport-tv-guide.live/images/leaguem-nba-26.png">                  <div class="col-inline">
                    <div class="text-nowrap">Toronto Raptors - New York Knicks</div>
                    <div class="text-nowrap"><b>NBA</b></div>
                  </div>
              </div>
           </div>
           
           <div class="col-xs-12 col-wrap text-right" id="ev3640102213016"></div>
        </div>
</a>   
<div class="dateSeparator">
                             <span class="date"> 3 Jan</span>
                             <div class="dateCenter">Monday</div>
                           </div>
 <a class="article" href="/event/live-am-football-nfl-minnesotavikings-greenbaypackers/779303?id=3640102022016">
        <div class="row">
           <div class="main col-xs-2 hidden-sm hidden-md hidden-lg pl-0  text-center">
             <div class="font-size-10"><img data-src="https://img.sport-tv-guide.live/images/t22.png" class="w-40 ls-is-cached lazyloaded" title="Am. Football" src="https://img.sport-tv-guide.live/images/t22.png"></div>
             01:20           </div>
           <div class="main time col-sm-2 hidden-xs">
             <div class="typeName col-wrap">Am. Football</div>
             <b>01:20</b>
             
           </div>
           <div class="col-xs-10">
              <div class="clearfix">
                  <img id="img-inline" class="pr-5 hidden-xs hidden-xxs lazyloaded" data-src="https://img.sport-tv-guide.live/images/leaguem-nfl-40.png" height="40" width="40" title="NFL" src="https://img.sport-tv-guide.live/images/leaguem-nfl-40.png">                  <div class="col-inline">
                    <div class="text-nowrap">Minnesota Vikings - Green Bay Packers</div>
                    <div class="text-nowrap"><b>NFL</b></div>
                  </div>
              </div>
           </div>
           
           <div class="col-xs-12 col-wrap text-right" id="ev3640102022016"></div>
        </div>

</a>   
</div>

【问题讨论】：

你能分享一个例外输出的例子吗？
当然@TomerS，我已经添加了我想要实现的电子表格屏幕截图，基本上是在B列中添加。

标签： python pandas dataframe web-scraping beautifulsoup

【解决方案1】：

如何实现？

稍微改变收集数据的方法并选择元素，以便它们出现在日期分隔符下方。

第 1 步

选择所有日期分隔符：

soup.select('.dateSeparator')

第 2 步

遍历它们中的每一个及其所有下一个兄弟，如果兄弟是<div>，则中断：

for item in date.find_next_siblings():
    if item.name == 'div':
        break

第 3 步

提取带有stripped_strings的文本并进行一些调整，因为结构并不总是相同的：

text = tuple(item.stripped_strings)
...

第 4 步

将信息存储在字典列表中并创建您的数据框。

pd.DataFrame(data)

示例

注意： 输出可能与预期略有不同，因为我在您添加屏幕截图之前创建了它，并且我看到了德国网站。但是方向要明确，方法要适应。

...
data = []
for date in soup.select('.dateSeparator'):
    
    for item in date.find_next_siblings():
        if item.name == 'div':
            break

        text = tuple(item.stripped_strings)
        data.append({
            'date':date.span.text.strip(),
            'time':text[0],
            'sport':text[1],
            'at':text[3].split('-')[-1] if len(text) > 4 else text[3].split(':')[-1].split('-')[-1],
            'teams':text[3] if len(text) > 4 else text[3].split(':')[-1],
            'event':text[4] if len(text) > 4 else text[3].split(':')[0] 
        })

pd.DataFrame(data)

输出

date	time	sport	at	teams	event
1 Jan	07:30	Tennis	Norwegen	Serbien - Norwegen	ATP-Cup-Teams
...	...	...	...	...	...
3 Jan	21:30	Basketball	Breogan	Obradoiro CAB - Breogan	ACB

【讨论】：

感谢您提供非常全面的答案，我一定会听从您关于如何收集数据的建议。这几乎可以直接将它输入到我的代码中，除了它每天都在复制事件，而不是在它们要播放的日期。我会尝试自己解决。
我会在我对其进行调整以使其符合我的需要时立即将其标记为正确，但您肯定让我走上了正确的道路。我投票赞成它的有用性。谢谢。
我一直在玩弄代码一段时间，但我没有任何运气产生我正在寻找的结果。如您所见，我希望返回 19 行，但结果是 42 行，在 1 月 3 日事件的情况下，较晚日期（1 月 2 日、1 月 3 日）的事件被重复和三次。知道如何解决吗？
@SpingoTakagi：嗯，好点。错过了检查是否有下一个div 和.dateSeparator --> 看看步骤#2，我也更新了示例。抱歉，我没注意检查了两次。