Web-Scraping Python，DataFrame 的索引问题

Question

我正在为 Spotify Charts 开发一个网络抓取工具，以每天提取前 200 首每日歌曲。我已尽一切努力提取我感兴趣的数据，包括排名、艺术家、曲目标题和流数。我坚持的是将所有内容放入 DataFrame 以导出为 CSV 到 excel。现在，当我打印我的 DataFrame 时，它​​将每个循环视为 1 行 4 列，而不是 200 行 4 列。

我不确定问题出在哪里，因为我几乎尝试了所有方法并尽可能多地进行了调查。我知道索引有问题，因为每个“应该是一行”都有相同的第一个“0”索引，当它们应该连续到 199 时。此外，我的 DataFrame 的列名在每个“应该是什么”之后不断重复一行”，所以我知道那里肯定有问题。

import requests
from bs4 import BeautifulSoup
from datetime import date, timedelta
from time import time
from time import sleep
from random import randint
import pandas as pd
import numpy as np

base_url = 'https://spotifycharts.com/regional/global/daily/'

r = requests.get(base_url)



soup = BeautifulSoup(r.text, 'html.parser')
chart = soup.find('table', {'class': 'chart-table'})
tbody = chart.find('tbody')
for tr in tbody.find_all('tr'):

    rank_text = []
    rank_text_elem = tr.find('td', {'class': 'chart-table- 
    position'})
    for item in rank_text_elem:
        rank_text = []
        rank_text.append(item)

    artist_text = []
    artist_text_elem = tr.find('td', {'class': 'chart-table- 
    track'}).find_all('span')
    for item in artist_text_elem:
        artist_text = []
        artist_text.append(item.text.replace('by ','').strip())

    title_text = []
    title_text_elem = tr.find('td', {'class': 'chart-table- 
    track'}).find_all('strong')
    for item in title_text_elem:
        title_text = []
        title_text.append(item.text)

    streams_text = []
    streams_text_elem = tr.find('td', {'class': 'chart-table-streams'})
    for item in streams_text_elem:
        streams_text = []
        streams_text.append(item)

# creating dataframe to store 4 variables 

    list_of_data = list(zip(rank_text, artist_text, title_text, 
    streams_text))

    df = pd.DataFrame(list_of_data, columns = 
    ['Rank','Artist','Title','Streams'])
    print(df)

基本上，我正在尝试创建一个数据框，以在每行中保存 4 个变量，用于 200 行，用于 spotify 全局图表的每个日期。请忽略我在顶部包含的一些模块和库，它们用于根据我已经弄清楚的动态 url 遍历历史数据的每一页。任何帮助是极大的赞赏！谢谢！

Answer 1

在for 循环之前创建列表all_rows。

在for 循环中，我将带有单行数据的列表添加到all_rows。

在for循环之后，我使用all_rows创建DataFrame

import requests
from bs4 import BeautifulSoup
import pandas as pd

base_url = 'https://spotifycharts.com/regional/global/daily/'

r = requests.get(base_url)

soup = BeautifulSoup(r.text, 'html.parser')
chart = soup.find('table', {'class': 'chart-table'})
tbody = chart.find('tbody')

all_rows = []

for tr in tbody.find_all('tr'):

    rank_text = tr.find('td', {'class': 'chart-table-position'}).text

    artist_text = tr.find('td', {'class': 'chart-table-track'}).find('span').text
    artist_text = artist_text.replace('by ','').strip()

    title_text = tr.find('td', {'class': 'chart-table-track'}).find('strong').text

    streams_text = tr.find('td', {'class': 'chart-table-streams'}).text

    all_rows.append( [rank_text, artist_text, title_text, streams_text] )

# after `for` loop

df = pd.DataFrame(all_rows, columns=['Rank','Artist','Title','Streams'])
print(df.head())

Answer 2

您可以使用pandas 和requests

import pandas as pd
import requests

headers = {'User-Agent': 'Mozilla/5.0'}
url ='https://spotifycharts.com/regional/global/daily/'

r = requests.get(url, headers = headers).content

table = pd.read_html(r)[0] #transfer html to pandas
table.dropna(axis = 1, how = 'all', inplace = True) #drop nan column
table[['Title','Artist']] = table['Unnamed: 3'].str.split(' by ',expand=True) #split title artist strings into two columns
del table['Unnamed: 3'] #remove combined column
table = table[['Track', 'Artist','Title', 'Unnamed: 4']] #re-order cols
table.columns= ['Rank', 'Artist','Title', 'Streams'] #rename cols
print(table)