如何遍历R中的多个URL并保存在数据框中

Question

我无法遍历多个 URL 并将其保存在数据框中。我分享了一次只能检索一个 url 并保存在数据框中的代码。

网址中发生变化的部分是网址末尾的数字，表示日期。 我正在尝试从例如 20190901 到 20190915 中抓取所有数据并将其存储在同一个数据框中。

代码如下：

    library(rvest)
    library(dplyr)

    # Specifying URL
    url <- 'https://classic.sportsbookreview.com/betting-odds/mlb-baseball/?date=20190901'
    
    # Reading the HTML code from website
    oddspage <- read_html(url)

    # Using CSS selectors to scrape away teams
    awayHtml <- html_nodes(oddspage,'.eventLine-value:nth-child(1) a')

    #Using CSS selectors to scrape scores
    awayScoreHtml <- html_nodes(oddspage,'.first.total')
    awayScore <- html_text(awayScoreHtml)
    awayScore <- as.numeric(awayScore)
    homeScoreHtml <- html_nodes(oddspage, '.score-periods+ .score-periods .total')
    homeScore <- html_text(homeScoreHtml)
    homeScore <- as.numeric(homeScore)

    # Converting away data to text
    away <- html_text(awayHtml)

    # Using CSS selectors to scrape home teams
    homeHtml <- html_nodes(oddspage,'.eventLine-value+ .eventLine-value a')

    # Converting home data to text
    home <- html_text(homeHtml)

    # Using CSS selectors to scrape Away Odds
    awayPinnacleHtml <- html_nodes(oddspage,'.eventLine-consensus+ .eventLine-book.eventLine-book-value:nth-child(1) b')
    awayBookmakerHtml <- html_nodes(oddspage,'.eventLine-book:nth-child(12) .eventLine-book-value:nth-child(1) b')

    # Converting Away Odds to Text
    awayPinnacle <- html_text(awayPinnacleHtml)
    awayBookmaker <- html_text(awayBookmakerHtml)

    # Converting Away Odds to numeric
    awayPinnacle <- as.numeric(awayPinnacle)
    awayBookmaker <- as.numeric(awayBookmaker)

    # Using CSS selectors to scrape Pinnacle Home Odds
    homePinnacleHtml <- html_nodes(oddspage,'.eventLine-consensus+ .eventLine-book .eventLine-book-value+ .eventLine-book-value b')
    homeBookmakerHtml <- html_nodes(oddspage,'.eventLine-book:nth-child(12) .eventLine-book-value:nth-child(2) b')

    # Converting Home Odds to Text
    homePinnacle <- html_text(homePinnacleHtml)
    homeBookmaker <- html_text(homeBookmakerHtml)

    # Converting Home Odds to Numeric
    homePinnacle <- as.numeric(homePinnacle)
    homeBookmaker <- as.numeric(homeBookmaker)


    # Create Data Frame
    df <- data.frame(away,home,awayScore,homeScore,awayPinnacle,homePinnacle,awayBookmaker,homeBookmaker)

    View(df)

我对编码非常陌生，但我无法成功应用类似问题中使用的任何技术。

Answer 1

把你所有的代码放在一个函数中，让date动态生成url：

get_data <- function(date) {
      url <- paste0('https://classic.sportsbookreview.com/betting-odds/mlb-baseball/?date=', date)
      #...Rest of the code as it is
      #...
}

使用sprintf创建一个日期向量

date_vec <- sprintf('201909%02d', 1:15)
date_vec
# [1] "20190901" "20190902" "20190903" "20190904" "20190905" "20190906"
# [7] "20190907" "20190908" "20190909" "20190910" "20190911" "20190912"
#[13] "20190913" "20190914" "20190915"

使用lapply 提取每个日期的数据并将它们组合起来。

all_data <- do.call(rbind, lapply(date_vec, get_data))

您也可以从purrr 使用map_df。

all_data <- purrr::map_df(date_vec, get_data)

但是，您可能需要在函数中为不返回特定字段的任何值的页面添加一些检查。