不同数量的节点

Question

我想从 www.airlinequality.com 页面获取一些航空公司评论，该页面提供了有关不同航班方面的信息。撰写飞行评论时，并非所有字段都是必填的。当不同的评论有不同数量的元素时，这会创建结构，这是我当前的代码无法处理的。

例如，我想从此页面获取评论： http://www.airlinequality.com/airline-reviews/austrian-airlines/page/1/

Seat Comfort 有 10 条评论，但 Inflight Entertainment 仅适用于 inf 8。最后，这会创建两个不同长度的向量，无法合并。

我的代码：

review_html_temp = read_html("http://www.airlinequality.com/airline-reviews/austrian-airlines/page/1/)

    review_seat_comfort = review_html_temp %>%
  html_nodes(xpath = './/table[@class = "review-ratings"]//td[@class = "review-rating-header seat_comfort"]/following-sibling::td/span[@class = "star fill"][last()]') %>%
  html_text() %>%
  str_replace_all(pattern = "[\r\n\t]" , "")

review_entertainment = review_html_temp %>%
  html_nodes(xpath = './/table[@class = "review-ratings"]//td[@class = "review-rating-header inflight_entertainment"]/following-sibling::td//span[@class = "star fill"][last()]') %>%
  html_text() %>%
  str_replace_all(pattern = "[\r\n\t]" , "")

有没有办法，当所有 10 条评论都不存在节点时，我如何用“”或 NA 填充娱乐价值？ 最终结果如下所示：

seat_comfort: "4" "5" "3" "3" "1" "4" "4" "3" "3" "3"
entertainment_system: "5" "1" NA "1" "1" "3" NA "3" "5" "1"

Answer 1

关键是html_nodes(...) %>% html_node(...)会返回一个条目对应html_nodes返回的每个节点 如果指定给html_node的路径是绝对的。 IIUC 这意味着html_node 将每个返回的节点视为其自己的根，并为每个根返回一个唯一节点（特别是为后续调用不匹配的节点返回NA）；以// 开始html_node 调用重置搜索并将根返回到整个页面根。我不是 100% 确定这种解释，但实际上这意味着以下内容可以工作（注意：我必须将页面下载为 HTML，因为该站点是动态加载的（至少对我而言）并且不是简单的@阅读987654328@)。

URL = '~/Desktop/airlines.html'
#get to table; we end at tbody here instead of tr
#  since we only want one entry for each "table" on the
#  page (i.e., for each review); if we add tr there,
#  the html_nodes call will give us an element for
#  _each row of each table_.
tbl = read_html(URL) %>% 
  html_nodes(xpath = '//table[@class="review-ratings"]/tbody')
#note the %s where we'll substitute the particular element we want
star_xp = paste0('tr/td[@class="%s"]/following-sibling::',
                 'td[@class="review-rating-stars stars"]',
                 '/span[@class="star fill"][last()]') 

tbl %>% 
  html_node(xpath = sprintf(star_xp, "review-rating-header seat_comfort")) %>% 
  html_text
#  [1] NA  "4" "5" "3" "3" "1" "4" "4" "3" "3" "3"

这很丑陋，但遵循我习惯看到的提取流程。我想下面的内容会更容易看maggrittr-y/，虽然有点非线性：

star_xp %>% sprintf("review-rating-header seat_comfort") %>%
  html_node(x = tbl, xpath = .) %>% html_text
#  [1] NA  "4" "5" "3" "3" "1" "4" "4" "3" "3" "3"

对于另一个：

star_xp %>% sprintf("review-rating-header inflight_entertainment") %>%
  html_node(x = tbl, xpath = .) %>% html_text
#  [1] NA  NA  "5" "1" "1" "1" "3" "3" "5" NA  "1"

Answer 2

IIUC，可以通过两个假设来获得所需的输出。首先，在每个表中，seat_comfort 类位于 inflight_entertainment 类之前。其次，任何两个具有相同类的连续节点都需要用NA 分隔。本质上，您不能有两个连续的 seat_comfort 类，而它们之间没有 inflight_entertainment 类。总之，你的td 标签应该是这样的：

<td class="review-rating-header seat_comfort">Seat Comfort</td>
<td class="review-rating-header inflight_entertainment">Inflight Entertainment</td>
<td class="review-rating-header seat_comfort">Seat Comfort</td>
<td class="review-rating-header inflight_entertainment">Inflight Entertainment</td>
...

但是，所提供页面的来源有几个重复 seat_comfort 类。因此，您可能必须循环使用inflight_entertainment 或seat_comfort 的所有节点，并填充具有相同类的两个连续标签的空白。下图为：

library(rvest)


URL <- 'http://www.airlinequality.com/airline-reviews/austrian-airlines/page/1/'
query <- './/table[@class="review-ratings"]//td[contains(@class, "review-rating-header seat_comfort") or contains(@class, "review-rating-header inflight_entertainment")]'
sub_query <- './/following-sibling::td//span[@class = "star fill"][last()]'

review_html_temp <- read_html(URL)

# Get all nodes with either seat_comfort or inflight_entertainment rows
all_nodes <- review_html_temp %>%
  html_nodes(xpath = query)

# Use the text at the first node as a variable to test conditions
current_string <- html_text(all_nodes[1])

# Get the first review value
first_review <- html_nodes(all_nodes[1], xpath = sub_query) %>%
  html_text()

# Loop through all nodes starting at the second node
# Check if the current node's text is the same as the global condition variable
# If so, prepend the review values with an NA
# Otherwise, return the review values
all_output <- lapply(all_nodes[2:length(all_nodes)], function(node) {
  node_text <- html_text(node)
  if (node_text == current_string) {
    current_string <<- node_text
    output <- html_nodes(node, xpath = sub_query) %>%
      html_text()
    c(NA, output)
  } else {
    current_string <<- node_text
    html_nodes(node, xpath = sub_query) %>%
      html_text()
  }
})

# Prepend the first review value to the output
all_output <- c(first_review, unlist(all_output))

# Select seat_comfort and inflight_entertainment
seat_comfort <- all_output[seq(1, length(all_output), 2)]
entertainment <- all_output[seq(2, length(all_output), 2)]

# Make a data.frame
data.frame(seat_comfort=seat_comfort,
           entertainment=entertainment,
           stringsAsFactors = F)

上面定义的数据框应该如下所示：

 seat_comfort entertainment
 4            <NA>         
 5            5            
 3            1            
 3            1            
 1            1            
 4            3            
 4            3            
 3            5            
 3            <NA>         
 3            1

我希望这会有所帮助。