协助创建词袋模型

Question

免责声明：这是家庭作业的一部分。

我有一组推文，我需要创建一个分类器来尝试预测他们的情绪。为此，我将创建一个词袋模型并将径向 SVM 核函数应用于数据。

这里是原始数据给你一个想法：

> original_tweets
# A tibble: 2,385 x 3
   tweet_id sentiment text                                                                                                                      
      <int> <chr>     <chr>                                                                                                                     
 1        1 positive  @TylerSkewes: It is almost 2014. Where are the self-driving cars so we don't have to worry about a DD tonight. Forreal tho
 2        2 positive  @WIRED: BMW builds a self-driving car -- that drifts I love this technology. Drive me to work baby!
 3        3 positive  Google better hurry up with that driverless car. Watching grandma do an 8 point turn to get in a parking spot is horrific.
 4        4 positive  I just waved thank you to this lady that let me merge on the highway and she gave me the finger. Need my self driving car.
 5        5 positive  I might be the only person who starts #cheering in their car when they see a @google car :) #happiness #feelslikeChristmas
 6        6 positive  I want the driverless car, and BAD. Seriously I would be happy if tomorrow morning there were no drivers behind the wheel.
 7        7 positive  I'm over here writing a 2000 word essay while *****s at Google are on driverless cars making ground breaking shit. Damn. _
 8        8 positive  Is it crazy to think that self driving cars will be the biggest innovation of the last few decades? 
 9        9 positive  Its very nice!RT @cdixon: It's awesome that Google is investing in futuristic stuff like AR glasses and self-driving cars.
10       10 positive  Look closely you will see the reflection of a google car !!!! Screen shot from google maps !!!!!
# ... with 2,375 more rows
> 

我稍微修改了一些术语，因为它们中有 URL，但你明白了。

我已将数据格式化为整洁的格式，并计算了每个术语的 TF-IDF 分数。对于我的特征空间，我选择了前 1000 个 IDF 得分最高的术语。

这是我的数据示例：

> feature_space
# A tibble: 3,000 x 7
   tweet_id sentiment word                   n     tf   idf tf_idf
      <int> <chr>     <chr>              <int>  <dbl> <dbl>  <dbl>
 1        1 positive  forreal                1 0.0435  7.78  0.338
 2        2 positive  drifts                 1 0.0476  7.78  0.370
 3        2 positive  rprjtelkg6             1 0.0476  7.78  0.370
 4        5 positive  cheering               1 0.0455  7.78  0.353
 5        5 positive  feelslikechristmas     1 0.0455  7.78  0.353
 6        7 positive  2000                   1 0.0476  7.78  0.370
 7        7 positive  *****s                 1 0.0476  7.78  0.370
 8        8 positive  decades                1 0.0417  7.78  0.324
 9        8 positive  vltlymug89             1 0.0417  7.78  0.324
10        9 positive  ar                     1 0.0476  7.78  0.370
# ... with 2,990 more rows

我想创建一个词袋模型，使用他们的 TF-IDF 分数来创建情绪分类器。对于这个模型，我知道我需要设置我的数据框，使得每条推文都是一行，而我的特征空间中每个可能的 TF-IDF 术语权重都是一列。

我很难弄清楚如何最好地改变 tibble 或数据框以将数据转换为这种格式。我尝试了 mutate() 和 join() 的各种组合，但从来都不是我想要的方式。

如何根据一组特征词快速将 3000 列或更多列添加到数据框或小标题中，并应用它们的 TF-IDF 值来填充这个稀疏数据结构？我不一定需要直接的代码答案，但在如何在 R 中实现这一点的正确方向上迈出的一步对我有很大帮助。

更新：我的词袋现在有一个空的小标题，我只需要填写数据中的非零 TF-DF 值。这里是：

    > bag_of_words
# A tibble: 2,385 x 3,002
   tweet_id sentiment forreal drifts rprjtelkg6 cheering feelslikechristmas `2000` *****s decades vltlymug89    ar closely reflection zg7hvvfgpn
      <int> <chr>       <dbl>  <dbl>      <dbl>    <dbl>              <dbl>  <dbl>  <dbl>   <dbl>      <dbl> <dbl>   <dbl>      <dbl>      <dbl>
 1        1 positive        0      0          0        0                  0      0      0       0          0     0       0          0          0
 2        2 positive        0      0          0        0                  0      0      0       0          0     0       0          0          0
 3        3 positive        0      0          0        0                  0      0      0       0          0     0       0          0          0
 4        4 positive        0      0          0        0                  0      0      0       0          0     0       0          0          0
 5        5 positive        0      0          0        0                  0      0      0       0          0     0       0          0          0
 6        6 positive        0      0          0        0                  0      0      0       0          0     0       0          0          0
 7        7 positive        0      0          0        0                  0      0      0       0          0     0       0          0          0
 8        8 positive        0      0          0        0                  0      0      0       0          0     0       0          0          0
 9        9 positive        0      0          0        0                  0      0      0       0          0     0       0          0          0
10       10 positive        0      0          0        0                  0      0      0       0          0     0       0          0          0
# ... with 2,375 more rows, and 2,987 more variables

Answer 1

好的，我想我有一个解决方案。我肯定会很好奇如何在没有 for 循环的情况下做到这一点，但我仍然对 apply() 编码风格不太满意。

这是我想出的：

#create bag of words model
#get tweet_id and sentiment
bag_of_words <- original_tweets %>%
  select(-one_of('text'))

#get words from feature space
feature_words <- feature_space$word

#generate empty columns
for(i in feature_words)
  bag_of_words[,i] <- 0

#fill in columns with values from feature space
for(i in 1:length(feature_words)) {
  word <- feature_space[i,]$word
  tweet <- feature_space[i,]$tweet_id
  score <- feature_space[i,]$tf_idf
  bag_of_words[tweet,word] <- score
}

检查输出，看起来不错：

> bag_of_words
# A tibble: 2,385 x 3,002
   tweet_id sentiment forreal drifts rprjtelkg6 cheering feelslikechristmas `2000` *****s decades vltlymug89    ar closely reflection zg7hvvfgpn
      <int> <chr>       <dbl>  <dbl>      <dbl>    <dbl>              <dbl>  <dbl>  <dbl>   <dbl>      <dbl> <dbl>   <dbl>      <dbl>      <dbl>
 1        1 positive    0.338  0          0        0                  0      0      0       0          0     0       0          0          0    
 2        2 positive    0      0.370      0.370    0                  0      0      0       0          0     0       0          0          0    
 3        3 positive    0      0          0        0                  0      0      0       0          0     0       0          0          0    
 4        4 positive    0      0          0        0                  0      0      0       0          0     0       0          0          0    
 5        5 positive    0      0          0        0.353              0.353  0      0       0          0     0       0          0          0    
 6        6 positive    0      0          0        0                  0      0      0       0          0     0       0          0          0    
 7        7 positive    0      0          0        0                  0      0.370  0.370   0          0     0       0          0          0    
 8        8 positive    0      0          0        0                  0      0      0       0.324      0.324 0       0          0          0    
 9        9 positive    0      0          0        0                  0      0      0       0          0     0.370   0          0          0    
10       10 positive    0      0          0        0                  0      0      0       0          0     0       0.370      0.370      0.370
# ... with 2,375 more rows, and 2,987 more variables

回想起来，我可能让自己变得比我需要的更难，但我绝对希望看到经验丰富的 R 兽医能想出的任何更有效的方法。干杯。