在一个人的推文中找到最受欢迎的词

Question

在一个项目中，我试图查询特定用户句柄的推文，并在用户的推文中找到最常用的词，并返回该最常用词的频率。

下面是我的代码：

  public String mostPopularWord()
  {
     this.removeCommonEnglishWords();
     this.sortAndRemoveEmpties();

     Map<String, Integer> termsCount = new HashMap<>();
     for(String term : terms)
     {
        Integer c = termsCount.get(term);
        if(c==null)
           c = new Integer(0);
        c++;
        termsCount.put(term, c);
     }
     Map.Entry<String,Integer> mostRepeated = null;
     for(Map.Entry<String, Integer> curr: termsCount.entrySet())
     {
         if(mostRepeated == null || mostRepeated.getValue()<curr.getValue())
             mostRepeated = curr;
     }

     //frequencyMax = termsCount.get(mostRepeated.getKey());

     try 
     {
        frequencyMax = termsCount.get(mostRepeated.getKey());
        return mostRepeated.getKey();
     } 
     catch (NullPointerException e) 
     {
        System.out.println("Cannot find most popular word from the tweets.");
     }

     return ""; 
  }

我也认为显示我在上面的方法中调用的前两个方法的代码会有所帮助，如下所示。它们都在同一个类中，定义如下：

  private Twitter twitter;
  private PrintStream consolePrint;
  private List<Status> statuses;
  private List<String> terms;
  private String popularWord;
  private int frequencyMax;

  @SuppressWarnings("unchecked")
  public void sortAndRemoveEmpties()
  {
     Collections.sort(terms);
     terms.removeAll(Arrays.asList("", null));
  }

  private void removeCommonEnglishWords()
  {          
     Scanner sc = null;

     try
     {
        sc = new Scanner(new File("commonWords.txt"));
     }
     catch(Exception e)
     {
        System.out.println("The file is not found");
     }

     List<String> commonWords = new ArrayList<String>(); 
     int count = 0;
     while(sc.hasNextLine())
     {
        count++;
        commonWords.add(sc.nextLine()); 
     }

     Iterator<String> termIt = terms.iterator();
     while(termIt.hasNext())
     {
        String term = termIt.next();
        for(String word : commonWords)
           if(term.equalsIgnoreCase(word))
              termIt.remove();
     }
  }

对于相当长的代码 sn-ps，我深表歉意。但令人沮丧的是，即使我的 removeCommonEnglish() 方法显然是正确的（在另一篇文章中讨论），当我运行 mostPopularWord() 时，它返回“the”，这显然是我常用的英语单词列表的一部分已经并打算从列表中删除。我可能做错了什么？

更新 1： 这是 commonWords 文件的链接： https://drive.google.com/file/d/1VKNI-b883uQhfKLVg-L8QHgPTLNb22uS/view?usp=sharing

更新 2：我在调试时注意到的一件事是 而（sc.hasNext（）） 在 removeCommonEnglishWords() 中被完全跳过。不过，我不明白为什么。

Answer 1

如果你像这样使用流会更简单：

String mostPopularWord() {
    return terms.stream()
            .collect(Collectors.groupingBy(s -> s, Collectors.counting()))
            .entrySet().stream()
            .sorted(Map.Entry.comparingByValue(Comparator.reverseOrder()))
            .findFirst()
            .map(Map.Entry::getKey)
            .orElse("");
}

Answer 2

我试过你的代码。这是你必须做的。替换removeCommonEnglishWords()中的以下部分

Iterator<String> termIt = terms.iterator();
while(termIt.hasNext())
{
   String term = termIt.next();
   for(String word : commonWords)
      if(!term.equalsIgnoreCase(word))
           reducedTerms.add( term );
}

用这个：

 List<String> reducedTerms = new ArrayList<>();
 for( String term : this.terms ) {
     if( !commonWords.contains( term ) ) reducedTerms.add( term );
 }

 this.terms = reducedTerms;

由于您没有提供该类，我创建了一个带有一些假设的类，但我认为这段代码会通过。

Answer 3

使用流的方法略有不同。

1. 这使用了使用流的相对常见的频率计数习语并将它们存储在地图中。
2. 然后它会进行简单的扫描以找到获得的最大计数并返回 该单词或字符串“No words found”。
3. 它还会过滤掉 Set<String> 中名为 ignore 的单词，因此您也需要创建它。

           import java.util.Arrays;
           import java.util.Comparator;
           import java.util.Map;
           import java.util.Map.Entry;
           import java.util.stream.Collectors;

            Set<String> ignore = Set.of("the", "of", "and", "a",
            "to", "in", "is", "that", "it", "he", "was",
            "you", "for", "on", "are", "as", "with",
            "his", "they", "at", "be", "this", "have",
            "via", "from", "or", "one", "had", "by",
            "but", "not", "what", "all", "were", "we",
            "RT", "I", "&", "when", "your", "can",
            "said", "there", "use", "an", "each",
            "which", "she", "do", "how", "their", "if",
            "will", "up", "about", "out", "many",
            "then", "them", "these", "so", "some",
            "her", "would", "make", "him", "into",
            "has", "two", "go", "see", "no", "way",
            "could", "my", "than", "been", "who", "its",
            "did", "get", "may", "…", "@", "??", "I'm",
            "me", "u", "just", "our", "like");


            Map.Entry<String, Long> entry = terms.stream()
                 .filter(wd->!ignore.contains(wd)).map(String::trim)
                .collect(Collectors.groupingBy(a -> a,
                        Collectors.counting()))
                .entrySet().stream()
                .collect(Collectors.maxBy(Comparator
                        .comparing(Entry::getValue)))
                .orElse(Map.entry("No words found", 0L));


              System.out.println(entry.getKey() + " " + entry.getValue());