【问题标题】:Counting word frequency and making a dictionary from it计算词频并从中制作字典
【发布时间】:2014-03-18 02:32:24
【问题描述】:

我想从文本文件中提取每个单词,并计算字典中的单词频率。

例如:'this is the textfile, and it is used to take words and count'

d = {'this': 1, 'is': 2, 'the': 1, ...} 

我不是那么远,但我只是看不到如何完成它。到目前为止我的代码:

import sys

argv = sys.argv[1]
data = open(argv)
words = data.read()
data.close()
wordfreq = {}
for i in words:
    #there should be a counter and somehow it must fill the dict.

【问题讨论】:

标签: python dictionary count readlines


【解决方案1】:
Write a Python program to create a list of strings by taking input from the user and then create  a dictionary containing each string along with their frequencies. (e.g. if the list is [‘apple’,  ‘banana’, ‘fig’, ‘apple’, ‘fig’, ‘banana’, ‘grapes’, ‘fig’, ‘grapes’, ‘apple’] then output should be  {'apple': 3, 'banana': 2, 'fig': 3, 'grapes': 2}.  

lst = []
d = dict()
print("ENTER ZERO NUMBER FOR EXIT !!!!!!!!!!!!")
while True:
    user = input('enter string element :: -- ')
    if user == "0":
        break
    else:
        lst.append(user)
print("LIST ELEMENR ARE :: ",lst)
l = len(lst)
for i in range(l) :
    c = 0
    for j in range(l) :
        if lst[i] == lst[j ]:
            c += 1
    d[lst[i]] = c
print("dictionary is  :: ",d)

【讨论】:

    【解决方案2】:

    def play_with_words(输入):

    input_split = input.split(",")
    input_split.sort()
    count = {}
    for i in input_split:
        if i in count:
            count[i] += 1
        else:
            count[i] = 1
    
    return count
    

    输入="我,我,这里,哪里,你,是"

    打印(play_with_words(输入))

    【讨论】:

      【解决方案3】:

      还有一个功能:

      def wcount(filename):
          counts = dict()
          with open(filename) as file:
              a = file.read().split()
              # words = [b.rstrip() for b in a]
          for word in a:
              if word in counts:
                  counts[word] += 1
              else:
                  counts[word] = 1
          return counts
      

      【讨论】:

        【解决方案4】:

        虽然使用@Michael 建议的collections 库中的Counter 是一种更好的方法,但我添加此答案只是为了改进您的代码。 (我相信这对于新的 Python 学习者来说是一个很好的答案。)

        从您的代码中的评论看来,您似乎想要改进您的代码。而且我认为您可以用文字阅读文件内容(虽然我通常避免使用read() 函数并使用for line in file_descriptor: 类型的代码)。

        由于words 是一个字符串,在for 循环中,for i in words: 循环变量i 不是一个词,而是一个字符。您正在迭代字符串中的字符,而不是迭代字符串words 中的单词。要理解这一点,请注意以下代码 sn-p:

        >>> for i in "Hi, h r u?":
        ...  print i
        ... 
        H
        i
        ,
         
        h
         
        r
         
        u
        ?
        >>> 
        

        因为逐字符而不是逐单词迭代给定字符串不是您想要实现的,所以要逐单词迭代您应该使用 Python 中字符串类中的 split 方法/函数。
        @ 987654321@ 方法 返回字符串中所有单词的列表, 使用 str 作为分隔符(如果未指定,则在所有空格上拆分),可选择将拆分次数限制为编号。

        注意下面的代码示例:

        拆分:

        >>> "Hi, how are you?".split()
        ['Hi,', 'how', 'are', 'you?']
        

        带分割的循环:

        >>> for i in "Hi, how are you?".split():
        ...  print i
        ... 
        Hi,
        how
        are
        you?
        

        它看起来像你需要的东西。除了单词Hi,,因为split() 默认情况下会被空格分割,所以Hi, 被保存为单个字符串(显然)你不希望这样。

        要计算文件中单词的频率,一个好的解决方案是使用正则表达式。但首先,为了简单起见,我将使用replace() 方法。 str.replace(old, new[, max]) 方法返回字符串的副本,其中出现的 old 已替换为 new,可选地将替换次数限制为最大。

        现在检查下面的代码示例,看看我的建议:

        >>> "Hi, how are you?".split()
        ['Hi,', 'how', 'are', 'you?'] # it has , with Hi
        >>> "Hi, how are you?".replace(',', ' ').split()
        ['Hi', 'how', 'are', 'you?'] # , replaced by space then split
        

        循环:

        >>> for word in "Hi, how are you?".replace(',', ' ').split():
        ...  print word
        ... 
        Hi
        how
        are
        you?
        

        现在,如何计算频率:

        一种方法是按照@Michael 的建议使用Counter,但要使用您希望从空字典开始的方法。执行以下代码示例:

        words = f.read()
        wordfreq = {}
        for word in .replace(', ',' ').split():
            wordfreq[word] = wordfreq.setdefault(word, 0) + 1
            #                ^^ add 1 to 0 or old value from dict 
        

        我在做什么?因为最初wordfreq 是空的,所以你不能第一次将它分配给wordfreq[word](它会引发关键异常错误)。所以我使用了setdefault dict 方法。

        dict.setdefault(key, default=None) 类似于 get(),但如果 key 不在 dict 中,则会设置 dict[key]=default。所以第一次当一个新词出现时,我使用setdefault在dict中设置0,然后添加1并分配给同一个dict。

        我已经使用with open 而不是单个open 编写了等效代码。

        with open('~/Desktop/file') as f:
            words = f.read()
            wordfreq = {}
            for word in words.replace(',', ' ').split():
                wordfreq[word] = wordfreq.setdefault(word, 0) + 1
        print wordfreq
        

        运行如下:

        $ cat file  # file is 
        this is the textfile, and it is used to take words and count
        $ python work.py  # indented manually 
        {'and': 2, 'count': 1, 'used': 1, 'this': 1, 'is': 2, 
         'it': 1, 'to': 1, 'take': 1, 'words': 1, 
         'the': 1, 'textfile': 1}
        

        使用re.split(pattern, string, maxsplit=0, flags=0)

        只需更改 for 循环:for i in re.split(r"[,\s]+", words):,它应该会产生正确的输出。

        编辑:最好找到所有字母数字字符,因为您可能有多个标点符号。

        >>> re.findall(r'[\w]+', words) # manually indent output  
        ['this', 'is', 'the', 'textfile', 'and', 
          'it', 'is', 'used', 'to', 'take', 'words', 'and', 'count']
        

        使用 for 循环:for word in re.findall(r'[\w]+', words):

        如果不使用read(),我将如何编写代码:

        文件是:

        $ cat file
        This is the text file, and it is used to take words and count. And multiple
        Lines can be present in this file.
        It is also possible that Same words repeated in with capital letters.
        

        代码是:

        $ cat work.py
        import re
        wordfreq = {}
        with open('file') as f:
            for line in f:
                for word in re.findall(r'[\w]+', line.lower()):
                    wordfreq[word] = wordfreq.setdefault(word, 0) + 1
          
        print wordfreq
        

        使用lower()将大写字母转换为小写字母。

        输出:

        $python work.py  # manually strip output  
        {'and': 3, 'letters': 1, 'text': 1, 'is': 3, 
         'it': 2, 'file': 2, 'in': 2, 'also': 1, 'same': 1, 
         'to': 1, 'take': 1, 'capital': 1, 'be': 1, 'used': 1, 
         'multiple': 1, 'that': 1, 'possible': 1, 'repeated': 1, 
         'words': 2, 'with': 1, 'present': 1, 'count': 1, 'this': 2, 
         'lines': 1, 'can': 1, 'the': 1}
        

        【讨论】:

        • 你对这一切如何结合在一起的阐述是愚蠢的,这应该是公认的答案。
        【解决方案5】:
        wordList = 'this is the textfile, and it is used to take words and count'.split()
        wordFreq = {}
        
        # Logic: word not in the dict, give it a value of 1. if key already present, +1.
        for word in wordList:
            if word not in wordFreq:
                wordFreq[word] = 1
            else:
                wordFreq[word] += 1
        
        print(wordFreq)
        

        【讨论】:

          【解决方案6】:

          您也可以使用 int 类型的默认字典。

           from collections import defaultdict
           wordDict = defaultdict(int)
           text = 'this is the textfile, and it is used to take words and count'.split(" ")
           for word in text:
              wordDict[word]+=1
          

          解释: 我们初始化一个默认字典,其值是 int 类型。这样,任何键的默认值都是 0,我们不需要检查字典中是否存在键。然后,我们将带有空格的文本拆分为单词列表。然后我们遍历列表并增加单词的计数。

          【讨论】:

            【解决方案7】:

            我的方法是从地面做一些事情:

            1. 从文本输入中删除标点符号。
            2. 列出单词。
            3. 删除空字符串。
            4. 遍历列表。
            5. 使每个新词成为字典中值为 1 的键。
            6. 如果一个词已经作为键存在,则将其值加一。

            text = '''this is the textfile, and it is used to take words and count'''
            word = '' #This will hold each word
            
            wordList = [] #This will be collection of words
            for ch in text: #traversing through the text character by character
            #if character is between a-z or A-Z or 0-9 then it's valid character and add to word string..
                if (ch >= 'a' and ch <= 'z') or (ch >= 'A' and ch <= 'Z') or (ch >= '0' and ch <= '9'): 
                    word += ch
                elif ch == ' ': #if character is equal to single space means it's a separator
                    wordList.append(word) # append the word in list
                    word = '' #empty the word to collect the next word
            wordList.append(word)  #the last word to append in list as loop ended before adding it to list
            print(wordList)
            
            wordCountDict = {} #empty dictionary which will hold the word count
            for word in wordList: #traverse through the word list
                if wordCountDict.get(word.lower(), 0) == 0: #if word doesn't exist then make an entry into dic with value 1
                    wordCountDict[word.lower()] = 1
                else: #if word exist then increament the value by one
                    wordCountDict[word.lower()] = wordCountDict[word.lower()] + 1
            print(wordCountDict)
            

            另一种方法:

            text = '''this is the textfile, and it is used to take words and count'''
            for ch in '.\'!")(,;:?-\n':
                text = text.replace(ch, ' ')
            wordsArray = text.split(' ')
            wordDict = {}
            for word in wordsArray:
                if len(word) == 0:
                    continue
                else:
                    wordDict[word.lower()] = wordDict.get(word.lower(), 0) + 1
            print(wordDict)
            

            【讨论】:

              【解决方案8】:
              #open your text book,Counting word frequency
              File_obj=open("Counter.txt",'r')
              w_list=File_obj.read()
              print(w_list.split())
              di=dict()
              for word in w_list.split():
              
              
                  if word in di:
                      di[word]=di[word] + 1
              
                  else:
                      di[word]=1
              
              
              
              max_count=max(di.values())
              largest=-1
              maxusedword=''
              for k,v in di.items():
                  print(k,v)
                  if v>largest:
                      largest=v
                      maxusedword=k
              
              print(maxusedword,largest)
              

              【讨论】:

                【解决方案9】:
                sentence = "this is the textfile, and it is used to take words and count"
                
                # split the sentence into words.
                # iterate thorugh every word
                
                counter_dict = {}
                for word in sentence.lower().split():
                # add the word into the counter_dict initalize with 0
                  if word not in counter_dict:
                    counter_dict[word] = 0
                # increase its count by 1   
                  counter_dict[word] =+ 1
                

                【讨论】:

                  【解决方案10】:

                  如果不想使用collections.Counter,可以自己写函数:

                  import sys
                  
                  filename = sys.argv[1]
                  fp = open(filename)
                  data = fp.read()
                  words = data.split()
                  fp.close()
                  
                  unwanted_chars = ".,-_ (and so on)"
                  wordfreq = {}
                  for raw_word in words:
                      word = raw_word.strip(unwanted_chars)
                      if word not in wordfreq:
                          wordfreq[word] = 0 
                      wordfreq[word] += 1
                  

                  为了更好的东西,看看正则表达式。

                  【讨论】:

                  • 如果 OP 希望他的词频不区分大小写,他应该将大小写转换为大写(或小写)
                  • 如果单词不在 wordfreq 中:wordfreq[word] = 1
                  • @Pranav,我将它设置为 0,因为它在下一行递增
                  【解决方案11】:

                  以下获取字符串,使用 split() 将其拆分为列表,for 循环列表并计数 用 Python 的计数函数 count() 计算句子中每一项的出现频率。这 单词 i 及其频率作为元组放置在一个空列表 ls 中,然后转换为 dict() 的键值对。

                  sentence = 'this is the textfile, and it is used to take words and count'.split()
                  ls = []  
                  for i in sentence:
                  
                      word_count = sentence.count(i)  # Pythons count function, count()
                      ls.append((i,word_count))       
                  
                  
                  dict_ = dict(ls)
                  
                  print dict_
                  

                  输出; {'and': 2, 'count': 1, 'used': 1, 'this': 1, 'is': 2, 'it': 1, 'to': 1, 'take': 1, ' words': 1, 'the': 1, 'textfile,': 1}

                  【讨论】:

                  • 这是非常低效的,因为它会为每个单词再次迭代整个字符串,而不是一次遍历。复杂度将是 O(n²) 而不是 O(n)。
                  【解决方案12】:
                  from collections import Counter
                  t = 'this is the textfile, and it is used to take words and count'
                  
                  dict(Counter(t.split()))
                  >>> {'and': 2, 'is': 2, 'count': 1, 'used': 1, 'this': 1, 'it': 1, 'to': 1, 'take': 1, 'words': 1, 'the': 1, 'textfile,': 1}
                  

                  或者在计数之前去掉标点符号会更好:

                  dict(Counter(t.replace(',', '').replace('.', '').split()))
                  >>> {'and': 2, 'is': 2, 'count': 1, 'used': 1, 'this': 1, 'it': 1, 'to': 1, 'take': 1, 'words': 1, 'the': 1, 'textfile': 1}
                  

                  【讨论】:

                    猜你喜欢
                    • 1970-01-01
                    • 1970-01-01
                    • 1970-01-01
                    • 1970-01-01
                    • 2022-07-16
                    • 2019-05-05
                    • 2010-09-17
                    • 2017-11-27
                    • 2013-05-24
                    相关资源
                    最近更新 更多