使用数据框对数据进行建模

Question

我正在尝试训练一个数据集来预测输入的文本是否来自科幻小说。我对python比较陌生，所以我不知道自己做错了什么。

代码：

#class17.py
"""
Created on Fri Nov 17 14:07:36 2017

@author: twaters

Read three science fiction novels
Predict a sentence or paragraph
see whether sentence/phrase/book is from a science fiction novel or not
"""

import nltk
import pandas as pd
import csv
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression

from sklearn import model_selection
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from nltk.corpus import stopwords

#nltk.download()


irobot = "C:/Users/twaters/Desktop/Assignments/SQL/Python/DA Project/irobot.txt"
enders_game = "C:/Users/twaters/Desktop/Assignments/SQL/Python/DA Project/endersgame.txt"
space_odyssey ="C:/Users/twaters/Desktop/Assignments/SQL/Python/DA Project/spaceodyssey.txt"
to_kill_a_mockingbird = "C:/Users/twaters/Desktop/Assignments/SQL/Python/DA Project/tokillamockingbird.txt"

sr = set(stopwords.words('english'))
freq = {}

def main():
    #read_novels()
    model_novels()


def read_novel(b, is_scifi):

    read_file = open(b)

    text = read_file.read()
    words = text.split()
    clean_tokens = words[:]
    filtered_list = []

    for word in clean_tokens:
        word = word.lower()
        if word not in sr:
            filtered_list.append(word)

    freq = nltk.FreqDist(clean_tokens)
    #print(filtered_list)
    for word in clean_tokens:
       count = freq.get(word,0)
       freq[word] = count + 1



    frequency_list = freq.keys()

    with open('C:/Users/twaters/Desktop/Assignments/SQL/Python/DA Project/novels_data.txt', 'w', encoding='utf-8') as csvfile:
        fieldnames = ['word','frequency','is_scifi']
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames, lineterminator = '\n')
        writer.writeheader()

        for words in frequency_list:
            writer.writerow({'word': words,'frequency': freq[words],'is_scifi':is_scifi})

    print("List compiled.")

def read_novels(): 

    read_novel(enders_game, 0)
    read_novel(space_odyssey, 0)
    read_novel(irobot, 0)
    read_novel(to_kill_a_mockingbird, 1)

def model_novels():

    df = pd.read_csv('C:/Users/twaters/Desktop/Assignments/SQL/Python/DA Project/novels_data.txt', 'rb', delimiter='\t', encoding='utf-8')
    print(df)

    #for index in range(2, df.shape[0], 100):
    df_subset = df.loc[1:]
    #print(df_subset)
    X = df_subset.loc[:, 'frequency':'is_scifi']
    Y = df_subset.loc[:, 'frequency':'is_scifi']
    testing_size = 0.2
    seed = 7
    X_train, X_validation, Y_train, Y_validation = model_selection.train_test_split(X, Y, test_size=testing_size, random_state=seed)

    selectedModel = LogisticRegression()
    selectedModel.fit(X_train, Y_train)  
    predictions = selectedModel.predict(X_validation)

#%%
#print("Accuracy Score:\n", accuracy_score(Y_validation, predictions))
#print("Confusion Matrix:\n",confusion_matrix(predictions, Y_validation))
#print("Class report:\n", classification_report(Y_validation, predictions))
#df_test = pd.read_csv('C:/Users/twaters/Desktop/Assignments/SQL/Python/DA Project/novels_data.txt', delimiter='\t')
#predictions_test = selectedModel.predict(df_test)
#test_frame = pd.DataFrame(predictions_test)
#test_frame.to_csv('C:/Users/twaters/Desktop/Assignments/SQL/Python/DA Project/novels_data_result.txt', sep='\t')

---

错误： Traceback（最近一次调用最后一次）：

文件“”，第 1 行，在 主要（）

文件“C:/Users/user/Desktop/Assignments/SQL/Python/DA Project/class17.py”，第 36 行，在 main 模型小说（）

文件“C:/Users/user/Desktop/Assignments/SQL/Python/DA Project/class17.py”，第 95 行，在 model_novels selectedModel.fit(X_train, Y_train)

文件“D:\Program Files (x86)\Anaconda\lib\site-packages\sklearn\linear_model\logistic.py”，第 1216 行，适合 order="C")

文件“D:\Program Files (x86)\Anaconda\lib\site-packages\sklearn\utils\validation.py”，第 573 行，在 check_X_y ensure_min_features、warn_on_dtype、估计器）

文件“D:\Program Files (x86)\Anaconda\lib\site-packages\sklearn\utils\validation.py”，第 453 行，在 check_array _assert_all_finite(数组)

文件“D:\Program Files (x86)\Anaconda\lib\site-packages\sklearn\utils\validation.py”，第 44 行，在 _assert_all_finite " 或对于 %r 来说太大的值。" % X.dtype)

ValueError：输入包含 NaN、无穷大或对于 dtype('float64') 来说太大的值。

如果您需要访问我正在读取的文件，我可以链接它们。

感谢您的帮助！

Answer 1

以下是stacktrace中你应该注意的点：

文件“C:/Users/user/Desktop/Assignments/SQL/Python/DA Project/class17.py”，第 95 行，model_novels selectedModel.fit(X_train, Y_train)

文件“D:\Program Files (x86)\Anaconda\lib\site-packages\sklearn\utils\validation.py”，第 44 行，在 _assert_all_finite 中“或对于 %r 而言太大的值。” % X.dtype)

这告诉我们格式化 X 存在问题，因此逻辑回归会接受它。

您应该检查 X_train 和 X 以查看它们是否包含错误值。

这个答案会给你一些关于如何做到这一点的指导。

Python pandas: check if any value is NaN in DataFrame