支持向量机的余弦相似核答案

【问题标题】：Cosine Similarity Kernel for SVM支持向量机的余弦相似核
【发布时间】：2019-05-16 09:24:11
【问题描述】：

所以我一直在研究这个聊天机器人项目，我将 SVM 用于它的 ML，我真的想使用余弦相似度作为内核。我尝试过使用 pykernel (as suggested from this post) 或来自不同来源的其他代码，但它仍然无法正常工作，我不知道为什么......

说我有train.py这样的代码

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
import pickle, csv, json, timeit, random, os, nltk
from nltk.stem.lancaster import LancasterStemmer
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split as tts
from sklearn.preprocessing import LabelEncoder as LE
from Sastrawi.Stemmer.StemmerFactory import StemmerFactory
from Sastrawi.StopWordRemover.StopWordRemoverFactory import StopWordRemoverFactory
import my_kernel

def preprocessing(text):
    factory1 = StopWordRemoverFactory()
    StopWord = factory1.create_stop_word_remover()
    text = StopWord.remove(text)
    factory2 = StemmerFactory()
    stemmer = factory2.create_stemmer()
    return (stemmer.stem(text))

le = LE()
tfv = TfidfVectorizer(min_df=1)

file = os.path.join(os.path.dirname(os.path.abspath(__file__)),"scraping","tes.json")
svm_pickle_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),"data","svm_model.pickle")

if os.path.exists(svm_pickle_path):
    os.remove(svm_pickle_path)

tit = [] # Title
cat = [] # Category
post = [] # Post

with open(file, "r") as sentences_file:
    reader = json.load(sentences_file)
    for row in reader:
        tit.append(preprocessing(row["Judul"]))
        cat.append(preprocessing(row["Kategori"]))
        post.append(preprocessing(row["Post"]))

tfv.fit(tit)
le.fit(cat)

features = tfv.transform(tit)
labels = le.transform(cat)

trainx, testx, trainy, testy = tts(features, labels, test_size=.30, random_state=42)

model = SVC(kernel=my_kernel, C=1.5)

f = open(svm_pickle_path, 'wb')
pickle.dump(model.fit(trainx, trainy), f)
f.close()

print("SVC training score:", model.score(testx, testy))

with open(svm_pickle_path, 'rb') as file:  
    pickle_model = pickle.load(file)

score = pickle_model.score(testx, testy)  
print("Test score: {0:.2f} %".format(100 * score))  
Ypredict = pickle_model.predict(testx)
print(Ypredict)

对于my_kernel.py 代码：

import numpy as np
import math
from numpy import linalg as LA

def my_kernel(X, Y):
    norm = LA.norm(X) * LA.norm(Y)
    return np.dot(X, Y.T)/norm

每次我运行程序都会显示这个

Traceback (most recent call last):

File "F:\env\chatbot\chatbotProj\chatbotProj\train.py", line 84, in <module>
pickle.dump(model.fit(trainx, trainy), f)

File "F:\env\lib\site-packages\sklearn\svm\base.py", line 212, in fit
fit(X, y, sample_weight, solver_type, kernel, random_seed=seed)

File "F:\env\lib\site-packages\sklearn\svm\base.py", line 252, in _dense_fit
X = self._compute_kernel(X)

File "F:\env\lib\site-packages\sklearn\svm\base.py", line 380, in _compute_kernel
kernel = self.kernel(X, self.__Xfit)

File "F:\env\chatbot\chatbotProj\chatbotProj\ChatbotCode\svm.py", line 31, in my_kernel
norm = LA.norm(X) * LA.norm(Y)

File "F:\env\lib\site-packages\numpy\linalg\linalg.py", line 2359, in norm
sqnorm = dot(x, x)

File "F:\env\lib\site-packages\scipy\sparse\base.py", line 478, in __mul__
raise ValueError('dimension mismatch')

ValueError: dimension mismatch

我是 python 和这个 SVM 领域的新手，有人知道哪里出了问题，或者可以推荐我如何更好、更清洁地编写余弦相似度内核吗？

哦，火车 X 的维度是 (193, 634)，火车 Y 是 (193, )，测试 X 是 (83, 634)，测试 Y 是 (83,)，来自 train_test_splitsklearn。

【问题讨论】：

标签： python machine-learning svm cosine-similarity kernel-density

【解决方案1】：

更新：我的朋友告诉我这是因为我有稀疏矩阵而不是简单的数组，所以我必须对其进行密集化并替换 my_kernel.py 代码成为这样

def my_kernel(X, Y):
    X=np.array(X.todense())
    Y=np.array(Y.todense())
    norm = LA.norm(X) * LA.norm(Y)
    return np.dot(X, Y.T)/norm

【讨论】：