pyspark 1.6.3 线性回归错误 float() 参数必须是字符串或数字答案

【问题标题】：pyspark 1.6.3 linear regression error float() argument must be a string or numberpyspark 1.6.3 线性回归错误 float() 参数必须是字符串或数字
【发布时间】：2017-12-23 19:41:03
【问题描述】：

我正在使用 pyspark 的线性回归，这是我的代码：

from pyspark.ml.regression import LabeledPoint,LinearRegressionWithSGD
from pyspark import SparkContext, SparkConf 
from pyspark.sql import SQLContext
from pyspark.ml.evaluation import RegressionEvaluator
import time
import csv

start_time = time.time()

conf = SparkConf().setAppName("project_spark").setMaster("local")
sc = SparkContext(conf=conf)
sqlc = SQLContext(sc)

X_train = sc.textFile('C:\Users\WINDOWS 8.1\Desktop\BoW_Train_int_1k.csv')
X_test = sc.textFile('C:\Users\WINDOWS 8.1\Desktop\BoW_Test_int_1k.csv')
y_train = sc.textFile('C:\Users\WINDOWS 8.1\Desktop\Train_Tags81_1k.csv')
y_test = sc.textFile('C:\Users\WINDOWS 8.1\Desktop\Test_Tags81_1k.csv')

X_train = X_train.map(lambda line: line.split(","))
X_test = X_test.map(lambda line: line.split(","))
y_train = y_train.map(lambda line: line.split(","))
y_test = y_test.map(lambda line: line.split(","))

training = LabeledPoint(y_train, X_train)
testing = LabeledPoint(y_test, X_test)

model = LinearRegressionWithSGD.train(training)
valuesAndPreds = (testing.map(lambda p: (p.label, model.predict(p.features))))

evaluator = RegressionEvaluator(metricName="rmse")
RMSE = evaluator.evaluate(valuesAndPreds)

print("Root Mean Squared Error = " + str(RMSE))
Time = time.time() - start_time
print("--- %s seconds ---" % Time)
spark.stop()

但此代码有错误 float() 参数必须是行中的字符串或数字

training = LabeledPoint(y_train, X_train)

那么，我该如何解决它

【问题讨论】：

请详细说明您想要达到的目标

标签： python pyspark linear-regression

【解决方案1】：

如果没有全貌，我猜你给LabeledPoint 提供了错误类型的参数。更具体地说，您的 y_train 和 y_test 从以下获取值：

...
y_train.map(lambda line: line.split(","))
y_test.map(lambda line: line.split(","))

每个都返回一个list，它与LabeledPoint label 参数不兼容。

所以： training = LabeledPoint(y_train, X_train) --> training = LabeledPoint([some, values], [some, other, values])

但是，取自 docs/source，LabeledPoint 期望第一个参数，即标签，可以转换为 float。

class LabeledPoint(object):

    """
    Class that represents the features and labels of a data point.

    :param label:
      Label for this data point.
    :param features:
      Vector of features for this point (NumPy array, list,
      pyspark.mllib.linalg.SparseVector, or scipy.sparse column matrix).

    .. note:: 'label' and 'features' are accessible as class attributes.

    .. versionadded:: 1.0.0
    """

    def __init__(self, label, features):
        self.label = float(label)
        self.features = _convert_to_vector(features)

因此，根据您的行的样子，可能会将您的代码更改为以下内容：

...
y_train.map(lambda line: line.split(",")[0])
...
y_test.map(lambda line: line.split(",")[0])

希望对你有帮助，祝你好运！

【讨论】：