拟合管道时 Pyspark ML 管道错误

Question

我第一次从三个 pyspark.ml.feature (tokenizer,CV,idf) 构建了一个流水线，所有丁字裤都运行良好，但第二次尝试它告诉我 Py4JJavaError：调用 o175.fit 时发生错误。 有谁知道这个错误的原因是什么谢谢

   import findspark
   findspark.init()
   import pyspark.sql.types as typ
   import pyspark as ps
   from pyspark.sql import SparkSession
   import pandas as pd
   import numpy as np
   import warnings
   from pyspark.sql import SQLContext
   sparkSession = SparkSession.builder \
       .master("local[2]") \
       .appName("Pyspark Sentiment") \
       .getOrCreate()
   df = sparkSession.read.load('data/Microblog_Trialdata.csv', 
                          format='com.databricks.spark.csv', 
                          header='true', 
                          inferSchema='true')
df=df.select("sentiment score","spans")
(train_set, val_set, test_set) = df.randomSplit([0.6, 0.2, 0.2], seed = 42)
from pyspark.ml.feature import HashingTF, IDF, Tokenizer ,CountVectorizer
from pyspark.ml.feature import StringIndexer
from pyspark.ml import Pipeline

tokenizer = Tokenizer(inputCol="spans", outputCol="words")
CV = CountVectorizer(vocabSize=2**11, inputCol="words", outputCol='cv_')
idf = IDF(inputCol='cv_', outputCol="features", minDocFreq=5) #minDocFreq: 

remove sparse terms


#model=CV.fit(data)
    #vo=model.vocabulary 
    #print(type(vo))
    pipeline = Pipeline(stages=[tokenizer, CV, idf])

    pipelineFit = pipeline.fit(train_set)
    train_df = pipelineFit.transform(train_set)
    val_df = pipelineFit.transform(val_set)
    train_df.select("cv_").show(5,truncate=False)
    train_df.show(5)

Answer 1

在 train_set 但在 val_set 中看不到的单词可能会导致错误。 Count Vectorizer 有 handleInvalid 选项来解决这个问题。

# this will ignore not seen words
CV = CountVectorizer(vocabSize=2**11, inputCol="words", outputCol='cv_',handleInvalid='skip')