在 python 中为文本分类管道生成 PMML

Question

我正在尝试为文本分类管道生成 PMML（使用 jpmml-sklearn）。代码的最后一行 - sklearn2pmml(Textpipeline, "TextMiningClassifier.pmml", with_repr = True) - 崩溃。

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import SGDClassifier
from sklearn2pmml import PMMLPipeline

categories = [
'alt.atheism',
'talk.religion.misc',
]

print("Loading 20 newsgroups dataset for categories:")
print(categories)
data = fetch_20newsgroups(subset='train', categories=categories)
print("%d documents" % len(data.filenames))
print("%d categories" % len(data.target_names))

Textpipeline = PMMLPipeline([
('vect', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf', SGDClassifier()),
])

Textpipeline.fit(data.data, data.target)

from sklearn2pmml import sklearn2pmml

sklearn2pmml(Textpipeline, "TextMiningClassifier.pmml", with_repr = True)

看起来 sklearn2pmml() 无法将 Textpipeline 作为输入。该代码适用于其他管道（此处的示例：https://github.com/jpmml/sklearn2pmml）但不适用于上面的文本分类管道。所以我的问题是：如何为文本分类问题生成 PMML？

我得到的错误：

    Jun 15, 2017 12:48:00 PM org.jpmml.sklearn.Main run
INFO: Parsing PKL..
Jun 15, 2017 12:48:01 PM org.jpmml.sklearn.Main run
INFO: Parsed PKL in 489 ms.
Jun 15, 2017 12:48:01 PM org.jpmml.sklearn.Main run
INFO: Converting..
Jun 15, 2017 12:48:01 PM sklearn2pmml.PMMLPipeline encodePMML
WARNING: The 'target_field' attribute is not set. Assuming y as the name of the target field
Jun 15, 2017 12:48:01 PM sklearn2pmml.PMMLPipeline initFeatures
WARNING: The 'active_fields' attribute is not set. Assuming [x1] as the names of active fields
Jun 15, 2017 12:48:01 PM org.jpmml.sklearn.Main run
SEVERE: Failed to convert
java.lang.IllegalArgumentException: The tokenizer object (null) is not Splitter
 at sklearn.feature_extraction.text.CountVectorizer.getTokenizer(CountVectorizer.java:263)
 at sklearn.feature_extraction.text.CountVectorizer.encodeDefineFunction(CountVectorizer.java:164)
 at sklearn.feature_extraction.text.CountVectorizer.encodeFeatures(CountVectorizer.java:124)
 at sklearn.pipeline.Pipeline.encodeFeatures(Pipeline.java:93)
 at sklearn2pmml.PMMLPipeline.encodePMML(PMMLPipeline.java:122)
 at org.jpmml.sklearn.Main.run(Main.java:144)
 at org.jpmml.sklearn.Main.main(Main.java:93)

Exception in thread "main" java.lang.IllegalArgumentException: The tokenizer object (null) is not Splitter
 at sklearn.feature_extraction.text.CountVectorizer.getTokenizer(CountVectorizer.java:263)
 at sklearn.feature_extraction.text.CountVectorizer.encodeDefineFunction(CountVectorizer.java:164)
 at sklearn.feature_extraction.text.CountVectorizer.encodeFeatures(CountVectorizer.java:124)
 at sklearn.pipeline.Pipeline.encodeFeatures(Pipeline.java:93)
 at sklearn2pmml.PMMLPipeline.encodePMML(PMMLPipeline.java:122)
 at org.jpmml.sklearn.Main.run(Main.java:144)
 at org.jpmml.sklearn.Main.main(Main.java:93)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Data\Anaconda2\lib\site-packages\sklearn2pmml\__init__.py", line 142, in sklearn2pmml
 raise RuntimeError("The JPMML-SkLearn conversion application has failed. The Java process should have printed more information about the failure into its standard output and/or error streams")
RuntimeError: The JPMML-SkLearn conversion application has failed. The Java process should have printed more information about the failure into its standard output and/or error streams

Answer 1

您需要使用与 PMML 兼容的文本标记化功能。默认实现是类sklearn2pmml.feature_extraction.text.Splitter：

from sklearn.feature_extraction.text import TfidfVectorizer 
from sklearn2pmml.feature_extraction.text import Splitter
vectorizer = TfidfVectorizer(analyzer = "word", token_pattern = None, tokenizer = Splitter())

JPMML 邮件列表中提供了更多详细信息和参考资料：https://groups.google.com/forum/#!topic/jpmml/wi-0rxzUn1o