Spark-nlp：无法从 pyspark 中的磁盘加载预训练的识别实体模型

Question

我设置了一个 spark 集群，并希望集成 spark-nlp 来运行命名实体识别。我需要从磁盘访问模型，而不是在运行时从 Internet 下载它。我已经从模型下载页面下载了recognize_entities_dl 模型，并将解压缩的文件放在 spark 应该 能够访问它的位置。当我运行以下代码时：

ner = NerDLModel.pretrained('/path/to/unzipped/files')

我看到Can not find the model to download please check the name! 消息，表明它找不到文件，后面跟着代码中的堆栈跟踪。我也尝试了PretrainedPipeline 类，结果相似。

关于它们的价值的一些重要细节：

火花版本：2.4.4

sparknlp 版本：2.3.3

Spark 在 Kubernetes pod 内的 docker 容器中运行。我可以执行到这个容器中并手动运行命令来重现问题。看起来_internal._GetResourceSize 正在返回-1，导致加载程序退出。我也收到了一些关于 http 的警告，但我想做的只是访问一个本地文件，所以不确定这与事情有什么关系：

>>> _internal._GetResourceSize('/path/in/container/recognize_entities_dl_en_2.1.0_2.4_1562946909722', 'en', remote_loc=None).apply()
19/12/02 20:29:03 WARN ApacheUtils: NoSuchMethodError was thrown when disabling normalizeUri. This indicates you are using an old version (< 4.5.8) of Apache http client. It is recommended to use http client version >= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information
19/12/02 20:29:03 WARN ApacheUtils: NoSuchMethodError was thrown when disabling normalizeUri. This indicates you are using an old version (< 4.5.8) of Apache http client. It is recommended to use http client version >= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information
'-1'
>>>

Answer 1

您正在尝试在注释器中加载预先训练的管道。有两种类型的预训练资源：模型和管道。预训练的模型可以加载到注释器中，稍后将在管道中使用，但是，预训练的管道可以轻松加载并在之后使用。

• 预训练管道示例（在线 - 需要互联网）：

import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP

SparkNLP.version()

val testData = spark.createDataFrame(Seq(
(1, "Google has announced the release of a beta version of the popular TensorFlow machine learning library"),
(2, "Donald John Trump (born June 14, 1946) is the 45th and current president of the United States")
)).toDF("id", "text")

// Pay attention, for loading a pre-trained pipeline we use PretrainedPipeline
val pipeline = PretrainedPipeline("recognize_entities_dl", lang="en")

val annotation = pipeline.transform(testData)

annotation.show()

/*
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP
2.4.0
testData: org.apache.spark.sql.DataFrame = [id: int, text: string]
pipeline: com.johnsnowlabs.nlp.pretrained.PretrainedPipeline = PretrainedPipeline(entity_recognizer_dl,en,public/models)
annotation: org.apache.spark.sql.DataFrame = [id: int, text: string ... 6 more fields]
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
| id|                text|            document|            sentence|               token|          embeddings|                 ner|       entities|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
|  1|Google has announ...|[[document, 0, 10...|[[document, 0, 10...|[[token, 0, 5, Go...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 5, Go...|
|  2|Donald John Trump...|[[document, 0, 92...|[[document, 0, 92...|[[token, 0, 5, Do...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 16, D...|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
*/

annotation.select("entities.result").show(false)

/*
+----------------------------------+
|result                            |
+----------------------------------+
|[Google, TensorFlow]              |
|[Donald John Trump, United States]|
+----------------------------------+
*/

• 预训练管道示例（离线加载保存的管道）：

import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP

SparkNLP.version()

val testData = spark.createDataFrame(Seq(
(1, "Google has announced the release of a beta version of the popular TensorFlow machine learning library"),
(2, "Donald John Trump (born June 14, 1946) is the 45th and current president of the United States")
)).toDF("id", "text")

// Here we are loading a pre-trained pipeline we already downloaded manually for offline use

val pipeline = PretrainedPipeline.load("/path/in/container/recognize_entities_dl_en_2.1.0_2.4_1562946909722")

val annotation = pipeline.transform(testData)

annotation.show()

/*
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP
2.4.0
testData: org.apache.spark.sql.DataFrame = [id: int, text: string]
pipeline: com.johnsnowlabs.nlp.pretrained.PretrainedPipeline = PretrainedPipeline(entity_recognizer_dl,en,public/models)
annotation: org.apache.spark.sql.DataFrame = [id: int, text: string ... 6 more fields]
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
| id|                text|            document|            sentence|               token|          embeddings|                 ner|       ner_converter|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
|  1|Google has announ...|[[document, 0, 10...|[[document, 0, 10...|[[token, 0, 5, Go...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 5, Go...|
|  2|Donald John Trump...|[[document, 0, 92...|[[document, 0, 92...|[[token, 0, 5, Do...|[[word_embeddings...|[[named_entity, 0...|[[chunk, 0, 16, D...|
+---+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
*/

annotation.select("entities.result").show(false)

/*
+----------------------------------+
|result                            |
+----------------------------------+
|[Google, TensorFlow]              |
|[Donald John Trump, United States]|
+----------------------------------+
*/

• 为 NerDLModel 加载预训练模型的示例

// Online
val ner = NerDLModel.pretrained(name="ner_dl", lang="en")
// Offline - manualy downloaded
val ner = NerDLModel.load("/path/ner_dl_en_2.4.0_2.4_1580251789753")

如果您对输入数据有任何疑问或问题，请告诉我，我会更新答案。

参考文献：

• https://github.com/JohnSnowLabs/spark-nlp
• https://nlp.johnsnowlabs.com/models
• https://github.com/JohnSnowLabs/spark-nlp-workshop