如何在自定义路径中保存/加载 tensorflow 集线器模块？

Question

tensorflow_hub 库维护者让用户可以轻松下载和使用预训练的 tensorflow 模块，例如：

import tensorflow_hub as hub

embed = hub.Module("https://tfhub.dev/google/universal-sentence-encoder/1")

但从sys.stderr 看来，它似乎是将模块本地保存到一个临时目录，即

信息：张量流：使用 /var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules 到 缓存模块。信息：张量流：初始化变量 模块/Embeddings_en/sharded_0:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_0 INFO:tensorflow:Initialize variable 模块/Embeddings_en/sharded_1:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_1 INFO:tensorflow:Initialize variable 模块/Embeddings_en/sharded_10:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Embeddings_en/sharded_10 INFO:tensorflow:Initialize 变量 模块/Embeddings_en/sharded_11:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Embeddings_en/sharded_11 INFO:tensorflow:Initialize 变量 模块/Embeddings_en/sharded_12:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Embeddings_en/sharded_12 INFO:tensorflow:Initialize 变量 模块/Embeddings_en/sharded_13:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_13 INFO:tensorflow:Initialize variable 模块/Embeddings_en/sharded_14:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Embeddings_en/sharded_14 INFO:tensorflow:Initialize 变量 模块/Embeddings_en/sharded_15:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Embeddings_en/sharded_15 INFO:tensorflow:Initialize 变量 模块/Embeddings_en/sharded_16:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Embeddings_en/sharded_16 INFO:tensorflow:Initialize 变量 模块/Embeddings_en/sharded_2:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_2 INFO:tensorflow:Initialize variable 模块/Embeddings_en/sharded_3:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_3 INFO:tensorflow:Initialize variable 模块/Embeddings_en/sharded_4:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_4 INFO:tensorflow:Initialize variable 模块/Embeddings_en/sharded_5:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_5 INFO:tensorflow:Initialize variable 模块/Embeddings_en/sharded_6:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Embeddings_en/sharded_6 INFO:tensorflow:Initialize 变量 模块/Embeddings_en/sharded_7:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_7 INFO:tensorflow:Initialize variable 模块/Embeddings_en/sharded_8:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_8 INFO:tensorflow:Initialize variable 模块/Embeddings_en/sharded_9:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Embeddings_en/sharded_9 INFO:tensorflow:Initialize 变量 module/Encoder_en/DNN/ResidualHidden_​​0/weights:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Encoder_en/DNN/ResidualHidden_​​0/weights 信息：张量流：初始化变量 module/Encoder_en/DNN/ResidualHidden_​​1/weights:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Encoder_en/DNN/ResidualHidden_​​1/weights 信息：张量流：初始化变量 module/Encoder_en/DNN/ResidualHidden_​​2/weights:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Encoder_en/DNN/ResidualHidden_​​2/weights 信息：张量流：初始化变量 module/Encoder_en/DNN/ResidualHidden_​​3/projection:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Encoder_en/DNN/ResidualHidden_​​3/projection 信息：张量流：初始化变量 module/Encoder_en/DNN/ResidualHidden_​​3/weights:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 Encoder_en/DNN/ResidualHidden_​​3/weights 信息：张量流：初始化变量 模块/SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_0/bias:0 来自 检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_0bia 信息：张量流：初始化变量 模块/SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_0/weights:0 从检查站 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_0/ 信息：张量流：初始化变量 模块/SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_1/bias:0 来自 检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_1/bia 信息：张量流：初始化变量 模块/SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_1/weights:0 从检查站 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_1/ 信息：张量流：初始化变量 模块/SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_2/bias:0 来自 检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_2/bia 信息：张量流：初始化变量 模块/SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_2/weights:0 从检查站 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_2/ 信息：张量流：初始化变量 module/SNLI/Classifier/LinearLayer/bias:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 带有 SNLI/Classifier/LinearLayer/bias INFO:tensorflow:Initialize 变量模块/SNLI/Classifier/LinearLayer/weights:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 带有 SNLI/Classifier/LinearLayer/weights INFO:tensorflow:Initialize 变量 module/SNLI/Classifier/tanh_layer_0/bias:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 带有 SNLI/Classifier/tanh_layer_0/bias INFO:tensorflow:Initialize 变量 module/SNLI/Classifier/tanh_layer_0/weights:0 来自检查点 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 带有 SNLI/Classifier/tanh_layer_0/weights INFO:tensorflow:Initialize 来自检查点的变量 module/global_step:0 b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' 与 global_step

机器重启后，模块被删除，再次运行hub.Module('...') 代码将重新下载模块。

是否可以将模块保存到自定义目录，然后从自定义目录加载？

如果可能，如何在自定义路径中保存/加载 tensorflow hub 模块？

Answer 1

确实可以指定/使用自定义目录 :) 有关说明，请参阅https://www.tensorflow.org/hub/basics 上名为“缓存模块”的部分

Answer 2

您可以从 url + '?tf-hub-format=compressed' 下载您需要的模型

我尝试下载 elmo 并且成功了

url = https://tfhub.dev/google/elmo/2 + '?tf-hub-format=compressed'

例如：https://tfhub.dev/google/elmo/2?tf-hub-format=compressed

模型将作为 tar 文件下载到您的计算机。

一旦你解压文件，它就会有 tfhub_module.pb

Answer 3

对于那些仍在寻找答案的人，特别是对于 ELMO，正如@harikris 所说，您可以使用

url = https://tfhub.dev/google/elmo/2 + '?tf-hub-format=compressed'

下载模块的格式，例如this link

下载后，将内容解压缩到一个文件夹中，然后在 hub.Module 代码中只需将您的 URL 替换为目录路径，如

中所述

https://www.tensorflow.org/hub/basics

Answer 4

如果没有设置TFHUB_CACHE_DIR环境变量值，默认缓存目录为：

cache_dir = os.path.join(tempfile.gettempdir(), "tfhub_modules")

下载的集线器就在里面。

Answer 5

• 获取 URL 并将其更改为：

https://tfhub.dev/google/universal-sentence-encoder/1

收件人：

https://storage.googleapis.com/tfhub-modules/google/universal-sentence-encoder/1.tar.gz

• 使用 Curl 或浏览器下载。
• 解压到首选位置（例如 /home/admin/embed/）
• 将您的代码更改为：

.

import tensorflow_hub as hub 
embed = hub.load('/home/admin/embed/')

Answer 6

1) 找到您的模型：例如https://tfhub.dev/google/imagenet/inception_v1/feature_vector/1

2) 获取真实下载路径：

将 tfhub.dev 替换为 storage.googleapis.com/tfhub-modules 并附加 .tar.gz 作为后缀。

3) 准备缓存：

在某些平台上，TF hub 会记录缓存目录，但有些平台 惯于。在代码中指定缓存位置要多得多 可靠的。只需在调用之前将以下代码放入您的文件中 tfhub 这样做。

os.environ["TFHUB_CACHE_DIR"] = '/tmp/tfhub'

参考：How to run TF hub locally without internet connection