Cloud-ML Job 没有这样的文件或目录答案

【问题标题】：Cloud-ML Job No such file or directoryCloud-ML Job 没有这样的文件或目录
【发布时间】：2019-04-08 21:40:14
【问题描述】：

我已向 cloud ml 提交了一个培训作业。但是，它找不到 csv 文件。它在桶里。这是代码。

# Use scikit-learn to grid search the batch size and epochs
import numpy
from sklearn.model_selection import GridSearchCV
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier

def create_model():
    model = Sequential()
    model.add(Dense(12, input_dim=11, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='nadam', metrics=['accuracy'])
    return model
seed = 7
numpy.random.seed(seed)

FIL = "gs://bubbly-hexagon-112008-ml/dataset/mixed.csv"
dataset = numpy.loadtxt(FIL, delimiter=",")
X = dataset[:,0:11]
Y = dataset[:,11]

model = KerasClassifier(build_fn=create_model, verbose=1)
batch_size = [10, 20, 40, 60, 80, 100]
epochs = [10, 50, 100, 500, 1000]
param_grid = dict(batch_size=batch_size, nb_epoch=epochs)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1)
grid_result = grid.fit(X, Y)
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

提交作业后我收到此错误。

Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in
run_globals File "/root/.local/lib/python2.7/
site-packages/trainer/task.py", line 18, in <module> dataset = numpy.loadtxt(FIL, delimiter=",") File "/root/.local/lib/python2.7/
site-packages/numpy/lib/npyio.py", line 803, in loadtxt fh = iter(open(fname, 'U')) IOError: [Errno 2] No such file or directory: 
'gs://bubbly-hexagon-112008-ml/dataset/mixed.csv'

-文件在指定的bucket中，权限包括cloud ml作为reader。

-我还使用gcloud beta ml init-project 来初始化项目。

-我创建了一个新存储桶并将文件放入其中，但出现了同样的错误。

-我的存储桶与我提交的作业位于同一区域。

谢谢

【问题讨论】：

标签： google-cloud-ml

【解决方案1】：

来自 tensorflow 的file_io 效果很好：

from tensorflow.python.lib.io import file_io
import numpy as np
import json

读取一个numpy数组：

with file_io.FileIO(path_npx, 'rb') as f:
    np_arr = np.load( BytesIO(f.read()) )
    print(np_arr)

读取 json 文件：

with file_io.FileIO(path_json, 'r') as f:
    print(json.loads(f.read()))

【讨论】：

【解决方案2】：

你不能直接从 gfs 中读取，你需要使用某种 io 库。

from io import BytesIO
import tensorflow as tf
import numpy as np
from tensorflow.python.lib.io import file_io

FIL = "gs://bubbly-hexagon-112008-ml/dataset/mixed.csv"
f = BytesIO(file_io.read_file_to_string(FIL, binary_mode=True))
data = np.load(f)

【讨论】：

【解决方案3】：

我不认为你可以直接用 numpy 读取 gcs 文件。

【讨论】：