如何使用python将文件夹从本地上传到GCP存储桶答案

【问题标题】：How to upload folder from local to GCP bucket using python如何使用python将文件夹从本地上传到GCP存储桶
【发布时间】：2022-06-21 15:41:24
【问题描述】：

我正在关注此链接并收到一些错误：

How to upload folder on Google Cloud Storage using Python API

我已将模型保存在容器环境中，我想从那里复制到 GCP 存储桶。

这是我的代码：

storage_client = storage.Client(project='*****')
def upload_local_directory_to_gcs(local_path, bucket, gcs_path):

   bucket = storage_client.bucket(bucket)

    assert os.path.isdir(local_path)
    for local_file in glob.glob(local_path + '/**'):
        
        print(local_file)


        
        print("this is bucket",bucket)
        blob = bucket.blob(gcs_path)
        print("here")
        blob.upload_from_filename(local_file)
        print("done")

path="/pythonPackage/trainer/model_mlm_demo" #this is local absolute path where my folder is. Folder name is **model_mlm_demo**
buc="py*****" #this is my GCP bucket address
gcs="model_mlm_demo2/" #this is the new folder that I want to store files in GCP

upload_local_directory_to_gcs(local_path=path, bucket=buc, gcs_path=gcs)

/pythonPackage/trainer/model_mlm_demo 里面有 3 个文件 config，model.bin 和 arguments.bin`

错误

代码没有抛出任何错误，但 GCP 存储桶中没有上传文件。它只是创建空文件夹。

【问题讨论】：

您可以分享 GCP Storage 中存储桶和空文件夹的屏幕截图吗？
@PriyashreeBhadra 上传
我在下面发布了一个答案。请检查确认。

标签： python-3.x google-cloud-platform

【解决方案1】：

我可以看到的错误是，您不需要将gs:// 作为存储桶参数传递。实际上，这是一个您可能需要查看的示例，

https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-python

def upload_blob(bucket_name, source_file_name, destination_blob_name):
    """Uploads a file to the bucket."""
    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"
    # The path to your file to upload
    # source_file_name = "local/path/to/file"
    # The ID of your GCS object
    # destination_blob_name = "storage-object-name"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(destination_blob_name)

    blob.upload_from_filename(source_file_name)

    print(
        "File {} uploaded to {}.".format(
            source_file_name, destination_blob_name
        )
    )

【讨论】：

即使我删除了 gs，我也看不到文件夹中的任何文件。奇怪的。和我上面写的代码几乎一样？
@MAC 我注意到您将目录传递给upload_from_filename，但您应该一一传递文件路径，而不是目录路径。

【解决方案2】：

我已经重现了您的问题，下面的代码 sn-p 工作正常。我已经根据您在问题中提到的文件夹和名称更新了代码。如果您有任何问题，请告诉我。

import os
import glob
from google.cloud import storage
storage_client = storage.Client(project='')

def upload_local_directory_to_gcs(local_path, bucket, gcs_path):

    bucket = storage_client.bucket(bucket)

    assert os.path.isdir(local_path)
    for local_file in glob.glob(local_path + '/**'):

        print(local_file)

        print("this is bucket", bucket)
        filename=local_file.split('/')[-1]
        blob = bucket.blob(gcs_path+filename)
        print("here")
        blob.upload_from_filename(local_file)
        print("done")


# this is local absolute path where my folder is. Folder name is **model_mlm_demo**
path = "/pythonPackage/trainer/model_mlm_demo"
buc = "py*****"  # this is my GCP bucket address
gcs = "model_mlm_demo2/"  # this is the new folder that I want to store files in GCP

upload_local_directory_to_gcs(local_path=path, bucket=buc, gcs_path=gcs)

【讨论】：

【解决方案3】：

我想出了一种使用子流程在 GCP 存储桶中上传模型工件的方法。

import subprocess

subprocess.call('gsutil cp -r source_folder_in_local gs://*****/folder_name', shell=True, stdout=subprocess.PIPE)

如果没有安装 gsutil。您可以使用此链接安装：

https://cloud.google.com/storage/docs/gsutil_install

【讨论】：

【解决方案4】：

这会在我们每次运行 mlfow 实验时复制所有文件夹。我只想上传由 mlflow 运行生成的最新工件。

这适用于 aws 但不适用于 gcp ： def 上传（s3_bucket_name =None，mlruns_direc=None）：尝试：输出 = subprocess.run(["aws", "s3", "sync", "{}".format(mlruns_direc), "s3://{}".format(s3_bucket_name)], 标准输出=子进程.PIPE，编码='utf-8') print("\n保存到桶：", s3_bucket_name) return f"完成上传：{output.stdout}"

except Exception as e:
    return f"Error Occured while uploading :{e.__str__()}"

【讨论】：