【问题标题】:Azure python storage block blob storage is eating all the memory upAzure python 存储块 blob 存储正在耗尽所有内存
【发布时间】:2019-08-22 14:48:05
【问题描述】:

我编写了一个 Python 脚本来自动构建 Azure VM 并从 KVM 上传到 Azure,但我遇到了一个无法解决的问题。 一旦构建了 VM,我就尝试使用 Azure Python 模块将磁盘上传到 Azure,问题是脚本实际上正在吃掉所有可用的 RAM。我尝试了几种编码方式,但总是以相同的结果结束。

   block_blob_service = BlockBlobService(vars.az_storage_acc_name, vars.az_sto_key)
    blob = open(args.pool_path + args.name + "-az"+'.vhd', 'r')
    print "Upload {} to Azure Blob service".format(args.name +"-az"+'.vhd')
    block_blob_service.create_blob_from_stream(vars.az_cnt, args.name +"-az"+'.vhd', blob)

我也尝试了以下方法:

stream = io.open('/path_to_vhd', 'rb')

BlockBlobService.create_blob_from_stream(vars.az_cnt, "test-stream.vhd", stream)

运气不好,每次启动 blob 创建但如果最终失败,因为没有可用的 RAM。

你有什么线索可以让我做到这一点吗?

【问题讨论】:

  • 根据您代码中的print 字样,您使用的是Python 2 吗?
  • 如果要使用block_blob_service.create_blob_from_stream方法上传大文件,请注意变量MIN_LARGE_BLOCK_UPLOAD_THRESHOLD,可以参考BlockBlobService的参考说明了解更多我认为对你有帮助。
  • 您是否在其他位置设置了MIN_LARGE_BLOCK_UPLOAD_THRESHOLD 变量?或设置其他变量MAX_BLOCK_SIZEMAX_SINGLE_PUT_SIZE?这将改变 sdk 操作。
  • 我实际上在使用 Python2。我不使用这个变量,但我会因为它似乎是关键解决方案;-)

标签: python azure memory storage ls


【解决方案1】:

感谢您的意见。

我不明白的是,到底有什么区别

block_blob_service.create_blob_from_stream

block_blob_service.create_blob_from_path

如果它试图将所有内容都保存在 RAM 中?

【讨论】:

  • 请参阅 create_blob_from_* 以了解使用自动分块和进度通知处理大型 blob 的创建和上传的高级函数。这两个的基本区别是一个取文件路径,另一个取已经打开的流。如果你想利用块上传的好处,请使用 MAX_BLOCK_SIZE 和 MAX_SINGLE_PUT_SIZE 也请参考本指南media.readthedocs.org/pdf/azure-storage/latest/…
  • 浏览此pdf并转到第24页了解详细说明。
  • 如果有帮助,请接受作为答案。它将帮助具有相同 ASK 的其他人。
【解决方案2】:

这需要将整个流保存在内存中,除非您的机器有最大 RAM 大小,否则此代码将无法运行,并且在某些时候会出现 systemoutofememory 异常。

我建议您以块的形式上传流,而不是一次性写入。

这是一个分块上传流的功能

def _upload_blob_chunks(blob_service, container_name, blob_name,
                        blob_size, block_size, stream, max_connections,
                        progress_callback, validate_content, lease_id, uploader_class,
                        maxsize_condition=None, if_modified_since=None, if_unmodified_since=None, if_match=None,
                        if_none_match=None, timeout=None,
                        content_encryption_key=None, initialization_vector=None, resource_properties=None):
    encryptor, padder = _get_blob_encryptor_and_padder(content_encryption_key, initialization_vector,
                                                       uploader_class is not _PageBlobChunkUploader)

    uploader = uploader_class(
        blob_service,
        container_name,
        blob_name,
        blob_size,
        block_size,
        stream,
        max_connections > 1,
        progress_callback,
        validate_content,
        lease_id,
        timeout,
        encryptor,
        padder
    )

    uploader.maxsize_condition = maxsize_condition

    # Access conditions do not work with parallelism
    if max_connections > 1:
        uploader.if_match = uploader.if_none_match = uploader.if_modified_since = uploader.if_unmodified_since = None
    else:
        uploader.if_match = if_match
        uploader.if_none_match = if_none_match
        uploader.if_modified_since = if_modified_since
        uploader.if_unmodified_since = if_unmodified_since

    if progress_callback is not None:
        progress_callback(0, blob_size)

    if max_connections > 1:
        import concurrent.futures
        from threading import BoundedSemaphore

        '''
        Ensures we bound the chunking so we only buffer and submit 'max_connections' amount of work items to the executor.
        This is necessary as the executor queue will keep accepting submitted work items, which results in buffering all the blocks if
        the max_connections + 1 ensures the next chunk is already buffered and ready for when the worker thread is available.
        '''
        chunk_throttler = BoundedSemaphore(max_connections + 1)

        executor = concurrent.futures.ThreadPoolExecutor(max_connections)
        futures = []
        running_futures = []

        # Check for exceptions and fail fast.
        for chunk in uploader.get_chunk_streams():
            for f in running_futures:
                if f.done():
                    if f.exception():
                        raise f.exception()
                    else:
                        running_futures.remove(f)

            chunk_throttler.acquire()
            future = executor.submit(uploader.process_chunk, chunk)

            # Calls callback upon completion (even if the callback was added after the Future task is done).
            future.add_done_callback(lambda x: chunk_throttler.release())
            futures.append(future)
            running_futures.append(future)

        # result() will wait until completion and also raise any exceptions that may have been set.
        range_ids = [f.result() for f in futures]
    else:
        range_ids = [uploader.process_chunk(result) for result in uploader.get_chunk_streams()]

    if resource_properties:
        resource_properties.last_modified = uploader.last_modified
        resource_properties.etag = uploader.etag

    return range_ids

供参考,您可以浏览下面的线程

https://github.com/Azure/azure-storage-python/blob/master/azure-storage-blob/azure/storage/blob/_upload_chunking.py

另外,同类型的请求也有类似的线程

how to transfer file to azure blob storage in chunks without writing to file using python

或者,您可以使用 powershell 将 VHD 上传到 vm 存储帐户,如下所示

$rgName = "myResourceGroup"
$urlOfUploadedImageVhd = "https://mystorageaccount.blob.core.windows.net/mycontainer/myUploadedVHD.vhd"
Add-AzVhd -ResourceGroupName $rgName -Destination $urlOfUploadedImageVhd `
    -LocalFilePath "C:\Users\Public\Documents\Virtual hard disks\myVHD.vhd"

这是相同的参考

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/upload-generalized-managed

希望对你有帮助。

【讨论】:

    猜你喜欢
    • 2011-05-06
    • 1970-01-01
    • 2021-08-31
    • 2017-11-10
    • 2017-12-14
    • 1970-01-01
    • 1970-01-01
    • 2015-08-22
    • 2019-07-28
    相关资源
    最近更新 更多