.net core大文件上传方法

Question

如何使用asp.net core web api上传大文件？超过 500mb 时，会出现以下异常消息。我还设置了“COMPlus_gcAllowVeryLargeObjects”环境变量和启动。

services.Configure<FormOptions>(options =>
    {
        options.BufferBodyLengthLimit = Int64.MaxValue;
        options.MemoryBufferThreshold = Int32.MaxValue;
        options.MultipartBodyLengthLimit = long.MaxValue;
        options.MultipartBoundaryLengthLimit = int.MaxValue;
        options.MultipartHeadersLengthLimit = Int32.MaxValue;
    });

下面是我的代码。有什么需要设置的吗？

[HttpPost("{id}")] 
[RequestFormLimits(MultipartBodyLengthLimit = long.MaxValue)]
[DisableRequestSizeLimit]
public ActionResult UploadLargeFiles(string id, [FromForm]IFormFile files)
{
    try
    {
        string fileName = files.FileName;
        int fileSize = Convert.ToInt32(files.Length);

        var uploadProvider = new JObject();
        var res = new JArray();

        var isExistence = _mailService.GetUploadFolder(id);
        if (isExistence != HttpStatusCode.OK)
        {
            var createFolder = _mailService.CreateUploadFolder(id);
            if (createFolder != HttpStatusCode.Created)
            {
                ModelState.AddModelError("OneDriveFolderError", "");
                return BadRequest(ModelState);
            }
        }
        if (files.Length > 0)
        {
            byte[] data = new byte[fileSize];

            var uploadSessionUrl = _mailService.CreateUploadSession(id, fileName);
            if (uploadSessionUrl != null)
            {
                uploadProvider = _mailService.UploadByteFile(id, uploadSessionUrl, data, fileName, fileSize);
                res.Add(uploadProvider);

                Array.Fill(data, (byte)0);
            }
            else
            {
                ModelState.AddModelError("sessionFail", "");
                return BadRequest(ModelState);
            }
        }

        var Link = this.SaveFileDownloadLink(res);
        return Ok(Link);
    }
    catch (ArgumentNullException e)
    {
        return NotFound(e.Message);
    }
}

以字节格式将文件上传到 OneDrive。 一个字节可以容纳 2 GB 或更多吗？

public JObject LargeFileUpload(string upn, string url, byte[] file, string fileName, int fileSize)
{
    int fragSize = 4 * 1024 * 1024; //4MB => 4 * 1024 * 1024;
    var byteRemaining = fileSize;
    var numFragments = ( byteRemaining / fragSize ) + 1;
    int i = 0;
    var responseCode = HttpStatusCode.OK;
    var jObject = new JObject();

    while (i < numFragments)
    {
        var chunkSize = fragSize;
        var start = i * fragSize;
        var end = i * fragSize + chunkSize - 1;
        var offset = i * fragSize;

        if (byteRemaining < chunkSize) {
            chunkSize = byteRemaining;
            end = fileSize - 1;
        }

        var contentRange = " bytes " + start + "-" + end + "/" + fileSize;

        using (var client = new HttpClient())
        {
            var content = new ByteArrayContent(file);
            content.Headers.Add("Content-Length", chunkSize.ToString());
            content.Headers.Add("Content-Range", contentRange);

            var response = client.PutAsync(url, content);
            var strData = response.Result.Content.ReadAsStringAsync().Result;
            responseCode = response.Result.StatusCode;

            
            if (responseCode == HttpStatusCode.Created)
            {
                JObject data = JObject.Parse(strData);
                string downloadUrl = data["@content.downloadUrl"].ToString();
                string itemId = data["id"].ToString();

                
                fileSize = fileSize / 1000;
                jObject = JObject.FromObject(new { name = fileName, id = itemId, url = downloadUrl, size = (double)fileSize });
            }
            
            else if (responseCode == HttpStatusCode.Conflict)
            {
                var restart = RestartByteFile(upn, url, fileName);
                responseCode = restart;
            }
        }
        byteRemaining = byteRemaining - chunkSize;
        i++;
    }

    if (responseCode == HttpStatusCode.Created) { return jObject; }
    else return jObject = JObject.FromObject(new { result = "Fail" });
}

Answer 1

所以，你的问题很明显。您正在尝试上传 500+MB 的文件。在您的控制器中，您正在获取文件的大小，然后执行以下操作：

byte[] data = new byte[fileSize];

您不想这样做...IFormFile 已经为您提供了一个可以读取的Stream。具体来说，任何从 IFormFile 实现的东西都必须实现这些方法：

void CopyTo(Stream target);
Task CopyToAsync(Stream target, CancellationToken cancellationToken = default(CancellationToken));
Stream OpenReadStream();

那么，如果你有一个流准备好了，你为什么要将该流复制到内存中？

解决此问题的快速而肮脏的方法是将IFormFile 转储到磁盘。然后打开磁盘上的文件并将其移动/流式传输到它需要去的地方：

[HttpPost("{id}")] 
[DisableRequestSizeLimit]
public async Task<IActionResult> UploadLargeFiles(string id, [FromForm]IFormFile file)
{
    FileInfo fi = null;
    try
    {
        // this is from my code, but you want to store this
        // somewhere on disk that works with your hosting setup.
        fi = _fileStorageService.GetTempFile();

        // open a stream for writing and copy it over
        using (var s = fi.OpenWrite())
        {
            await file.CopyToAsync(s).ConfigureAwait(false);
        }
    }
    catch { fi?.Delete(); fi = null; }
    
    if(fi == null) { /* return a 500 error */ }

    // TODO: At this point, you have a file that you can open and
    // stream or move to where ever you want

    // Make sure that when you are done with the file, that you
    // delete it if you no longer need it.
}

所以现在您拥有该文件，它在您的手中。您现在可以将其发送到 BackgoundService 或其他 HostedService ——甚至发送到另一个瞬态、作用域或单例服务进行处理。天空才是极限。

此方法可能不是最有效的，因为您实际上是将文件从磁盘的一部分复制到另一部分。

您可以走捷径，将OpenReadStream 或IFormFile 返回的流传递给一个进程，该进程将该流发送到另一个服务，有点像我们使用CopyToAsync() 将它发送到磁盘上的文件。

请记住，一旦此端点完成，IFormFile 将被释放。因此，您必须在此执行的上下文中完成处理。

归根结底，无论您最终做什么，都不要将文件读入内存。它已经驻留在IFormFile 中。没有理由在内存中再次复制它。