【问题标题】:"Chunked" MemoryStream“分块”内存流
【发布时间】:2009-07-29 21:20:42
【问题描述】:

我正在寻找 MemoryStream 的实现,它不会将内存分配为一个大块,而是一组块。我想在内存中存储几 GB 的数据(64 位)并避免内存碎片的限制。

【问题讨论】:

  • 你不需要内存映射文件吗?
  • RAM 中的磁盘之类的东西?
  • 他只想要一个 MemoryStream,它不会在一个大的连续块中分配所需的内存。可以理解,出于同样的原因(碎片),我不得不在 C 中围绕内存池编写一个包装器。但我还没有在 C# 中看到过。
  • 另一个原因是 MemoryStream 有 2gb 的限制。另外值得一提的是,任何超过 85k 的数组都会卡在大型对象堆上,这可能会导致碎片。
  • @nos 我很惊讶每个人似乎都错过了msdn.microsoft.com/en-us/library/…,这是解决这个问题的完美选择。

标签: c# .net


【解决方案1】:

您需要先确定是否是虚拟地址碎片问题。

如果你使用的是 64 位机器(你似乎表明你是),我严重怀疑它是。每个 64 位进程几乎都有可用的整个 64 位虚拟内存空间,您唯一担心的是虚拟地址空间碎片而不是物理内存碎片(这是操作系统必须担心的)。操作系统内存管理器已经在后台对内存进行了分页。在可预见的未来,您不会在物理内存用完之前用完虚拟地址空间。在我们都退休之前,这不太可能发生变化。

如果你有一个 32 位地址空间,那么在 GB ramge 中分配连续的大块内存,你很快就会遇到碎片问题。 CLR 中没有库存块分配内存流。在 ASP.NET 的底层有一个(出于其他原因),但它是不可访问的。如果您必须走这条路,您最好还是自己编写一个,因为您的应用程序的使用模式不太可能与许多其他应用程序相似,并且尝试将您的数据放入 32 位地址空间可能会成为您的性能瓶颈。

如果您要处理 GB 的数据,我强烈建议您使用 64 位进程。无论您有多聪明,它都会比 32 位地址空间碎片的手动解决方案做得更好。

【讨论】:

  • 谢谢chuckj,你说得对。我正在运行 64 位 ASP.NET 应用程序,通常这不是问题。但出于开发目的(32 位 VS 内部 Web 服务器),它会自动回退到 32 位模式,我只想加载 1.2 GB 流。我已经实现了一个定制的解决方案(基本上根据业务标准将 MemoryStream 拆分为许多较小的流),这成功了。
【解决方案2】:

类似这样的:

class ChunkedMemoryStream : Stream
{
    private readonly List<byte[]> _chunks = new List<byte[]>();
    private int _positionChunk;
    private int _positionOffset;
    private long _position;

    public override bool CanRead
    {
        get { return true; }
    }

    public override bool CanSeek
    {
        get { return true; }
    }

    public override bool CanWrite
    {
        get { return true; }
    }

    public override void Flush() { }

    public override long Length
    {
        get { return _chunks.Sum(c => c.Length); }
    }

    public override long Position
    {
        get
        {
            return _position;
        }
        set
        {
            _position = value;

            _positionChunk = 0;

            while (_positionOffset != 0)
            {
                if (_positionChunk >= _chunks.Count)
                    throw new OverflowException();

                if (_positionOffset < _chunks[_positionChunk].Length)
                    return;

                _positionOffset -= _chunks[_positionChunk].Length;
                _positionChunk++;
            }
        }
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        int result = 0;
        while ((count != 0) && (_positionChunk != _chunks.Count))
        {
            int fromChunk = Math.Min(count, _chunks[_positionChunk].Length - _positionOffset);
            if (fromChunk != 0)
            {
                Array.Copy(_chunks[_positionChunk], _positionOffset, buffer, offset, fromChunk);
                offset += fromChunk;
                count -= fromChunk;
                result += fromChunk;
                _position += fromChunk;
            }

            _positionOffset = 0;
            _positionChunk++;
        }
        return result;
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        long newPos = 0;

        switch (origin)
        {
            case SeekOrigin.Begin:
                newPos = offset;
                break;
            case SeekOrigin.Current:
                newPos = Position + offset;
                break;
            case SeekOrigin.End:
                newPos = Length - offset;
                break;
        }

        Position = Math.Max(0, Math.Min(newPos, Length));
        return newPos;
    }

    public override void SetLength(long value)
    {
        throw new NotImplementedException();
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        while ((count != 0) && (_positionChunk != _chunks.Count))
        {
            int toChunk = Math.Min(count, _chunks[_positionChunk].Length - _positionOffset);
            if (toChunk != 0)
            {
                Array.Copy(buffer, offset, _chunks[_positionChunk], _positionOffset, toChunk);
                offset += toChunk;
                count -= toChunk;
                _position += toChunk;
            }

            _positionOffset = 0;
            _positionChunk++;
        }

        if (count != 0)
        {
            byte[] chunk = new byte[count];
            Array.Copy(buffer, offset, chunk, 0, count);
            _chunks.Add(chunk);
            _positionChunk = _chunks.Count;
            _position += count;
        }
    }
}

class Program
{
    static void Main(string[] args)
    {
        ChunkedMemoryStream cms = new ChunkedMemoryStream();

        Debug.Assert(cms.Length == 0);
        Debug.Assert(cms.Position == 0);

        cms.Position = 0;

        byte[] helloworld = Encoding.UTF8.GetBytes("hello world");

        cms.Write(helloworld, 0, 3);
        cms.Write(helloworld, 3, 3);
        cms.Write(helloworld, 6, 5);

        Debug.Assert(cms.Length == 11);
        Debug.Assert(cms.Position == 11);

        cms.Position = 0;

        byte[] b = new byte[20];
        cms.Read(b, 3, (int)cms.Length);
        Debug.Assert(b.Skip(3).Take(11).SequenceEqual(helloworld));

        cms.Position = 0;
        cms.Write(Encoding.UTF8.GetBytes("seeya"), 0, 5);

        Debug.Assert(cms.Length == 11);
        Debug.Assert(cms.Position == 5);

        cms.Position = 0;
        cms.Read(b, 0, (byte) cms.Length);
        Debug.Assert(b.Take(11).SequenceEqual(Encoding.UTF8.GetBytes("seeya world")));

        Debug.Assert(cms.Length == 11);
        Debug.Assert(cms.Position == 11);

        cms.Write(Encoding.UTF8.GetBytes(" again"), 0, 6);

        Debug.Assert(cms.Length == 17);
        Debug.Assert(cms.Position == 17);

        cms.Position = 0;
        cms.Read(b, 0, (byte)cms.Length);
        Debug.Assert(b.Take(17).SequenceEqual(Encoding.UTF8.GetBytes("seeya world again")));

    }
}
【解决方案3】:

Bing 团队发布了RecyclableMemoryStream 并写了关于它的here。他们列举的好处是:

  1. 使用池化缓冲区消除大对象堆分配
  2. 第 2 代 GC 发生的次数要少得多,并且由于 GC 而暂停的时间要少得多
  3. 通过限定池大小来避免内存泄漏
  4. 避免内存碎片
  5. 提供出色的可调试性
  6. 提供性能跟踪指标

【讨论】:

  • 遗憾的是 RecyclableMemoryStream 仅适用于 .Net Framework 4.5
【解决方案4】:

我在我的应用程序中发现了类似的问题。我已经阅读了大量的压缩数据,并且在使用 MemoryStream 时遇到了 OutOfMemoryException。我已经基于字节数组的集合编写了自己的“分块”内存流实现。如果你知道如何让这个内存流更有效,请写信告诉我。

    public sealed class ChunkedMemoryStream : Stream
{
    #region Constants

    private const int BUFFER_LENGTH = 65536;
    private const byte ONE = 1;
    private const byte ZERO = 0;

    #endregion

    #region Readonly & Static Fields

    private readonly Collection<byte[]> _chunks;

    #endregion

    #region Fields

    private long _length;

    private long _position;
    private const byte TWO = 2;

    #endregion

    #region C'tors

    public ChunkedMemoryStream()
    {
        _chunks = new Collection<byte[]> { new byte[BUFFER_LENGTH], new byte[BUFFER_LENGTH] };
        _position = ZERO;
        _length = ZERO;
    }

    #endregion

    #region Instance Properties

    public override bool CanRead
    {
        get { return true; }
    }

    public override bool CanSeek
    {
        get { return true; }
    }

    public override bool CanWrite
    {
        get { return true; }
    }

    public override long Length
    {
        get { return _length; }
    }

    public override long Position
    {
        get { return _position; }
        set
        {
            if (!CanSeek)
                throw new NotSupportedException();

            _position = value;

            if (_position > _length)
                _position = _length - ONE;
        }
    }


    private byte[] CurrentChunk
    {
        get
        {
            long positionDividedByBufferLength = _position / BUFFER_LENGTH;
            var chunkIndex = Convert.ToInt32(positionDividedByBufferLength);
            byte[] chunk = _chunks[chunkIndex];
            return chunk;
        }
    }

    private int PositionInChunk
    {
        get
        {
            int positionInChunk = Convert.ToInt32(_position % BUFFER_LENGTH);
            return positionInChunk;
        }
    }

    private int RemainingBytesInCurrentChunk
    {
        get
        {
            Contract.Ensures(Contract.Result<int>() > ZERO);
            int remainingBytesInCurrentChunk = CurrentChunk.Length - PositionInChunk;
            return remainingBytesInCurrentChunk;
        }
    }

    #endregion

    #region Instance Methods

    public override void Flush()
    {
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        if (offset + count > buffer.Length)
            throw new ArgumentException();

        if (buffer == null)
            throw new ArgumentNullException();

        if (offset < ZERO || count < ZERO)
            throw new ArgumentOutOfRangeException();

        if (!CanRead)
            throw new NotSupportedException();

        int bytesToRead = count;
        if (_length - _position < bytesToRead)
            bytesToRead = Convert.ToInt32(_length - _position);

        int bytesreaded = 0;
        while (bytesToRead > ZERO)
        {
            // get remaining bytes in current chunk
            // read bytes in current chunk
            // advance to next position
            int remainingBytesInCurrentChunk = RemainingBytesInCurrentChunk;
            if (remainingBytesInCurrentChunk > bytesToRead)
                remainingBytesInCurrentChunk = bytesToRead;
            Array.Copy(CurrentChunk, PositionInChunk, buffer, offset, remainingBytesInCurrentChunk);
            //move position in source
            _position += remainingBytesInCurrentChunk;
            //move position in target
            offset += remainingBytesInCurrentChunk;
            //bytesToRead is smaller
            bytesToRead -= remainingBytesInCurrentChunk;
            //count readed bytes;
            bytesreaded += remainingBytesInCurrentChunk;
        }
        return bytesreaded;
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        switch (origin)
        {
            case SeekOrigin.Begin:
                Position = offset;
                break;
            case SeekOrigin.Current:
                Position += offset;
                break;
            case SeekOrigin.End:
                Position = Length + offset;
                break;
        }
        return Position;
    }

    private long Capacity
    {
        get
        {
            int numberOfChunks = _chunks.Count;


            long capacity = numberOfChunks * BUFFER_LENGTH;
            return capacity;
        }
    }

    public override void SetLength(long value)
    {
        if (value > _length)
        {
            while (value > Capacity)
            {
                var item = new byte[BUFFER_LENGTH];
                _chunks.Add(item);
            }
        }
        else if (value < _length)
        {
            var decimalValue = Convert.ToDecimal(value);
            var valueToBeCompared = decimalValue % BUFFER_LENGTH == ZERO ? Capacity : Capacity - BUFFER_LENGTH;
            //remove data chunks, but leave at least two chunks
            while (value < valueToBeCompared && _chunks.Count > TWO)
            {
                byte[] lastChunk = _chunks.Last();
                _chunks.Remove(lastChunk);
            }
        }
        _length = value;
        if (_position > _length - ONE)
            _position = _length == 0 ? ZERO : _length - ONE;
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        if (!CanWrite)
            throw new NotSupportedException();

        int bytesToWrite = count;

        while (bytesToWrite > ZERO)
        {
            //get remaining space in current chunk
            int remainingBytesInCurrentChunk = RemainingBytesInCurrentChunk;

            //if count of bytes to be written is fewer than remaining
            if (remainingBytesInCurrentChunk > bytesToWrite)
                remainingBytesInCurrentChunk = bytesToWrite;

            //if remaining bytes is still greater than zero
            if (remainingBytesInCurrentChunk > ZERO)
            {
                //write remaining bytes to current Chunk

                Array.Copy(buffer, offset, CurrentChunk, PositionInChunk, remainingBytesInCurrentChunk);

                //change offset of source array
                offset += remainingBytesInCurrentChunk;
                //change bytes to write
                bytesToWrite -= remainingBytesInCurrentChunk;
                //change length and position
                _length += remainingBytesInCurrentChunk;
                _position += remainingBytesInCurrentChunk;
            }

            if (Capacity == _position)
                _chunks.Add(new byte[BUFFER_LENGTH]);
        }
    }

    /// <summary>
    ///     Gets entire content of stream regardless of Position value and return output as byte array
    /// </summary>
    /// <returns>byte array</returns>
    public byte[] ToArray()
    {
        var outputArray = new byte[Length];
        if (outputArray.Length != ZERO)
        {
            long outputPosition = ZERO;
            foreach (byte[] chunk in _chunks)
            {
                var remainingLength = (Length - outputPosition) > chunk.Length
                                          ? chunk.Length
                                          : Length - outputPosition;
                Array.Copy(chunk, ZERO, outputArray, outputPosition, remainingLength);
                outputPosition = outputPosition + remainingLength;
            }
        }
        return outputArray;
    }

    /// <summary>
    ///     Method set Position to first element and write entire stream to another
    /// </summary>
    /// <param name="stream">Target stream</param>
    public void WriteTo(Stream stream)
    {
        Contract.Requires(stream != null);

        Position = ZERO;
        var buffer = new byte[BUFFER_LENGTH];
        int bytesReaded;
        do
        {
            bytesReaded = Read(buffer, ZERO, BUFFER_LENGTH);
            stream.Write(buffer, ZERO, bytesReaded);
        } while (bytesReaded > ZERO);
    }

    #endregion
}

【讨论】:

    【解决方案5】:

    这是一个完整的实现:

    /// <summary>
    /// Defines a MemoryStream that does not sit on the Large Object Heap, thus avoiding memory fragmentation.
    /// </summary>
    /// <seealso cref="Stream" />
    public sealed class ChunkedMemoryStream : Stream
    {
        /// <summary>
        /// Defines the default chunk size. Currently defined as 0x10000.
        /// </summary>
        public const int DefaultChunkSize = 0x10000; // needs to be < 85000
        private const int _lohSize = 85000;
    
        private List<byte[]> _chunks = new List<byte[]>();
        private long _position;
        private int _chunkSize;
        private int _lastChunkPos;
        private int _lastChunkPosIndex;
    
        /// <summary>
        /// Initializes a new instance of the <see cref="ChunkedMemoryStream" /> class based on the specified byte array.
        /// </summary>
        /// <param name="chunkSize">Size of the underlying chunks.</param>
        /// <param name="buffer">The array of unsigned bytes from which to create the current stream.</param>
        public ChunkedMemoryStream(int chunkSize = DefaultChunkSize, byte[] buffer = null)
        {
            FreeOnDispose = true;
            ChunkSize = chunkSize;
            _chunks.Add(new byte[chunkSize]);
            if (buffer != null)
            {
                Write(buffer, 0, buffer.Length);
                Position = 0;
            }
        }
    
        /// <summary>
        /// Gets or sets a value indicating whether to free the underlying chunks on dispose.
        /// </summary>
        /// <value>
        ///   <c>true</c> if the underlying chunks must be freed on disposal; otherwise, <c>false</c>.
        /// </value>
        public bool FreeOnDispose { get; set; }
    
        /// <summary>
        /// Releases the unmanaged resources used by the <see cref="Stream" /> and optionally releases the managed resources.
        /// </summary>
        /// <param name="disposing">true to release both managed and unmanaged resources; false to release only unmanaged resources.</param>
        protected override void Dispose(bool disposing)
        {
            if (FreeOnDispose)
            {
                if (_chunks != null)
                {
                    _chunks = null;
                    _chunkSize = 0;
                    _position = 0;
                }
            }
            base.Dispose(disposing);
        }
    
        /// <summary>
        /// When overridden in a derived class, clears all buffers for this stream and causes any buffered data to be written to the underlying device.
        /// This implementation does nothing.
        /// </summary>
        public override void Flush()
        {
            // do nothing
        }
    
        /// <summary>
        /// When overridden in a derived class, reads a sequence of bytes from the current stream and advances the position within the stream by the number of bytes read.
        /// </summary>
        /// <param name="buffer">An array of bytes. When this method returns, the buffer contains the specified byte array with the values between <paramref name="offset" /> and (<paramref name="offset" /> + <paramref name="count" /> - 1) replaced by the bytes read from the current source.</param>
        /// <param name="offset">The zero-based byte offset in <paramref name="buffer" /> at which to begin storing the data read from the current stream.</param>
        /// <param name="count">The maximum number of bytes to be read from the current stream.</param>
        /// <returns>
        /// The total number of bytes read into the buffer. This can be less than the number of bytes requested if that many bytes are not currently available, or zero (0) if the end of the stream has been reached.
        /// </returns>
        /// <exception cref="ArgumentNullException"><paramref name="buffer" /> is null.</exception>
        /// <exception cref="ArgumentOutOfRangeException"><paramref name="offset" /> or <paramref name="count" /> is negative.</exception>
        /// <exception cref="ArgumentException">The sum of <paramref name="offset" /> and <paramref name="count" /> is larger than the buffer length.</exception>
        /// <exception cref="ObjectDisposedException">Methods were called after the stream was closed.</exception>
        public override int Read(byte[] buffer, int offset, int count)
        {
            if (buffer == null)
                throw new ArgumentNullException(nameof(buffer));
    
            if (offset < 0)
                throw new ArgumentOutOfRangeException(nameof(offset));
    
            if (count < 0)
                throw new ArgumentOutOfRangeException(nameof(count));
    
            if ((buffer.Length - offset) < count)
                throw new ArgumentException(null, nameof(count));
    
            CheckDisposed();
    
            var chunkIndex = (int)(_position / ChunkSize);
            if (chunkIndex == _chunks.Count)
                return 0;
    
            var chunkPos = (int)(_position % ChunkSize);
            count = (int)Math.Min(count, Length - _position);
            if (count == 0)
                return 0;
    
            var left = count;
            var inOffset = offset;
            var total = 0;
            do
            {
                var toCopy = Math.Min(left, ChunkSize - chunkPos);
                Buffer.BlockCopy(_chunks[chunkIndex], chunkPos, buffer, inOffset, toCopy);
                inOffset += toCopy;
                left -= toCopy;
                total += toCopy;
                if ((chunkPos + toCopy) == ChunkSize)
                {
                    if (chunkIndex == (_chunks.Count - 1))
                    {
                        // last chunk
                        break;
                    }
                    chunkPos = 0;
                    chunkIndex++;
                }
                else
                {
                    chunkPos += toCopy;
                }
            }
            while (left > 0);
            _position += total;
            return total;
        }
    
        /// <summary>
        /// Reads a byte from the stream and advances the position within the stream by one byte, or returns -1 if at the end of the stream.
        /// </summary>
        /// <returns>
        /// The unsigned byte cast to an Int32, or -1 if at the end of the stream.
        /// </returns>
        /// <exception cref="ObjectDisposedException">Methods were called after the stream was closed.</exception>
        public override int ReadByte()
        {
            CheckDisposed();
            if (_position >= Length)
                return -1;
    
            var ret = _chunks[(int)(_position / ChunkSize)][_position % ChunkSize];
            _position++;
            return ret;
        }
    
        /// <summary>
        /// When overridden in a derived class, sets the position within the current stream.
        /// </summary>
        /// <param name="offset">A byte offset relative to the <paramref name="origin" /> parameter.</param>
        /// <param name="origin">A value of type <see cref="SeekOrigin" /> indicating the reference point used to obtain the new position.</param>
        /// <returns>The new position within the current stream.</returns>
        /// <exception cref="ObjectDisposedException">Methods were called after the stream was closed.</exception>
        public override long Seek(long offset, SeekOrigin origin)
        {
            CheckDisposed();
            switch (origin)
            {
                case SeekOrigin.Begin:
                    Position = offset;
                    break;
    
                case SeekOrigin.Current:
                    Position += offset;
                    break;
    
                case SeekOrigin.End:
                    Position = Length + offset;
                    break;
            }
            return Position;
        }
    
        private void CheckDisposed()
        {
            if (_chunks == null)
                throw new ObjectDisposedException(null, "Cannot access a disposed stream.");
        }
    
        /// <summary>
        /// When overridden in a derived class, sets the length of the current stream.
        /// </summary>
        /// <param name="value">The desired length of the <paramref name="value" /> stream in bytes.</param>
        /// <exception cref="ArgumentOutOfRangeException"><paramref name="value" /> is out of range.</exception>
        /// <exception cref="ObjectDisposedException">Methods were called after the stream was closed.</exception>
        public override void SetLength(long value)
        {
            CheckDisposed();
            if (value < 0)
                throw new ArgumentOutOfRangeException(nameof(value));
    
            if (value > Length)
                throw new ArgumentOutOfRangeException(nameof(value));
    
            var needed = value / ChunkSize;
            if ((value % ChunkSize) != 0)
            {
                needed++;
            }
    
            if (needed > int.MaxValue)
                throw new ArgumentOutOfRangeException(nameof(value));
    
            if (needed < _chunks.Count)
            {
                var remove = (int)(_chunks.Count - needed);
                for (var i = 0; i < remove; i++)
                {
                    _chunks.RemoveAt(_chunks.Count - 1);
                }
            }
            _lastChunkPos = (int)(value % ChunkSize);
        }
    
        /// <summary>
        /// Converts the current stream to a byte array.
        /// </summary>
        /// <returns>
        /// An array of bytes
        /// </returns>
        public byte[] ToArray()
        {
            CheckDisposed();
            var bytes = new byte[Length];
            var offset = 0;
            for (var i = 0; i < _chunks.Count; i++)
            {
                var count = (i == (_chunks.Count - 1)) ? _lastChunkPos : _chunks[i].Length;
                if (count > 0)
                {
                    Buffer.BlockCopy(_chunks[i], 0, bytes, offset, count);
                    offset += count;
                }
            }
            return bytes;
        }
    
        /// <summary>
        /// When overridden in a derived class, writes a sequence of bytes to the current stream and advances the current position within this stream by the number of bytes written.
        /// </summary>
        /// <param name="buffer">An array of bytes. This method copies <paramref name="count" /> bytes from <paramref name="buffer" /> to the current stream.</param>
        /// <param name="offset">The zero-based byte offset in <paramref name="buffer" /> at which to begin copying bytes to the current stream.</param>
        /// <param name="count">The number of bytes to be written to the current stream.</param>
        /// <exception cref="ArgumentException">The sum of <paramref name="offset" /> and <paramref name="count" /> is greater than the buffer length.</exception>
        /// <exception cref="ArgumentNullException"><paramref name="buffer" /> is null.</exception>
        /// <exception cref="ArgumentOutOfRangeException"><paramref name="offset" /> or <paramref name="count" /> is negative.</exception>
        /// <exception cref="ObjectDisposedException">Methods were called after the stream was closed.</exception>
        public override void Write(byte[] buffer, int offset, int count)
        {
            if (buffer == null)
                throw new ArgumentNullException(nameof(buffer));
    
            if (offset < 0)
                throw new ArgumentOutOfRangeException(nameof(offset));
    
            if (count < 0)
                throw new ArgumentOutOfRangeException(nameof(count));
    
            if ((buffer.Length - offset) < count)
                throw new ArgumentException(null, nameof(count));
    
            CheckDisposed();
    
            var chunkPos = (int)(_position % ChunkSize);
            var chunkIndex = (int)(_position / ChunkSize);
            if (chunkIndex == _chunks.Count)
            {
                _chunks.Add(new byte[ChunkSize]);
            }
    
            var left = count;
            var inOffset = offset;
            do
            {
                var copied = Math.Min(left, ChunkSize - chunkPos);
                Buffer.BlockCopy(buffer, inOffset, _chunks[chunkIndex], chunkPos, copied);
                inOffset += copied;
                left -= copied;
                if ((chunkPos + copied) == ChunkSize)
                {
                    chunkIndex++;
                    chunkPos = 0;
                    if (chunkIndex == _chunks.Count)
                    {
                        _chunks.Add(new byte[ChunkSize]);
                    }
                }
                else
                {
                    chunkPos += copied;
                }
            }
            while (left > 0);
    
            _position += count;
            if (chunkIndex == (_chunks.Count - 1))
            {
                if (chunkIndex > _lastChunkPosIndex || (chunkIndex == _lastChunkPosIndex && chunkPos > _lastChunkPos))
                {
                    _lastChunkPos = chunkPos;
                    _lastChunkPosIndex = chunkIndex;
                }
            }
        }
    
        /// <summary>
        /// Writes a byte to the current position in the stream and advances the position within the stream by one byte.
        /// </summary>
        /// <param name="value">The byte to write to the stream.</param>
        /// <exception cref="ObjectDisposedException">Methods were called after the stream was closed.</exception>
        public override void WriteByte(byte value)
        {
            CheckDisposed();
            var chunkIndex = (int)(_position / ChunkSize);
            var chunkPos = (int)(_position % ChunkSize);
    
            if (chunkPos > (ChunkSize - 1))
            {
                chunkIndex++;
                chunkPos = 0;
                if (chunkIndex == _chunks.Count)
                {
                    _chunks.Add(new byte[ChunkSize]);
                }
            }
    
            _chunks[chunkIndex][chunkPos++] = value;
            _position++;
            if (chunkIndex == (_chunks.Count - 1))
            {
                if (chunkIndex > _lastChunkPosIndex || (chunkIndex == _lastChunkPosIndex && chunkPos > _lastChunkPos))
                {
                    _lastChunkPos = chunkPos;
                    _lastChunkPosIndex = chunkIndex;
                }
            }
        }
    
        /// <summary>
        /// Writes to the specified stream.
        /// </summary>
        /// <param name="stream">The stream.</param>
        /// <exception cref="ArgumentNullException"><paramref name="stream" /> is null.</exception>
        public void WriteTo(Stream stream)
        {
            if (stream == null)
                throw new ArgumentNullException(nameof(stream));
    
            CheckDisposed();
            for (var i = 0; i < _chunks.Count; i++)
            {
                var count = i == (_chunks.Count - 1) ? _lastChunkPos : _chunks[i].Length;
                stream.Write(_chunks[i], 0, count);
            }
        }
    
        /// <summary>
        /// When overridden in a derived class, gets a value indicating whether the current stream supports reading.
        /// </summary>
        public override bool CanRead => true;
    
        /// <summary>
        /// When overridden in a derived class, gets a value indicating whether the current stream supports seeking.
        /// </summary>
        public override bool CanSeek => true;
    
        /// <summary>
        /// When overridden in a derived class, gets a value indicating whether the current stream supports writing.
        /// </summary>
        public override bool CanWrite => true;
    
        /// <summary>
        /// When overridden in a derived class, gets the length in bytes of the stream.
        /// </summary>
        /// <exception cref="ObjectDisposedException">Methods were called after the stream was closed.</exception>
        public override long Length
        {
            get
            {
                CheckDisposed();
                if (_chunks.Count == 0)
                    return 0;
    
                return (long)(_chunks.Count - 1) * ChunkSize + _lastChunkPos;
            }
        }
    
        /// <summary>
        /// Gets or sets the size of the underlying chunks. Cannot be greater than or equal to 85000.
        /// </summary>
        /// <value>
        /// The chunks size.
        /// </value>
        /// <exception cref="ArgumentOutOfRangeException"><paramref name="value" /> is out of range.</exception>
        public int ChunkSize
        {
            get => _chunkSize;
            set
            {
                if (value <= 0 || value >= _lohSize)
                    throw new ArgumentOutOfRangeException(nameof(value));
    
                _chunkSize = value;
            }
        }
    
        /// <summary>
        /// When overridden in a derived class, gets or sets the position within the current stream.
        /// </summary>
        /// <exception cref="ArgumentOutOfRangeException"><paramref name="value" /> is out of range.</exception>
        /// <exception cref="ObjectDisposedException">Methods were called after the stream was closed.</exception>
        public override long Position
        {
            get
            {
                CheckDisposed();
                return _position;
            }
            set
            {
                CheckDisposed();
                if (value < 0)
                    throw new ArgumentOutOfRangeException(nameof(value));
    
                if (value > Length)
                    throw new ArgumentOutOfRangeException(nameof(value));
    
                _position = value;
            }
        }
    }
    

    【讨论】:

    • Length 属性中存在一个错误 - 在乘法之前缺少转换为 long。否则,如果你有例如52795 个块,您将获得负长度。应该是:return (long)(_chunks.Count - 1) * ChunkSize + _lastChunkPos;
    • @Nyuno - 绝对。我已经修复了它并对 C# 进行了一些现代化改造
    【解决方案6】:

    在处理超过 2GB 的内存块时,您应该使用 UnmanagedMemoryStream,因为 MemoryStream 被限制为 2GB,而 UnmanagedMemoryStream 就是为了解决这个问题。

    【讨论】:

    • 你有最大2GB的官方来源吗?我想了解更多。 MSDN 文档没有做出这样的声明。
    • 最大 2gb 是基于 .NET 中支持的最大字节数组为 0x7FFFFFC7 字节。
    【解决方案7】:

    SparseMemoryStream 在 .NET 中执行此操作,但它深埋在内部类库中 - 当然,源代码是可用的,因为 Microsoft 将其全部作为开源代码放在那里。

    你可以在这里获取它的代码:http://www.dotnetframework.org/default.aspx/4@0/4@0/DEVDIV_TFS/Dev10/Releases/RTMRel/wpf/src/Base/MS/Internal/IO/Packaging/SparseMemoryStream@cs/1305600/SparseMemoryStream@cs

    话虽如此,我强烈建议不要按原样使用它——至少删除所有对孤立存储的调用,因为这似乎是框架打包 API 中没有尽头的错误*的原因。

    (*: 除了在流中传播数据之外,如果它变得太大,它基本上会出于某种原因重新发明交换文件——在用户的隔离存储中同样如此——而且巧合的是,大多数 MS 产品允许基于 .NET 的加载项没有以您可以访问独立存储的方式设置其应用程序域——例如,VSTO 加载项因遭受此问题而臭名昭著。)

    【讨论】:

      【解决方案8】:

      分块流的另一种实现可以被视为库存 MemoryStream 的替代品。此外,它允许在 LOH 上分配一个大字节数组,该数组将用作“块”池,在所有 ChunkedStream 实例之间共享...

      https://github.com/ImmortalGAD/ChunkedStream

      【讨论】:

        猜你喜欢
        • 1970-01-01
        • 1970-01-01
        • 2021-03-09
        • 2016-02-13
        • 2016-10-01
        • 2012-04-27
        • 2012-12-22
        • 2010-10-10
        • 1970-01-01
        相关资源
        最近更新 更多