将窄 (char) 输入流重新解释为宽 (wchar_t) 流

Question

我收到了一个包含 UTF-16 编码字符串的 std::istream。想象一下这样打开的 UTF-16 编码文本文件：

std::ifstream file( "mytext_utf16.txt", std::ios::binary );

我想将此流传递给采用std::wistream& 参数的函数。我无法将文件流类型更改为 std::wifstream。

问题：标准库或 boost 库中是否有任何工具可以让我将 istream 重新解释为 wistream？

我正在想象一个类似于std::wbuffer_convert 的适配器类，只是它不应该进行任何编码转换。基本上，对于从适配器类读取的每个 wchar_t，它应该只从关联的 istream 中读取两个字节，并将 reinterpret_cast 它们读取到 wchar_t。

我创建了一个使用 boost::iostreams 的实现，可以像这样使用并且像魅力一样工作：

std::ifstream file( "mytext_utf16.txt", std::ios::binary );

// Create an instance of my adapter class.
reinterpret_as_wide_stream< std::ifstream > wfile( &file );

// Read a wstring from file, using the adapter.
std::wstring str;
std::get_line( wfile, str );    

那我为什么要问呢？因为我喜欢重用现有代码而不是重新发明轮子。

Answer 1

由于还没有其他答案，我发布了使用 Boost.Iostreams 库的解决方案。虽然它很简单，但我仍然认为应该有一个更简单的解决方案。

首先，我们创建一个模板类来模拟 Boost.Iostreams device 概念并用作相关窄设备的适配器。它将 read、write 和 seek 操作转发到相关设备，但会调整流位置和大小值以适应不同设备之间的大小差异。窄和宽字符类型。

“basic_reinterpret_device.h”

#pragma once
#include <boost/iostreams/traits.hpp>
#include <boost/iostreams/read.hpp>
#include <boost/iostreams/write.hpp>
#include <boost/iostreams/seek.hpp>

// CategoryT: boost.iostreams device category tag
// DeviceT  : type of associated narrow device
// CharT    : (wide) character type of this device adapter 
template< typename CategoryT, typename DeviceT, typename CharT >
class basic_reinterpret_device
{
public:
    using category = CategoryT;               // required by boost::iostreams device concept
    using char_type = CharT;                  // required by boost::iostreams device concept
    using associated_device = DeviceT;
    using associated_char_type = typename boost::iostreams::char_type_of< DeviceT >::type;
    static_assert( sizeof( associated_char_type ) == 1, "Associated device must have a byte-sized char_type" );

    // Default constructor.
    basic_reinterpret_device() = default;

    // Construct from a narrow device
    explicit basic_reinterpret_device( DeviceT* pDevice ) :
        m_pDevice( pDevice ) {}

    // Get the asociated device.
    DeviceT* get_device() const { return m_pDevice; }

    // Read up to n characters from the underlying data source into the buffer s, 
    // returning the number of characters read; return -1 to indicate EOF
    std::streamsize read( char_type* s, std::streamsize n )
    {
        ThrowIfDeviceNull();

        std::streamsize bytesRead = boost::iostreams::read( 
            *m_pDevice, 
            reinterpret_cast<associated_char_type*>( s ), 
            n * sizeof( char_type ) );

        if( bytesRead == static_cast<std::streamsize>( -1 ) )  // EOF
            return bytesRead;
        return bytesRead / sizeof( char_type );
    }

    // Write up to n characters from the buffer s to the output sequence, returning the 
    // number of characters written.
    std::streamsize write( const char_type* s, std::streamsize n )
    {
        ThrowIfDeviceNull();

        std::streamsize bytesWritten = boost::iostreams::write(
            *m_pDevice, 
            reinterpret_cast<const associated_char_type*>( s ), 
            n * sizeof( char_type ) );

        return bytesWritten / sizeof( char_type );
    }

    // Advances the read/write head by off characters, returning the new position, 
    // where the offset is calculated from:
    //  - the start of the sequence if way == ios_base::beg
    //  - the current position if way == ios_base::cur
    //  - the end of the sequence if way == ios_base::end
    std::streampos seek( std::streamoff off, std::ios_base::seekdir way ) 
    {
        ThrowIfDeviceNull();

        std::streampos newPos = boost::iostreams::seek( *m_pDevice, off * sizeof( char_type ), way );
        return newPos / sizeof( char_type );
    }

protected:
    void ThrowIfDeviceNull()
    {
        if( ! m_pDevice )
            throw std::runtime_error( "basic_reinterpret_device - no associated device" );
    }

private:
    DeviceT* m_pDevice = nullptr;
};

为了简化此模板的使用，我们为最常见的 Boost.Iostreams 设备标签创建了一些别名模板。基于这些我们创建别名模板来构建标准兼容的流缓冲区和流。

“reinterpret_stream.h”

#pragma once
#include "basic_reinterpret_device.h"

#include <boost/iostreams/categories.hpp>
#include <boost/iostreams/traits.hpp>
#include <boost/iostreams/stream.hpp>
#include <boost/iostreams/stream_buffer.hpp>

struct reinterpret_device_tag : virtual boost::iostreams::source_tag, virtual boost::iostreams::sink_tag {};
struct reinterpret_source_seekable_tag : boost::iostreams::device_tag, boost::iostreams::input_seekable {};
struct reinterpret_sink_seekable_tag : boost::iostreams::device_tag, boost::iostreams::output_seekable {};


template< typename DeviceT, typename CharT >
using reinterpret_source = basic_reinterpret_device< boost::iostreams::source_tag, DeviceT, CharT >;

template< typename DeviceT, typename CharT >
using reinterpret_sink = basic_reinterpret_device< boost::iostreams::sink_tag, DeviceT, CharT >;

template< typename DeviceT, typename CharT >
using reinterpret_device = basic_reinterpret_device< reinterpret_device_tag, DeviceT, CharT >;

template< typename DeviceT, typename CharT >
using reinterpret_device_seekable = basic_reinterpret_device< boost::iostreams::seekable_device_tag, DeviceT, CharT >;

template< typename DeviceT, typename CharT >
using reinterpret_source_seekable = 
    basic_reinterpret_device< reinterpret_source_seekable_tag, DeviceT, CharT >;

template< typename DeviceT, typename CharT >
using reinterpret_sink_seekable = 
    basic_reinterpret_device< reinterpret_sink_seekable_tag, DeviceT, CharT >;


template< typename DeviceT >
using reinterpret_as_wistreambuf = boost::iostreams::stream_buffer< reinterpret_source_seekable< DeviceT, wchar_t > >;

template< typename DeviceT >
using reinterpret_as_wostreambuf = boost::iostreams::stream_buffer< reinterpret_sink_seekable< DeviceT, wchar_t > >;

template< typename DeviceT >
using reinterpret_as_wstreambuf = boost::iostreams::stream_buffer< reinterpret_device_seekable< DeviceT, wchar_t > >;


template< typename DeviceT >
using reinterpret_as_wistream = boost::iostreams::stream< reinterpret_source_seekable< DeviceT, wchar_t > >;

template< typename DeviceT >
using reinterpret_as_wostream = boost::iostreams::stream< reinterpret_sink_seekable< DeviceT, wchar_t > >;

template< typename DeviceT >
using reinterpret_as_wstream = boost::iostreams::stream< reinterpret_device_seekable< DeviceT, wchar_t > >;

用法示例：

#include "reinterpret_stream.h"

void read_something_as_utf16( std::istream& input )
{
    reinterpret_as_wistream< std::istream > winput( &input );
    std::wstring wstr;
    std::getline( winput, wstr );   
}

void write_something_as_utf16( std::ostream& output )
{
    reinterpret_as_wostream< std::ostream > woutput( &output );
    woutput << L"сайт вопросов и ответов для программистов";
}

Answer 2

正在进行中

这不是您应该使用的，但可能是您可以开始的提示，如果您还没有考虑过这样做的话。如果这没有帮助，或者当您可以找到更好的解决方案时，我很高兴删除或扩展此答案。

据我了解，您想读取一个 UTF-8 文件，然后简单地将每个单个字符转换为 wchar_t。

如果标准设施做的太多了，你就不能写你自己的方面吗？

#include <codecvt>
#include <locale>
#include <fstream>
#include <cwchar>
#include <iostream>
#include <fstream>

class MyConvert
{
 public:
  using state_type = std::mbstate_t;
  using result = std::codecvt_base::result;
  using From = char;
  using To = wchar_t;
  bool always_noconv() const throw() {
    return false;
  }
  result in(state_type& __state, const From* __from,
    const From* __from_end, const From*& __from_next,
    To* __to, To* __to_end, To*& __to_next) const
  {
    while (__from_next != __from_end) {
      *__to_next = static_cast<To>(*__from_next);
      ++__to_next;
      ++__from_next;
    }
    return result::ok;
  }
  result out(state_type& __state, const To* __from,
      const To* __from_end, const To*& __from_next,
      From* __to, From* __to_end, From*& __to_next) const
  {
    while (__from_next < __from_end) {
      std::cout << __from << " " << __from_next << " " << __from_end << " " << (void*)__to << 
        " " << (void*)__to_next << " " << (void*)__to_end << std::endl;
      if (__to_next >= __to_end) {
        std::cout << "partial" << std::endl;
        std::cout << "__from_next = " << __from_next << " to_next = " <<(void*) __to_next << std::endl;
        return result::partial;
      }
      To* tmp = reinterpret_cast<To*>(__to_next);
      *tmp = *__from_next;
      ++tmp;
      ++__from_next;
      __to_next = reinterpret_cast<From*>(tmp);
    }
    return result::ok;
  }
};

int main() {
  std::ofstream of2("test2.out");
  std::wbuffer_convert<MyConvert, wchar_t> conv(of2.rdbuf());
  std::wostream wof2(&conv);
  wof2 << L"сайт вопросов и ответов для программистов";
  wof2.flush();
  wof2.flush();
}

这不是你应该在你的代码中使用的东西。如果方向正确，您需要阅读文档，包括此方面所需的内容、所有这些指针的含义以及您需要如何写入它们。

如果你想使用这样的东西，你需要考虑你应该为构面使用哪些模板参数（如果有的话）。

更新我现在更新了我的代码。外功能现在更接近我想的我们想要的。它并不漂亮，只是一个测试代码，我仍然不确定为什么 __from_next 没有更新（或保留）。

目前的问题是我们无法写入流。使用 gcc，我们刚刚脱离了 wbuffer_convert 的同步，对于 clang，我们得到了一个 SIGILL。