Pandas 到 Oracle 数据库 - 元音变音编码答案

【问题标题】：Pandas to Oracle Database - encoding for umlautePandas 到 Oracle 数据库 - 元音变音编码
【发布时间】：2018-03-16 17:17:00
【问题描述】：

我正在使用 python 3.6 读取一些 xml 数据并将其存储在 pandas.Dataframe 中。数据包含在 python 中正确打印的带有变音符号的字符串。

我与 Oracle 数据库的连接创建如下：

config.encoding = 'iso-8859-1'
es = "oracle://" + schema + ":" + schema + "@(DESCRIPTION = (LOAD_BALANCE=on) (FAILOVER=ON) (CHARACTERSET = AL16UTF16) (ADDRESS = (PROTOCOL = TCP)(HOST = " + host + ")(PORT = " + port + ")) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = " + service + ")))"
engine = create_engine(es, encoding= config.encoding)
con = engine.connect()
df.to_sql("test_table", con, if_exists = 'replace')

我尝试了两种方式将数据写入数据库：

1) 没有直接编码我得到以下错误：

Traceback (most recent call last):
File "...", line 59, in <module>
df.to_sql("test_table", con, if_exists = 'replace')
File "D:\Python\Python36\lib\site-packages\pandas\core\generic.py", line 1534, in to_sql
chunksize=chunksize, dtype=dtype)
File "D:\Python\Python36\lib\site-packages\pandas\io\sql.py", line 473, in to_sql
chunksize=chunksize, dtype=dtype)
File "D:\Python\Python36\lib\site-packages\pandas\io\sql.py", line 1156, in to_sql
table.insert(chunksize)
File "D:\Python\Python36\lib\site-packages\pandas\io\sql.py", line 670, in insert
self._execute_insert(conn, keys, chunk_iter)
File "D:\Python\Python36\lib\site-packages\pandas\io\sql.py", line 645, in _execute_insert
conn.execute(self.insert_statement(), data)
File "D:\Python\Python36\lib\site-packages\sqlalchemy\engine\base.py", line 948, in execute
return meth(self, multiparams, params)
File "D:\Python\Python36\lib\site-packages\sqlalchemy\sql\elements.py", line 269, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "D:\Python\Python36\lib\site-packages\sqlalchemy\engine\base.py", line 1060, in _execute_clauseelement
compiled_sql, distilled_params
File "D:\Python\Python36\lib\site-packages\sqlalchemy\engine\base.py", line 1200, in _execute_context
context)
File "D:\Python\Python36\lib\site-packages\sqlalchemy\engine\base.py", line 1416, in _handle_dbapi_exception
util.reraise(*exc_info)
File "D:\Python\Python36\lib\site-packages\sqlalchemy\util\compat.py", line 187, in reraise
raise value
File "D:\Python\Python36\lib\site-packages\sqlalchemy\engine\base.py", line 1170, in _execute_context
context)
File "D:\Python\Python36\lib\site-packages\sqlalchemy\dialects\oracle\cx_oracle.py", line 852, in do_executemany
cursor.executemany(statement, parameters)
UnicodeEncodeError: 'ascii' codec can't encode character '\xfc' in position 3: ordinal not in range(128)

2) 我使用 str.encode(config.encoding) 对每个相关条目进行了编码。在这种情况下，我没有收到错误，但元音符号似乎没有正确存储。当使用相同的连接和 pandas.from_sql(...) 再次读取它们时，输出如下所示：

status
Erf?llt

此外，当使用 SQLDeveloper 或 Data Grip 查看数据库时，元音变音显示为问号。

数据库设置如下所示：

NLS_RDBMS_VERSION   12.1.0.2.0
NLS_NCHAR_CONV_EXCP FALSE
NLS_LENGTH_SEMANTICS    BYTE
NLS_COMP    BINARY
NLS_DUAL_CURRENCY   $
NLS_TIMESTAMP_TZ_FORMAT DD-MON-RR HH.MI.SSXFF AM TZR
NLS_TIME_TZ_FORMAT  HH.MI.SSXFF AM TZR
NLS_TIMESTAMP_FORMAT    DD-MON-RR HH.MI.SSXFF AM
NLS_TIME_FORMAT HH.MI.SSXFF AM
NLS_SORT    BINARY
NLS_DATE_LANGUAGE   AMERICAN
NLS_DATE_FORMAT DD-MON-RR
NLS_CALENDAR    GREGORIAN
NLS_NUMERIC_CHARACTERS  .,
NLS_NCHAR_CHARACTERSET  AL16UTF16
NLS_CHARACTERSET    WE8ISO8859P15
NLS_ISO_CURRENCY    AMERICA
NLS_CURRENCY    $
NLS_TERRITORY   AMERICA
NLS_LANGUAGE    AMERICAN

【问题讨论】：

标签： python sql oracle pandas

【解决方案1】：

我找到了基于类似问题的答案。

在创建连接/引擎之前，我将 NLS_LANG 设置为数据库的设置：

os.environ["NLS_LANG"] = 'AMERICAN_AMERICA.WE8ISO8859P1'

这避免了导致错误的 cx_oracle 进行的转换。

【讨论】：