Python：在 Ubuntu 上覆盖 os.path.supports_unicode_filenames

Question

我在 Ubuntu 服务器上运行 Python Web 应用程序，而我在 OS X 上进行本地开发。

我在希伯来语中使用了很多 unicode 字符串，包括处理图像的文件名，因此它们将与希伯来字符一起保存在文件系统中。

我的 Ubuntu 服务器已针对 UTF-8 进行了完全配置 - 我在文件系统（此应用程序之外）上还有其他带有希伯来语名称、希伯来语命名目录等的图像。

但是，我的应用程序在尝试在 Ubuntu（但不是 OS X）上保存具有希伯来文文件名的图像时返回错误。

错误是：

UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)

在大量调查之后，就我所见，我找到了最后一个可能的原因：

# Inside my virtualenv, Mac OS X
>>> import os.path
>>> os.path.supports_unicode_filenames
>>> True

# Inside my virtualenv, Ubuntu 12.04
>>> import os.path
>>> os.path.supports_unicode_filenames
>>> False

为了好奇，这里是我的 Ubuntu 语言环境设置：

locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

更新：添加代码和示例字符串：

# a string, of the type I would get for instance.product.name, as used below.
u'\\u05e7\\u05e8\\u05d5\\u05d1-\\u05e8\\u05d7\\u05d5\\u05e7'


#utils.py
# I get an image object from django, and I run this function so django 
# can use the generated filepath for the image.
def get_upload_path(instance, filename):

    tmp = filename.split('.')
    extension = '.' + tmp[-1]

    if instance.__class__.__name__ == 'MyClass':

        seo_filename = unislugify(instance.product.name)
        # unislugify takes a string and strips spaces, etc.
        value = IMAGES_PRODUCT_DIR + seo_filename + extension

    else:

        value = IMAGES_GENERAL_DIR + unislugify(filename)

    return value

示例堆栈跟踪：

UnicodeEncodeError: 'ascii' codec can't encode characters in position 60-66: ordinal not in range(128)

Stacktrace (most recent call last):

  File "django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "django/contrib/admin/options.py", line 366, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)

  File "django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)

  File "django/views/decorators/cache.py", line 89, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)

  File "django/contrib/admin/sites.py", line 196, in inner
    return view(request, *args, **kwargs)

  File "django/utils/decorators.py", line 25, in _wrapper
    return bound_func(*args, **kwargs)

  File "django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)

  File "django/utils/decorators.py", line 21, in bound_func
    return func(self, *args2, **kwargs2)

  File "django/db/transaction.py", line 209, in inner
    return func(*args, **kwargs)

  File "django/contrib/admin/options.py", line 1055, in change_view
    self.save_related(request, form, formsets, True)

  File "django/contrib/admin/options.py", line 733, in save_related
    self.save_formset(request, form, formset, change=change)

  File "django/contrib/admin/options.py", line 721, in save_formset
    formset.save()

  File "django/forms/models.py", line 497, in save
    return self.save_existing_objects(commit) + self.save_new_objects(commit)

  File "django/forms/models.py", line 628, in save_new_objects
    self.new_objects.append(self.save_new(form, commit=commit))

  File "django/forms/models.py", line 731, in save_new
    obj.save()

  File "django/db/models/base.py", line 463, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)

  File "django/db/models/base.py", line 551, in save_base
    result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)

  File "django/db/models/manager.py", line 203, in _insert
    return insert_query(self.model, objs, fields, **kwargs)

  File "django/db/models/query.py", line 1593, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)

  File "django/db/models/sql/compiler.py", line 909, in execute_sql
    for sql, params in self.as_sql():

  File "django/db/models/sql/compiler.py", line 872, in as_sql
    for obj in self.query.objs

  File "django/db/models/fields/files.py", line 249, in pre_save
    file.save(file.name, file, save=False)

  File "django/db/models/fields/files.py", line 86, in save
    self.name = self.storage.save(name, content)

  File "django/core/files/storage.py", line 44, in save
    name = self.get_available_name(name)

  File "django/core/files/storage.py", line 70, in get_available_name
    while self.exists(name):

  File "django/core/files/storage.py", line 230, in exists
    return os.path.exists(self.path(name))

  File "python2.7/genericpath.py", line 18, in exists
    os.stat(path)

Answer 1

os.path.supports_unicode_filenames 在除 darwin 之外的 posix 系统上是 always False，这是因为它们并不真正关心文件名的编码，它只是一个字节序列。语言环境设置指定了如何解释这些字节，这就是为什么当语言环境设置不正确时，您最终会在终端中出现损坏的字符。

您如何运行您的网络应用程序？如果您使用 cgi 或 wsgi 通过 Web 服务器（apache？）运行它，语言环境可能不是您在 shell 中看到的，因此这可能是 python 尝试使用 ascii 编解码器对路径名进行编码的原因。

为使其正常工作，您可以在打开文件时手动将路径名编码为utf-8。

编辑：
所以失败是调用os.stat，使用unicode字符串调用它，尝试根据默认编码（sys.getdefaultencoding()）将其转换为字节字符串，在uWSGI环境中似乎总是@987654327 @ 使用 python2 时。要解决此问题，您可以确保将任何 unicode 字符串编码为 utf-8，然后才能将其传递给 os.stat。

Answer 2

感谢大家的帮助。我仍然没有用 uWSGI 解决这个问题。

但是，这是我“配置”uWSGI 的最后一根稻草，我回到 gunicorn 作为应用服务器，一切正常。我当然想使用 uWSGI，因为它是一个雄心勃勃的项目，但归根结底，我是一名开发人员而不是系统管理员，而且 gunicorn 更容易在常见用例中工作。