Python 的 shelve.open 可以嵌套调用吗？

Question

我正在尝试编写一个使用shelve 持久存储返回值的记忆库。如果我有调用其他 memoized 函数的 memoized 函数，我想知道如何正确打开架子文件。

import shelve
import functools


def cache(filename):
    def decorating_function(user_function):
        def wrapper(*args, **kwds):
            key = str(hash(functools._make_key(args, kwds, typed=False)))
            with shelve.open(filename, writeback=True) as cache:
                if key in cache:
                    return cache[key]
                else:
                    result = user_function(*args, **kwds)
                    cache[key] = result
                    return result

        return functools.update_wrapper(wrapper, user_function)

    return decorating_function


@cache(filename='cache')
def expensive_calculation():
    print('inside function')
    return


@cache(filename='cache')
def other_expensive_calculation():
    print('outside function')
    return expensive_calculation()

other_expensive_calculation()

除非这不起作用

$ python3 shelve_test.py
outside function
Traceback (most recent call last):
  File "shelve_test.py", line 33, in <module>
    other_expensive_calculation()
  File "shelve_test.py", line 13, in wrapper
    result = user_function(*args, **kwds)
  File "shelve_test.py", line 31, in other_expensive_calculation
    return expensive_calculation()
  File "shelve_test.py", line 9, in wrapper
    with shelve.open(filename, writeback=True) as cache:
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/shelve.py", line 239, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/shelve.py", line 223, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
_gdbm.error: [Errno 35] Resource temporarily unavailable

您对此类问题的解决方案有何建议。

Answer 1

不，您可能没有嵌套具有相同文件名的 shelve 实例。

搁置模块不支持对搁置对象的并发读/写访问。 （多个同时读取访问是安全的。）当一个程序有一个架子可供写入时，任何其他程序都不应该打开它来读取或写入。可以使用 Unix 文件锁定来解决这个问题，但这在 Unix 版本之间有所不同，并且需要了解所使用的数据库实现。

https://docs.python.org/3/library/shelve.html#restrictions

Answer 2

而不是尝试嵌套调用 open （正如您所发现的那样，它不起作用），您可以让您的装饰器维护对 shelve.open 返回的句柄的引用，然后如果它存在并且仍然打开，将其重新用于后续调用：

import shelve
import functools

def _check_cache(cache_, key, func, args, kwargs):
    if key in cache_:
        print("Using cached results")
        return cache_[key]
    else:
        print("No cached results, calling function")
        result = func(*args, **kwargs)
        cache_[key] = result
        return result

def cache(filename):
    def decorating_function(user_function):
        def wrapper(*args, **kwds):
            args_key = str(hash(functools._make_key(args, kwds, typed=False)))
            func_key = '.'.join([user_function.__module__, user_function.__name__])
            key = func_key + args_key
            handle_name = "{}_handle".format(filename)
            if (hasattr(cache, handle_name) and
                not hasattr(getattr(cache, handle_name).dict, "closed")
               ):
                print("Using open handle")
                return _check_cache(getattr(cache, handle_name), key, 
                                    user_function, args, kwds)
            else:
                print("Opening handle")
                with shelve.open(filename, writeback=True) as c:
                    setattr(cache, handle_name, c)  # Save a reference to the open handle
                    return _check_cache(c, key, user_function, args, kwds)

        return functools.update_wrapper(wrapper, user_function)
    return decorating_function


@cache(filename='cache')
def expensive_calculation():
    print('inside function')
    return


@cache(filename='cache')
def other_expensive_calculation():
    print('outside function')
    return expensive_calculation()

other_expensive_calculation()
print("Again")
other_expensive_calculation()

输出：

Opening handle
No cached results, calling function
outside function
Using open handle
No cached results, calling function
inside function
Again
Opening handle
Using cached results

编辑：

您也可以使用WeakValueDictionary 来实现装饰器，这样看起来更易读：

from weakref import WeakValueDictionary

_handle_dict = WeakValueDictionary()
def cache(filename):
    def decorating_function(user_function):
        def wrapper(*args, **kwds):
            args_key = str(hash(functools._make_key(args, kwds, typed=False)))
            func_key = '.'.join([user_function.__module__, user_function.__name__])
            key = func_key + args_key
            handle_name = "{}_handle".format(filename)
            if handle_name in _handle_dict:
                print("Using open handle")
                return _check_cache(_handle_dict[handle_name], key, 
                                    user_function, args, kwds)
            else:
                print("Opening handle")
                with shelve.open(filename, writeback=True) as c:
                    _handle_dict[handle_name] = c
                    return _check_cache(c, key, user_function, args, kwds)

        return functools.update_wrapper(wrapper, user_function)
    return decorating_function

一旦没有其他对句柄的引用，它将从字典中删除。由于我们的句柄仅在对装饰函数的最外层调用结束时才超出范围，因此当句柄打开时，我们将始终在 dict 中有一个条目，而在它关闭后则没有条目。

Answer 3

您打开文件两次，但从未真正关闭它以更新文件以用于任何用途。最后使用f.close ()。