我有一个长期运行的Python服务器,希望能够在不重新启动服务器的情况下升级服务。最好的方法是什么?

if foo.py has changed:
    unimport foo  <-- How do I do this?
    import foo
    myfoo = foo.Foo()

当前回答

编辑(答案V2)

前面的解决方案只适用于获取重置信息,但它不会更改所有引用(超过重新加载,但小于所需)。为了设置所有的引用,我必须进入垃圾收集器,并在那里重写引用。现在它就像一个咒语!

请注意,如果GC被关闭,或者如果重新加载不受GC监视的数据,这将不起作用。如果您不想打乱GC,原始答案可能就足够了。

新代码:

import importlib
import inspect
import gc
from enum import EnumMeta
from weakref import ref


_readonly_attrs = {'__annotations__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__',
               '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__func__', '__ge__', '__get__',
               '__getattribute__', '__globals__', '__gt__', '__hash__', '__init__', '__init_subclass__',
               '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__',
               '__reduce__', '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__sizeof__', '__str__',
               '__subclasshook__', '__weakref__', '__members__', '__mro__', '__itemsize__', '__isabstractmethod__',
               '__basicsize__', '__base__'}


def reset_module(module, inner_modules_also=True):
    """
    This function is a stronger form of importlib's `reload` function. What it does, is that aside from reloading a
    module, it goes to the old instance of the module, and sets all the (not read-only) attributes, functions and classes
    to be the reloaded-module's
    :param module: The module to reload (module reference, not the name)
    :param inner_modules_also: Whether to treat ths module as a package as well, and reload all the modules within it.
    """

    # For the case when the module is actually a package
    if inner_modules_also:
        submods = {submod for _, submod in inspect.getmembers(module)
                   if (type(submod).__name__ == 'module') and (submod.__package__.startswith(module.__name__))}
        for submod in submods:
            reset_module(submod, True)

    # First, log all the references before reloading (because some references may be changed by the reload operation).
    module_tree = _get_tree_references_to_reset_recursively(module, module.__name__)

    new_module = importlib.reload(module)
    _reset_item_recursively(module, module_tree, new_module)


def _update_referrers(item, new_item):
    refs = gc.get_referrers(item)

    weak_ref_item = ref(item)
    for coll in refs:
        if type(coll) == dict:
            enumerator = coll.keys()
        elif type(coll) == list:
            enumerator = range(len(coll))
        else:
            continue

        for key in enumerator:

            if weak_ref_item() is None:
                # No refs are left in the GC
                return

            if coll[key] is weak_ref_item():
                coll[key] = new_item

def _get_tree_references_to_reset_recursively(item, module_name, grayed_out_item_ids = None):
    if grayed_out_item_ids is None:
        grayed_out_item_ids = set()

    item_tree = dict()
    attr_names = set(dir(item)) - _readonly_attrs
    for sub_item_name in attr_names:

        sub_item = getattr(item, sub_item_name)
        item_tree[sub_item_name] = [sub_item, None]

        try:
            # Will work for classes and functions defined in that module.
            mod_name = sub_item.__module__
        except AttributeError:
            mod_name = None

        # If this item was defined within this module, deep-reset
        if (mod_name is None) or (mod_name != module_name) or (id(sub_item) in grayed_out_item_ids) \
                or isinstance(sub_item, EnumMeta):
            continue

        grayed_out_item_ids.add(id(sub_item))
        item_tree[sub_item_name][1] = \
            _get_tree_references_to_reset_recursively(sub_item, module_name, grayed_out_item_ids)

    return item_tree


def _reset_item_recursively(item, item_subtree, new_item):

    # Set children first so we don't lose the current references.
    if item_subtree is not None:
        for sub_item_name, (sub_item, sub_item_tree) in item_subtree.items():

            try:
                new_sub_item = getattr(new_item, sub_item_name)
            except AttributeError:
                # The item doesn't exist in the reloaded module. Ignore.
                continue

            try:
                # Set the item
                _reset_item_recursively(sub_item, sub_item_tree, new_sub_item)
            except Exception as ex:
                pass

    _update_referrers(item, new_item)

原来的答案

正如在@bobince的回答中所写的,如果在另一个模块中已经有对该模块的引用(特别是如果它是用As关键字导入的,如import numpy As np),该实例将不会被覆盖。

当应用需要配置模块的“全新”状态的测试时,这对我来说是相当有问题的,所以我写了一个名为reset_module的函数,它使用importlib的reload函数并递归地覆盖所有声明的模块属性。它已经用Python版本3.6进行了测试。

import importlib
import inspect
from enum import EnumMeta

_readonly_attrs = {'__annotations__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__',
               '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__func__', '__ge__', '__get__',
               '__getattribute__', '__globals__', '__gt__', '__hash__', '__init__', '__init_subclass__',
               '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__',
               '__reduce__', '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__sizeof__', '__str__',
               '__subclasshook__', '__weakref__', '__members__', '__mro__', '__itemsize__', '__isabstractmethod__',
               '__basicsize__', '__base__'}


def reset_module(module, inner_modules_also=True):
    """
    This function is a stronger form of importlib's `reload` function. What it does, is that aside from reloading a
    module, it goes to the old instance of the module, and sets all the (not read-only) attributes, functions and classes
    to be the reloaded-module's
    :param module: The module to reload (module reference, not the name)
    :param inner_modules_also: Whether to treat ths module as a package as well, and reload all the modules within it.
    """

    new_module = importlib.reload(module)

    reset_items = set()

    # For the case when the module is actually a package
    if inner_modules_also:
        submods = {submod for _, submod in inspect.getmembers(module)
                   if (type(submod).__name__ == 'module') and (submod.__package__.startswith(module.__name__))}
        for submod in submods:
            reset_module(submod, True)

    _reset_item_recursively(module, new_module, module.__name__, reset_items)


def _reset_item_recursively(item, new_item, module_name, reset_items=None):
    if reset_items is None:
        reset_items = set()

    attr_names = set(dir(item)) - _readonly_attrs

    for sitem_name in attr_names:

        sitem = getattr(item, sitem_name)
        new_sitem = getattr(new_item, sitem_name)

        try:
            # Set the item
            setattr(item, sitem_name, new_sitem)

            try:
                # Will work for classes and functions defined in that module.
                mod_name = sitem.__module__
            except AttributeError:
                mod_name = None

            # If this item was defined within this module, deep-reset
            if (mod_name is None) or (mod_name != module_name) or (id(sitem) in reset_items) \
                    or isinstance(sitem, EnumMeta):  # Deal with enums
                continue

            reset_items.add(id(sitem))
            _reset_item_recursively(sitem, new_sitem, module_name, reset_items)
        except Exception as ex:
            raise Exception(sitem_name) from ex

注意事项:请谨慎使用!在非外围模块(例如,定义外部使用的类的模块)上使用这些可能会导致Python中的内部问题(例如pickle / unpickling问题)。

其他回答

Python不会在重载时重新计算子模块地址,如果它在sys.modules中

这里有一个变通办法,虽然不完美,但有效。

# Created by BaiJiFeiLong@gmail.com at 2022/2/19 18:50
import importlib
import types

import urllib.parse
import urllib.request


def reloadModuleWithChildren(mod):
    mod = importlib.reload(mod)
    for k, v in mod.__dict__.items():
        if isinstance(v, types.ModuleType):
            setattr(mod, k, importlib.import_module(v.__name__))


fakeParse = types.ModuleType("urllib.parse")
realParse = urllib.parse

urllib.parse = fakeParse
assert urllib.parse is fakeParse

importlib.reload(urllib)
assert urllib.parse is fakeParse
assert getattr(urllib, "parse") is fakeParse

reloadModuleWithChildren(urllib)
assert urllib.parse is not fakeParse
assert urllib.parse is realParse

2018-02-01

模块foo必须提前导入成功。 从importlib导入重载,重载(foo)

31.5. import - Python 3.6.4文档的实现

如果你遇到以下错误,这个答案可能会帮助你得到一个解决方案:

回溯(最近一次调用): 文件“FFFF”,第1行 NameError: name 'YYYY'没有定义

OR

回溯(最近一次调用): 文件“FFFF”,第1行 文件"/usr/local/lib/python3.7/importlib/__init__.py",第140行,重新加载 抛出TypeError("reload()参数必须是一个模块") reload()参数必须是一个模块

如果你有一个像下面这样的导入,你可能需要使用sys。模块来获取你想要重载的模块:

  import importlib
  import sys

  from YYYY.XXX.ZZZ import CCCC
  import AAA.BBB.CC


  def reload(full_name)
    if full_name in sys.modules:
      importlib.reload(sys.modules[full_name])


  reload('YYYY.XXX.ZZZ') # this is fine in both cases
  reload('AAA.BBB.CC')  

  importlib.reload(YYYY.XXX.ZZZ) # in my case: this fails
  importlib.reload(AAA.BBB.CC)   #             and this is ok

主要的问题是进口。重载只接受模块而不接受字符串。

如果你不在服务器上,但正在开发,需要频繁地重新加载一个模块,这里有一个不错的技巧。

首先,确保您使用的是Jupyter Notebook项目中的优秀IPython shell。在安装Jupyter之后,您可以使用ipython或Jupyter控制台启动它,甚至更好的是Jupyter qtconsole,这将为您提供一个在任何操作系统中都具有代码完成功能的漂亮的彩色控制台。

现在在shell中输入:

%load_ext autoreload
%autoreload 2

现在,每次运行脚本时,模块都会被重新加载。

除了2,还有其他自动重载魔法的选项:

%autoreload
Reload all modules (except those excluded by %aimport) automatically now.

%autoreload 0
Disable automatic reloading.

%autoreload 1
Reload all modules imported with %aimport every time before executing the Python code typed.

%autoreload 2
Reload all modules (except those excluded by %aimport) every time before
executing the Python code typed.

当然,它也可以在木星笔记本上工作。

下面的代码允许你兼容Python 2/3:

try:
    reload
except NameError:
    # Python 3
    from imp import reload

你可以在两个版本中使用它作为reload(),这让事情变得更简单。