【问题标题】:is this even possible? send commands/objects from one python shell to another?这甚至可能吗?将命令/对象从一个 python shell 发送到另一个?
【发布时间】:2020-08-05 01:00:39
【问题描述】:

我有一个问题,经过一番挖掘后我无法真正解决,这也不是我的专业领域,所以我什至不知道我在寻找什么。

我想知道是否可以将两个 python shell“链接”在一起?

这是实际用例...

我正在使用一个程序,该程序在 GUI 中内置了自己的专用 python shell。当您在内部 python shell 中运行命令时,GUI 会实时更新以反映您运行的命令。

问题是,脚本环境很糟糕。它基本上是一个与 shell 相邻的文本板,只是不断地复制和粘贴,不可能真正实现真正的开发。

我想要做的是打开我的 IDE(VSCode/Spyder),这样我就可以拥有一个合适的环境,但能够在我的 IDE 中运行以某种方式发送到软件内部 python shell 的命令。

是否有可能以某种方式检测软件中的打开外壳并在两个 python 实例之间连接/链接或建立管道?所以我可以在两者之间传递命令/python对象,并且基本上每个变量的状态都相同?

我最接近看到我想要的东西是使用multiprocessing 模块。或者socketpexpect

Passing data between separately running Python scripts

How to share variables across scripts in python?

即使它只是一种可行的通信方式,也只是希望能够在适当的开发环境中使用该软件。

老实说,我真的不知道自己在做什么,并希望在这里得到一些帮助..

【问题讨论】:

  • 您可以查看subprocess 在您的 2 个进程之间打开管道
  • 合法的方式是使用microservice 架构,但我假设您希望使用更简单的东西。
  • @IainShelvington 你能扩展一下吗?我对这种事情不是很有经验。这是否允许我从我的 IDE 发送可以在软件中评估的命令和/或 python 对象?这样是否有可能在每个变量中具有相同的变量状态?
  • @Felipe Lol 老实说,我来自土木工程背景。我做的 90% 的 Python 操作都是数据分析。我在这里有点过头了:D
  • 我错了,Blender 作为模块不支持打开的 Gui。它似乎与blender的控制台版本具有相同的限制。

标签: python sockets multiprocessing pipe


【解决方案1】:

所有的部分都放在一起了!

  • 多线程,使文件处理系统和 Python 交互式 shell 可以同时工作。
  • 变量在交互式 shell 和文件之间更新。换句话说,文件和交互式 shell 共享变量、函数、类等。
  • shell 和文件之间的即时更新。

import threading
import platform
import textwrap
import traceback
import hashlib
import runpy
import code
import time
import sys
import os


def clear_console():
    """ Clear your console depending on OS. """

    if platform.system() == "Windows":
        os.system("cls")
    elif platform.system() in ("Darwin", "Linux"):
        os.system("clear")


def get_file_md5(file_name):
    """ Grabs the md5 hash of the file. """

    with open(file_name, "rb") as f:
        return hashlib.md5(f.read()).hexdigest()


def track_file(file_name, one_way=False):
    """ Process external file. """

    # Grabs current md5 of file.
    md5 = get_file_md5(file_name)

    # Flag for the first run.
    first_run = True

    # If the event is set, thread gracefully quits by exiting loop.
    while not event_close_thread.is_set():

        time.sleep(0.1)

        # Gets updated (if any) md5 hash of file.
        md5_current = get_file_md5(file_name)
        if md5 != md5_current or first_run:
            md5 = md5_current

            # Executes the content of the file.
            try:
                # Gather the threads global scope to update the main thread's scope.
                thread_scope = runpy.run_path(file_name, init_globals=globals())

                if not one_way:
                    # Updates main thread's scope with other thread..
                    globals().update(thread_scope)

                # Prints updated only after first run.
                if not first_run:
                    print(f'\n{"="*20} File {file_name} updated! {"="*20}\n>>> ', end="")
                else:
                    first_run = False

            except:
                print(
                    f'\n{"="*20} File {file_name} threw error! {"="*20}\n {traceback.format_exc()}\n>>> ',
                    end="",
                )


def track(file_name):
    """ Initializes tracking thread (must be started with .start()). """

    print(f'{"="*20} File {file_name} being tracked! {"="*20}')
    return threading.Thread(target=track_file, args=(file_name,)).start()


if __name__ == "__main__":
    clear_console()

    # Creates a thread event for garbage collection, and file lock.
    event_close_thread = threading.Event()

    banner = textwrap.dedent(
        f"""\
        {"="*20} Entering Inception Shell {"="*20}\n
        This shell allows the sharing of the global scope between
        Python files and the Python interactive shell. To use:

        \t >>> track("script.py", one_way=False)

        On update of the file 'script.py' this shell will execute the
        file (passing the shells global variables to it), and then, if
        one_way is False, update its own global variables to that of the
        file's execution.
        """
    )

    # Begins interactive shell.
    code.interact(banner=banner, readfunc=None, local=globals(), exitmsg="")

    # Gracefully exits the thread.
    event_close_thread.set()

    # Exits shell.
    print(f'\n{"="*20} Exiting Inception Shell {"="*20}')
    exit()

一个班轮:

exec("""\nimport threading\nimport platform\nimport textwrap\nimport traceback\nimport hashlib\nimport runpy\nimport code\nimport time\nimport sys\nimport os\n\n\ndef clear_console():\n    \"\"\" Clear your console depending on OS. \"\"\"\n\n    if platform.system() == "Windows":\n        os.system("cls")\n    elif platform.system() in ("Darwin", "Linux"):\n        os.system("clear")\n\n\ndef get_file_md5(file_name):\n    \"\"\" Grabs the md5 hash of the file. \"\"\"\n\n    with open(file_name, "rb") as f:\n        return hashlib.md5(f.read()).hexdigest()\n\n\ndef track_file(file_name, one_way=False):\n    \"\"\" Process external file. \"\"\"\n\n    # Grabs current md5 of file.\n    md5 = get_file_md5(file_name)\n\n    # Flag for the first run.\n    first_run = True\n\n    # If the event is set, thread gracefully quits by exiting loop.\n    while not event_close_thread.is_set():\n\n        time.sleep(0.1)\n\n        # Gets updated (if any) md5 hash of file.\n        md5_current = get_file_md5(file_name)\n        if md5 != md5_current or first_run:\n            md5 = md5_current\n\n            # Executes the content of the file.\n            try:\n                # Gather the threads global scope to update the main thread's scope.\n                thread_scope = runpy.run_path(file_name, init_globals=globals())\n\n                if not one_way:\n                    # Updates main thread's scope with other thread..\n                    globals().update(thread_scope)\n\n                # Prints updated only after first run.\n                if not first_run:\n                    print(f'\\n{"="*20} File {file_name} updated! {"="*20}\\n>>> ', end="")\n                else:\n                    first_run = False\n\n            except:\n                print(\n                    f'\\n{"="*20} File {file_name} threw error! {"="*20}\\n {traceback.format_exc()}\\n>>> ',\n                    end="",\n                )\n\n\ndef track(file_name):\n    \"\"\" Initializes tracking thread (must be started with .start()). \"\"\"\n\n    print(f'{"="*20} File {file_name} being tracked! {"="*20}')\n    return threading.Thread(target=track_file, args=(file_name,)).start()\n\n\nif __name__ == "__main__":\n    clear_console()\n\n    # Creates a thread event for garbage collection, and file lock.\n    event_close_thread = threading.Event()\n\n    banner = textwrap.dedent(\n        f\"\"\"\\\n        {"="*20} Entering Inception Shell {"="*20}\\n\n        This shell allows the sharing of the global scope between\n        Python files and the Python interactive shell. To use:\n\n        \\t >>> track("script.py", one_way=False)\n\n        On update of the file 'script.py' this shell will execute the\n        file (passing the shells global variables to it), and then, if\n        one_way is False, update its own global variables to that of the\n        file's execution.\n        \"\"\"\n    )\n\n    # Begins interactive shell.\n    code.interact(banner=banner, readfunc=None, local=globals(), exitmsg="")\n\n    # Gracefully exits the thread.\n    event_close_thread.set()\n\n    # Exits shell.\n    print(f'\\n{"="*20} Exiting Inception Shell {"="*20}')\n    exit()\n""")

为您的 Blender shell 尝试以下操作:

import threading
import traceback
import hashlib
import runpy
import time


def get_file_md5(file_name):
    """ Grabs the md5 hash of the file. """

    with open(file_name, "rb") as f:
        return hashlib.md5(f.read()).hexdigest()


def track_file(file_name, one_way=False):
    """ Process external file. """

    # Grabs current md5 of file.
    md5 = get_file_md5(file_name)

    # Flag for the first run.
    first_run = True

    # If the event is set, thread gracefully quits by exiting loop.
    while not event_close_thread.is_set():

        time.sleep(0.1)

        # Gets updated (if any) md5 hash of file.
        md5_current = get_file_md5(file_name)
        if md5 != md5_current or first_run:
            md5 = md5_current

            # Executes the content of the file.
            try:
                # Gather the threads global scope to update the main thread's scope.
                thread_scope = runpy.run_path(file_name, init_globals=globals())

                if not one_way:
                    # Updates main thread's scope with other thread..
                    globals().update(thread_scope)

                # Prints updated only after first run.
                if not first_run:
                    print(
                        f'\n{"="*20} File {file_name} updated! {"="*20}\n>>> ', end=""
                    )
                else:
                    first_run = False

            except:
                print(
                    f'\n{"="*20} File {file_name} threw error! {"="*20}\n {traceback.format_exc()}\n>>> ',
                    end="",
                )


def track(file_name):
    """ Initializes tracking thread (must be started with .start()). """

    print(f'{"="*20} File {file_name} being tracked! {"="*20}')
    return threading.Thread(target=track_file, args=(file_name,)).start()


if __name__ == "__main__":
    # Creates a thread event for garbage collection, and file lock.
    event_close_thread = threading.Event()

    # Gracefully exits the thread.
    event_close_thread.set()

一个班轮:

exec("""\nimport threading\nimport traceback\nimport hashlib\nimport runpy\nimport time\n\n\ndef get_file_md5(file_name):\n    \"\"\" Grabs the md5 hash of the file. \"\"\"\n\n    with open(file_name, "rb") as f:\n        return hashlib.md5(f.read()).hexdigest()\n\n\ndef track_file(file_name, one_way=False):\n    \"\"\" Process external file. \"\"\"\n\n    # Grabs current md5 of file.\n    md5 = get_file_md5(file_name)\n\n    # Flag for the first run.\n    first_run = True\n\n    # If the event is set, thread gracefully quits by exiting loop.\n    while not event_close_thread.is_set():\n\n        time.sleep(0.1)\n\n        # Gets updated (if any) md5 hash of file.\n        md5_current = get_file_md5(file_name)\n        if md5 != md5_current or first_run:\n            md5 = md5_current\n\n            # Executes the content of the file.\n            try:\n                # Gather the threads global scope to update the main thread's scope.\n                thread_scope = runpy.run_path(file_name, init_globals=globals())\n\n                if not one_way:\n                    # Updates main thread's scope with other thread..\n                    globals().update(thread_scope)\n\n                # Prints updated only after first run.\n                if not first_run:\n                    print(\n                        f'\\n{"="*20} File {file_name} updated! {"="*20}\\n>>> ', end=""\n                    )\n                else:\n                    first_run = False\n\n            except:\n                print(\n                    f'\\n{"="*20} File {file_name} threw error! {"="*20}\\n {traceback.format_exc()}\\n>>> ',\n                    end="",\n                )\n\n\ndef track(file_name):\n    \"\"\" Initializes tracking thread (must be started with .start()). \"\"\"\n\n    print(f'{"="*20} File {file_name} being tracked! {"="*20}')\n    return threading.Thread(target=track_file, args=(file_name,)).start()\n\n\nif __name__ == "__main__":\n    # Creates a thread event for garbage collection, and file lock.\n    event_close_thread = threading.Event()\n\n    # Gracefully exits the thread.\n    event_close_thread.set()\n""")

【讨论】:

  • 你为什么要重新发明轮子而不使用标准库? (docs.python.org/3.8/library/…)
  • 上面的代码only 使用了来自 Python 标准库的模块。关于您提供的链接,您链接到multiprocessing 库中的“Listeners and Clients”部分——这都是与网络相关的,所以坦率地说,我对你的建议感到困惑.
  • 谢谢..现在没有时间测试这个,但我会在接下来几天的某个时间..非常感谢您的意见
  • @MatteoRagni 是否愿意提供一个可能实现我想要的行为的代码示例?
  • lmao 不会说谎,这太野蛮了……并且与普通的 python 外壳一样具有绝对的魅力……但是在搅拌机内部外壳中我得到一个错误,请参阅 img:imgur.com/a/0w7J174 .. 你可以下载 Blender/如果你真的想要,我可以给你发送 code.py,但我认为你已经做得足够了......从技术上讲,你的解决方案有效,但不在 Blender 中......我将查看 Matteo 的答案以查看如果它有效并且会接受一些东西
【解决方案2】:

TL.DR;

这是一个复杂的请求。我不认为可以通过技巧来实现。或者你拥有运行blender的进程(意味着你导入它的api),或者你附加到进程(使用gdb,但我不知道你是否可以使用你想要的IDE)或者你使用IDE包括pydevd。即便如此,我也不知道你能达到多少。

同步两个python进程并非易事。答案说明了一点。


PyDev.Debugger

您想找到一种方法来同步位于不同 python 实例中的两个 python 对象。我认为解决您的问题的唯一真正方法是设置pydevd 服务器并连接到它。如果您使用受支持的 IDE 之一(例如 PyDEV 或 PyCharm)会更简单,因为它们已经准备好执行此操作:

pydev 所做的工作不是微不足道的,存储库是一个相当大的项目。这是你最好的选择,但我不能保证它会奏效。


进程通信

通常的通信解决方案将不起作用,因为它们会在后台序列化和反序列化(pickle 和 unpickle)数据。让我们举个例子,在搅拌机进程中实现一个服务器,它接收任意代码作为字符串并执行它,发回代码的最后结果。结果将作为 Python 对象被客户端接收,因此您可以使用 IDE 接口对其进行检查,甚至可以在其上运行一些代码。有限制:

  • 并非所有内容都可以被客户端接收(例如,类定义必须存在于客户端中)
  • 只有可拾取的对象可以在连接上传输
  • 客户端和服务器中的对象不同:如果没有额外的(相当复杂的)逻辑,客户端上所做的修改将不会应用到服务器上

这是应该在您的 Blender 实例上运行的服务器

from multiprocessing.connection import Listener
from threading import Thread
import pdb
import traceback

import ast
import copy


# This is your configuration, chose a strong password and only open
# on localhost, because you are opening an arbitrary code execution
# server. It is not much but at least we cover something.
port = 6000
address = ('127.0.0.1', port)
authkey = b'blender'
# If you want to run it from another machine, you must set the address
# to '0.0.0.0' (on Linux, on Windows is not accepted and you have to
# specify the interface that will accept the connection) 


# Credits: https://stackoverflow.com/a/52361938/2319299
# Awesome piece of code, it is a carbon copy from there

def convertExpr2Expression(Expr):
    r"""
    Convert a "subexpression" of a piece of code in an actual
    Expression in order to be handled by eval without a syntax error

    :param Expr: input expression
    :return: an ast.Expression object correctly initialized
    """
    Expr.lineno = 0
    Expr.col_offset = 0
    result = ast.Expression(Expr.value, lineno=0, col_offset=0)
    return result


def exec_with_return(code):
    r"""
    We need an evaluation with return value. The only two function 
    that are available are `eval` and `exec`, where the first evaluates
    an expression, returning the result and the latter evaluates arbitrary code
    but does not return.

    Those two functions intercept the commands coming from the client and checks
    if the last line is an expression. All the code is executed with an `exec`,
    if the last one is an expression (e.g. "a = 10"), then it will return the 
    result of the expression, if it is not an expression (e.g. "import os")
    then it will only `exec` it.

    It is bindend with the global context, thus it saves the variables there.

    :param code: string of code
    :return: object if the last line is an expression, None otherwise
    """
    code_ast = ast.parse(code)
    init_ast = copy.deepcopy(code_ast)
    init_ast.body = code_ast.body[:-1]
    last_ast = copy.deepcopy(code_ast)
    last_ast.body = code_ast.body[-1:]
    exec(compile(init_ast, "<ast>", "exec"), globals())
    if type(last_ast.body[0]) == ast.Expr:
        return eval(compile(convertExpr2Expression(last_ast.body[0]), "<ast>", "eval"), globals())
    else:
        exec(compile(last_ast, "<ast>", "exec"), globals())

# End of carbon copy code


class ArbitraryExecutionServer(Thread):
    r"""
    We create a server execute arbitrary piece of code (the most dangerous
    approach ever, but needed in this case) and it is capable of sending
    python object. There is an important thing to keep in mind. It cannot send
    **not pickable** objects, that probably **include blender objects**!

    This is a dirty server to be used as an example, the only way to close 
    it is by sending the "quit" string on the connection. You can envision
    your stopping approach as you wish

    It is a Thread object, remeber to initialize it and then call the
    start method on it.

    :param address: the tuple with address interface and port
    :param authkey: the connection "password"
    """

    QUIT = "quit" ## This is the string that closes the server

    def __init__(self, address, authkey):
        self.address = address
        self.authkey = authkey
        super().__init__()

    def run(self):
        last_input = ""
        with Listener(self.address, authkey=self.authkey) as server:
            with server.accept() as connection:
                while last_input != self.__class__.QUIT:
                    try:
                        last_input = connection.recv()
                        if last_input != self.__class__.QUIT:
                            result = exec_with_return(last_input) # Evaluating remote input                       
                            connection.send(result)
                    except:
                        # In case of an error we return a formatted string of the exception
                        # as a little plus to understand what's happening
                        connection.send(traceback.format_exc())


if __name__ == "__main__":
    server = ArbitraryExecutionServer(address, authkey)
    server.start() # You have to start the server thread
    pdb.set_trace() # I'm using a set_trace to get a repl in the server.
                    # You can start to interact with the server via the client
    server.join() # Remember to join the thread at the end, by sending quit

虽然这是您的 VSCode 中的客户端

import time
from multiprocessing.connection import Client


# This is your configuration, should be coherent with 
# the one on the server to allow the connection
port = 6000
address = ('127.0.0.1', port)
authkey = b'blender'


class ArbitraryExecutionClient:
    QUIT = "quit"

    def __init__(self, address, authkey):
        self.address = address
        self.authkey = authkey
        self.connection = Client(address, authkey=authkey)

    def close(self):
        self.connection.send(self.__class__.QUIT)
        time.sleep(0.5)  # Gives some time before cutting connection
        self.connection.close()

    def send(self, code):
        r"""
        Run an arbitrary piece of code on the server. If the
        last line is an expression a python object will be returned.
        Otherwise nothing is returned
        """
        code = str(code)
        self.connection.send(code)
        result = self.connection.recv()
        return result

    def repl(self):
        r"""
        Run code in a repl loop fashion until user enter "quit". Closing
        the repl will not close the connection. It must be manually 
        closed.
        """
        last_input = ""
        last_result = None
        while last_input != self.__class__.QUIT:
            last_input = input("REMOTE >>> ")
            if last_input != self.__class__.QUIT:
                last_result = self.send(last_input)
                print(last_result)
        return last_result


if __name__ == "__main__":
    client = ArbitraryExecutionClient(address, authkey)
    import pdb; pdb.set_trace()
    client.close()

在脚本的底部还有如何在将pdb 设置为“repl”时启动它们。 使用此配置,您可以在服务器上从客户端运行任意代码(实际上这是一个极其危险的场景,但对于您非常具体的情况是有效的,或者更好“主要要求”)。

让我们深入了解我预期的限制。

你可以在服务器上定义一个类Foo

[client] >>> client = ArbitraryExecutionClient(address, authkey)
[client] >>> client.send("class Foo: pass")

[server] >>> Foo
[server] <class '__main__.Foo'>

你可以在服务器上定义一个名为“foo”的对象,但是你会立即收到一个错误,因为类Foo在本地实例中不存在(这次使用repl):

[client] >>> client.repl()
[client] REMOTE >>> foo = Foo()
[client] None
[client] REMOTE >>> foo
[client] *** AttributeError: Can't get attribute 'Foo' on <module '__main__' from 'client.py'>

出现此错误是因为在本地实例中没有Foo 类的声明,因此无法正确解开接收到的对象(所有 Blender 对象都会出现此问题。请注意,如果对象在某种程度上是可导入的,它可能仍然有效,我们稍后会看到这种情况)。

不接收错误的唯一方法是预先在客户端上声明该类,但它们不会是同一个对象,正如您可以通过查看它们的 id 看到的那样:

[client] >>> class Foo: pass
[client] >>> client.send("foo")
[client] <__main__.Foo object at 0x0000021E2F2F3488>

[server] >>> foo
[server] <__main__.Foo object at 0x00000203AE425308>

它们的 id 不同,因为它们存在于不同的内存空间中:它们是完全不同的实例,你必须手动同步它们上的每个操作!

如果类定义在某种程度上是可导入的并且对象是可挑选的,你可以避免重复类定义,据我所知它将自动导入:

[client] >>> client.repl()
[client] REMOTE >>> import numpy as np
[client] None
[client] REMOTE >>> ary = np.array([1, 2, 3])
[client] None
[client] REMOTE >>> ary
[client] [1 2 3]
[client] REMOTE >>> quit
[client] array([1, 2, 3])
[client] >>> ary = client.send("ary")
[client] >>> ary
[client] array([1, 2, 3])
[client] >>> type(ary)
[client] <class 'numpy.ndarray'>

我们从未在客户端上导入numpy,但我们已正确接收到对象。但是如果我们将本地实例修改为远程实例会怎样呢?

[client] >>> ary[0] = 10
[client] >>> ary
[client] array([10,  2,  3])
[client] >>> client.send("ary")
[client] array([1, 2, 3])

[server] >>> ary
[server] array([1, 2, 3])

我们没有同步对象内部的修改。

如果一个对象不可拾取怎么办?我们可以使用server 变量进行测试,该对象是Thread 并包含一个连接,它们都是不可选择的(意味着您不能将它们作为字节列表提供可逆的表示):

[server] >>> import pickle
[server] >>> pickle.dumps(server)
[server] *** TypeError: can't pickle _thread.lock objects

我们也可以在客户端看到错误,尝试接收它:

[client] >>> client.send("server")
[client] ... traceback for "TypeError: can't pickle _thread.lock objects" exception ...

我认为这个问题没有“简单”的解决方案,但我认为有一些库(如pydevd)实现了克服这个问题的完整协议。

我希望现在我的 cmets 更清楚了。

【讨论】:

  • 我会在早上测试这个第一件事......非常感谢你的努力
  • 我已经在搅拌机 2.82a-win64 中对其进行了测试,并且可以正常工作(进程之间存在通信)。但它不允许您交换 bpy 对象,因为它们不可拾取,如帖子中所述。
  • 是的...该死的大声笑...我非常感谢你们付出的努力,我不知道该接受谁,因为两者在技术上都解决了我的问题,但事实证明还有一个附加的皱起 bpy 对象不可腌制,我不知道......
  • 如果您被困在两者之间,请接受@MatteoRagni 的回答。 :) 他提供了一个专门涉及 Blender 的解决方案,这是值得注意的。坦率地说,直到现在我才意识到 Blender 有一个 API —— 只是在原始帖子下阅读你们 cmets。
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2016-05-05
相关资源
最近更新 更多