无法使用 cx_Freeze 制作独立的二进制scrapy spider

Question

关于我的工作环境的简短描述：win 7 x64，python 2.7 x64，scrapy 0.22，cx_Freeze 4.3.2。

首先，我开发了一个简单的爬虫，它运行良好。然后，使用核心的scrapy API，我创建了一个外部脚本main.py，它可以运行spider，它也可以根据需要运行。这是脚本的代码：

# external main.py using scrapy core API, 'test' is just replaced name of my project
from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy import log, signals
from test.spiders.testSpider import TestSpider
from test import settings, pipelines
from scrapy.utils.project import get_project_settings

spider = TestSpider(domain='test.com')
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run()

所以现在我正在尝试使用 setup.py 使用 cx_Freeze 为所有这些制作二进制文件，就像在另一个主题 here 中一样。代码如下：

from cx_Freeze import setup, Executable

includes = ['scrapy', 'pkg_resources', 'lxml.etree', 'lxml._elementpath']

build_options = {'compressed' : True,
                'optimize' : 2,
                'namespace_packages' : ['zope', 'scrapy', 'pkg_resources'],
                'includes' : includes,
                'excludes' : []}

executable = Executable(script='main.py',
                        copyDependentFiles=True,
                        includes=includes)

setup(name='Stand-alone scraper',
      version='0.1',
      description='Stand-alone scraper',
      options= {'build_exe': build_options},
      executables=[executable])

它通常编译成 exe 文件。当我尝试运行它时，问题就开始了：

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\cx_Freeze\initscripts\Console.py", line 27, in       <module>
    exec code in m.__dict__
  File "main.py", line 2, in <module>
    from scrapy.crawler import Crawler
  File "C:\Python27\lib\site-packages\scrapy\__init__.py", line 6, in <module>
    __version__ = pkgutil.get_data(__package__, 'VERSION').strip()
  File "C:\Python27\lib\pkgutil.py", line 591, in get_data
    return loader.get_data(resource_name)
IOError: [Errno 2] No such file or directory: 'scrapy\\VERSION'

我解决了这个问题，只是将 scrapy\version 文件从原始源 (python\lib\site-packages\scrapy) 移动到 build-folder 中的 library.zip\scapy。在第二次运行 main.exe 后，我收到了另一条消息：

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\cx_Freeze\initscripts\Console.py", line 27, in <module>
    exec code in m.__dict__
  File "main.py", line 11, in <module>
    crawler = Crawler(settings)
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 20, in __init__
    self.stats = load_object(settings['STATS_CLASS'])(self)
  File "C:\Python27\lib\site-packages\scrapy\utils\misc.py", line 42, in load_object
    raise ImportError("Error loading object '%s': %s" % (path, e))
ImportError: Error loading object 'scrapy.statscol.MemoryStatsCollector': No module named statscol

我没有找到任何解决方案，只是尝试从我的 main.py 中的错误消息中导入模块。简而言之-它没有用。每次新导入我都会收到一条带有另一个模块的新消息（我总共尝试导入 15 个 :)）模块，直到在密码学中出现有关 aes 模块的错误。 我也尝试使用 cx_freeze 替代品，如 py2exe 和 pyinstaller，但结果相同。

谁能帮我解决这个问题？ 感谢您阅读到此为止。

Answer 1

用这个替换你的 cx_Freeze 代码。

import sys 
    from cx_Freeze import setup, Executable 
    build_exe_options = {"packages": ["os","twisted","scrapy","test"], "excludes": ["tkinter"],"include_msvcr":True} 

    base = None
    setup(  name = "MyScript", 
            version = "0.1",
            description = "Demo", 
            options = {"build_exe": build_exe_options}, 
            executables = [Executable("C:\\MyScript", base=base)]) 

代码的不同之处在于我已经包含了整个包，因此您可以访问它们的所有功能。