Scrapy API - 传递自定义记录器

Question

我正在使用 API 从脚本（Python 3.5、Scrapy 1.5）运行 Scrapy。

主脚本调用一个函数来处理它的日志记录：

def main(target_year):
    project = os.path.splitext(os.path.basename(os.path.abspath(__file__)))[0]
    iso_run_date = datetime.date.today().isoformat()
    logger = utils.get_logger(project, iso_run_date)

    scraping.run(project, iso_run_date, target_year)

这是文件“utils.py”中的函数，带有一个用于格式化的附加类，它使用 Python 的日志记录库创建一个记录器：

class UTCFormatter(logging.Formatter):
    converter = time.gmtime


def get_logger(project, iso_run_date):
    ip_address_param = 'ip'
    logger = logging.getLogger(project)
    logger.setLevel(logging.DEBUG)
    file_handler = logging.FileHandler(os.path.abspath(os.path.join(
        'log', '{}_{}.log'.format(project, iso_run_date))))
    file_handler.setLevel(logging.DEBUG)
    formatter = UTCFormatter(
        fmt=('[%(asctime)s.%(msecs)03dZ] %({})s %(name)s %(levelname)s: '
             '%(message)s').format(ip_address_param),
        datefmt='%Y-%m-%dT%H:%M:%S')
    file_handler.setFormatter(formatter)
    logger.addHandler(file_handler)
    logger = logging.LoggerAdapter(
        logger, {ip_address_param: socket.gethostbyname(socket.gethostname())})
    return logger

这里是 Scrapy 目录中的“__init__.py”文件：

@twisted.internet.defer.inlineCallbacks
def crawl(crawler_process, project, iso_run_date, target_year):
    yield crawler_process.crawl(project, iso_run_date, target_year)


def run(project, iso_run_date, target_year):
    os.environ.setdefault(
        'SCRAPY_SETTINGS_MODULE', 'scraping.scraping.settings')
    crawler_process = scrapy.crawler.CrawlerProcess(
        scrapy.utils.project.get_project_settings())
    crawl(crawler_process, project, iso_run_date, target_year)
    crawler_process.start()

当我执行脚本时，我从输出日志文件中的主脚本中获取日志，但从 Scrapy 中没有。

当我在我的蜘蛛中添加这个时：

self.logger.debug('Test')

我收到此错误：

--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.5/logging/__init__.py", line 980, in emit
    msg = self.format(record)
  File "/usr/lib/python3.5/logging/__init__.py", line 830, in format
    return fmt.format(record)
  File "/usr/lib/python3.5/logging/__init__.py", line 570, in format
    s = self.formatMessage(record)
  File "/usr/lib/python3.5/logging/__init__.py", line 539, in formatMessage
    return self._style.format(record)
  File "/usr/lib/python3.5/logging/__init__.py", line 383, in format
    return self._fmt % record.__dict__
KeyError: 'ip'
Call stack:
  File "XXXXX.py", line 105, in <module>
    main(target_year)
  File "XXXXX.py", line 23, in main
    scraping.run(project, iso_run_date, target_year)
  File "/home/XYZ/virtualenvs/scraping/project/scraping/__init__.py", line 27, in run
    crawler_process.start()
  File "/home/XYZ/virtualenvs/scraping/lib/python3.5/site-packages/scrapy/crawler.py", line 291, in start
    reactor.run(installSignalHandlers=False)  # blocking call
  File "/home/XYZ/virtualenvs/scraping/lib/python3.5/site-packages/twisted/internet/base.py", line 1261, in run
    self.mainLoop()
  File "/home/XYZ/virtualenvs/scraping/lib/python3.5/site-packages/twisted/internet/base.py", line 1270, in mainLoop
    self.runUntilCurrent()
  File "/home/XYZ/virtualenvs/scraping/lib/python3.5/site-packages/twisted/internet/base.py", line 896, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/home/XYZ/virtualenvs/scraping/lib/python3.5/site-packages/scrapy/utils/reactor.py", line 41, in __call__
    return self._func(*self._a, **self._kw)
  File "/home/XYZ/virtualenvs/scraping/lib/python3.5/site-packages/scrapy/core/engine.py", line 127, in _next_request
    request = next(slot.start_requests)
  File "/home/XYZ/virtualenvs/scraping/project/scraping/scraping/spiders/XXXXX.py", line 47, in start_requests
    self.logger.debug('Test')
Message: 'Test'
Arguments: ()

当我在我的主脚本中使用 basicConfig 时一切正常，看起来 Scrapy 会选择这个基本的记录器。但由于额外的格式，看起来我需要使用更高级的代码进行日志记录。

我希望能够从我的主脚本中定义一个自定义记录器，如代码所示，并让 Scrapy 对相同的输出文件使用相同的格式，而不必再次重新定义所有这些。这可能吗？

Answer 1

我找到了一种似乎可行的方法。

主脚本不变：

def main(target_year):
    project = os.path.splitext(os.path.basename(os.path.abspath(__file__)))[0]
    iso_run_date = datetime.date.today().isoformat()
    logger = utils.get_logger(project, iso_run_date)

文件“utils.py”现在只使用 logging.basicConfig()，就像我之前做的那样。

def get_logger(project, iso_run_date):
    logging.basicConfig(
        filename=os.path.abspath(os.path.join('log', '{0}_{1}.log'.format(project, iso_run_date))),
        format='[%(asctime)s.%(msecs)03dZ] {0} %(name)s %(levelname)s: %(message)s'.format(socket.gethostbyname(socket.gethostname())),
        datefmt='%Y-%m-%dT%H:%M:%S',
        level=logging.DEBUG)
    logging.Formatter.converter = time.gmtime
    return logging.getLogger(project)

文件“__init__.py”也没有改变：

@twisted.internet.defer.inlineCallbacks
def crawl(crawler_process, project, iso_run_date, target_year):
    yield crawler_process.crawl(project, iso_run_date, target_year)


def run(project, iso_run_date, target_year):
    os.environ.setdefault(
        'SCRAPY_SETTINGS_MODULE', 'scraping.scraping.settings')
    crawler_process = scrapy.crawler.CrawlerProcess(
        scrapy.utils.project.get_project_settings())
    crawl(crawler_process, project, iso_run_date, target_year)
    crawler_process.start()

现在，所有日志消息都以相同的自定义格式输出到同一个日志文件中。另外：

• 主脚本中的日志使用变量“project”作为名称。
• 来自蜘蛛的日志使用蜘蛛名称作为名称。
• 来自 Scrapy 组件的日志使用 Scrapy 组件的名称。