如何在 celery 中添加日志处理程序？答案

【问题标题】：How do I add log handler in celery?如何在 celery 中添加日志处理程序？
【发布时间】：2018-02-06 04:57:01
【问题描述】：

所以我有我的 tasks.py 这里是一个摘录：

import builtins
import logging
import os
import urllib
import inspect

from celery import Celery
from common.environment_helper import EnvironmentHelper
from config import log

# Logging functionality
logger = logging.getLogger(__name__)
EnvironmentHelper.set_environment(logger)


app = Celery('tasks', broker=BROKER__URL, broker_transport_options=BROKER_TRANSPORT_OPTIONS)
app.conf.CELERY_ACCEPT_CONTENT = ['json']
app.conf.CELERY_TASK_SERIALIZER = 'json'
app.conf.CELERYD_PREFETCH_MULTIPLIER = 1
app.log.already_setup=True
app.conf.CELERY_ENABLE_REMOTE_CONTROL = False
app.conf.CELERY_DEFAULT_QUEUE = queue_name

@app.task
def convert_file(file_conversion_json):
    file_conversion_json_copy = file_conversion_json.copy()

所以基本上我想从字典中获取一个值并将其添加到我的日志中。通过使用以下代码，我已经在我的实际应用程序中成功地做到了这一点：

import uuid
import logging
import flask


# Generate a new request ID, optionally including an original request ID
def generate_request_id(original_id):
    new_id = uuid.uuid4()

    if not original_id:
        return new_id
    else:
        new_id = original_id
    return new_id

# Returns the current request ID or a new one if there is none
# In order of preference:
#   * If we've already created a request ID and stored it in the flask.g context local, use that
#   * If a client has passed in the X-Request-Id header, create a new ID with that prepended
#   * Otherwise, generate a request ID and store it in flask.g.request_id
def request_id():
    if getattr(flask.g, 'request_id', None):
        return flask.g.request_id

    headers = flask.request.headers
    original_request_id = headers.get("X-Request-Id")
    new_uuid = generate_request_id(original_request_id)
    flask.g.request_id = new_uuid

    return new_uuid


class RequestIdFilter(logging.Filter):
    # This is a logging filter that makes the request ID available for use in
    # the logging format. Note that we're checking if we're in a request
    # context, as we may want to log things before Flask is fully loaded.
    def filter(self, record):
        record.request_id = request_id() if flask.has_request_context() else ''
        return True

............

# log.py
import logging.config
import os

LOGGER_CONFIGURATION = {
    'version': 1,
    'filters': {
        'request_id': {
            '()': 'config.utils.RequestIdFilter',
        },
    },
    'formatters': {
        'standard': {
            'format': '%(asctime)s - %(name)s.%(module)s.%(funcName)s:%(lineno)d - %(levelname)s - %(request_id)s - %(message)s',
        },
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'level': 'INFO',
            'filters': ['request_id'],
            'formatter': 'standard'
        }
    },
    'loggers': {
        '': {
            'handlers': ['console'],
            'level':'INFO',
        },
        'app': {
            'handlers': ['console'],
            'level':'INFO',
        },
    }
}

logging.config.dictConfig(LOGGER_CONFIGURATION)

但它不适用于芹菜，我不知道如何像这样动态添加变量。有什么建议吗？

【问题讨论】：

标签： python asynchronous logging celery celery-task

【解决方案1】：

所以我想如果我将这个添加到我的任务中：

logFormatter = logging.Formatter('%(asctime)s - %(name)s.%(module)s.%(funcName)s:%(lineno)d - %(levelname)s - Request ID:'+ file_conversion_json['x_request_id'] + ' - %(message)s')
rootLogger = logging.getLogger()
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
rootLogger.addHandler(consoleHandler)

它会将我想要的内容添加到日志输出中，但现在它复制了我的日志消息，我在日志输出中获得了每条消息两次......

【讨论】：