Airflow Elastic 日志配置答案

【问题标题】：Airflow Elastic logs configurationAirflow Elastic 日志配置
【发布时间】：2019-02-19 15:43:14
【问题描述】：

我在 Apache Airflow 中设置 Elastic 日志记录时遇到了一些问题。从 1.10 版开始，弹性日志记录已添加到配置中。

查看airflow.cfg 文件时，我们有两个与Elastic 相关的部分：

# Airflow can store logs remotely in AWS S3, Google Cloud Storage or Elastic Search.
# Users must supply an Airflow connection id that provides access to the storage
# location. If remote_logging is set to true, see UPDATING.md for additional
# configuration requirements.
remote_logging = True
remote_log_conn_id =
remote_base_log_folder =
encrypt_s3_logs = False

[elasticsearch]
elasticsearch_host = xxx.xxx.xxx.xxx
elasticsearch_log_id_template = {dag_id}-{task_id}-{execution_date}-{try_number}
elasticsearch_end_of_log_mark = end_of_log

现在我不确定如何设置它。查看airflow_local_settings.py 文件时，我们可以看到以下代码：

if REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('s3://'):
        DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['s3'])
elif REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('gs://'):
        DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['gcs'])
elif REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('wasb'):
        DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['wasb'])
elif REMOTE_LOGGING and ELASTICSEARCH_HOST:
        DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['elasticsearch'])

所以从逻辑上讲，如果我将远程日志记录设置为 True 并将弹性的主机/IP 放在弹性部分它应该可以工作。目前没有从气流实例生成日志。

【问题讨论】：

标签： elasticsearch airflow

【解决方案1】：

根据Airflow ElasticsearchTaskHandler doc

    ElasticsearchTaskHandler is a python log handler that
    reads logs from Elasticsearch. Note logs are not directly
    indexed into Elasticsearch. Instead, it flushes logs
    into local files. Additional software setup is required
    to index the log into Elasticsearch, such as using
    Filebeat and Logstash.

不幸的是，这个日志处理程序不会直接将日志刷新到您的 ES。

【讨论】：