如何使用 elasticsearch 配置 Jaeger？答案

【问题标题】：How to configure Jaeger with elasticsearch?如何使用 elasticsearch 配置 Jaeger？
【发布时间】：2019-01-18 00:40:49
【问题描述】：

我已尝试执行此 docker 命令以使用 elasticsearch 设置 Jaeger 代理和 jaeger 收集器。

sudo docker run \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-e SPAN_STORAGE_TYPE=elasticsearch \
--name=jaeger \
jaegertracing/all-in-one:latest

但是这个命令给出了以下错误。如何使用 ElasticSearch 配置 Jaeger？

"msg":"Failed to init storage factory","error":"health check timeout: no Elasticsearch node available","errorVerbose":"no Elasticsearch node available\

【问题讨论】：

为什么一体机的解决方案不行？

标签： docker elasticsearch opentracing jaeger distributed-tracing

【解决方案1】：

在搜索了一段时间的解决方案后，我找到了一个 docker-compose.yml 文件，其中包含 Jaeger Query、Agent、collector 和 Elasticsearch 配置。

docker-compose.yml

  version: "3"

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.3.1
    networks:
      - elastic-jaeger
    ports:
      - "127.0.0.1:9200:9200"
      - "127.0.0.1:9300:9300"
    restart: on-failure
    environment:
      - cluster.name=jaeger-cluster
      - discovery.type=single-node
      - http.host=0.0.0.0
      - transport.host=127.0.0.1
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
      - xpack.security.enabled=false
    volumes:
      - esdata:/usr/share/elasticsearch/data

  jaeger-collector:
    image: jaegertracing/jaeger-collector
    ports:
      - "14269:14269"
      - "14268:14268"
      - "14267:14267"
      - "9411:9411"
    networks:
      - elastic-jaeger
    restart: on-failure
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
    command: [
      "--es.server-urls=http://elasticsearch:9200",
      "--es.num-shards=1",
      "--es.num-replicas=0",
      "--log-level=error"
    ]
    depends_on:
      - elasticsearch

  jaeger-agent:
    image: jaegertracing/jaeger-agent
    hostname: jaeger-agent
    command: ["--collector.host-port=jaeger-collector:14267"]
    ports:
      - "5775:5775/udp"
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5778:5778"
    networks:
      - elastic-jaeger
    restart: on-failure
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
    depends_on:
      - jaeger-collector

  jaeger-query:
    image: jaegertracing/jaeger-query
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - no_proxy=localhost
    ports:
      - "16686:16686"
      - "16687:16687"
    networks:
      - elastic-jaeger
    restart: on-failure
    command: [
      "--es.server-urls=http://elasticsearch:9200",
      "--span-storage.type=elasticsearch",
      "--log-level=debug"
    ]
    depends_on:
      - jaeger-agent

volumes:
  esdata:
    driver: local

networks:
  elastic-jaeger:
    driver: bridge

docker-compose.yml 文件安装 elasticsearch、Jaeger 收集器、查询和代理。

先安装docker和docker compose https://docs.docker.com/compose/install/#install-compose

然后，依次执行这些命令

 
1. sudo docker-compose up -d elasticsearch

2. sudo docker-compose up -d 

3. sudo docker ps -a

启动所有 docker 容器 - Jaeger 代理、收集器、查询和弹性搜索。

sudo docker start 容器 ID

访问 -> http://localhost:16686/

【讨论】：

在服务 jager-agent 下的 docker-compose 文件中，复制网络条目。
当我尝试运行 jaeger-agent 时，容器会产生以下错误：Error: unknown flag: --collector.host-port

【解决方案2】：

正如我在上面对 OP 的第一个答案的评论中提到的，在完全按照给定的方式运行 docker-compose 时出现错误：

Error: unknown flag: --collector.host-port

我认为这个 CLI 标志已经被 Jaeger 的人弃用了，因为这个答案被写出来了。所以我在 jaeger-agent 文档中翻了一下：

我做了几个小的修改就可以使用它：

我将端口范围 "14250:14250" 添加到 jaeger-collector 端口
我将 jaeger-agent 命令输入更新为：command: ["--reporter.grpc.host-port=jaeger-collector:14250"]
最后，我将image 标签中的弹性搜索版本更新为他们目前可用的最新版本（尽管我怀疑这是必需的）。

更新后的 docker-compose.yaml：

version: "3"

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.3
    networks:
      - elastic-jaeger
    ports:
      - "127.0.0.1:9200:9200"
      - "127.0.0.1:9300:9300"
    restart: on-failure
    environment:
      - cluster.name=jaeger-cluster
      - discovery.type=single-node
      - http.host=0.0.0.0
      - transport.host=127.0.0.1
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
      - xpack.security.enabled=false
    volumes:
      - esdata:/usr/share/elasticsearch/data

  jaeger-collector:
    image: jaegertracing/jaeger-collector
    ports:
      - "14269:14269"
      - "14268:14268"
      - "14267:14267"
      - "14250:14250"
      - "9411:9411"
    networks:
      - elastic-jaeger
    restart: on-failure
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
    command: [
      "--es.server-urls=http://elasticsearch:9200",
      "--es.num-shards=1",
      "--es.num-replicas=0",
      "--log-level=error"
    ]
    depends_on:
      - elasticsearch

  jaeger-agent:
    image: jaegertracing/jaeger-agent
    hostname: jaeger-agent
    command: ["--reporter.grpc.host-port=jaeger-collector:14250"]
    ports:
      - "5775:5775/udp"
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5778:5778"
    networks:
      - elastic-jaeger
    restart: on-failure
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
    depends_on:
      - jaeger-collector

  jaeger-query:
    image: jaegertracing/jaeger-query
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - no_proxy=localhost
    ports:
      - "16686:16686"
      - "16687:16687"
    networks:
      - elastic-jaeger
    restart: on-failure
    command: [
      "--es.server-urls=http://elasticsearch:9200",
      "--span-storage.type=elasticsearch",
      "--log-level=debug"
    ]
    depends_on:
      - jaeger-agent

volumes:
  esdata:
    driver: local

networks:
  elastic-jaeger:
    driver: bridge

【讨论】：

【解决方案3】：

如果您想使用 Elasticsearch 和 Kibana 部署 Jaeger 以快速验证和检查堆栈，例如实物或者Minikube，下面的sn-p可能对你有帮助。

#######################
## Add jaegertracing helm repo
#######################
helm repo add jaegertracing
https://jaegertracing.github.io/helm-charts

#######################
## Create a target namespace
#######################
kubectl create namespace observability

#######################
## Check and use the jaegertracing helm chart
#######################
helm search repo jaegertracing
helm install -n observability jaeger-operator jaegertracing/jaeger-operator

#######################
## Use the elasticsearch all-in-one operator
#######################
kubectl apply -f https://download.elastic.co/downloads/eck/1.1.2/all-in-one.yaml

#######################
## Create an elasticsearch deployment
#######################
cat <<EOF | kubectl apply -n observability -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.7.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
EOF

PASSWORD=$(kubectl get secret -n observability quickstart-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode)
kubectl create secret -n observability generic jaeger-secret --from-literal=ES_PASSWORD=${PASSWORD} --from-literal=ES_USERNAME=elastic

#######################
## Kibana to visualize the trace data
#######################

cat <<EOF | kubectl apply -n observability -f -
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 7.7.0
  count: 1
  elasticsearchRef:
    name: quickstart
EOF

kubectl port-forward -n observability service/quickstart-kb-http 5601
## To get the pw
kubectl get secret quickstart-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode; echo

login:
https://localhost:5601
username: elastic
pw: <see above to outcome of the command>

#######################
## Deploy a jaeger tracing application
#######################
cat <<EOF | kubectl apply -n observability -f -
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  agent:
    strategy: DaemonSet
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: https://quickstart-es-http:9200
        tls:
          ca: /es/certificates/ca.crt
        num-shards: 1
        num-replicas: 0
    secretName: jaeger-secret
  volumeMounts:
    - name: certificates
      mountPath: /es/certificates/
      readOnly: true
  volumes:
    - name: certificates
      secret:
        secretName: quickstart-es-http-certs-public
EOF

## to visualize it
kubectl --namespace observability port-forward simple-prod-query-<POP ID> 16686:16686

#######################
## To test the setup
## Of course if you set it up to another namespace it will work, the only thing that matters is the collector URL and PORT
#######################
cat <<EOF | kubectl apply -n observability -f -
apiVersion: v1
kind: List
items:
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: jaeger-k8s-example
    labels:
      app: jaeger-k8s-example
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: jaeger-k8s-example
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          app: jaeger-k8s-example
      spec:
        containers:
          - name: jaeger-k8s-example
            env:
            - name: JAEGER_COLLECTOR_URL
              value: "simple-prod-collector.observability.svc.cluster.local"
            - name: JAEGER_COLLECTOR_PORT
              value: "14268"
            image: norbertfenk/jaeger-k8s-example:latest
            imagePullPolicy: IfNotPresent
EOF

【讨论】：

【解决方案4】：

如果需要在 Kubernetes 集群中设置 Jaeger 作为 helm chart，可以使用这个：https://github.com/jaegertracing/helm-charts/tree/master/charts/jaeger 它可以部署 Elasticsearch 或 Cassandara 作为存储后端。这只是将正确的值传递到图表的问题：

storage:
  type: elasticsearch

本节以 helm 命令为例： https://github.com/jaegertracing/helm-charts/tree/master/charts/jaeger#installing-the-chart-using-a-new-elasticsearch-cluster

【讨论】：

【解决方案5】：

对于使用 OpenTelemetry、Jaeger 和 Elasticsearch 的人来说，这就是方法。

注意使用的图片是jaegertracing/jaeger-opentelemetry-collector 和jaegertracing/jaeger-opentelemetry-agent。

version: '3.8'

services:
  collector:
    image: otel/opentelemetry-collector:latest
    command: ["--config=/conf/opentelemetry-collector.config.yaml", "--log-level=DEBUG"]
    volumes:
      - ./opentelemetry-collector.config.yaml:/conf/opentelemetry-collector.config.yaml
    ports:
      - "9464:9464"
      - "55680:55680"
      - "55681:55681"
    depends_on:
      - jaeger-collector

  jaeger-collector:
    image: jaegertracing/jaeger-opentelemetry-collector
    command: ["--es.num-shards=1", "--es.num-replicas=0", "--es.server-urls=http://elasticsearch:9200", "--collector.zipkin.host-port=:9411"]
    ports:
      - "14250"
      - "14268"
      - "9411"
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - LOG_LEVEL=debug
    restart: on-failure
    depends_on:
      - elasticsearch

  jaeger-agent:
    image: jaegertracing/jaeger-opentelemetry-agent
    command: ["--config=/config/otel-agent-config.yml", "--reporter.grpc.host-port=jaeger-collector:14250"]
    volumes:
      - ./:/config/:ro
    ports:
      - "6831/udp"
      - "6832/udp"
      - "5778"
    restart: on-failure
    depends_on:
      - jaeger-collector

  jaeger-query:
    image: jaegertracing/jaeger-query
    command: ["--es.num-shards=1", "--es.num-replicas=0", "--es.server-urls=http://elasticsearch:9200"]
    ports:
      - "16686:16686"
      - "16687"
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - LOG_LEVEL=debug
    restart: on-failure
    depends_on:
      - elasticsearch

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.9.0
    environment:
      - discovery.type=single-node
    ports:
      - "9200/tcp"

那么就需要

docker-compose up -d

参考：https://github.com/jaegertracing/jaeger/blob/master/crossdock/jaeger-opentelemetry-docker-compose.yml

【讨论】：

我收到No elasticsearch node 错误，无法通过elasticsearch 健康检查，收集器/代理不断重启，这是与嗅探相关的问题吗？
@John_J 我写答案时应该固定图像版本。我认为它现在不适用于最新版本。