Keda ScaledJob kafka-trigger 缩放不正确答案

【问题标题】：Keda ScaledJob kafka-trigger scaling incorrectlyKeda ScaledJob kafka-trigger 缩放不正确
【发布时间】：2022-10-02 11:29:03
【问题描述】：

科达配置如下

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: job-message-consumer
spec:
  jobTargetRef:
    parallelism: 1
    template:
      spec:
        containers:
          - name: {{ .Values.JobMessageConsumer.name }}
            image: {{ .Values.image.repository }}
            args:
              - python
              - manage.py
              - jobs_consumer
            imagePullPolicy: {{ .Values.image.pullPolicy }}
            env:
              - name: SSL_CA_LOCATION
                value: {{ .Values.kafka.sslCaDirMount }}/ca.crt
              - name: SSL_CA_DIR_MOUNT
                value: {{ .Values.kafka.sslCaDirMount }}
              - name: SSL_SECRET_NAME
                value: {{ .Values.kafka.sslSecretName }}
              - name: SERVICE_CONSUMER_GROUP_ID
                value: {{ .Values.kafka.CONSUMER_GROUP_ID }}
              - name: KAFKA_SERVICE_JOBS_TOPIC
                value: {{ .Values.job_consumer_service.consumption_topic }}
              {{- include \"env\" . | nindent 14 }}
            resources:
              limits:
                cpu: \"2\"
                memory: \"4Gi\"
              requests:
                cpu: \"500m\"
                memory: \"500Mi\"
            volumeMounts:
              - name: user-certs
                mountPath: {{ .Values.kafka.sslUserCertDir }}
              - name: ca-certs
                mountPath: {{ .Values.kafka.sslCaDirMount }}
        volumes:
          - name: user-certs
            secret:
              secretName: {{ .Release.Name }}-kafka-user-certs
          - name: ca-certs
            secret:
              secretName: kafka-cluster-ca-cert
  pollingInterval: 20
  maxReplicaCount: 100
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 5
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: \"kafka-kafka-bootstrap.kafka:9093\"
        consumerGroup: \"JobConsumer\"
        topic: {{ .Values.job_consumer_service.consumption_topic }}
        lagThreshold: \"1\"
        offsetResetPolicy: latest
      authenticationRef:
        name: {{ .Release.Name }}-keda-trigger-auth-kafka-credential

keda-jobs 需要运行很长时间才能完成任务，因此是 ScaledJob。但是使用上述 ScaledJobs 配置，keda 作业每 20 秒缩放一次。我希望 keda 作业仅基于 kafka-trigger 上的消息而不是基于 pollinginterval 进行扩展。谁可以帮我这个事

标签： python kubernetes apache-kafka keda keda-scaledjob

【解决方案1】：

请将此参数添加到triggers 部分的kafka 触发器中：

        scaleToZeroOnInvalidOffset: "true"

如果在2.7.0 及更高版本中添加的scaleToZeroOnInvalidOffset 参数设置为"true"，则当该分区没有有效偏移量[1] 时，该分区的消费者将缩放为zero。

[1] - https://keda.sh/docs/2.7/scalers/apache-kafka/

【讨论】：