如何解决 Argo 输出参数大小限制？

Question

我有一个循环 JSON 数组的 Argo 工作流程。当列表变得太大时，我会收到如下错误：

time="some-time" level=fatal msg="Pod \"some-pod-name\" is invalid: metadata.annotations: Too long: must have at most 262144 characters"

或者，in newer versions of Argo：

Output is larger than the maximum allowed size of 256 kB, only the last 256 kB were saved

如何在不达到大小限制的情况下遍历这个大型 JSON 数组？

我的工作流程看起来有点像这样，但 JSON 数组更大：

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-sequence-
spec:
  entrypoint: loops-sequence
  templates:
  - name: loops-sequence
    steps:
    - - name: get-items
        template: get-items
    - - name: sequence-param
        template: echo
        arguments:
          parameters:
          - name: item
            value: "{{item}}"
        withParam: "{{steps.get-items.outputs.parameters.items}}"
  - name: get-items
    container:
      image: alpine:latest
      command: ["/bin/sh", "-c"]
      args: ["echo '[\"a\", \"b\", \"c\"]' > /tmp/items"]
    outputs:
      parameters:
      - name: items
        valueFrom:
          path: /tmp/items
  - name: echo
    inputs:
      parameters:
      - name: item
    container:
      image: stedolan/jq:latest
      command: [echo, "{{inputs.parameters.item}}"]

Answer 1

不要将 JSON 数组写入输出参数，而是将其写入工件并将其长度写入输出参数。然后您可以使用withSequence 循环遍历数组索引并从 JSON 工件中检索相应的项目。

这是一个包含硬编码 JSON 数组和项目计数的示例。你的get-items 步骤肯定会更复杂。

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-sequence-
spec:
  entrypoint: loops-sequence
  templates:
  - name: loops-sequence
    steps:
    - - name: get-items
        template: get-items
    - - name: sequence-param
        template: echo
        arguments:
          parameters:
          - name: index
            value: "{{item}}"
          artifacts:
          - name: items
            from: "{{steps.get-items.outputs.artifacts.items}}"
        withSequence:
          count: "{{steps.get-items.outputs.parameters.count}}"
  - name: get-items
    container:
      image: alpine:latest
      command: ["/bin/sh", "-c"]
      args: ["echo '[\"a\", \"b\", \"c\"]' > /tmp/items && echo '3' > /tmp/count"]
    outputs:
      artifacts:
      - name: items
        path: /tmp/items
      parameters:
      - name: count
        valueFrom:
          path: /tmp/count
  - name: echo
    inputs:
      parameters:
      - name: index
      artifacts:
      - name: items
        path: /tmp/items
    container:
      image: stedolan/jq:latest
      command: [sh, -c]
      args: ["cat /tmp/items | jq '.[{{inputs.parameters.index}}]'"]