【问题标题】:How can I parse a Json file in linux using bash or jq for setting a results如何使用 bash 或 jq 在 linux 中解析 Json 文件以设置结果
【发布时间】:2019-11-12 08:17:49
【问题描述】:

在我的 linux 机器上,我正在尝试解析“Results.json”文件。我想从中搜索和 grep 的两个字符串是

  1. “first_review”:真
  2. “firstReview”:真

如果其中任何一个(或两者)为真,那么我想将结果打印为 'fail' 并且只有当两者都为 'false' 时,我才想将结果打印为通过。

我怎样才能正确地做到这一点,请提出建议。

这里是 Json 文件(results.json)

{
  "scan-dir": "/var/local",
  "scan_time_ms": "20394 ms",
  "by": "a user",
  "project": "local",
  "date": "2019-06-30T10:48:07.270Z",
  "g_version": "2.2.16",
  "q_version": "1.0.112",
  "directories_scanned": 25,
  "files_scanned": 41,
  "packages_found": 3,
  "packages": [
    {
      "name": "flow",
      "version": "unknown",
      "path": "development_environment/flow",
      "source": "__init__.py",
      "file_path": "development_environment/flow/__init__.py",
      "analyzer": "package-finder",
      "license_files": [
        {
          "file": "development_environment/flow/__init__.py",
          "legal": false,
          "legal_category": "Other",
          "contains_keywords": true,
          "blocks": [
            {
              "text": "# -*- coding: utf-8 -*-\n#\n# ",
              "matches": null
            },
            {
              "text": "Licensed",
              "matches": "KEYWORD"
            },
            {
              "text": " to the ",
              "matches": null
            },
            {
              "text": "Apache Software Foundation (ASF) under one",
              "matches": "ACCEPTABLE"
            },
            {
              "text": "\n# or more contributor ",
              "matches": null
            },
            {
              "text": "license",
              "matches": "KEYWORD"
            },
            {
              "text": " agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF ",
              "matches": null
            },
            {
              "text": "licenses",
              "matches": "KEYWORD"
            },
            {
              "text": " this file\n# to you under",
              "matches": null
            },
            {
              "text": " the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.",
              "matches": "Apache-2.0"
            }
          ],
          "license_types": [
            "Apache-2.0"
          ],
          "license_approval_flags": {
            "approved": true,
            "first_review": false,
            "needs_legal_approval": false,
            "prohibited": false,
            "no_license_info": false
          },
          "affects_package_license_types": true
        },
        {
          "file": "development_environment/flow/base_flow.py",
          "legal": false,
          "legal_category": "Other",
          "contains_keywords": true,
          "blocks": [
            {
              "text": "# -*- coding: utf-8 -*-\n#\n# ",
              "matches": null
            },
            {
              "text": "Licensed",
              "matches": "KEYWORD"
            },
            {
              "text": " to the ",
              "matches": null
            },
            {
              "text": "Apache Software Foundation (ASF) under one",
              "matches": "ACCEPTABLE"
            },
            {
              "text": "\n# or more contributor ",
              "matches": null
            },
            {
              "text": "license",
              "matches": "KEYWORD"
            },
            {
              "text": " agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF ",
              "matches": null
            },
            {
              "text": "licenses",
              "matches": "KEYWORD"
            },
            {
              "text": " this file\n# to you under",
              "matches": null
            },
            {
              "text": " the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.",
              "matches": "Apache-2.0"
            },
            {
              "text": "\n\n\"\"\"Base classes for flow and flowBag.\"\"\"\n\nfrom abc import ABCMeta, abstractmethod\n\n\nclass Baseflow(metaclass=ABCMeta):\n    \"\"\"\n    Base flow object that both the Simpleflow and flow inherit.\n    \"\"\"\n\n    @property\n    @abstractmethod\n    def flow_id(self):\n        \"\"\"\n        :return: the flow ID\n        :rtype: unicode\n        \"\"\"\n        raise NotImplementedError()\n\n    @property\n    @abstractmethod\n    def task_ids(self):\n        \"\"\"\n        :return: A list of task IDs that are in this flow\n        :rtype: List[unicode]\n        \"\"\"\n        raise NotImplementedError()\n\n    @property\n    @abstractmethod\n    def full_filepath(self):\n        \"\"\"\n        :return: The absolute path to the file that contains this flow's definition\n        :rtype: unicode\n        \"\"\"\n        raise NotImplementedError()\n\n    @property\n    @abstractmethod\n    def concurrency(self):\n        \"\"\"\n        :return: maximum number of tasks that can run simultaneously from this flow\n        :rtype: int\n        \"\"\"\n        raise NotImplementedError()\n\n    @abstractmethod\n    def is_paused(self):\n        \"\"\"\n        :return: whether this flow is paused or not\n        :rtype: bool\n        \"\"\"\n        raise NotImplementedError()\n\n    @abstractmethod\n    def pickle_id(self):\n        \"\"\"\n        :return: The pickle ID for this flow, if it has one. Otherwise None.\n        :rtype: unicode\n        \"\"\"\n        raise NotImplementedError\n\n\nclass BaseflowBag:\n    \"\"\"\n    Base object that both the SimpleflowBag and flowBag inherit.\n    \"\"\"\n    @property\n    @abstractmethod\n    def flow_ids(self):\n        \"\"\"\n        :return: a list of flow IDs in this bag\n        :rtype: List[unicode]\n        \"\"\"\n        raise NotImplementedError()\n\n    @abstractmethod\n    def get_flow(self, flow_id):\n        \"\"\"\n        :return: whether the task exists in this bag\n        :rtype: tzlflow.flow.base_flow.Baseflow\n        \"\"\"\n        raise NotImplementedError()",
              "matches": null
            }
          ],
          "license_types": [
            "Apache-2.0"
          ],
          "license_approval_flags": {
            "approved": true,
            "first_review": false,
            "needs_legal_approval": false,
            "prohibited": false,
            "no_license_info": false
          },
          "affects_package_license_types": true
        }
      ],
      "license_types": [
        "Apache-2.0"
      ],
      "license_approval_flags": {
        "approved": true,
        "first_review": false,
        "needs_legal_approval": false,
        "prohibited": false,
        "no_license_info": false,
        "reference_only": false
      },
      "license_approval_status": "Approved",
      "package_hash": "c7d6757c6c814a22b44d8299cace3ec1",
      "Treat": {
        "TreatReviewed": false,
        "possiblyRelatedPackages": [],
        "searchedName": "flow",
        "searchedVersion": "unknown",
        "allLicensesMatch": false,
        "firstReview": true,
        "firstReviewReason": "New package and not exempt"
      }
    },
    {
      "name": "testscan-results",
      "version": "unknown",
      "language": "unknown",
      "analyzer": "License verify",
      "path": "testscan-results",
      "file_path": "testscan-results/License-Report.html",
      "license_files": [
        {
          "file": "testscan-results/License-Report.html",
          "hash": "db02a6be21775d25af4cfdb993442a8e",
          "legal": true,
          "legal_category": "License",
          "contains_keywords": true,
          "blocks": [
            {
              "text": "<style>h2 strong{color: red;} pre{padding-left:1em; background-color:cornsilk; border:1px solid black; margin: 0 2em}</style><h1 id=\"",
              "matches": null
            },
            {
              "text": "nolicenseswerefoundwhichrequirereview",
              "matches": "KEYWORD"
            },
            {
              "text": "\">No ",
              "matches": null
            },
            {
              "text": "licenses",
              "matches": "KEYWORD"
            },
            {
              "text": " were found which require review.</h1>",
              "matches": null
            }
          ],
          "license_types": [
            "KEYWORD",
            "UNKNOWN"
          ],
          "license_approval_flags": {
            "approved": false,
            "first_review": true,
            "needs_legal_approval": false,
            "prohibited": false,
            "no_license_info": false
          },
          "affects_package_license_types": true
        },
        {
          "file": "test_scan-results/License-Report.md",
          "hash": "b85fd1e259874c81c31253e352068b6f7",
          "legal": true,
          "legal_category": "License",
          "contains_keywords": true,
          "blocks": [
            {
              "text": "# No ",
              "matches": null
            },
            {
              "text": "licenses",
              "matches": "KEYWORD"
            },
            {
              "text": " were found which require review.",
              "matches": null
            }
          ],
          "license_types": [
            "KEYWORD",
            "UNKNOWN"
          ],
          "license_approval_flags": {
            "approved": false,
            "first_review": true,
            "needs_legal_approval": false,
            "prohibited": false,
            "no_license_info": false
          },
          "affects_package_license_types": true
        }
      ],
      "license_types": [
        "KEYWORD",
        "UNKNOWN"
      ],
      "license_approval_flags": {
        "approved": false,
        "first_review": true,
        "needs_legal_approval": false,
        "prohibited": false,
        "no_license_info": false,
        "reference_only": false
      },
      "license_approval_status": "Needs Review",
      "package_hash": "11d905fbde2bbd890c5d3e677704185a"
    },
    {
      "name": "unknown",
      "version": "unknown",
      "language": "unknown",
      "analyzer": "License verify",
      "path": "development_environment/README.md",
      "file_path": "development_environment/README.md",
      "license_files": [
        {
          "file": "development_environment/README.md",
          "legal": false,
          "legal_category": "Other",
          "contains_keywords": false,
          "blocks": [
            {
              "text": "# tzlflow EnV\n\nA development environment for ",
              "matches": null
            },
            {
              "text": "Flow deploy upon tzl Cloud utilising shared Database i",
              "matches": "ACCEPTABLE"
            },
            {
              "text": ".e.",
              "matches": null
            },
            {
              "text": " PostGresql ",
              "matches": "PostgreSQL"
            },
            {
              "text": "and Messenging Services i.e. Redis.  Intention is that developers can get up and running quickly with working tzlflow environment (precanned to G+EHR software stack).\n\n## Development and Testing\nSetting up a development environment utilising developer free tzl Cloud. \n\n### Pre-requistes\n- Sign up for [tzl Cloud Account](",
              "matches": null
            },

            {
              "text": ")\n-- tzl Cloud CLI\n-- kubectl \n- Access to [repo](",
              "matches": null
            },
            {
              "text": ")\n- Docker\n- Helm\n\nFollowing two are not pre-req but you may decide to 'bring your own'\n- Postgres DB\n- Redis \nIf using your own following comment instructions in values.yaml to configure. \n\n### Clone repo\n\n- git clone  git@",
              "matches": null
            },

            {
              "text": ":TzlCo/Tzl-tools\n- cd Tzl-tools\n\n### Update configuration\n\nUnder",
              "matches": null
            },
            {
              "text": " postgresql ",
              "matches": "PostgreSQL"
            },

            {
              "text": " PostgreSQL ",
              "matches": "PostgreSQL"
            },
            {
              "text": "port\n  service:\n    port: 30406\n\n  ##",
              "matches": null
            },
            {
              "text": " PostgreSQL ",
              "matches": "PostgreSQL"
            },
            {
              "text": "User to create.\n  postgresUser: tzl_cloud_xxx\n  ##\n  ##",
              "matches": null
            },
            {
              "text": " PostgreSQL ",
              "matches": "PostgreSQL"
            },
            {
              "text": "Password for the new user.\n  ## If not set, a random 10 characters password will be used.\n  postgresPassword: xxxx\n  ##\n  ##",
              "matches": null
            },
            {
              "text": " PostgreSQL ",
              "matches": "PostgreSQL"
            },
            {
              "text": "Database to create.\n  postgresDatabase: tzlclouddb\n```\n\nand Redis\n\n```redis:\n  ##\n  ## Use the redis chart dependency.\n  ## Set to false if bringing your own redis.\n  enabled: false\n  ##\n  ## If you are bringing your own redis, you can set the host in redisHost.\n  redisHost: xxxx.databases.appdomain.cloud\n  ##\n  ## Redis password\n  ##\n  password: xxxxx\n  username: admin\n  ##\n  ## Master configuration\n  master:\n    #Redis Port - missing from chart.\n    port: 31932\n    \n```\n\n### flow Git Repo configuration\n\nFollow instructions to generate ssh key for git repo.\n",
              "matches": null
            },

            {
              "text": "#deploy-key\nIf you are using a private Git repo, you can set `flows.gitSecret` to the name of a secret you created containing private keys and a `known_hosts` file.\n\nFor example, this will create a secret named `my-git-secret` from your ed25519 key and known_hosts file stored in your home directory:  `kubectl create secret generic flow-git-secret --from-file=gitSshKey=g+ehr_flows --from-file=known_hosts=known_hosts --from-file=gitSshKey.pub=g+ehr_flows.pub`\n\n\n\n###  Deploy\n\nLog onto tzl Cloud using CLI\n```tzlcloud login -a ",
              "matches": null
            },

            {
              "text": " -r us-south -g Tzl-Dev --sso```\n\nDownload the kubeconfig files for your cluster.\n```tzlcloud ks cluster-config --cluster Tzl_dev```\n\nUsing the output from the previous step, set the KUBECONFIG environment variable. The command looks similar to the following example:\n```export KUBECONFIG=/Users/$USER/.tzl/plugins/container-service/clusters/Tzl_dev/kube-config-dal10-Tzl_dev.yml```\n\nUsing Helm deploy tzlflow\n```helm install --namespace \"default\" --name \"tzlflow\" stable/tzlflow -f values.yaml```\n\nTreat access to tzlflow admin\n```export POD_NAME=$(kubectl get pods --namespace default -l \"component=web,app=tzlflow\" -o jsonpath=\"{.items[0].metadata.name}\")\n   echo http://127.0.0.1:8080\n   kubectl port-forward --namespace default $POD_NAME 8080:8080```\n   \n\n\n# TOD0 \n- python script to automate deploy. \n- Steps to integrate with different dependency e.g. postgres or redis.\n- flow script locations.\n- instructions to use github repo as flow location.\n",
              "matches": null
            }
          ],
          "license_types": [
            "PostgreSQL"
          ],
          "license_approval_flags": {
            "approved": true,
            "first_review": false,
            "needs_legal_approval": false,
            "prohibited": false,
            "no_license_info": false
          },
          "affects_package_license_types": true
        }
      ],
      "license_types": [
        "PostgreSQL"
      ],
      "license_approval_flags": {
        "approved": true,
        "first_review": false,
        "needs_legal_approval": false,
        "prohibited": false,
        "no_license_info": false,
        "reference_only": false
      },
      "license_approval_status": "Approved",
      "package_hash": "2776a8d434cd903a7cd441c9dcbdd4ed"
    }
  ],
  "Treat_checked": true
}

以下是我在每种方法上尝试过但无法进一步取得进展的方法:

  1. 首先尝试使用 jq 并可以解析路径的 json 并获取值,即。所需节点的“真”或“假”,但不知道如何使用这些真值和假值进行下一步。

    用过类似的东西:

    cat results.json |jq '.packages[0].license_approval_flags.first_review'

  2. 在另一个想法中想要 Cat 然后 grep json 文件内容,对于上述两个必需的完整字符串值(“first_review”:true 和“firstReview”:true),以便如果得到任何结果为“true”,那么我想要将“结果”设置/打印为“失败”,但无法得到任何正确的响应,因为我认为字符串的引号没有在我的 grep 搜索中被过滤。

    使用的grep命令如下:

    grep -o '"[firstReview = true"]\+"' results.json

  3. 然后想使用 Python 解析 json,然后完成我想要的工作,但无法超越此页面“如何将数据从 json 解析到 python”,这似乎详细说明了我可能会使用的东西,但我还没有python所需的基础知识。

这可能并不难,因为我通过互联网阅读,但我还无法完成。

我怎样才能正确地做到这一点,请提出建议。

【问题讨论】:

  • 为什么要随意选择.packages数组的第一个元素?这很重要还是只是一种方法?
  • 要求很不明确。如果您遵循minimal reproducible example 指南,将会有很大帮助。显示预期输出的几个最小示例将非常有帮助。
  • .packages 数组具有我的测试所需的节点。因此正在使用它。将来会使用示例指南,谢谢。

标签: json linux bash jq jsonparser


【解决方案1】:

据我了解,这是针对该问题的 解决方案。该解决方案非常简短,因为它不知道两个键的位置,但您可能需要根据您的详细要求对其进行修改:

# Check if both are false
reduce (.. | objects) as $o ({};
  if $o.firstReview == false then .firstReview = false else . end
   | if $o.first_review == false then .first_review = false else . end )
| if length == 2 then "Pass" else "Fail" end

这个解决方案有点微妙,因为它在应用于对象时依赖于length 的语义,但它可能在感兴趣的键的位置方面具有鲁棒性的优势。

按包解决方案

如果您想要每个“包”的结果,只需添加包装器.packages | map(_),即用上述过滤器替换_,产生:

.packages
| map(reduce (.. | objects) as $o ({};
      if $o.firstReview == false then .firstReview = false else . end
       | if $o.first_review == false then .first_review = false else . end )
    | if length == 2 then "Pass" else "Fail" end
)

【讨论】:

  • 说实话,我还没有习惯如何添加包装器。但是,在使用此解决方案测试 mu json 时,对于两个测试文件,我得到了预期的“失败”结果,但对于第三个文件,我期望它是“通过”,因为所有值都设置为“假”,但我得到了'失败'。
  • 感谢您的回答,Peak。这是我使用它的方式,cat Results_All-False_So_PASS.json |jq 'reduce (.. | objects) as $o ({}; &gt; if .firstReview == false then . + {firstReview} &gt; elif .first_review == false then . + {first_review} &gt; else . &gt; end) &gt; | if length == 2 then "Pass" else "Fail" end' "Fail" 在下面使用 Jeff 的解决方案时,我得到了第三个 Json 作为 Pass(正如我所期望的那样)。这里有点困惑。
  • @Vizag - 您遗漏了一些 $o。请将您的 jq 过滤器与此页面上的过滤器进行比较。
  • 知道了,谢谢。您能否指出任何示例,关于如何使用您之前建议的这种包装器。再次感谢。
【解决方案2】:

你可以这样做:

cat results.json |
  jq '.packages[0].Treat.firstReview
       or .packages[0].license_approval_flags.first_review
      | if . == true then "Fail" else "Pass" end'
false

【讨论】:

  • 顺便说一句,使用&lt;results.json而不是cat results.json |更有效;这样jq 直接从文件中读取,而不是从本身读取文件的单独程序的输出中读取。对于jq 来说差别不大,但对于像sorttail 这样可以从文件中跳转或并行读取不同部分中受益的工具,差别可能是巨大的,所以这是一个好习惯。
【解决方案3】:

如果您只想在 json 中查找任何具有 true 值的命名属性,您有很多选择。要获得快速而肮脏的解决方案,您可以这样做:

if any(..|objects|.first_review,.firstReview; . == true) then "Fail" else "Pass" end

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 2021-02-01
    • 1970-01-01
    • 1970-01-01
    • 2022-01-10
    • 2019-12-01
    • 1970-01-01
    相关资源
    最近更新 更多