【问题标题】:Create new bash var from value of dict bash var从 dict bash var 的值创建新的 bash var
【发布时间】:2023-03-24 12:32:01
【问题描述】:

我的环境创建了一个如下所示的变量:

SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}

编辑 by Ed Morton:根据the OPs comment below,这是他在上面试图描述为示例输入的内容:

$ SM_TRAINING_ENV='{"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}'

$ echo "$SM_TRAINING_ENV"
{"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}

如何创建一个等于SM_TRAINING_ENV["hyperparameters"]["model_dir"] 值的新bash 变量?

为了完整起见,我尝试了echo ${SM_TRAINING_ENV} | jq . 之类的简单操作,但我尝试的所有操作都不断出错。

编辑:我被告知这个值不是正确的 json,所以重新措辞这个问题。我认为环境将其设置为 python 字典的值,所以 jq 似乎不可用。删除了json 标签。也许这是awk 的工作?

如果我假设结构不会随着正则表达式模式s3.*?model 而改变,但我不确定如何将正则表达式模式设置为新变量,那么看起来我可以匹配我想要的值。

【问题讨论】:

  • 在这里提问之前您搜索了什么? man jq
  • 我一直在尝试 jq 和 regex,不知道如何正确使用它。不断收到错误jq: error: additional_framework_parameters/0 is not defined at <top-level>, line 1:
  • 您仍然应该展示您尝试过的内容,以使问题完整。
  • 使用该信息编辑
  • 不清楚a variable that looks like this 是什么意思,因为该语句后面的文本看起来不像变量。您是想说您有一个变量(例如,名为var),如果您执行echo "$var",则输出SM_TRAINING_ENV={"add...sh"},或者您有一个名为SM_TRAINING_ENV 的变量,如果您执行echo "$SM_TRAINING_ENV",则输出{"add...sh"} 或还有什么?

标签: bash dictionary indexing awk environment-variables


【解决方案1】:

这并不是真正的索引,但如果顺序始终相同,它就可以工作:

NEW_VAR=$(echo $SM_TRAINING_ENV | egrep -o s3.*?model | head -1)

虽然更喜欢不依赖于顺序的东西。

【讨论】:

    【解决方案2】:

    首先,您需要引用 JSON 值,以便在值中包含双引号。

    SM_TRAINING_ENV='{"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}'
    

    然后您可以使用jq 实用程序来提取您想要的值。

    new_var=$(echo "$SM_TRAINING_ENV" | jq '.hyperparameters.model_dir')
    

    【讨论】:

    • 感谢您的回复,如何通过在原始变量周围加上引号来创建新变量?
    • 如果你看到echo "$SM_TRAINING_ENV",你会看到双引号。然后他们做对了。如果他们没有正确执行,就无法修复它,因为信息会丢失。
    • 那么分配变量的代码需要像我展示的那样修复。
    • 你不能在你的代码中修复它,它必须在那里修复。
    • 他们正在创建无效的 JSON。
    猜你喜欢
    • 2013-08-10
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2016-11-07
    • 2012-08-07
    • 2016-09-17
    • 2015-05-08
    相关资源
    最近更新 更多