我希望提供一个结构化的配置文件,它对于非技术用户来说尽可能容易编辑(不幸的是它必须是一个文件),所以我想使用YAML。然而,我找不到任何方法从Unix shell脚本解析这个。


当前回答

我曾经使用python将yaml转换为json,并在jq中进行处理。

python -c "import yaml; import json; from pathlib import Path; print(json.dumps(yaml.safe_load(Path('file.yml').read_text())))" | jq '.'

其他回答

我知道这是非常具体的,但我认为我的回答可能对某些用户有帮助。 如果你的机器上安装了node和npm,你可以使用js-yaml。 首次安装:

npm i -g js-yaml
# or locally
npm i js-yaml

然后在bash脚本中

#!/bin/bash
js-yaml your-yaml-file.yml

如果你正在使用jq,你也可以这样做

#!/bin/bash
json="$(js-yaml your-yaml-file.yml)"
aproperty="$(jq '.apropery' <<< "$json")"
echo "$aproperty"

因为js-yaml将yaml文件转换为json字符串文字。然后,您可以在unix系统中的任何json解析器中使用该字符串。

如果您知道您感兴趣的标记和您期望的yaml结构,那么在Bash中编写一个简单的yaml解析器并不难。

在下面的示例中,解析器将一个结构化YAML文件读入环境变量、数组和关联数组。

注意:这个解析器的复杂性与YAML文件的结构有关。对于YAML文件的每个结构化组件,都需要一个单独的子例程。高度结构化的YAML文件可能需要更复杂的方法,例如通用的递归下降解析器。

圣诞节。yaml文件:

# Xmas YAML example
---
 # Values
 pear-tree: partridge
 turtle-doves: 2.718
 french-hens: 3

 # Array
 calling-birds:
   - huey
   - dewey
   - louie
   - fred

 # Structure
 xmas-fifth-day:
   calling-birds: four
   french-hens: 3
   golden-rings: 5
   partridges:
     count: 1
     location: "a pear tree"
   turtle-doves: two

解析器使用mapfile将文件作为数组读入内存,然后循环遍历每个标记并创建环境变量。

梨树、斑鸠和法国母鸡:最终成为简单的环境变量 呼叫鸟:变成一个数组 xmas-fifth-day:结构被表示为一个关联数组,但是如果您没有使用Bash 4.0或更高版本,您可以将这些数组编码为环境变量。 注释和空白将被忽略。

#!/bin/bash
# -------------------------------------------------------------------
# A simple parser for the xmas.yaml file
# -------------------------------------------------------------------
# 
# xmas.yaml tags
#  #                        - Ignored
#                           - Blank lines are ignored
#  ---                      - Initialiser for days-of-xmas 
#   pear-tree: partridge    - a string
#   turtle-doves: 2.718     - a string, no float type in Bash
#   french-hens: 3          - a number
#   calling-birds:          - an array of strings
#     - huey                - calling-birds[0]
#     - dewey
#     - louie
#     - fred
#   xmas-fifth-day:         - an associative array
#     calling-birds: four   - a string
#     french-hens: 3        - a number
#     golden-rings: 5       - a number
#     partridges:           - changes the key to partridges.xxx
#       count: 1            - a number
#       location: "a pear tree" - a string
#     turtle-doves: two     - a string
# 
# This requires the following routines
# ParseXMAS
#   parses #, ---, blank line
#   unexpected tag error
#   calls days-of-xmas
#
# days-of-xmas
#   parses pear-tree, turtle-doves, french-hens
#   calls calling-birds
#   calls xmas-fifth-day
# 
# calling-birds
#   elements of the array
#
# xmas-fifth-day
#   parses calling-birds, french-hens, golden-rings, turtle-doves
#   calls partridges
# 
# partridges
#   parses partridges.count, partridges.location
#

function ParseXMAS()
{

  # days-of-xmas
  #   parses pear-tree, turtle-doves, french-hens
  #   calls calling-birds
  #   calls xmas-fifth-day
  # 
  function days-of-xmas()
  {
    unset PearTree TurtleDoves FrenchHens

    while [ $CURRENT_ROW -lt $ROWS ]
    do
      LINE=( ${CONFIG[${CURRENT_ROW}]} )
      TAG=${LINE[0]}
      unset LINE[0]

      VALUE="${LINE[*]}"

      echo "  days-of-xmas[${CURRENT_ROW}] ${TAG}=${VALUE}"

      if [ "$TAG" = "pear-tree:" ]
      then
        declare -g PearTree=$VALUE
      elif [ "$TAG" = "turtle-doves:" ]
      then
        declare -g TurtleDoves=$VALUE
      elif [ "$TAG" = "french-hens:" ]
      then
        declare -g FrenchHens=$VALUE
      elif [ "$TAG" = "calling-birds:" ]
      then
        let CURRENT_ROW=$(($CURRENT_ROW + 1))
        calling-birds
        continue
      elif [ "$TAG" = "xmas-fifth-day:" ]
      then
        let CURRENT_ROW=$(($CURRENT_ROW + 1))
        xmas-fifth-day
        continue
      elif [ -z "$TAG" ] || [ "$TAG" = "#" ]
      then
        # Ignore comments and blank lines
        true
      else
        # time to bug out
        break
      fi

      let CURRENT_ROW=$(($CURRENT_ROW + 1))
    done
  }

  # calling-birds
  #   elements of the array
  function calling-birds()
  {
    unset CallingBirds

    declare -ag CallingBirds

    while [ $CURRENT_ROW -lt $ROWS ]
    do
      LINE=( ${CONFIG[${CURRENT_ROW}]} )
      TAG=${LINE[0]}
      unset LINE[0]

      VALUE="${LINE[*]}"

      echo "    calling-birds[${CURRENT_ROW}] ${TAG}=${VALUE}"

      if [ "$TAG" = "-" ]
      then
        CallingBirds[${#CallingBirds[*]}]=$VALUE
      elif [ -z "$TAG" ] || [ "$TAG" = "#" ]
      then
        # Ignore comments and blank lines
        true
      else
        # time to bug out
        break
      fi

      let CURRENT_ROW=$(($CURRENT_ROW + 1))
    done
  }

  # xmas-fifth-day
  #   parses calling-birds, french-hens, golden-rings, turtle-doves
  #   calls fifth-day-partridges
  # 
  function xmas-fifth-day()
  {
    unset XmasFifthDay

    declare -Ag XmasFifthDay

    while [ $CURRENT_ROW -lt $ROWS ]
    do
      LINE=( ${CONFIG[${CURRENT_ROW}]} )
      TAG=${LINE[0]}
      unset LINE[0]

      VALUE="${LINE[*]}"

      echo "    xmas-fifth-day[${CURRENT_ROW}] ${TAG}=${VALUE}"

      if [ "$TAG" = "calling-birds:" ]
      then
        XmasFifthDay[CallingBirds]=$VALUE
      elif [ "$TAG" = "french-hens:" ]
      then
        XmasFifthDay[FrenchHens]=$VALUE
      elif [ "$TAG" = "golden-rings:" ]
      then
        XmasFifthDay[GOLDEN-RINGS]=$VALUE
      elif [ "$TAG" = "turtle-doves:" ]
      then
        XmasFifthDay[TurtleDoves]=$VALUE
      elif [ "$TAG" = "partridges:" ]
      then
        let CURRENT_ROW=$(($CURRENT_ROW + 1))
        partridges
        continue
      elif [ -z "$TAG" ] || [ "$TAG" = "#" ]
      then
        # Ignore comments and blank lines
        true
      else
        # time to bug out
        break
      fi
 
      let CURRENT_ROW=$(($CURRENT_ROW + 1))
    done
  }

  function partridges()
  {
    while [ $CURRENT_ROW -lt $ROWS ]
    do
      LINE=( ${CONFIG[${CURRENT_ROW}]} )
      TAG=${LINE[0]}
      unset LINE[0]

      VALUE="${LINE[*]}"

      echo "      partridges[${CURRENT_ROW}] ${TAG}=${VALUE}"

      if [ "$TAG" = "count:" ]
      then
        XmasFifthDay[PARTRIDGES.COUNT]=$VALUE
      elif [ "$TAG" = "location:" ]
      then
        XmasFifthDay[PARTRIDGES.LOCATION]=$VALUE
      elif [ -z "$TAG" ] || [ "$TAG" = "#" ]
      then
        # Ignore comments and blank lines
        true
      else
        # time to bug out
        break
      fi
 
      let CURRENT_ROW=$(($CURRENT_ROW + 1))
    done
  }

  # ===================================================================
  # Load the configuration file

  mapfile CONFIG < xmas.yaml

  let ROWS=${#CONFIG[@]}
  let CURRENT_ROW=0

  # +
  # #
  #
  # ---
  # -
  while [ $CURRENT_ROW -lt $ROWS ]
  do
    LINE=( ${CONFIG[${CURRENT_ROW}]} )
    TAG=${LINE[0]}
    unset LINE[0]

    VALUE="${LINE[*]}"

    echo "[${CURRENT_ROW}] ${TAG}=${VALUE}"

    if [ "$TAG" = "---" ]
    then
        let CURRENT_ROW=$(($CURRENT_ROW + 1))
        days-of-xmas
        continue
    elif [ -z "$TAG" ] || [ "$TAG" = "#" ]
    then
        # Ignore comments and blank lines
        true
    else
        echo "Unexpected tag at line $(($CURRENT_ROW + 1)): <${TAG}>={${VALUE}}"
        break
    fi

    let CURRENT_ROW=$(($CURRENT_ROW + 1))
  done
}

echo =========================================
ParseXMAS

echo =========================================
declare -p PearTree
declare -p TurtleDoves
declare -p FrenchHens
declare -p CallingBirds
declare -p XmasFifthDay

这将产生以下输出

=========================================
[0] #=Xmas YAML example
[1] ---=
  days-of-xmas[2] #=Values
  days-of-xmas[3] pear-tree:=partridge
  days-of-xmas[4] turtle-doves:=2.718
  days-of-xmas[5] french-hens:=3
  days-of-xmas[6] =
  days-of-xmas[7] #=Array
  days-of-xmas[8] calling-birds:=
    calling-birds[9] -=huey
    calling-birds[10] -=dewey
    calling-birds[11] -=louie
    calling-birds[12] -=fred
    calling-birds[13] =
    calling-birds[14] #=Structure
    calling-birds[15] xmas-fifth-day:=
  days-of-xmas[15] xmas-fifth-day:=
    xmas-fifth-day[16] calling-birds:=four
    xmas-fifth-day[17] french-hens:=3
    xmas-fifth-day[18] golden-rings:=5
    xmas-fifth-day[19] partridges:=
      partridges[20] count:=1
      partridges[21] location:="a pear tree"
      partridges[22] turtle-doves:=two
    xmas-fifth-day[22] turtle-doves:=two
=========================================
declare -- PearTree="partridge"
declare -- TurtleDoves="2.718"
declare -- FrenchHens="3"
declare -a CallingBirds=([0]="huey" [1]="dewey" [2]="louie" [3]="fred")
declare -A XmasFifthDay=([CallingBirds]="four" [PARTRIDGES.LOCATION]="\"a pear tree\"" [FrenchHens]="3" [GOLDEN-RINGS]="5" [PARTRIDGES.COUNT]="1" [TurtleDoves]="two" )

如果你需要一个单一的值,你可以使用一个工具将你的YAML文档转换为JSON并提供给jq,例如yq。

sample.yaml的内容:

---
bob:
  item1:
    cats: bananas
  item2:
    cats: apples
  thing:
    cats: oranges

例子:

$ yq -r '.bob["thing"]["cats"]' sample.yaml 
oranges

现在做这件事的一个快速方法(以前的方法对我没用):

sudo wget https://github.com/mikefarah/yq/releases/download/v4.4.1/yq_linux_amd64 -O /usr/bin/yq &&\
sudo chmod +x /usr/bin/yq

示例asd.yaml:

a_list:
  - key1: value1
    key2: value2
    key3: value3

解析:根

user@vm:~$ yq e '.' asd.yaml                                                                                                         
a_list:
  - key1: value1
    key2: value2
    key3: value3

解析key3:

user@vm:~$ yq e '.a_list[0].key3' asd.yaml                                                                                             
value3

我刚刚写了一个解析器,我称之为Yay!(Yaml不是Yamlesque!)它解析Yamlesque, Yaml的一个小子集。因此,如果您正在为Bash寻找一个100%兼容的YAML解析器,那么这不是它。但是,为了引用OP,如果您想要一个结构化的配置文件,使非技术用户能够尽可能容易地编辑它,并且是类似yaml的,那么您可能会对它感兴趣。

它受到前面答案的启发,但编写了关联数组(是的,它需要Bash 4.x)而不是基本变量。它以一种允许在不事先了解键的情况下解析数据的方式进行操作,从而可以编写数据驱动的代码。

除了键/值数组元素外,每个数组都有一个包含键名列表的键数组、一个包含子数组名称的子数组和一个引用其父数组的父键。

这是Yamlesque的一个例子:

root_key1: this is value one
root_key2: "this is value two"

drink:
  state: liquid
  coffee:
    best_served: hot
    colour: brown
  orange_juice:
    best_served: cold
    colour: orange

food:
  state: solid
  apple_pie:
    best_served: warm

root_key_3: this is value three

下面是一个如何使用它的例子:

#!/bin/bash
# An example showing how to use Yay

. /usr/lib/yay

# helper to get array value at key
value() { eval echo \${$1[$2]}; }

# print a data collection
print_collection() {
  for k in $(value $1 keys)
  do
    echo "$2$k = $(value $1 $k)"
  done

  for c in $(value $1 children)
  do
    echo -e "$2$c\n$2{"
    print_collection $c "  $2"
    echo "$2}"
  done
}

yay example
print_collection example

输出:

root_key1 = this is value one
root_key2 = this is value two
root_key_3 = this is value three
example_drink
{
  state = liquid
  example_coffee
  {
    best_served = hot
    colour = brown
  }
  example_orange_juice
  {
    best_served = cold
    colour = orange
  }
}
example_food
{
  state = solid
  example_apple_pie
  {
    best_served = warm
  }
}

下面是解析器:

yay_parse() {

   # find input file
   for f in "$1" "$1.yay" "$1.yml"
   do
     [[ -f "$f" ]] && input="$f" && break
   done
   [[ -z "$input" ]] && exit 1

   # use given dataset prefix or imply from file name
   [[ -n "$2" ]] && local prefix="$2" || {
     local prefix=$(basename "$input"); prefix=${prefix%.*}
   }

   echo "declare -g -A $prefix;"

   local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo @|tr @ '\034')
   sed -n -e "s|^\($s\)\($w\)$s:$s\"\(.*\)\"$s\$|\1$fs\2$fs\3|p" \
          -e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p" "$input" |
   awk -F$fs '{
      indent       = length($1)/2;
      key          = $2;
      value        = $3;

      # No prefix or parent for the top level (indent zero)
      root_prefix  = "'$prefix'_";
      if (indent ==0 ) {
        prefix = "";          parent_key = "'$prefix'";
      } else {
        prefix = root_prefix; parent_key = keys[indent-1];
      }

      keys[indent] = key;

      # remove keys left behind if prior row was indented more than this row
      for (i in keys) {if (i > indent) {delete keys[i]}}

      if (length(value) > 0) {
         # value
         printf("%s%s[%s]=\"%s\";\n", prefix, parent_key , key, value);
         printf("%s%s[keys]+=\" %s\";\n", prefix, parent_key , key);
      } else {
         # collection
         printf("%s%s[children]+=\" %s%s\";\n", prefix, parent_key , root_prefix, key);
         printf("declare -g -A %s%s;\n", root_prefix, key);
         printf("%s%s[parent]=\"%s%s\";\n", root_prefix, key, prefix, parent_key);
      }
   }'
}

# helper to load yay data file
yay() { eval $(yay_parse "$@"); }

在链接的源文件中有一些文档,下面是对代码功能的简短解释。

yay_parse函数首先定位输入文件或退出,退出状态为1。接下来,它确定数据集前缀,要么显式指定,要么从文件名派生。

它将有效的bash命令写入其标准输出,如果执行该输出,则定义表示输入数据文件内容的数组。第一个定义了顶级数组:

echo "declare -g -A $prefix;"

注意,数组声明是关联的(-A),这是Bash版本4的一个特性。声明也是全局的(-g),所以它们可以在函数中执行,但像yay helper一样可用于全局作用域:

yay() { eval $(yay_parse "$@"); }

最初使用sed处理输入数据。它删除不匹配Yamlesque格式规范的行,然后用ASCII文件分隔符分隔有效的Yamlesque字段,并删除值字段周围的任何双引号。

 local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo @|tr @ '\034')
 sed -n -e "s|^\($s\)\($w\)$s:$s\"\(.*\)\"$s\$|\1$fs\2$fs\3|p" \
        -e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p" "$input" |

这两种表达是相似的;它们的不同之处在于第一个选择了带引号的值,而第二个选择了不带引号的值。

使用文件分隔符(28/十六进制12/八进制034)是因为,作为一个不可打印字符,它不太可能出现在输入数据中。

结果通过管道传输到awk中,每次处理一行输入。它使用FS字符将每个字段分配给一个变量:

indent       = length($1)/2;
key          = $2;
value        = $3;

所有行都有缩进(可能为零)和键,但它们并不都有值。它为包含前导空白的第一个字段的长度除以2的行计算缩进级别。没有缩进的顶级项位于缩进级别0。

接下来,它计算出为当前项使用什么前缀。这是添加到键名中以创建数组名的内容。顶级数组有一个root_prefix,它被定义为数据集名称和一个下划线:

root_prefix  = "'$prefix'_";
if (indent ==0 ) {
  prefix = "";          parent_key = "'$prefix'";
} else {
  prefix = root_prefix; parent_key = keys[indent-1];
}

parent_key是位于当前行缩进级别之上的缩进级别的键,表示当前行所属的集合。集合的键/值对将存储在一个数组中,其名称定义为前缀和parent_key的连接。

对于顶层(缩进级别0),数据集前缀被用作父键,因此它没有前缀(它被设置为“”)。所有其他数组都以根前缀作为前缀。

接下来,将当前键插入到包含键的(awk-internal)数组中。该数组在整个awk会话中持续存在,因此包含先前行插入的键。键以其缩进作为数组索引插入数组。

keys[indent] = key;

因为这个数组包含前几行的键,所以任何缩进级别大于当前行缩进级别的键都将被移除:

 for (i in keys) {if (i > indent) {delete keys[i]}}

这将留下包含从根缩进级别0到当前行的键链的keys数组。它删除前一行缩进比当前行更深时保留的过时键。

最后一部分输出bash命令:不带值的输入行开始一个新的缩进级别(在YAML中是一个集合),带值的输入行向当前集合添加一个键。

集合的名称是当前行的前缀和parent_key的组合。

当一个键有一个值时,具有该值的键会被赋给当前集合,如下所示:

printf("%s%s[%s]=\"%s\";\n", prefix, parent_key , key, value);
printf("%s%s[keys]+=\" %s\";\n", prefix, parent_key , key);

第一个语句输出将值赋给一个以键命名的关联数组元素的命令,第二个语句输出将键添加到集合的空格分隔键列表的命令:

<current_collection>[<key>]="<value>";
<current_collection>[keys]+=" <key>";

当一个键没有值时,一个新的集合像这样开始:

printf("%s%s[children]+=\" %s%s\";\n", prefix, parent_key , root_prefix, key);
printf("declare -g -A %s%s;\n", root_prefix, key);

第一个语句输出将新集合添加到当前集合的空格分隔子列表的命令,第二个语句输出为新集合声明一个新的关联数组的命令:

<current_collection>[children]+=" <new_collection>"
declare -g -A <new_collection>;

yay_parse的所有输出都可以通过bash eval或源内置命令解析为bash命令。