README.md

README

Demonstrates some examples of using yq to process yaml files

It also can process xml here

Contents

• README
  • Contents
  • Preeqs
  • Simple
  • Anchors
  • Merging
  • Validation
  • Replace values
  • Looping overs arrays in bash
  • Conversion
    • JSON
  • XML
  • CSV
  • Resources

Preeqs

brew info yq

brew install yq
brew upgrade yq

# using version 4.34.1 at time of writing
yq --version 

Simple

# read and process
yq eval ./examples/simple.yaml 

# render with debugging 
yq eval -v ./examples/simple.yaml   

# extract a field
yq eval .image ./examples/simple.yaml 

Anchors

# show internal representation of yaml
yq eval ./examples/anchors.yaml

# playing with anchors (THIS NO LONGER WORKS)
yq '. | explode(.)' ./examples/anchors.yaml  

Merging

yq '. *= load("./examples/simple.yaml")' ./examples/anchors.yaml

Show an example of docker compose overrides.

yq '. *= load("./docker-compose/docker-compose.override.yaml")' ./docker-compose/docker-compose.yaml

Validation

yq -P ./examples/broken.yaml                

Replace values

# replace a simple value
yq e '(.replacethis) |= "REPLACED"' ./examples/replace.yaml

# replace multiple object values
yq e '(.image) += {"repository": "docker.io/library/verdaccio.prometheus", "tag": "latest"}' ./examples/replace.yaml

# replace a substring
yq e '(.embedded) |= sub("\${APIKEY}", "theapikey")' ./examples/replace.yaml

# replace a value in an array
yq e '(.service[]) |= sub("\${APIKEY}", "theapikey")' ./examples/replace.yaml

# add a key value to an array
yq e '.extraEnvVars += [{"name": "VERDACCIO_PORT", "value": "4873"}]' ./examples/replace.yaml

# replace a value in a map
yq e '(.datasources.jsonData[]) |= sub("\${AWS_ROLE_ARN}", "arn:aws:iam::0000000000000:role/myrole")' ./examples/replace.yaml

# replace a value in a map for a specific key
yq e '(.datasources.jsonData["assumeRoleArn"]) |= "arn:aws:iam::0000000000000:role/myrole"' ./examples/replace.yaml

Looping overs arrays in bash

Take an array defined in yaml and loop over it in bash.

# run the example script
./read_into_bash_array.sh

Conversion

JSON

# read in json and output yaml
yq e --output-format=yaml ./json/images.json   

XML

When converting to xml we're flattening attributes into the node heirarchy.

# xml to json (prettify the xml)
cat ./xml/results.hurl.xml | xmllint --format - | sponge ./xml/results.hurl.xml
yq e --output-format=json ./xml/results.hurl.xml

# get most recent stats from test results (TODO: replace the jq bit with yq)
yq e --output-format=json ./xml/results.hurl.xml | jq -c '.testsuites.testsuite[-1] | {"total":."+@tests", "errors": ."+@errors","failures":."+@failures"}'

CSV

# simple conversion of csv file to json
yq -o=json ./csv/simple.csv 

# it seems to be broken at converting strings with : in them even if they are quoted.  
yq -o=json ./csv/broken.csv 
yq -o=json ./csv/bug.csv 

Resources

• mikefarah/yq repo here
• How It Works here
• Anchor and Alias Operators here
• Multiply (Merge) here
• The yq gitbook has lots of good examples here
• yq: Mastering YAML Processing in Command Line here
• Tips, Tricks, Troubleshooting here