BASH JQ - Playing with Records

Preface

Goal: Utilizing JQ in BASH for string processing.

Instead of using pure bash, we can directly extract data in json format using jq.

For convenience, we can combine the output of jq command, to bash for further processing.

Reference Reading

• jq Manual

Source Examples

You can obtain source examples here:

• /songs/jq

Common Use Case

Task: Get the unique tag string

Please read overview for more detail.

Data Structure Support

We are going to use external json file.

Prepopulated Data

Songs and Poetry

The data is simply a json file.

• gitlab.com/…/bash-jq/songs.json

{
  "songs": [
    { "title": "Cantaloupe Island",
      "tags": ["60s", "jazz"]},
    { "title": "Let It Be",
      "tags": ["60s", "rock"]},
    { "title": "Knockin' on Heaven's Door",
      "tags": ["70s", "rock"]},
    { "title": "Emotion",
      "tags": ["70s", "pop"]},
    { "title": "The River"}
  ]
}

JQ in BASH Solution

The Answer

We are going to combine JQ with other tools in BASH.

jq -r "(.songs[].tags | \
       select( . != null ))[]" \
    songs.json \
  | sort \
  | uniq \
  | tr '\n' ' '

Enough with introduction, at this point we should go straight to coding.

Environment

No need any special setup. Just run and voila..!

1: Array in JQ

We are going to check how far associative array in bash, can handle data structure.

Simple Array

Consider begin with simple array.

• gitlab.com/…/bash-jq/01-tags.sh

json='{
  "tags": [
    "rock", "jazz", "rock", "pop", "pop"
  ]
}'
tags=$(echo $json | jq -r .tags[])
echo ${tags[@]}

With output result as below:

❯ ./01-tags.sh
rock jazz rock pop pop

No problem so far.

Associative Array

Instead of using interal data, we better go straight to json file as below:

• gitlab.com/…/bash-jq/song.json

{
  "title": "Cantaloupe Island",
  "tags": ["60s", "jazz"]
}

Now consider this form.

• gitlab.com/…/bash-jq/02-song.sh

jq -r .title song.json
jq -r .tags song.json

With output result as below:

❯ ./02-song.sh
Cantaloupe Island
[
  "60s",
  "jazz"
]

We still have no problem with code above. But unfortunately we cannot go further than this.

2: Songs JSON

We need to prepare our song json.

Records in Songs JSON

The json data is simply as shown below:

• gitlab.com/…/bash-jq/songs.json

{
  "songs": [
    { "title": "Cantaloupe Island",
      "tags": ["60s", "jazz"]},
    { "title": "Let It Be",
      "tags": ["60s", "rock"]},
    { "title": "Knockin' on Heaven's Door",
      "tags": ["70s", "rock"]},
    { "title": "Emotion",
      "tags": ["70s", "pop"]},
    { "title": "The River"}
  ]
}

Testing JQ Result

Now we can have a very short code. With the result exactly the same as above code:

• gitlab.com/…/bash-jq/03-songs.sh

title=$(jq -r .songs[1].title songs.json)
echo $title

tags=( $(jq -r .songs[1].tags[] songs.json) )
echo ${#tags[@]}
echo ${tags[@]}

With the result as below lines of string.

❯ ./03-songs.sh
Let It Be
2
60s rock

3: Finishing The Task

Extract, Flatten, Unique

Extracting Data Structure

Based on the result above, we can go further, extracting the tags data.

We need to filter lines without tags. And get rid of it.

• gitlab.com/…/bash-jq/04-extract.sh

path='.songs[].tags'
remove='select( . != null )'
jq -r "$path | $remove" songs.json

[] With the result of array of array, as shown below.

❯ ./04-extract.sh
[
  "60s",
  "jazz"
]
[
  "60s",
  "rock"
]
[
  "70s",
  "rock"
]
[
  "70s",
  "pop"
]

Notice how we remove the unwanted record.

  remove='select( . != null )'

Flatten

Now we can normalize, the separated values. Flatten all values into just single array.

• gitlab.com/…/bash-jq/05-flatten.sh

Using two loops. One after another.

path='.songs[].tags'
remove='select( . != null )'

tags=$(jq -r "($path | $remove)[]" songs.json)
echo ${tags[@]}

With the result of a flattened array shown below.

❯ ./05-flatten.sh
60s jazz 60s rock 70s rock 70s pop

Flatten Rewrite

We can also rewrite the variable in bash, so the code can become more readable. Although it is not necessary.

• gitlab.com/…/bash-jq/06-flatten.sh

join='., " "'
path='.songs[].tags'
remove='select( . != null )'

filter="($path | $remove)[] | $join"
tags=$(jq -r "$filter" songs.json)
echo ${tags[@]}

With the result exactly the same as previous flattened array.

Unique

Instead of solving the approach using pure jq, I prefer to use other tools as necessary.

• gitlab.com/…/bash-jq/06-uniq.sh

jq -r "(.songs[].tags | \
       select( . != null ))[]" \
    songs.json \
  | sort \
  | uniq \
  | tr '\n' ' '

With the result similar as below array:

❯ ./06-uniq.sh
60s 70s jazz pop rock

It is so easy to learn, if you willing to read the offical documentation.

That is all.

What is Next 🤔?

Consider continue reading [ Awk - Playing with Records ].