jq

Very often you will get content in JSON format. If you want to process this in bash scripting you will soon find out, it is very hard to do. JSON can have lots of optional line breaks and whitespaces, and in order to find a specific element you often need to almost parse the structure. But worry no more:

jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.

jq manual

https://stedolan.github.io/jq/manual/v1.6/

Identity

Just copies all that is there, but pretty print

echo '[{"name":"John","age":42, "colors": ["red", "green", "blue"]},{"name":"John","age":42, "colors": ["red", "green", "blue"]}]' \
| jq "."

[
{
"name": "John",
"age": 42,
"colors": [
"red",
"green",
"blue"
]
},
{
"name": "John",
"age": 42,
"colors": [
"red",
"green",
"blue"
]
}
]

Get a single field

echo '{ "name": "John", "age": 42, "address": { "street": "Main street", "zipcode": 12345 }}' \
| jq ".address.zipcode"

12345

Arrays

You have an array / list of objects

Get all the elements of the array

echo '[ {"name":"John","age":42, "colors": ["red", "green", "blue"]}, {"name":"Jane","age":35, "colors": ["yellow", "red", "blue"]}]' \
| jq ".[]"

{
"name": "John",
"age": 42,
"colors": [
"red",
"green",
"blue"
]
}

{
"name": "Jane",
"age": 35,
"colors": [
"yellow",
"red",
"blue"
]
}

Get only the first element

echo '[ {"name":"John","age":42, "colors": ["red", "green", "blue"]}, {"name":"Jane","age":35, "colors": ["yellow", "red", "blue"]}]' \
| jq ".[1]"

Get the last element

echo '[ {"name":"John","age":42, "colors": ["red", "green", "blue"]}, {"name":"Jane","age":35, "colors": ["yellow", "red", "blue"]}]' \
| jq ".[-1]"

Get the first 3 element

echo '[ {"name":"John","age":42, "colors": ["red", "green", "blue"]}, {"name":"Jane","age":35, "colors": ["yellow", "red", "blue"]}]' \
jq ".[0:2]"

Multiple fields

echo '{"name":"John","age":42, "colors": ["red", "green", "blue"]}' \
| jq ".name, .age"

"John"
42

Somewhere in the object is an array

First copy some fields + the array (boring)

echo '{"name":"John","age":42, "colors": ["red", "green", "blue"]}' \
| jq "{name, age, colors: .colors }"

Now have for each entry in the array an extra output line

echo '{"name":"John","age":42, "colors": ["red", "green", "blue"]}' \
| jq "{name, age, colors: .colors[] }"

(:source lang=Bash:) [@
{
"name": "John",
"age": 42,
"colors": "red"
}

{
"name": "John",
"age": 42,
"colors": "green"
}

{
"name": "John",
"age": 42,
"colors": "blue"
}

If you want to iterate about an json array use this to get each entry of the Array in one line

jq --compact-output | while read line
do
echo "$line" | jq
done

Pipe

Have several commands and the output of the first, goes to the second, ... Like a Unix "|"

Array 1 by 1, from the result only take 2 fields

echo '[{"name":"John","age":42, "colors": ["red", "green", "blue"]},{"name":"Jane","age":35, "colors": ["yellow", "red", "blue"]}]' \
| jq ".[] | .name, .age"

Build new objects

Take the fields with values and build a new object. This builds the same object again

echo '{"name":"John","age":42, "colors": ["red", "green", "blue"]}' \
| jq "{name, age}"

This

jq "{name, age}"

is a short form of

{ "name": .name, "age": .age, }

The long form is required if you need to access fields inside the data, e.g. person.age. As the short form would try to name the field otherwise with a dot in it and fail.

You can also take field name and field value separately

echo '[{"name":"John","age":42}]' \
| jq "{name: .name, age: .age}"

And of course change them

echo '[{"name":"John","age":42, "colors": ["red", "green", "blue"]},{"name":"Jane","age":35, "colors": ["yellow", "red", "blue"]}]' \
| jq ".[] | { "TheName": .name, age: 43 }"

Filtering (like with grep)

echo '[{"name":"John","age":42, "colors": ["red", "green", "blue"]},{"name":"Jane","age":35, "colors": ["yellow", "red", "blue"]}]' \
| jq ".[] | select(.name == \"Jane\")"

{
"name": "Jane",
"age": 35,
"colors": [
"yellow",
"red",
"blue"
]
}