JSON in Bash and CLI with jq

The JSON data format has become a ubiquitous tool for interchanging and storing human-readable data. In particular, it is very convenient when it comes to storing user-defined settings and properties. For example, in rFaaS, our RDMA-accelerated serverless platform, we have to store multi-parameter device configurations for the local and remote endpoints. Thus, we want to have the configuration in JSON as this format is flexible, widely supported, and easy to parse for humans:

{
  "devices": [
    {
      "name": "rocep7s0",
      "ip_address": "192.168.0.18",
      "port": 10005,
      "max_inline_data": 128,
      "default_receive_buffer_size": 32
    },
    {
      "name": "rocep6s0f0",
      "ip_address": "192.168.1.31",
      "port": 10000,
      "max_inline_data": 128,
      "default_receive_buffer_size": 32
    }
  ]
}

While JSON is natively supported in many languages, and even C++ has a multitude of easy-to-use or high-performance JSON libraries, there’s no native way of using JSONs in Bash scripts. Bash syntax and capabilities are a bit ancient, but automated testing and benchmarking scripts could benefit from using the same configuration format. Fortunately, there’s an easy and lightweight tool for that - jq. jq is a very simple querying tool that allows reading and processing all complex and nested JSON structures. Below, I will present a selection of queries in jq that demonstrate the basic usage and capabilities of this simple yet powerful and useful tool.

The simplest query displays all JSON contents.

~ jq '.' configuration/devices.json

Another one can be used to create a JSON object from scratch.

~ jq --null-input '{}'

Selecting JSON object.

~ jq '.["devices"]' configuration/devices.json
[
  {
    "name": "rocep7s0",
    "ip_address": "192.168.0.18",
    "port": 10005,
    "max_inline_data": 128,
    "default_receive_buffer_size": 32
  },
  {
    "name": "rocep6s0f0",
    "ip_address": "192.168.1.31",
    "port": 10000,
    "max_inline_data": 128,
    "default_receive_buffer_size": 32
  }
]

Selecting object in an array through direct indexing.

~ jq '.["devices"][0]' configuration/devices.json
{
  "name": "rocep7s0",
  "ip_address": "192.168.0.18",
  "port": 10005,
  "max_inline_data": 128,
  "default_receive_buffer_size": 32
}

Requesting a specific field in the object.

~ jq '.["devices"][0]["ip_address"]' configuration/devices.json
"192.168.0.18"

The quotation marks are usually not needed but we don’t need sed or tr to remove them.

~ jq -r '.["devices"][0]["ip_address"]' configuration/devices.json
192.168.0.18

What if we want to access multiple fields with a single query?

~ jq -r '.["devices"][0] | .ip_address,.port' configuration/devices.json
192.168.0.18
10005

Unnecessary whitespace can be removed too.

~ jq -j '.["devices"][0] | .ip_address,.port' configuration/devices.json
192.168.0.1810005

However, this output is far from ideal. Instead, we might want to get a CSV-like or a semicolon-separated output. We can use string interpolation for that.

~ jq -j '.["devices"][0] | "\(.ip_address);\(.port)"' configuration/devices.json
192.168.0.18;10005

Thus, we can process device addresses with this one-liner in Bash.

~ IFS=";" read ip port <<< $(jq -j '.["devices"][0] | "\(.ip_address);\(.port)"' configuration/devices.json)
~ echo $ip $port
192.168.0.18 10005

jq is not only about simple queries and text parsing. The tool allows more complex operations, including finding an object with a specific value. In our case, we want to find the address of a device with a given name, without knowing its position in the devices array:

~ jq -r '.devices[] | select(.name=="rocep7s0") | .ip_address' configuration/devices.json
192.168.0.18

Furthermore, objects can be freely modified. In this example, we add a new simple object to the array.

~ jq '.devices += [{"name": "new_dev"}]' configuration/devices.json

Do you need to expand your JSON with new values that are defined at runtime? You can tell jq to treat certain values as arguments:

~ jq --arg addr $ADDRESS '.devices[0]["ip_address"] = $addr' configuration/devices.json

Do you need to merge two JSON files or copy values from one JSON to another? Use the argfile option. In this example, we add elements of an array from the file devices.json to the array in configuration/devices.json.

~ jq --argfile file1 devices.json '.devices += $file1.devices' configuration/devices.json

More advanced queries are available in jq, including scripting with select, map, and user-defined functions.