qq

qq is a multi-format transcoder and query tool powered by jq syntax. It lets you query and convert between configuration and data formats without needing separate tools for each one.

Supported formats (read/write): .json, .yaml/.yml, .toml, .xml, .hcl/.tf, .csv, .tsv, .ini, .properties, .gron, .html, .jsonl/.ndjson/.jsonlines, .jsonc, .parquet, .msgpack/.mpk, .cbor, .avro, .base64/.b64, .txt/.text,

Read-only: .proto, .env

Transcoding Between Formats

Any input format can be transcoded to any output format in a single command. Use -i/--input and -o/--output to specify formats explicitly, or let qq detect them from the file extension.

# YAML → JSON
qq '.' config.yaml -o json

# TOML → YAML
qq '.' pyproject.toml -o yaml

# CSV → Parquet
qq '.' data.csv -o parquet

# Avro → JSON (schema embedded in OCF file)
qq '.' events.avro

# JSON → CBOR
qq '.' payload.json -o cbor

# XML → TOML
qq '.root' config.xml -o toml

# Terraform HCL → JSON (useful for scripting)
qq '.' infra.tf -o json

# Pipe formats freely
cat file.msgpack | qq -i msgpack -o yaml

Usage

Here's some example usage, this emphasises the interactive mode for demonstration, but qq is designed for usage in shell scripts.

# JSON is default in and output.
cat file.${ext} | qq -i ${ext}

# Extension is parsed, no need for input flag
qq '.' file.xml

# random example: query xml, grep with gron using qq io and output as json
qq file.xml -o gron | grep -vE "sweet.potatoes" | qq -i gron

# get some content from a site with html input
curl motherfuckingwebsite.com | bin/qq -i html '.html.body.ul.li[0]'

# interactive query builder mode on target file
qq . file.json --interactive

# streaming mode - works with JSON, YAML, CSV and more
qq --stream 'select(length == 2)' large.json

# slurp mode - read multiple inputs into an array
echo -e '{"id":1}\n{"id":2}' | qq -s 'map(.id)'

# exit-status - use in conditionals
echo '{"active":true}' | qq -e '.active' && echo "is active"

Git

You can also use it for cleaner diffing of configuration files by adding to your git/config file a snippet such as

  [diff "csv"]
  textconv = "f(){ in=\"$1\"; \
      if command -v qq   >/dev/null 2>&1; then qq --monochrome-output --output gron --input csv  \"$in\" 2>/dev/null | sort && exit 0; fi; \
      cat \"$in\"; \
    }; f"
  [diff "env"]
    textconv = "f(){ qq --monochrome-output --output gron --input env  \"$1\" 2>/dev/null | sort || cat \"$1\"; }; f"
  [diff "html"]
    textconv = "f(){ qq --monochrome-output --output gron --input html \"$1\" 2>/dev/null | sort || cat \"$1\"; }; f"
  [diff "ini"]
    textconv = "f(){ qq --monochrome-output --output gron --input ini  \"$1\" 2>/dev/null | sort || cat \"$1\"; }; f"
  [diff "toml"]
    textconv = "f(){ qq --monochrome-output --output gron --input toml \"$1\" 2>/dev/null | sort || cat \"$1\"; }; f"

and to git/attributes correspondingly

*.csv diff=csv
*.env diff=env
*.html diff=html
*.ini diff=ini
*.toml diff=toml

Installation

From brew:

brew install jfryy/tap/qq

From AUR (ArchLinux):

yay qq-git

From source (requires go >=1.22.4)

make install

Download at releases here.

Docker quickstart:

# install the image
docker pull jfryy/qq

# run an example
echo '{"foo":"bar"}' | docker run -i jfryy/qq '.foo = "bazz"' -o tf

Background

qq is inspired by fq and jq. jq is a powerful and succinct query tool, sometimes I would find myself needing to use another bespoke tool for another format than json, whether its something dedicated with json query built in or a simple converter from one configuration format to json to pipe into jq. qq aims to be a handly utility on the terminal or in shell scripts that can be used for most interaction with structured formats in the terminal. It can transcode configuration formats interchangeably between one-another with the power of jq and it has an an interactive repl (with automcomplete) to boot so you can have an interactive experience when building queries optionally. Many thanks to the authors of the libraries used in this project, especially jq, gojq, gron and fq for direct usage and/or inspiration for the project.

Features

• Support a wide range of configuration formats and transform them interchangeably between each other.
• Quick and comprehensive querying of configuration formats without needing a pipeline of dedicated tools.
• Provide an interactive mode for building queries with autocomplete and realtime rendering preview.
• Streaming mode (--stream) (identical to jq's --stream), plus extended support for JSONL, YAML, CSV, TSV, and line-delimited formats - all emit path-value pairs for memory-efficient processing of large files.
• qq is broad, but performant encodings are still a priority, execution is quite fast despite covering a broad range of codecs. qq performs comparitively with dedicated tools for a given format.

Contributions

All contributions are welcome to qq, especially for upkeep/optimization/addition of new encodings.

Thanks and Acknowledgements / Related Projects

This tool would not be possible without the following projects, this project is arguably more of a composition of these projects than a truly original work, with glue code, some dedicated encoders/decoders, and the interactive mode being original work.
Nevertheless, I hope this project can be useful to others, and I hope to contribute back to the community with this project.

• gojq: gojq is a pure Go implementation of jq. It is used to power the query engine of qq.
• fq : fq is a jq like tool for querying a wide array of binary formats.
• jq: jq is a lightweight and flexible command-line JSON processor.
• gron: gron transforms JSON into discrete assignments that are easy to grep.
• yq: yq is a lightweight and flexible command-line YAML (and much more) processor.
• goccy: goccy has quite a few encoders and decoders for various formats, and is used in the project for some encodings.
• go-toml: go-toml is a TOML parser for Golang with reflection.