Uji - checklist generator and tracking system

uji is a tool to generate checklists from template files and hold on to
those lists forever or until the cows come home, whichever happens later.

uji is useful for cases where a user has

• large checklists with overlapping items, e.g. the same sets of test to be
  performed on different hardware
• the checklists are recurring, e.g. they need to be performed on every
  package update
• not all checklist items will be ticked every time but the user needs to
  remember which ones were ticked off
• there is other free-form information to be stored alongside the
  checklists

The main purpose of uji is to simplify the answer to "Wait, didn't I test
this?"

• uji is not a todo list
• uji is not a test suite
• uji is not something you can integrate into a CI

If you have a test that can be automated, uji is not the right tool. uji
is to track manual tests that cannot be automated. If you want to generate
test summaries, uji is not the right tool. uji tracks manual tests.

uji is optimized for writing, not reading. 99% of the logs tracked with uji
will never be read. Hence uji is built for minimal friction to write the
tests - basically you need git, python and an editor. The central file is
a markdown text file that you can edit and annotate (almost) at will.
It's all stored in git, so backup, sharing, and collaboration is trivial.
It's markdown, so you push the file anywhere (gitlab, github, ...) and it'll
probably look nice in the browser.

uji just preps the file for you and (in the future) provides a set of
CLI tools to automate some tasks around maintaining the test files.

"uji" is Indonesian for "test".

See uji on readthedocs for more
documentation.

Use case

Upgrading software packages usually requires multiple manual tests. Ideally
you have a checklist to tick off the things you tested. Next time you
upgrade that same package, the same checklist applies. Ideally you tick off
all the checklist items every time but we both know you don't. Maybe
hardware is missing, or the moon phase is wrong for tedious work, or, well,
so many reasons.

And then suddenly, a few months later a bug reports appears. And now the
question is: "wait, didn't I test this?"

And uji should be able answer that question.

How it works

The core of uji is a set of test templates in YAML format. These templates
are combined to a full test document (in markdown) on invoking uji new.
That file together with the log files required for the various tests are
stored in a git tree. As the tests are performed the user ticks them off in
the .md file and eventually commits them to git.

The next test run does the same, uji new creates a new directory, rinse,
wash, repeat.

So when the question "did I test this?" arises, you can go back, check the
respective log set and check - is the box for that test ticked off?
And if it is and it's still broken - well, you should have the various log
files in that same directory to figure out where the differences are.

None of this is novel of course, uji is just a commandline wrapper to make
that proces simpler.

Installation

pip install --user uji

Usage

Look at the example.yaml file for an example test
configuration.

$ mkdir my-test-results && cd my-test-results
$ git init
$ wget -o mypackage.yaml https://raw.githubusercontent.com/whot/uji/master/examples/example.yaml

Now you're set up. Edit the mypackage.yaml file and add your tests.
Once ready, git commit mypackage.yaml because you want this to be
preserved.

And when you're ready to start a test run:

$ cd my-test-results
$ uji new mypackage.yaml
Your test records and log files are
  mypackage-2019-12-10.0/
  mypackage-2019-12-10.0/logitech_keyboard
  mypackage-2019-12-10.0/t450_keyboard
  mypackage-2019-12-10.0/t460_keyboard
  mypackage-2019-12-10.0/example.md
  mypackage-2019-12-10.0/example.yaml
  mypackage-2019-12-10.0/logitech_g500s
  mypackage-2019-12-10.0/macbook_keyboard
  mypackage-2019-12-10.0/logitech_mx_anywhere_2s
  mypackage-2019-12-10.0/logitech_keyboard/test_hotplug_libinput
  mypackage-2019-12-10.0/logitech_keyboard/test_usb_logs
  mypackage-2019-12-10.0/logitech_keyboard/log_input_devices
  mypackage-2019-12-10.0/logitech_keyboard/test_hotplug_libinput/dmesg
  mypackage-2019-12-10.0/logitech_keyboard/test_usb_logs/lsusb −v
  mypackage-2019-12-10.0/logitech_keyboard/log_input_devices/libinput record −−all
  mypackage-2019-12-10.0/logitech_keyboard/log_input_devices/∕proc∕bus∕input∕devices
  mypackage-2019-12-10.0/t450_keyboard/log_input_devices
  mypackage-2019-12-10.0/t450_keyboard/log_input_devices/libinput record −−all
  mypackage-2019-12-10.0/t450_keyboard/log_input_devices/∕proc∕bus∕input∕devices
  mypackage-2019-12-10.0/t460_keyboard/log_input_devices
  mypackage-2019-12-10.0/t460_keyboard/log_input_devices/libinput record −−all
  mypackage-2019-12-10.0/t460_keyboard/log_input_devices/∕proc∕bus∕input∕devices
  mypackage-2019-12-10.0/logitech_g500s/log_input_devices
  mypackage-2019-12-10.0/logitech_g500s/log_input_devices/libinput record −−all
  mypackage-2019-12-10.0/logitech_g500s/log_input_devices/∕proc∕bus∕input∕devices
  mypackage-2019-12-10.0/macbook_keyboard/test_usb_logs
  mypackage-2019-12-10.0/macbook_keyboard/log_input_devices
  mypackage-2019-12-10.0/macbook_keyboard/test_usb_logs/lsusb −v
  mypackage-2019-12-10.0/macbook_keyboard/log_input_devices/libinput record −−all
  mypackage-2019-12-10.0/macbook_keyboard/log_input_devices/∕proc∕bus∕input∕devices
  mypackage-2019-12-10.0/logitech_mx_anywhere_2s/log_input_devices
  mypackage-2019-12-10.0/logitech_mx_anywhere_2s/log_input_devices/libinput record −−all
  mypackage-2019-12-10.0/logitech_mx_anywhere_2s/log_input_devices/∕proc∕bus∕input∕devices
Run "git commit" to commit the changes, or "git reset" to throw them away
$ git commit -am 'mypackage: new test log set'
$ vim mypackage-2019-12-04.0/mypackage.md
# tick off the tests as you confirm them

$ cp /proc/bus/input/devices mypackage-2019-12-10.0/logitech_keyboard/log_input_devices/∕proc∕bus∕input∕devices
$ lsusb &> mypackage-2019-12-10.0/macbook_keyboard/test_usb_logs/lsusb −v

# copy the other files into the right file names

$ git commit -am 'mypackage: test log set done'

And that's it for now.

See uji on readthedocs for more
documentation.

License

uji is licensed under the MIT license. See LICENSE for more info.