Ondatra: Open Network Device Automated Test Runner and API

Ondatra is a framework for writing and running tests against both real and
containerized network devices.

For an introduction to Ondatra, take the
Ondatra Tour.

Building Ondatra

To build and execute Ondatra's unit tests, run the following:

go generate ./...
go build ./...
go test $(go list ./... | grep -v /integration)

Providing a Binding

The Ondatra binding is the API layer through which Ondatra connects to and
controls the devices in your test environment. For an Ondatra test to run in
your environment, you must provide an implementation of the Binding interface.
For testing on KNE, Ondatra comes bundled with a binding
implementation for accessing KNE on your local machine. For testing on your
physical devices,
read how to author your own Binding implementation.

Testbed File

To run an Ondatra test, the user must specify the testbed of resources that
the Ondatra test runner should reserve in advance. The testbed is specified in
an external text file in protobuf text format. The protobuf Testbed message is
defined in proto/testbed.proto, and an example testbed
can be found in
knebind/integration/testbed.textproto.
As the proto definition and example show, testbed consists of the DUTs (devices
under test), ATEs (automated test equipment), the links between them, as well as
properties of the DUTs, ATEs, and links. It is the job of the Reserve method
in the binding implementation to locate available resources that match the
abstract topology and criteria specified in the testbed file.

Writing an Ondatra Test

Ondatra provides a set of
fluent interfaces for
configuring and interacting with network devices. The interfaces are divided
into several API packages:

• Config provides
  an API to set native config on devices via vendor-specific (non-gNMI)
  protocols.
• Console provides
  an API to interact with the serial console of a device.
• Debug provides an
  API to add breakpoints to the test to debug its execution.
• Event Listener
  provides an API to attach listeners that are called at events during the
  test execution.
• gNMI provides an
  API for querying telemetry and setting the state of the device via gNMI.
• Netutil provides
  a collection of network-related helper methods for testing.
• OTG provides an API
  to generate traffic using Open Traffic Generator.
• Raw provides
  low-level access to the raw device APIs, to be used when the other
  higher-level APIs are not sufficient.
• Report provides
  an API to add properties to, and extract properties from, the JUnit XML test
  report.

See the
full API reference documentation.

Running an Ondatra Test

An Ondatra test is a Go test, and so is run with go test, albeit with some
additional flags to control the execution of the test:

• -testbed (required): Path to the testbed text proto file.
• -wait_time (optional): Maximum amount of time the test should wait until
  the testbed is ready. If not specified, the binding chooses the amount of
  time to wait.
• -run_time (optional): Timeout of the test run, excluding the wait time
  for the testbed to be ready. If not specified, no limit is imposed.
• -xml (optional): File path to write JUnit XML test results; disables
  normal Go test logging.
• -debug (optional): Whether the test is run in debug mode.
• -reserve (optional): Reservation id or a mapping of device and port IDs
  to names; allowed only in debug mode

To run a subset of the test cases of an Ondatra test, use the Go test -run
flag. While the -run flag accepts an arbitrary
Go regexp, to match a single test case it is
usually sufficient to just pass the name of the test function:

$ go test -testbed=testbed.textproto -config=config.yaml -run=$RUN_THIS_TEST_CASE

Read the
Go doc on subtests
for more details on matching subtests with the -run flag.

Debugging an Ondatra Test

To run an Ondatra test in debug mode, pass the -debug flag to go test. Debug
mode allows you to insert breakpoints in your code using the
Debug API.

Debug mode also offers a menu option to pause the test immediately after the
testbed is reserved. This is useful if you want to manually inspect the testbed
before the test cases run, or to run a separate test execution against with the
same reservation.

To debug a test against a pre-allocated reservation, set the -reserve flag to
the reservation ID:

$ go test -testbed=testbed.textproto -config=config.yaml -debug -reserve=123abc

To debug the test against specifically-named devices, set the -reserve flag to
a comma-separated list of id=name strings at the command line, where ports are
named with the syntax deviceID:portID=portName:

$ go test -testbed=testbed.textproto -config=config.yaml -debug \
    -reserve=dut=mydevice,dut:port1=Ethernet1/1,ate=myixia,ate:port2=2/3

Logging Verbosity

Ondatra always sets the Go test -v flag to true for verbose test output, so
there is no need to set this flag explicitly in your test invocations.

Ondatra uses glog for its own
logging. By default, glog logs to a temporary dir, but setting the
-alsologtostderr flag will output those logs to stderr. You can increase the
verbosity glog by setting its -v flag to a positive integer. Increasing the
value of v produces increasingly verbose and granular details in the logs, as
outlined in this table:

Because go test has its own -v flag, setting glog's -v value must be
preceded by -args to avoid it being interpreted by go test itself. For
example:

go test -alsologtostderr -args -v=1

See the
glog package documentation
for more information on all the glog flags.

XML Test Report

Ondatra has the ability to output test results in JUnit XML format. If you pass
-xml=[path] to your go test invocation, Ondatra will use
go-junit-report to translate the
Go test log to an XML file at the provided path. To attach properties to the XML
report and to programmatically parse the XML file use the Ondatra
Report API.

Testing on KNE

You don't have to code your own binding implementation before getting started
with Ondatra, because Ondatra comes packaged with a binding for
KNE, and an
example Ondatra test that uses that
binding. See the knebind README for more on how to use the
KNE binding and run the example test.