RT Framework
============

.. image:: https://circleci.com/gh/docker/rt-framework/tree/master.svg?style=svg&circle-token=0b5c6e136c8182e68ebb687ee88b78facf1ac36c
    :target: https://circleci.com/gh/docker/rt-framework/tree/master

**A regression testing framework**

This project contains a generic, cross-platform regression test
framework.

The generic regression test framework is implemented in Python 
whereas the test are written as shell scripts. The library code is in
the ``rt`` directory and some common utilities and helper programs
are contained in the ``rt/lib`` directory.

The RT Framework consists of 3 utilities:

* ``rt-local`` - a local test runner
* ``rt-remote`` - a tool for running rt-local on a remote host
* ``rt-vm`` - a tool for creating virtual machines to use with rt-remote

For more details, see the documentation in ``./docs/USER_GUIDE.md``.

Installation
------------

To install all the utilities
::
  pip install 'rt-framework[vm,remote]' --extra-index-url https://sSKjtzxT1CdrpaZaEdcw@repo.fury.io/docker-io/


To install ``rt-local`` only:
::
  pip install rt-framework --extra-index-url https://sSKjtzxT1CdrpaZaEdcw@repo.fury.io/docker-io/

To install from this source directory:
::
   pip install --upgrade .[vm,remote]


Development
-----------

To run the test suite please use:
::
  make test


Prerequisites
-------------

On a Mac and Linux, this should pretty much just work out of the box.

On Windows, you need to have python installed as well as some form of
``bash``. The tests must be run from a ``bash`` shell and ``bash.exe``
must be in your path. The simplest way is to install via
`chocolatey <https://chocolatey.org/>`_
::
  choco install git python

Quickstart
----------

The regression test framework allows running tests on a local host (or
inside a VM) as well as against a suitably configured remote host.

To run tests locally, simply execute the ``rt-local run`` command.  It
will executed all the test cases in the supplied cases directory.
This defaults to ``./cases``

To list all current tests run ``rt-local list``, or to get a one line
summary for each test use ``rt-local info``.

When running tests, by default a line per test is printed on the
console with a pass/fail indication.  Detailed logs, by default, are
stored in ``./_results/<UUID>/``. In that directory, ``TESTS.log``
contains detailed logs of all tests, ``TESTS.csv`` contains a line per
test and ``SUMMARY.csv`` contains a one line summary of the all tests
run. The directory also contains a log file for each tests, with the
same contents as ``TESTS.log``.

If you prefer a bit more information in the log files use:
::
  rt-local -x run

This executes the tests with ``-x`` and thus logs all commands executed.

For a CI system, where the output is displayed on a web page use:
::
  rt-local -x -vvv run

This prints the same information logged to the log file to the console.

To run test on a remote host use the ``rt-remote`` command. You must
specify the type of the remote host (``osx`` or ``win``) as well as the
address.  You must also supply user name and password.  If ``-t osx`` is
used you must also supply the location of the SSH private key to be
used to communciate with the remote machine.
::
  ./rt-remote -t win -r 172.16.10.131 -u dave -p secret -- <rt-local arguments>


This:

* copies the test framework and tests from the local host to the remote host.
* executes the test on remote host
* copies the results back to the local host

Results can be found in the ``./_results`` directory, relative to where the
command was run from.