Merge branch 'catchify'
Switch from CppUnit to Catch for the unit tests. Closes https://github.com/wxWidgets/wxWidgets/pull/577
This commit is contained in:
Vadim Zeitlin
@@ -1,144 +1,94 @@
|
||||
How to write unit tests for wxWidgets
|
||||
=====================================
|
||||
|
||||
Unit tests for wxWidgets are written using small cppunit framework. To compile
|
||||
(but not to run) them you need to have it installed. Hence the first part of
|
||||
this note explains how to do it while the second one explains how to write the
|
||||
test.
|
||||
wxWidgets unit tests use [Catch](http://catch-lib.net/) framework. It is
|
||||
included in wxWidgets as a submodule, so you will need to run
|
||||
|
||||
I. CppUnit Installation
|
||||
-----------------------
|
||||
$ git submodule update --init 3rdparty/catch
|
||||
|
||||
1. Get it from http://www.sourceforge.net/projects/cppunit
|
||||
(latest version as of the time of this writing is 1.10.2)
|
||||
to get it before the first use. Catch is header-only and doesn't need to be
|
||||
compiled.
|
||||
|
||||
2. Build the library:
|
||||
- Under Windows using VC++:
|
||||
- build everything in CppUnitLibraries.dsw work space
|
||||
- add include and lib subdirectories of the directory
|
||||
where you installed cppunit to the compiler search path
|
||||
using "Tools|Options" menu in VC IDE
|
||||
Testing with Catch
|
||||
------------------
|
||||
|
||||
- Under Unix: run `configure && make && make install` as usual
|
||||
**WARNING**: Most of the existing tests are currently still written in the
|
||||
CppUnit style, please do _not_ follow them when writing new tests, the old
|
||||
style is too complex and unnecessary.
|
||||
|
||||
Writing tests with Catch is almost embarrassingly simple: you need to just
|
||||
add a new test case and use Catch assertion macros inside it, e.g.
|
||||
|
||||
TEST_CASE("MyNewTest", "[my][new][another-tag]")
|
||||
{
|
||||
wxString s("Hello, world!");
|
||||
CHECK( s.BeforeFirst(",") == "Hello" );
|
||||
CHECK( s.AfterLast(" ") == "world!" );
|
||||
}
|
||||
|
||||
This is all, the new test will be automatically run when you run the full test
|
||||
suite or you can run just it using
|
||||
|
||||
$ ./test MyNewTest
|
||||
|
||||
(see below for more about running tests).
|
||||
|
||||
See [Catch tutorial](https://github.com/philsquared/Catch/blob/v1.11.0/docs/tutorial.md)
|
||||
for more information.
|
||||
|
||||
|
||||
II. Writing tests with CppUnit
|
||||
------------------------------
|
||||
Tests physical structure
|
||||
------------------------
|
||||
|
||||
1. Create a new directory tests/foo
|
||||
All (i.e. both GUI and non-GUI) unit tests are under `tests` subdirectory. When
|
||||
adding a new test, try to find an existing file to add it to. If there are no
|
||||
applicable files, try to add a new file to an existing directory. If there is
|
||||
no applicable directory neither, create a new one and put the new file there
|
||||
(i.e. do _not_ put new files directly under `tests`). If your test is small,
|
||||
consider adding it to `tests/misc/misctests.cpp`.
|
||||
|
||||
2. Write a cpp file for the test copying, if you want,
|
||||
from one of the existing tests. The things to look for:
|
||||
If you add a new file, you need to update `tests/test.bkl` and add a
|
||||
`<sources>` tag for your new file.bkl. Make sure it's in the correct section:
|
||||
the one starting `<exe id="test_gui"` for a gui test, the one starting `<exe
|
||||
id="test" template="wx_sample_console` otherwise. After modifying this file,
|
||||
rerun bakefile to regenerate the tests make- and project files:
|
||||
|
||||
a) #include "wx/cppunit.h" instead of directly including CppUnit headers
|
||||
|
||||
b) don't put too many things in one test case nor in one method of a test
|
||||
case as it makes understanding what exactly failed harder later
|
||||
|
||||
c) 'register' your tests as follows so that the test program will find and
|
||||
execute them:
|
||||
|
||||
// register in the unnamed registry so that these tests are run by default
|
||||
CPPUNIT_TEST_SUITE_REGISTRATION(MBConvTestCase);
|
||||
|
||||
// also include in its own registry so that these tests can be run alone
|
||||
CPPUNIT_TEST_SUITE_NAMED_REGISTRATION(MBConvTestCase, "MBConvTestCase");
|
||||
|
||||
Read CppUnit documentation for more.
|
||||
|
||||
d) wxUIActionSimulator can be used when user input is required, for example
|
||||
clicking buttons or typing text. A simple example of this can be found
|
||||
in controls/buttontest.cpp. After simulating some user input always
|
||||
wxYield to allow event processing. When writing a test using
|
||||
wxUIActionSimulator always add the test using WXUISIM_TEST rather than
|
||||
CPPUNIT_TEST as then it won't run on unsupported platforms. The test itself
|
||||
must also be wrapped in a #if wxUSE_UIACTIONSIMULATOR block.
|
||||
|
||||
e) There are a number of classes that are available to help with testing GUI
|
||||
elements. Firstly throughout the test run there is a frame of type
|
||||
wxTestableFrame that you can access through `wxTheApp->GetTopWindow()`. This
|
||||
class adds two new functions, GetEventCount, which takes an optional
|
||||
wxEventType. It then returns the number of events of that type that it has
|
||||
received since the last call. Passing nothing returns the total number of
|
||||
event received since the last call. Also there is OnEvent, which counts the
|
||||
events based on type that are passed to it. To make it easy to count events
|
||||
there is also a new class called EventCounter which takes a window and event
|
||||
type and connects the window to the top level wxTestableFrame with the specific
|
||||
event type. It disconnects again once it is out of scope. It simply reduces
|
||||
the amount of typing required to count events.
|
||||
|
||||
3. add a `<sources>` tag for your source file to tests/test.bkl. Make sure it's
|
||||
in the correct section: the one starting `<exe id="test_gui"` for a gui test,
|
||||
the one starting `<exe id="test" template="wx_sample_console` otherwise.
|
||||
$ cd build/bakefiles
|
||||
$ bakefile_gen -b ../../tests/test.bkl
|
||||
|
||||
|
||||
III. Running the tests
|
||||
----------------------
|
||||
Writing GUI-specific tests
|
||||
--------------------------
|
||||
|
||||
1. Regenerate the make/project files from test.bkl using bakefile_gen, e.g.:
|
||||
cd build/bakefiles
|
||||
bakefile_gen -b ../../tests/test.bkl
|
||||
and if you're on a unix system re-run configure.
|
||||
`wxUIActionSimulator` can be used when user input is required, for example
|
||||
clicking buttons or typing text. A simple example of this can be found in
|
||||
`tests/controls/buttontest.cpp`. After simulating some user input always
|
||||
call `wxYield()` to allow event processing. When writing a test using
|
||||
`wxUIActionSimulator` wrap it in `#if wxUSE_UIACTIONSIMULATOR` block.
|
||||
|
||||
2. Build the test program using one of the make/project files in the tests
|
||||
subdirectory.
|
||||
|
||||
3. Run the test program by using the command 'test' for the console tests,
|
||||
'test_gui' for the gui ones. With no arguments, all the default set of tests
|
||||
(all those registered with CPPUNIT_TEST_SUITE_REGISTRATION) are run.
|
||||
Or to list the test suites without running them:
|
||||
test -l or test_gui -l
|
||||
|
||||
4. Tests that have been registered under a name using
|
||||
CPPUNIT_TEST_SUITE_NAMED_REGISTRATION can also be run separately. For
|
||||
example:
|
||||
test_gui ButtonTestCase
|
||||
or to list the tests done by a particular testcase:
|
||||
test -L MBConvTestCase
|
||||
|
||||
5. Fault navigation.
|
||||
VC++ users can run the programs as a post build step (Projects/Settings/
|
||||
Post-build step) to see the test results in an IDE window. This allows
|
||||
errors to be jumped to in the same way as for compiler errors, for
|
||||
example by pressing F4 or highlighting the error and pressing return.
|
||||
|
||||
Similarly for makefile users: makefiles can be modified to execute the
|
||||
test programs as a final step. Then you can navigate to any errors in the
|
||||
same way as for compiler errors, if your editor supports that.
|
||||
|
||||
Another alternative is to run the tests manually, redirecting the output
|
||||
to a file. Then use your editor to jump to any failures. Using Vim, for
|
||||
example, ':cf test.log' would take you to the first error in test.log, and
|
||||
':cn' to the next.
|
||||
|
||||
If you would like to set a breakpoint on a failing test using a debugger,
|
||||
put the breakpoint on the function 'CppUnit::Asserter::fail()'. This will
|
||||
stop on each failing test.
|
||||
There are a number of classes that are available to help with testing GUI
|
||||
elements. Firstly throughout the test run there is a frame of type
|
||||
`wxTestableFrame` that you can access through `wxTheApp->GetTopWindow()`. This
|
||||
class adds two new functions, `GetEventCount()`, which takes an optional
|
||||
`wxEventType`. It then returns the number of events of that type that it has
|
||||
received since the last call. Passing nothing returns the total number of event
|
||||
received since the last call. Also there is `OnEvent()`, which counts the events
|
||||
based on type that are passed to it. To make it easy to count events there is
|
||||
also a new class called `EventCounter` which takes a window and event type and
|
||||
connects the window to the top level `wxTestableFrame` with the specific event
|
||||
type. It disconnects again once it is out of scope. It simply reduces the
|
||||
amount of typing required to count events.
|
||||
|
||||
|
||||
IV. Notes
|
||||
---------
|
||||
Running the tests
|
||||
-----------------
|
||||
|
||||
1. You can register your tests (or a subset of them) just under a name, and not
|
||||
in the unnamed registry if you don't want them to be executed by default.
|
||||
Run the main test suite by using the command `test` for the console tests,
|
||||
or `test_gui` for the GUI ones. With no arguments, all the default set of tests
|
||||
(all those registered without `[hide]` tag) are run.
|
||||
|
||||
2. If you are going to register your tests both in the unnamed registry
|
||||
and under a name, then use the name that the tests have in the 'test -l'
|
||||
listing.
|
||||
To list the test suites without running them use `-l` command-line option.
|
||||
|
||||
3. Tests which fail can be temporarily registered under "fixme" while the
|
||||
problems they expose are fixed, instead of the unnamed registry. That
|
||||
way they can easily be run, but they do not make regression testing with
|
||||
the default suite more difficult. E.g.:
|
||||
|
||||
// register in the unnamed registry so that these tests are run by default
|
||||
//CPPUNIT_TEST_SUITE_REGISTRATION(wxRegExTestCase);
|
||||
CPPUNIT_TEST_SUITE_NAMED_REGISTRATION(wxRegExTestCase, "fixme");
|
||||
|
||||
// also include in its own registry so that these tests can be run alone
|
||||
CPPUNIT_TEST_SUITE_NAMED_REGISTRATION(wxRegExTestCase, "wxRegExTestCase");
|
||||
|
||||
4. Tests which take a long time to execute can be registered under "advanced"
|
||||
instead of the unnamed registry. The default suite should execute reasonably
|
||||
quickly. To run the default and advanced tests together:
|
||||
test "" advanced
|
||||
To run a particular test case, use `./test NameTestCase`. To run all tests
|
||||
using the specified tag, use `./test [tag_name]` (the square brackets may need
|
||||
to be escaped from your shell).
|
||||
|
||||
Reference in New Issue
Block a user