Amebis/wxWidgets
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace CppUnit with Catch for unit tests

Browse Source 
Drop the legacy CppUnit testing framework used for the unit tests.
Replacing it with Catch has the advantage of not requiring CppUnit
libraries to be installed on the system in order to be able to run
tests (Catch is header-only and a copy of it is now included in the
main repository itself) and, in the future, of being able to write
the tests in a much more natural way.

For now, however, avoid changing the existing tests code as much as
[reasonably] possible to avoid introducing bugs in them and provide
the CppUnit compatibility macros in the new wx/catch_cppunit.h header
which allow to preserve the 99% of the existing code unchanged. Some
of the required changes are:

 - Decompose asserts using "a && b" conditions into multiple asserts
   checking "a" and "b" independently. This would have been better
   even with CppUnit (to know which part of condition exactly failed)
   and is required with Catch.

 - Use extra parentheses around such conditions when they can't be
   easily decomposed in the arrays test, due to the use of macros.
   This is not ideal from the point of view of messages given when
   the tests fail but will do for now.

 - Rewrite asserts using "a || b" as a combination of condition
   checks and assert macros. Again, this is better anyhow, and is
   required with Catch. Incidentally, this allowed to fix a bug in
   the "exec" unit test which didn't leave enough time for the new
   process to be launched before trying to kill it.

 - Remove multiple CPPUNIT_TEST_SUITE_NAMED_REGISTRATION() macros,
   our emulation of this macro can be used only once.

 - Provide string conversions using Catch-specific StringMaker for
   a couple of types.

 - Replace custom wxImage comparison with a Catch-specific matcher
   class.

 - Remove most of test running logic from test.cpp, in particular don't
   parse command line ourselves any longer but use Catch built-in
   command line parser. This is a source of a minor regression:
   previously, both "Foo" and "FooTestCase" could be used as the name of
   the test to run, but now only the latter is accepted.
This commit is contained in:
Vadim Zeitlin
2017-11-01 19:15:24 +01:00
parent 5520d56222
commit e70fc11ef1
85 changed files with 876 additions and 1718 deletions
Download Patch File Download Diff File Expand all files Collapse all files

196
docs/contributing/how-to-write-unit-tests.md
View File

@@ -1,144 +1,94 @@
How to write unit tests for wxWidgets
=====================================
Unit tests for wxWidgets are written using small cppunit framework. To compile
(but not to run) them you need to have it installed. Hence the first part of
this note explains how to do it while the second one explains how to write the
test.
wxWidgets unit tests use [Catch](http://catch-lib.net/) framework. It is
included in wxWidgets as a submodule, so you will need to run
I. CppUnit Installation
-----------------------
$ git submodule update --init 3rdparty/catch
1. Get it from http://www.sourceforge.net/projects/cppunit
(latest version as of the time of this writing is 1.10.2)
to get it before the first use. Catch is header-only and doesn't need to be
compiled.
2. Build the library:
- Under Windows using VC++:
- build everything in CppUnitLibraries.dsw work space
- add include and lib subdirectories of the directory
where you installed cppunit to the compiler search path
using "Tools|Options" menu in VC IDE
Testing with Catch
------------------
- Under Unix: run `configure && make && make install` as usual
**WARNING**: Most of the existing tests are currently still written in the
CppUnit style, please do _not_ follow them when writing new tests, the old
style is too complex and unnecessary.
Writing tests with Catch is almost embarrassingly simple: you need to just
add a new test case and use Catch assertion macros inside it, e.g.
TEST_CASE("MyNewTest", "[my][new][another-tag]")
{
wxString s("Hello, world!");
CHECK( s.BeforeFirst(",") == "Hello" );
CHECK( s.AfterLast(" ") == "world!" );
}
This is all, the new test will be automatically run when you run the full test
suite or you can run just it using
$ ./test MyNewTest
(see below for more about running tests).
See [Catch tutorial](hhttps://github.com/philsquared/Catch/blob/v1.11.0/docs/tutorial.md)
for more information.
II. Writing tests with CppUnit
------------------------------
Tests physical structure
------------------------
1. Create a new directory tests/foo
All (i.e. both GUI and non-GUI) unit tests are under `tests` subdirectory. When
adding a new test, try to find an existing file to add it to. If there are no
applicable files, try to add a new file to an existing directory. If there is
no applicable directory neither, create a new one and put the new file there
(i.e. do _not_ put new files directly under `tests`). If your test is small,
consider adding it to `tests/misc/misctests.cpp`.
2. Write a cpp file for the test copying, if you want,
from one of the existing tests. The things to look for:
If you add a new file, you need to update `tests/test.bkl` and add a
`<sources>` tag for your new file.bkl. Make sure it's in the correct section:
the one starting `<exe id="test_gui"` for a gui test, the one starting `<exe
id="test" template="wx_sample_console` otherwise. After modifying this file,
rerun bakefile to regenerate the tests make- and project files:
a) #include "wx/cppunit.h" instead of directly including CppUnit headers
b) don't put too many things in one test case nor in one method of a test
case as it makes understanding what exactly failed harder later
c) 'register' your tests as follows so that the test program will find and
execute them:
// register in the unnamed registry so that these tests are run by default
CPPUNIT_TEST_SUITE_REGISTRATION(MBConvTestCase);
// also include in its own registry so that these tests can be run alone
CPPUNIT_TEST_SUITE_NAMED_REGISTRATION(MBConvTestCase, "MBConvTestCase");
Read CppUnit documentation for more.
d) wxUIActionSimulator can be used when user input is required, for example
clicking buttons or typing text. A simple example of this can be found
in controls/buttontest.cpp. After simulating some user input always
wxYield to allow event processing. When writing a test using
wxUIActionSimulator always add the test using WXUISIM_TEST rather than
CPPUNIT_TEST as then it won't run on unsupported platforms. The test itself
must also be wrapped in a #if wxUSE_UIACTIONSIMULATOR block.
e) There are a number of classes that are available to help with testing GUI
elements. Firstly throughout the test run there is a frame of type
wxTestableFrame that you can access through `wxTheApp->GetTopWindow()`. This
class adds two new functions, GetEventCount, which takes an optional
wxEventType. It then returns the number of events of that type that it has
received since the last call. Passing nothing returns the total number of
event received since the last call. Also there is OnEvent, which counts the
events based on type that are passed to it. To make it easy to count events
there is also a new class called EventCounter which takes a window and event
type and connects the window to the top level wxTestableFrame with the specific
event type. It disconnects again once it is out of scope. It simply reduces
the amount of typing required to count events.
3. add a `<sources>` tag for your source file to tests/test.bkl. Make sure it's
in the correct section: the one starting `<exe id="test_gui"` for a gui test,
the one starting `<exe id="test" template="wx_sample_console` otherwise.
$ cd build/bakefiles
$ bakefile_gen -b ../../tests/test.bkl
III. Running the tests
----------------------
Writing GUI-specific tests
--------------------------
1. Regenerate the make/project files from test.bkl using bakefile_gen, e.g.:
cd build/bakefiles
bakefile_gen -b ../../tests/test.bkl
and if you're on a unix system re-run configure.
`wxUIActionSimulator` can be used when user input is required, for example
clicking buttons or typing text. A simple example of this can be found in
`tests/controls/buttontest.cpp`. After simulating some user input always
call `wxYield()` to allow event processing. When writing a test using
`wxUIActionSimulator` wrap it in `#if wxUSE_UIACTIONSIMULATOR` block.
2. Build the test program using one of the make/project files in the tests
subdirectory.
3. Run the test program by using the command 'test' for the console tests,
'test_gui' for the gui ones. With no arguments, all the default set of tests
(all those registered with CPPUNIT_TEST_SUITE_REGISTRATION) are run.
Or to list the test suites without running them:
test -l or test_gui -l
4. Tests that have been registered under a name using
CPPUNIT_TEST_SUITE_NAMED_REGISTRATION can also be run separately. For
example:
test_gui ButtonTestCase
or to list the tests done by a particular testcase:
test -L MBConvTestCase
5. Fault navigation.
VC++ users can run the programs as a post build step (Projects/Settings/
Post-build step) to see the test results in an IDE window. This allows
errors to be jumped to in the same way as for compiler errors, for
example by pressing F4 or highlighting the error and pressing return.
Similarly for makefile users: makefiles can be modified to execute the
test programs as a final step. Then you can navigate to any errors in the
same way as for compiler errors, if your editor supports that.
Another alternative is to run the tests manually, redirecting the output
to a file. Then use your editor to jump to any failures. Using Vim, for
example, ':cf test.log' would take you to the first error in test.log, and
':cn' to the next.
If you would like to set a breakpoint on a failing test using a debugger,
put the breakpoint on the function 'CppUnit::Asserter::fail()'. This will
stop on each failing test.
There are a number of classes that are available to help with testing GUI
elements. Firstly throughout the test run there is a frame of type
`wxTestableFrame` that you can access through `wxTheApp->GetTopWindow()`. This
class adds two new functions, `GetEventCount()`, which takes an optional
`wxEventType`. It then returns the number of events of that type that it has
received since the last call. Passing nothing returns the total number of event
received since the last call. Also there is `OnEvent()`, which counts the events
based on type that are passed to it. To make it easy to count events there is
also a new class called `EventCounter` which takes a window and event type and
connects the window to the top level `wxTestableFrame` with the specific event
type. It disconnects again once it is out of scope. It simply reduces the
amount of typing required to count events.
IV. Notes
---------
Running the tests
-----------------
1. You can register your tests (or a subset of them) just under a name, and not
in the unnamed registry if you don't want them to be executed by default.
Run the main test suite by using the command `test` for the console tests,
or `test_gui` for the GUI ones. With no arguments, all the default set of tests
(all those registered without `[hide]` tag) are run.
2. If you are going to register your tests both in the unnamed registry
and under a name, then use the name that the tests have in the 'test -l'
listing.
To list the test suites without running them use `-l` command-line option.
3. Tests which fail can be temporarily registered under "fixme" while the
problems they expose are fixed, instead of the unnamed registry. That
way they can easily be run, but they do not make regression testing with
the default suite more difficult. E.g.:
// register in the unnamed registry so that these tests are run by default
//CPPUNIT_TEST_SUITE_REGISTRATION(wxRegExTestCase);
CPPUNIT_TEST_SUITE_NAMED_REGISTRATION(wxRegExTestCase, "fixme");
// also include in its own registry so that these tests can be run alone
CPPUNIT_TEST_SUITE_NAMED_REGISTRATION(wxRegExTestCase, "wxRegExTestCase");
4. Tests which take a long time to execute can be registered under "advanced"
instead of the unnamed registry. The default suite should execute reasonably
quickly. To run the default and advanced tests together:
test "" advanced
To run a particular test case, use `./test NameTestCase`. To run all tests
using the specified tag, use `./test [tag_name]` (the square brackets may need
to be escaped from your shell).