reorganise test suite compartments
this change is prerequisite to allow linking against different scopes (#938)
This commit is contained in:
parent
ed601d9eba
commit
7c9ab5fba2
33 changed files with 6105 additions and 21 deletions
|
|
@ -16,11 +16,11 @@ running test cases
|
||||||
|
|
||||||
Tests are the only form of documentation, known to provides some resilience against
|
Tests are the only form of documentation, known to provides some resilience against
|
||||||
becoming outdated. Tests help to focus on the usage, instead of engaging in spurious
|
becoming outdated. Tests help to focus on the usage, instead of engaging in spurious
|
||||||
implementation details. Developers are highly encourraged to write the tests _before_
|
implementation details. Developers are highly encouraged to write the tests _before_
|
||||||
the actual implementation, or at least alongside and interleaved with expanding the
|
the actual implementation, or at least alongside and interleaved with expanding the
|
||||||
feature set of the actual code.
|
feature set of the actual code.
|
||||||
There may be exceptions to this rule. Not every single bit needs to be covered by tests.
|
There may be exceptions to this rule. Not every single bit needs to be covered by tests.
|
||||||
Some features are highly cross-cutting and exceptionally difficult to cover with thests.
|
Some features are highly cross-cutting and exceptionally difficult to cover with tests.
|
||||||
And sometimes, just an abstract specification is a better choice.
|
And sometimes, just an abstract specification is a better choice.
|
||||||
|
|
||||||
As a rule of thumb, consider to write test code which is easy to read and understand,
|
As a rule of thumb, consider to write test code which is easy to read and understand,
|
||||||
|
|
@ -33,37 +33,44 @@ Test Structure
|
||||||
- simple test cases may be written as stand-alone application, while the more tightly
|
- simple test cases may be written as stand-alone application, while the more tightly
|
||||||
integrated test cases can be written as classes within the Lumiera application framework.
|
integrated test cases can be written as classes within the Lumiera application framework.
|
||||||
- test cases should use the +CHECK+ macro of NoBug to verify test results, since the normal
|
- test cases should use the +CHECK+ macro of NoBug to verify test results, since the normal
|
||||||
assertions may be deconfigured for optimised release builds.
|
assertions may be de-configured for optimised release builds.
|
||||||
- our test runner script 'test.sh' provides mechanisms for checking for expected output
|
- our test runner script 'test.sh' provides mechanisms to check for expected output
|
||||||
|
|
||||||
Several levels of aggregation are available. At the lowest level, a test typically runs
|
Several levels of aggregation are available. At the lowest level, a test typically runs
|
||||||
several functions within the same test fixture. This allows to create a ``narrative''
|
several functions within the same test fixture. This allows to create a ``narrative''
|
||||||
in the code: first do this, than do that, and now that, and now this should happen...
|
in the code: first do this, than do that, and now that, and now this should happen...
|
||||||
Generally speaking, it is up to the individual test to take care or isolate himself
|
Generally speaking, it is up to the individual test to take care or isolate himself
|
||||||
from any _dependencies_. Test code and application code uses the same mechanisms
|
from any _dependencies_. Test code and application code uses the same mechanisms
|
||||||
for accessing other components within the application. Up to now (2012), there
|
for accessing other components within the application. Up to now (2014), there
|
||||||
was no need for any kind of _dependency injection_, nor did we face any
|
was no need for any kind of _dependency injection_, nor did we face any
|
||||||
difficulties with tainted state.
|
difficulties with tainted state.
|
||||||
|
|
||||||
Test classes are organised into a tree closely mirroring the main application source
|
Test classes are organised into a tree closely mirroring the main application source
|
||||||
code tree. Large sections of this test tree are linked together into *test runners*
|
code tree. Large sections of this test tree are linked together into *test libraries*.
|
||||||
|
Some of these are linked against a specific (sub)scope of the application, like e.g.
|
||||||
|
only against the support library, the application framework or the backend. Since we
|
||||||
|
use _strict dependencies_, this linking step will spot code not being placed at the
|
||||||
|
correct scope within the whole system. As a final step, the build system creates a
|
||||||
|
*test runner* application (`target/test-suite`), which links dynamically against _all_
|
||||||
|
the test libraries and thus against all application dependencies.
|
||||||
Individual test classes integrate into this framework by placing a simple declaration
|
Individual test classes integrate into this framework by placing a simple declaration
|
||||||
(actually using the `LAUNCHER` macro), which typically also defines some tags and
|
(actually using the `LAUNCHER` macro), which typically also defines some tags and
|
||||||
classification alongside.
|
classification alongside.
|
||||||
Using command line parameters for invocation of the test runners, it is possible to
|
This way, using command line parameters for invocation of the test runners, it is possible
|
||||||
run some category or especially tagged test classes, or to invoke just a single test class
|
to run some category or especially tagged test classes, or to invoke just a single test class
|
||||||
in isolation (using the ID, which is also the class name).
|
in isolation (using the ID, which is also the class name).
|
||||||
|
|
||||||
The next level of aggregation is provided by the top level test collection definitions
|
The next level of aggregation is provided by the top level test collection definitions
|
||||||
located in the 'test/' subdirectory. For running these collections as automatic tests within
|
located in the 'test/' subdirectory. For running these collections as automatic tests within
|
||||||
the build process, we use Cehteh's simple `test.sh` shell script.
|
the build process, we use Cehteh's `test.sh` shell script.
|
||||||
|
|
||||||
|
|
||||||
Tools and conventions
|
Tools and conventions
|
||||||
---------------------
|
---------------------
|
||||||
Test code and application code has to be kept separate; the application may be built without any
|
Test code and application code has to be kept separate; the application may be built without any
|
||||||
tests, since test code has a tendency to bloat the executables, especially in debug mode. Generic
|
tests, since test code has a tendency to bloat the executables, especially in debug mode. As an
|
||||||
test support code may be included in the library, and it is common for core components to offer
|
exception, generic _test support code_ may be included in the library, and it is common for core
|
||||||
dedicated test support and diagnostic features as part of the main application.
|
components to offer dedicated _test support_ and _diagnostic features_ as part of the main application.
|
||||||
|
|
||||||
Conventions for the Buildsystem
|
Conventions for the Buildsystem
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
@ -84,7 +91,7 @@ to help with automating the build and test execution, test code should adhere to
|
||||||
* subtrees of test classes (C++) are linked into one shared library per subtree. In the final linking step,
|
* subtrees of test classes (C++) are linked into one shared library per subtree. In the final linking step,
|
||||||
these are linked together into a single testrunner, which is also linked against the application core.
|
these are linked together into a single testrunner, which is also linked against the application core.
|
||||||
The resulting executable 'test-suite' is able to invoke any of the test classes in
|
The resulting executable 'test-suite' is able to invoke any of the test classes in
|
||||||
isolation, or a group / categroy of tests. +
|
isolation, or a group / category of tests. +
|
||||||
* simple plain-C tests (names starting with 'test-*') are grouped into several directories thematically,
|
* simple plain-C tests (names starting with 'test-*') are grouped into several directories thematically,
|
||||||
and linked according to the application layer. Each of those simple tests needs to be self contained
|
and linked according to the application layer. Each of those simple tests needs to be self contained
|
||||||
and provide a main method.
|
and provide a main method.
|
||||||
|
|
@ -92,9 +99,10 @@ to help with automating the build and test execution, test code should adhere to
|
||||||
Internal testsuite runner
|
Internal testsuite runner
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
The class `test::Suite` (as used by 'tests/testrunner.cpp') helps building an executable which will run _all registered
|
The class `test::Suite` (as used by 'tests/testrunner.cpp') helps building an executable which will run _all registered
|
||||||
test case objects,_ or some group of such testcases. Each test case implements a simple interface and thus provides
|
test case objects,_ or some group of such test cases. Each test case implements a simple interface and thus provides
|
||||||
a `run (args)` function, moreover, it registers itself immediately alongside with his definition; this works by the
|
a `run (args)` function, moreover, it registers itself immediately alongside with his definition; this works by the
|
||||||
usual trick of defining a static class object and calling some registration function from the constructor of this static var. See the following
|
usual trick of defining a static class object and calling some registration function from the constructor of this static var.
|
||||||
|
See the following
|
||||||
|
|
||||||
.hello-world-test example
|
.hello-world-test example
|
||||||
[source,C]
|
[source,C]
|
||||||
|
|
@ -130,16 +138,17 @@ namespace test {
|
||||||
|
|
||||||
.Notes:
|
.Notes:
|
||||||
* type Arg is compatible to `std::vector<string> &`
|
* type Arg is compatible to `std::vector<string> &`
|
||||||
* this vector may be `arg.size()==0`, which means no comandline args available.
|
* this vector may be `arg.size()==0`, which means no commandline args available.
|
||||||
* these args may contain further arguments passed from system commandline (or the testsuite definition).
|
* these args may contain further arguments passed from system commandline (or the testsuite definition).
|
||||||
* the test can/should produce output that can be checked with 'test.sh'
|
* the test can/should produce output that can be checked with 'test.sh'
|
||||||
* the macro `LAUNCHER` expands to `Launch<HelloWorld_test> run_HelloWorld_test("HelloWorld_test","unit function common");`
|
* the macro `LAUNCHER` expands to +
|
||||||
|
`Launch<HelloWorld_test> run_HelloWorld_test("HelloWorld_test","unit function common");`
|
||||||
* note the second parameter to the macro (or the `Laucher`-ctor) is a space-delimited list of group names
|
* note the second parameter to the macro (or the `Laucher`-ctor) is a space-delimited list of group names
|
||||||
* thus any test can declare itself as belonging to some groups, and we can create a `test::Suite` for each group if we want.
|
* thus any test can declare itself as belonging to some groups, and we can create a `test::Suite` for each group if we want.
|
||||||
|
|
||||||
invoking a testrunner executable
|
invoking a testrunner executable
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
The class `test::TestOption` predefines a boost-commandlineparser to support the following optons:
|
The class `test::TestOption` predefines a boost-commandlineparser to support the following options:
|
||||||
|
|
||||||
[width="90%",cols="<.<,^4"]
|
[width="90%",cols="<.<,^4"]
|
||||||
|====
|
|====
|
||||||
|
|
@ -149,7 +158,9 @@ The class `test::TestOption` predefines a boost-commandlineparser to support the
|
||||||
|`[testID]` | (optional) one single testcase. If missing, all testcases of the group will be invoked
|
|`[testID]` | (optional) one single testcase. If missing, all testcases of the group will be invoked
|
||||||
|`--describe`| print all registered tests to stdout in a format suited for use with test.sh
|
|`--describe`| print all registered tests to stdout in a format suited for use with test.sh
|
||||||
|====
|
|====
|
||||||
Further commandline arguments are deliverd to a single testcase only if you specify a `testID`. Otherwise, all commandline arguments remaining after options parsing will be discarded and all tests of the suite will be run with an commandline vector of `size()==0`
|
Further commandline arguments are deliverd to a single testcase only if you specify a `testID`.
|
||||||
|
Otherwise, all commandline arguments remaining after options parsing will be discarded and all tests of the suite
|
||||||
|
ill be run with an commandline vector of `size()==0`
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -190,7 +201,7 @@ Writing test collection definitions
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
The definitions for test collections usable with `test.sh` are written in files named '##name.tests'
|
The definitions for test collections usable with `test.sh` are written in files named '##name.tests'
|
||||||
in the 'tests/' directory dir, where ## is a number defining the order of the various test files.
|
in the 'tests/' directory dir, where ## is a number defining the order of the various test files.
|
||||||
Of course, ``name'' should be a descriptive name about whats going to be tested. Each test collection
|
Of course, ``name'' should be a descriptive name about what is going to be tested. Each test collection
|
||||||
may invoke _only a single binary_ -- yet it may define numerous test cases, each invoking this binary
|
may invoke _only a single binary_ -- yet it may define numerous test cases, each invoking this binary
|
||||||
while supplementing different arguments. Combined with the ability of our test runner executables to
|
while supplementing different arguments. Combined with the ability of our test runner executables to
|
||||||
invoke individual test classes, this allows for fine grained test case specifications.
|
invoke individual test classes, this allows for fine grained test case specifications.
|
||||||
|
|
|
||||||
1518
tests/00helloworld.tests
Normal file
1518
tests/00helloworld.tests
Normal file
File diff suppressed because it is too large
Load diff
1518
tests/12metaprogramming.tests
Normal file
1518
tests/12metaprogramming.tests
Normal file
File diff suppressed because it is too large
Load diff
1518
tests/15library.tests
Normal file
1518
tests/15library.tests
Normal file
File diff suppressed because it is too large
Load diff
1518
tests/25fundamental.tests
Normal file
1518
tests/25fundamental.tests
Normal file
File diff suppressed because it is too large
Load diff
1
tests/basics/DIR_INFO
Normal file
1
tests/basics/DIR_INFO
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
basics and fundamentals testsuite
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
/*
|
/*
|
||||||
errortest.c - executable for running bug regression tests
|
HelloBug(test) - placeholder for running bug regression tests
|
||||||
|
|
||||||
Copyright (C) Lumiera.org
|
Copyright (C) Lumiera.org
|
||||||
2008, Christian Thaeter <ct@pipapo.org>
|
2008, Christian Thaeter <ct@pipapo.org>
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue