After the individual tests, we calculate the averaged delta over the
whole test suite, to detect changes to the overall timings. As it turned out,
using the error propagation for the calculation of the averaged delta
yields the right tolerance band to ignore random fluctuations but
trigger alarm on real changes.
Moreover, add several further timing test cases
to verify the calibration via "platform model" works as intended
Since the platform calibration inevitably incurs some additional error band,
a linear regresssion over the time series of measurements can additionally be used
to spot ongoing systematic changes below this general error band, while
leveling out local statistical fluctuations.
...mostly routine after solving the tricky design challenge
- for usage, instantiate the template DataFile with a Storage record
- object is created with filename, and immediately slurps in existing data
- data storage is optimised for readability (not speed); newest value at top
Note: some kind of testcase is "hidden" in this changeset only;
next changeset will remove research-experiment.hpp
rationale: the purpose is to read back our own values,
yet it should be reasonably standard, to allow investigating
and tweaking values with a spreadsheet
- first line is a header line and used to verify the number of columns
- one record per line, embedded line breaks prohibited
- fields separated by comma, semicolon tolerated
- fields are trimmed and may be empty
- a field may be double quoted
- only quoted fields may contain whitespace or comma
- no escaping of quotes, i.e. no quotes within quotes
- build the parametrisation from several config sources
- read and interpret the test specifications in a subdirectory structure
- as a result build a complete test suite definition as a sequence of test steps
- setup a stage, also based on the parametrisation
- actually perform the test suite and capture results
- generate a test report
