benn/lumiera_
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Scheduler-test: draft a structure to formalise these investigations

Browse source 
- the goal is to run a binary search
- the search condition should be factored out
- thus some kind of framework or DSL is required,
  to separate the technicalities of the measurement
  from the specifics of the actual test case.
This commit is contained in:
Fischlurch 2024-01-02 21:46:44 +01:00
parent 29699991a0
commit 6f4bd150fd
5 changed files with 211 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

15
tests/vault/gear/scheduler-stress-test.cpp
View file

@ -27,6 +27,7 @@
#include "lib/test/run.hpp"
#include "test-chain-load.hpp"
#include "stress-test-rig.hpp"
#include "vault/gear/scheduler.hpp"
#include "lib/time/timevalue.hpp"
#include "lib/format-string.hpp"
@ -344,6 +345,20 @@ SHOW_EXPR(refTime);
sdev = sqrt (sdev/REPETITIONS);
cout << rowFmt % stress % (avg/1000) % (sdev/1000) % (avgd/1000) % (double(misses)/REPETITIONS) % (expTime/1000) <<endl;
}
///////////////////////////////////////////////////////////////////////////////////////////////////WIP : draft for testbench-DSL
struct Setup : StressRig
{
usec LOAD_BASE = 500us;
uint CONCURRENCY = 4;
auto testLoad() { return TestChainLoad<>{64}.configureShape_chain_loadBursts(); }
};
auto [stress,delta,time] = StressRig::with<Setup>().searchBreakingPoint();
///////////////////////////////////////////////////////////////////////////////////////////////////WIP : draft for testbench-DSL
SHOW_EXPR(stress)
SHOW_EXPR(delta)
SHOW_EXPR(time)
}

153
tests/vault/gear/stress-test-rig.hpp Normal file
View file

@ -0,0 +1,153 @@
/*
STRESS-TEST-RIG.hpp - setup for stress and performance investigation
Copyright (C) Lumiera.org
2024, Hermann Vosseler <Ichthyostega@web.de>
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of
the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
*/
/** @file stress-test-rig.hpp
** A test bench to conduct performance measurement series. Outfitted especially
** to determine runtime behaviour of the Scheduler and associated parts of the
** Lumiera Engine through systematic execution of load scenarios.
**
** # Scheduler Stress Testing
**
** The point of departure for any stress testing is to show that the subject will
** break in controlled ways only. For the Scheduler this can easily be achieved by
** overloading until job deadlines are broken. Much more challenging however is the
** task to find out about the boundary of regular scheduler operation. This realm
** can be defined by the ability of the scheduler to follow and conform to the
** timings set out explicitly in the schedule. Obviously, short and localised
** load peaks can be accommodated, yet once a persistent backlog builds up,
** the schedule starts to slip and the calculation process will flounder.
**
** A method to determine such a _»breaking point«_ in a systematic way is based on
** building a [synthetic calculation load](\ref test-chain-load.hpp) and establish
** the timings of a test schedule based on a simplified model of expected computation
** expense. By scaling and condensing these schedule timings, a loss of control can
** be provoked, and determined by statistical observation: since the process of
** scheduling contains an essentially random component, persistent overload will be
** indicated by an increasing variance of the overall runtime, and a departure from
** the nominal runtime of the executed schedule.
**
** ## Setup
** To perform this test scheme, an operational Scheduler is required, and an instance
** of the TestChainLoad must be provided, configured with desired load properties.
** The _stressFactor_ of the corresponding generated schedule will be the active parameter
** of this test, performing a binary search for the _breaking point._
**
** ## Observation tools
**
** @see TestChainLoad_test
** @see SchedulerStress_test
*/
#ifndef VAULT_GEAR_TEST_STRESS_TEST_RIG_H
#define VAULT_GEAR_TEST_STRESS_TEST_RIG_H
#include "vault/common.hpp"
//#include "test-chain-load.hpp"
//#include "lib/test/transiently.hpp"
#include "vault/gear/scheduler.hpp"
#include "lib/time/timevalue.hpp"
//#include "lib/iter-explorer.hpp"
#include "lib/format-string.hpp"
#include "lib/format-cout.hpp"//////////////////////////TODO RLY?
//#include "lib/util.hpp"
//#include <functional>
//#include <utility>
//#include <memory>
//#include <string>
//#include <vector>
#include <tuple>
//#include <array>
namespace vault{
namespace gear {
namespace test {
using util::_Fmt;
// using util::min;
// using util::max;
// using util::isnil;
// using util::limited;
// using util::unConst;
// using util::toString;
// using util::isLimited;
// using lib::time::Time;
// using lib::time::TimeValue;
// using lib::time::FrameRate;
// using lib::time::Duration;
// using lib::test::Transiently;
// using lib::meta::_FunRet;
// using std::string;
// using std::function;
// using std::make_pair;
using std::make_tuple;
// using std::forward;
// using std::string;
// using std::swap;
// using std::move;
namespace err = lumiera::error; //////////////////////////TODO RLY?
namespace { // Default definitions ....
}
namespace stress_test_rig {
template<class CONF>
class BreakingPointBench
{
public:
auto
searchBreakingPoint()
{
double finalStress{0};
double avgDelta{0};
double avgTime{0};
return make_tuple (finalStress, avgDelta, avgTime);
}
};
}
class StressRig
: util::NonCopyable
{
public:
using usec = std::chrono::microseconds;
template<class CONF>
static auto
with()
{
return stress_test_rig::BreakingPointBench<CONF>{};
}
};
}}} // namespace vault::gear::test
#endif /*VAULT_GEAR_TEST_STRESS_TEST_RIG_H*/

22
tests/vault/gear/test-chain-load.hpp
View file

@ -69,6 +69,28 @@
** [retrieved](\ref TestChainLoad::getHash). The node structure can then be traversed or
** [scheduled as Render Jobs](\ref TestChainLoad::scheduleJobs).
**
** ## Test support
** A tool for generating roughly calibrated [computational load](\ref ComputationalLoad) is included,
** to be controlled by the Node::weight stepping. Load can either be generated by arithmetic (hash)
** computations, or by accessing and adding memory in a private allocated data block. To make this
** load controllable, the instance is configured with a _time base_ setting, with sensible settings
** between 50µs to 100ms; moreover, a _calibration run_ is necessary once per runtime (static variables);
** the actual invocation uses a multiple of this base setting, as determined by the Node::weight.
**
** This allows to render individual steps in the calculation graph much more expensive than the
** associated organisational code and setup, and thus creates the foundation for building realistic
** test scenarios. For a given graph instance, a [runtime weight](\ref TestChainLoad::levelScheduleSequence)
** can be estimated heuristically, expressed in multiples of the Node::weight ≡ 1, taking into account
** the possible parallelisation _per level of nodes_ described by an idealised model.
**
** For the actual test run, a ScheduleCtx is built, using an actual scheduler instance. Specialised
** render job functors are provided to perform incremental job planning and invocation of individual
** nodes in the graph as computation steps, optionally with a computation load. The scheduler is
** triggered by inserting the initial planning job in a _pre roll phase,_ blocking the main thread
** until a callback job is invoked, which is set as final dependency behind the exit node of the
** complete graph, returning a observed runtime in microseconds from the nominal start point of
** the schedule.
**
** ## Observation tools
** The generated topology can be visualised as a graph, using the Graphviz-DOT language.
** Nodes are rendered from bottom to top, organised into strata according to the time-level

9
wiki/renderengine.html
View file

@ -7327,7 +7327,7 @@ The last point highlights a //circular structure:// the planning job itself was
The way other parts of the system are built, requires us to obtain a guaranteed knowledge of some job's termination. It is possible to obtain that knowledge with some limited delay, but it nees to be absoultely reliable (violations leading to segfault). The requirements stated above assume this can be achieved through //jobs with guaranteed execution.// Alternatively we could consider installing specific callbacks -- in this case the scheduler itself has to guarantee the invocation of these callbacks, even if the corresponding job fails or is never invoked. It doesn't seem there is any other option.
</pre>
</div>
<div title="SchedulerTest" creator="Ichthyostega" modifier="Ichthyostega" created="202312281814" modified="202312281836" tags="Rendering operational draft" changecount="4">
<div title="SchedulerTest" creator="Ichthyostega" modifier="Ichthyostega" created="202312281814" modified="202401022341" tags="Rendering operational draft" changecount="9">
<pre>With the Scheduler testing effort [[#1344|https://issues.lumiera.org/ticket/1344]], several goals are pursued
* by exposing the new scheduler implementation to excessive overload, its robustness can be assessed and defects can be spotted
* with the help of a systematic, calibrated load, characteristic performance limits and breaking points can be established
@ -7356,6 +7356,13 @@ The Scheduler thus follows the connectivity pattern, and defining a specific pat
* the node graph holds a fixed preconfigured number of nodes
* the //width// is likewise preconfigured, i.e. the number of nodes on the same »level« and typically started at the same time
* generation is automatically terminated with an exit node, joining all remaining chains
! Scheduler Stress Testing
The point of departure for any stress testing is to show that the subject is resilient and will break in controlled ways only. Since the Scheduler relies on the limited computational resources available in the system, overload is easy to provoke by adding too many render jobs; delay will build up until some job's deadline is violated, at which point the Scheduler will drop this job (and any follow-up jobs with unmet dependencies). Much more challenging however is the task to find out about //the boundary of regular scheduler operation.// The domain of regular operation can be defined by the ability of the scheduler to follow and conform to the timings set out explicitly in the schedule. Obviously, short and localised load peaks can be accommodated, yet once a persistent backlog builds up, the schedule starts to slip and the calculation process will flounder.
A method to determine such a »''breaking point''« in a systematic way relies on the //synthetic calculation load// mentioned above, the {{{TestChainLoad}}}. Using a simplified model of expected computation expense, a timing grid for scheduling can be established, which is then //scaled and condensed// to provoke loss of control in the Scheduler -- a condition that can be determined by statistical observation: since the process of scheduling contains an element of //essential randomness,// persistent overload will be indicated by an increasing variance of the overall runtime, and a departure from the nominal runtime of the executed schedule. This consideration leads to a formalised condition, which can be used to control a //binary search// to find the breaking point in terms of a //critical stress factor.//
Observing this breaking point in correlation with various load patterns will unveil performance characteristics and weak spots of the implementation.
</pre>
</div>
<div title="SchedulerWorker" creator="Ichthyostega" modifier="Ichthyostega" created="202309041605" modified="202312281745" tags="Rendering operational spec draft" changecount="21">

13
wiki/thinkPad.ichthyo.mm
View file

@ -107818,6 +107818,19 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<icon BUILTIN="idea"/>
</node>
</node>
<node BACKGROUND_COLOR="#eef0c5" COLOR="#990000" CREATED="1704217480508" ID="ID_653759226" MODIFIED="1704237587427" TEXT="diese Methode in eine Test-Komponente &#xfc;berf&#xfc;hren">
<icon BUILTIN="pencil"/>
<node CREATED="1704232960613" ID="ID_689181948" MODIFIED="1704232994486" STYLE="bubble" TEXT="&#xbb;StressTestRig&#xab;">
<font BOLD="true" NAME="SansSerif" SIZE="14"/>
</node>
<node BACKGROUND_COLOR="#eef0c5" COLOR="#990000" CREATED="1704228416349" ID="ID_1774406683" MODIFIED="1704237584351" TEXT="Entwurf">
<icon BUILTIN="pencil"/>
<node CREATED="1704228425177" ID="ID_758715159" MODIFIED="1704228439379" TEXT="nicht zu viel Fachlichkeit explizit in das Test-Setup einbauen"/>
<node CREATED="1704228440158" ID="ID_18825123" MODIFIED="1704228452703" TEXT="ein Baukasten-System anstreben (f&#xfc;r weitere Tests)"/>
<node CREATED="1704237555914" ID="ID_1779084715" MODIFIED="1704237568213" TEXT="funktionale Konfigurations-Hooks"/>
<node CREATED="1704237569280" ID="ID_227955379" MODIFIED="1704237574469" TEXT="Basis-Setup einmischen"/>
</node>
</node>
</node>
</node>
<node BACKGROUND_COLOR="#eef0c5" COLOR="#990000" CREATED="1702945804704" ID="ID_996535642" MODIFIED="1703423856728" TEXT="Beobachtungen aus erweiterten Testl&#xe4;ufen">