Scheduler-test: define criterion for breaking point
This statistical criterion defines when to count observed Scheduler performance as loosing control. The test is comprised of three observations, which all must be confirmed: - an individual run counts as accidentally failed when the execution slips away by more than 2ms with respect to the defined overall schedule. When more than 55% of all observed runs are considered as failed, the first condition is met - moreover, the observed standard derivation must also surpass the same limit of > 2ms, which indicates that the Scheduling mechanism is under substantial strain (on average, the slip is ~ 200µs) - the third condition is that the ''averaged delta'' has surpassed 4ms, which is 2 times the basic failure indicator. These conditions are based on watching the Scheduler in operation; typically all three conditions slip away by large margin after a very narrow yet critical increase in the stress level. Using three conditions together should improve robustness; often the problems to keep up with the schedule build up over some parameter range, yet the actual decision should be based on complete loss of control.
This commit is contained in:
parent
fda34a42ca
commit
54e489b9b6
2 changed files with 25 additions and 10 deletions
|
|
@ -49,7 +49,19 @@
|
|||
** To perform this test scheme, an operational Scheduler is required, and an instance
|
||||
** of the TestChainLoad must be provided, configured with desired load properties.
|
||||
** The _stressFactor_ of the corresponding generated schedule will be the active parameter
|
||||
** of this test, performing a binary search for the _breaking point._
|
||||
** of this test, performing a binary search for the _breaking point._ The Measurement
|
||||
** attempts to narrow down to the point of massive failure, when the ability to somehow
|
||||
** cope with the schedule completely break down. Based on watching the Scheduler in
|
||||
** operation, the detection was linked to three conditions, which typically will
|
||||
** be triggered together, and within a narrow and reproducible parameter range:
|
||||
** - an individual run counts as _accidentally failed_ when the execution slips
|
||||
** away by more than 2ms with respect to the defined overall schedule. When more
|
||||
** than 55% of all observed runs are considered as failed, the first condition is met
|
||||
** - moreover, the observed ''standard derivation'' must also surpass the same limit
|
||||
** of > 2ms, which indicates that the Scheduling mechanism is under substantial
|
||||
** strain; in regular operation, the slip is rather ~ 200µs.
|
||||
** - the third condition is that the ''averaged delta'' has surpassed 4ms,
|
||||
** which is 2 times the basic failure indicator.
|
||||
**
|
||||
** ## Observation tools
|
||||
**
|
||||
|
|
@ -154,7 +166,7 @@ namespace test {
|
|||
auto sqr = [](auto n){ return n*n; };
|
||||
Res res;
|
||||
auto& [sf,pf,sdev,avgD,avgT,expT] = res;
|
||||
sf = stressFac;
|
||||
sf = stressFac;
|
||||
expT = testSetup.getExpectedEndTime() / 1000;
|
||||
std::array<double, CONF::REPETITIONS> runTime;
|
||||
for (uint i=0; i<CONF::REPETITIONS; ++i)
|
||||
|
|
@ -182,7 +194,9 @@ namespace test {
|
|||
bool
|
||||
decideBreakPoint (Res& res)
|
||||
{
|
||||
return true; //////TODO booooo
|
||||
return res.percentOff > CONF::TRIGGER_FAIL
|
||||
and res.stdDev > CONF::TRIGGER_SDEV
|
||||
and res.avgDelta > CONF::TRIGGER_DELTA;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -196,10 +210,11 @@ namespace test {
|
|||
UNIMPLEMENTED ("invoke a library implementation of binary search");
|
||||
}
|
||||
|
||||
|
||||
_Fmt fmtRun_ {"....·%-2d: Δ=%4.1f t=%4.1f %s %s"}; // i % Δ % t % t>avg? % fail?
|
||||
_Fmt fmtStep_{ "%4.2f| : ∅Δ=%4.1f±%-4.2f ∅t=%4.1f %%%3.1f -- expect:%4.1fms"}; // stress % ∅Δ % σ % ∅t % fail % t-expect
|
||||
_Fmt fmtResVal_{"%9s: %5.2f%s"};
|
||||
_Fmt fmtResSDv_{"%9s= %5.2f ±%4.2f%s"};
|
||||
_Fmt fmtResVal_{"%9s: %5.2f%s"};
|
||||
|
||||
void
|
||||
showRun(uint i, double delta, double t, bool over, bool fail)
|
||||
|
|
@ -279,10 +294,10 @@ namespace test {
|
|||
|
||||
usec LOAD_BASE = 500us;
|
||||
uint CONCURRENCY = work::Config::getDefaultComputationCapacity();
|
||||
double FAIL_LIMIT = 2.0; ///< delta-limit when to count a run as failure
|
||||
double TRIGGER_FAIL = 0.55; ///< %-fact: criterion-1 failures above this rate
|
||||
double TRIGGER_SDEV = FAIL_LIMIT; ///< in ms : criterion-2 standard derivation
|
||||
double TRIGGER_DELTA = 4.0; ///< in ms : criterion-3 delta above this limit
|
||||
double FAIL_LIMIT = 2.0; ///< delta-limit when to count a run as failure
|
||||
double TRIGGER_FAIL = 0.55; ///< %-fact: criterion-1 failures above this rate
|
||||
double TRIGGER_SDEV = FAIL_LIMIT; ///< in ms : criterion-2 standard derivation
|
||||
double TRIGGER_DELTA = 2*FAIL_LIMIT; ///< in ms : criterion-3 delta above this limit
|
||||
bool showRuns = false; ///< print a line for each individual run
|
||||
bool showStep = true; ///< print a line for each binary search step
|
||||
bool showRes = true; ///< print result data
|
||||
|
|
|
|||
|
|
@ -107789,12 +107789,12 @@ Date:   Thu Apr 20 18:53:17 2023 +0200<br/>
|
|||
<node CREATED="1704205086169" ID="ID_903640160" MODIFIED="1704205137872" TEXT="wenn's woanders bricht schauen wir uns das dann an"/>
|
||||
</node>
|
||||
</node>
|
||||
<node BACKGROUND_COLOR="#f0d5c5" COLOR="#990033" CREATED="1704130775228" ID="ID_1335708273" MODIFIED="1704130795750" TEXT="Untersuchung: den »breaking point« identifizieren....?">
|
||||
<node COLOR="#435e98" CREATED="1704130775228" ID="ID_1335708273" MODIFIED="1704312564086" TEXT="Untersuchung: den »breaking point« identifizieren....?">
|
||||
<icon BUILTIN="help"/>
|
||||
<node CREATED="1704206510327" ID="ID_40492887" MODIFIED="1704206560860" TEXT="erster Eindruck: bei steigendem Druck geht Zeit asymptotisch gegen einen Sockelwert"/>
|
||||
<node CREATED="1704206563512" ID="ID_366217988" MODIFIED="1704206579674" TEXT="Problem dabei: starke zufällige Schwankungen"/>
|
||||
<node CREATED="1704213495794" ID="ID_13093420" MODIFIED="1704213595391" TEXT="⟹ Statistik ����☢"/>
|
||||
<node CREATED="1704213601579" ID="ID_1669745150" MODIFIED="1704215362689" TEXT="mehrere Kriterien sprechen zugleich an">
|
||||
<node BACKGROUND_COLOR="#c8c0b6" COLOR="#435e98" CREATED="1704213601579" ID="ID_1669745150" MODIFIED="1704312567285" TEXT="mehrere Kriterien sprechen zugleich an">
|
||||
<icon BUILTIN="forward"/>
|
||||
<node CREATED="1704213628183" ID="ID_4871192" MODIFIED="1704215332007" TEXT="bei ~ stress = 0.5"/>
|
||||
<node CREATED="1704213650125" ID="ID_1392777284" MODIFIED="1704215323229" TEXT="mehr als 55% »Ausrutscher«"/>
|
||||
|
|
|
|||
Loading…
Reference in a new issue