benn/lumiera_
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
lumiera_/tests/vault/gear/stress-test-rig.hpp

499 lines
19 KiB
C++
Raw Normal View History

Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
/*
STRESS-TEST-RIG.hpp - setup for stress and performance investigation
Copyright (C) Lumiera.org
2024, Hermann Vosseler <Ichthyostega@web.de>
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of
the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
*/
/** @file stress-test-rig.hpp
** A test bench to conduct performance measurement series. Outfitted especially
** to determine runtime behaviour of the Scheduler and associated parts of the
** Lumiera Engine through systematic execution of load scenarios.
**
** # Scheduler Stress Testing
**
** The point of departure for any stress testing is to show that the subject will
** break in controlled ways only. For the Scheduler this can easily be achieved by
** overloading until job deadlines are broken. Much more challenging however is the
** task to find out about the boundary of regular scheduler operation. This realm
** can be defined by the ability of the scheduler to follow and conform to the
** timings set out explicitly in the schedule. Obviously, short and localised
** load peaks can be accommodated, yet once a persistent backlog builds up,
** the schedule starts to slip and the calculation process will flounder.
**
** A method to determine such a _»breaking point«_ in a systematic way is based on
** building a [synthetic calculation load](\ref test-chain-load.hpp) and establish
** the timings of a test schedule based on a simplified model of expected computation
** expense. By scaling and condensing these schedule timings, a loss of control can
** be provoked, and determined by statistical observation: since the process of
** scheduling contains an essentially random component, persistent overload will be
** indicated by an increasing variance of the overall runtime, and a departure from
** the nominal runtime of the executed schedule.
**
** ## Setup
** To perform this test scheme, an operational Scheduler is required, and an instance
** of the TestChainLoad must be provided, configured with desired load properties.
** The _stressFactor_ of the corresponding generated schedule will be the active parameter
Scheduler-test: define criterion for breaking point This statistical criterion defines when to count observed Scheduler performance as loosing control. The test is comprised of three observations, which all must be confirmed: - an individual run counts as accidentally failed when the execution slips away by more than 2ms with respect to the defined overall schedule. When more than 55% of all observed runs are considered as failed, the first condition is met - moreover, the observed standard derivation must also surpass the same limit of > 2ms, which indicates that the Scheduling mechanism is under substantial strain (on average, the slip is ~ 200µs) - the third condition is that the ''averaged delta'' has surpassed 4ms, which is 2 times the basic failure indicator. These conditions are based on watching the Scheduler in operation; typically all three conditions slip away by large margin after a very narrow yet critical increase in the stress level. Using three conditions together should improve robustness; often the problems to keep up with the schedule build up over some parameter range, yet the actual decision should be based on complete loss of control.
2024-01-03 21:02:23 +01:00
** of this test, performing a binary search for the _breaking point._ The Measurement
** attempts to narrow down to the point of massive failure, when the ability to somehow
** cope with the schedule completely break down. Based on watching the Scheduler in
** operation, the detection was linked to three conditions, which typically will
** be triggered together, and within a narrow and reproducible parameter range:
** - an individual run counts as _accidentally failed_ when the execution slips
** away by more than 2ms with respect to the defined overall schedule. When more
** than 55% of all observed runs are considered as failed, the first condition is met
** - moreover, the observed ''standard derivation'' must also surpass the same limit
** of > 2ms, which indicates that the Scheduling mechanism is under substantial
** strain; in regular operation, the slip is rather ~ 200µs.
** - the third condition is that the ''averaged delta'' has surpassed 4ms,
** which is 2 times the basic failure indicator.
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
**
** ## Observation tools
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
** As a complement to the bench::BreakingPoint tool, another tool is provided to
** run a specific Scheduler setup while varying a single control parameter within
** defined limits. This produces a set of (x,y) data, which can be used to search
** for correlations or build a linear regression model to describe the Scheduler's
** behaviour as function of the control parameter. The typical use case would be
** to use the input length (number of Jobs) as control parameter, leading to a
** model for the Scheduler's expense.
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
**
** @see TestChainLoad_test
** @see SchedulerStress_test
Scheduler-test: extract search algo into lib
2024-01-04 02:03:05 +01:00
** @see binary-search.hpp
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
*/
#ifndef VAULT_GEAR_TEST_STRESS_TEST_RIG_H
#define VAULT_GEAR_TEST_STRESS_TEST_RIG_H
#include "vault/common.hpp"
Scheduler-test: extract search algo into lib
2024-01-04 02:03:05 +01:00
#include "lib/binary-search.hpp"
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
//#include "test-chain-load.hpp"
//#include "lib/test/transiently.hpp"
#include "vault/gear/scheduler.hpp"
#include "lib/time/timevalue.hpp"
//#include "lib/iter-explorer.hpp"
Scheduler-test: binary search over continuous domain - textbook implementation - capture results from visited points - average results form the last three points to damp statistic fluctuations
2024-01-03 22:48:49 +01:00
#include "lib/meta/function.hpp"
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
#include "lib/format-string.hpp"
#include "lib/format-cout.hpp"//////////////////////////TODO RLY?
Scheduler-test: simple implementation of range coverage - fill the range randomly with probe points - use the node count as independent parameter - measurement method *works as intended* - results indeed show a linear relationship Results are ''interesting'' however, since the (par,time) points seem to be arranged into two lines, implying that about half of the runs were somehow ''degraded'' and performed way slower.
2024-02-24 04:17:05 +01:00
#include "lib/util.hpp"
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
//#include <functional>
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
#include <utility>
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
//#include <memory>
//#include <string>
Scheduler-test: simplify binary search implementation While the idea with capturing observation values is nice, it definitively does not belong into a library impl of the search algorithm, because this is usage specific and grossly complicates the invocation. Rather, observation data can be captured by side-effect from the probe-λ holding the actual measurement run.
2024-01-04 01:32:11 +01:00
#include <vector>
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
#include <tuple>
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
#include <array>
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
namespace vault{
namespace gear {
namespace test {
using util::_Fmt;
Scheduler-test: binary search over continuous domain - textbook implementation - capture results from visited points - average results form the last three points to damp statistic fluctuations
2024-01-03 22:48:49 +01:00
using util::min;
using util::max;
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
// using util::isnil;
// using util::limited;
// using util::unConst;
// using util::toString;
// using util::isLimited;
// using lib::time::Time;
// using lib::time::TimeValue;
// using lib::time::FrameRate;
// using lib::time::Duration;
// using lib::test::Transiently;
// using lib::meta::_FunRet;
// using std::string;
// using std::function;
Scheduler-test: binary search over continuous domain - textbook implementation - capture results from visited points - average results form the last three points to damp statistic fluctuations
2024-01-03 22:48:49 +01:00
using std::make_pair;
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
using std::make_tuple;
// using std::forward;
// using std::string;
// using std::swap;
Scheduler-test: simplify binary search implementation While the idea with capturing observation values is nice, it definitively does not belong into a library impl of the search algorithm, because this is usage specific and grossly complicates the invocation. Rather, observation data can be captured by side-effect from the probe-λ holding the actual measurement run.
2024-01-04 01:32:11 +01:00
using std::vector;
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
using std::move;
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
namespace err = lumiera::error; //////////////////////////TODO RLY?
namespace { // Default definitions ....
}
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
/** configurable template framework for running Scheduler Stress tests */
class StressRig
: util::NonCopyable
{
public:
/***********************************************************************//**
* Entrance Point: build a stress test measurement setup using a dedicated
* \a TOOL class, takes the configuration \a CONF as template parameter
* and which is assumed to inherit (indirectly) from StressRig.
* @tparam CONF specialised subclass of StressRig with customisation
* @return a builder to configure and then launch the actual test
*/
template<class CONF>
static auto
with()
{
return Launcher<CONF>{};
}
/* ======= default configuration (inherited) ======= */
using usec = std::chrono::microseconds;
usec LOAD_BASE = 500us;
usec BASE_EXPENSE = 0us;
bool SCHED_NOTIFY = true;
bool SCHED_DEPENDS = false;
uint CONCURRENCY = work::Config::getDefaultComputationCapacity();
bool INSTRUMENTATION = true;
double EPSILON = 0.01; ///< error bound to abort binary search
double UPPER_STRESS = 0.6; ///< starting point for the upper limit, likely to fail
double FAIL_LIMIT = 2.0; ///< delta-limit when to count a run as failure
double TRIGGER_FAIL = 0.55; ///< %-fact: criterion-1 failures above this rate
double TRIGGER_SDEV = FAIL_LIMIT; ///< in ms : criterion-2 standard derivation
double TRIGGER_DELTA = 2*FAIL_LIMIT; ///< in ms : criterion-3 average delta above this limit
bool showRuns = false; ///< print a line for each individual run
bool showStep = true; ///< print a line for each binary search step
bool showRes = true; ///< print result data
bool showRef = true; ///< calculate single threaded reference time
static uint constexpr REPETITIONS{20};
BlockFlowAlloc bFlow{};
EngineObserver watch{};
Scheduler scheduler{bFlow, watch};
protected:
/** Extension point: build the computation topology for this test */
auto
Scheduler-test: simple implementation of range coverage - fill the range randomly with probe points - use the node count as independent parameter - measurement method *works as intended* - results indeed show a linear relationship Results are ''interesting'' however, since the (par,time) points seem to be arranged into two lines, implying that about half of the runs were somehow ''degraded'' and performed way slower.
2024-02-24 04:17:05 +01:00
testLoad(size_t nodes =64)
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
{
Scheduler-test: simple implementation of range coverage - fill the range randomly with probe points - use the node count as independent parameter - measurement method *works as intended* - results indeed show a linear relationship Results are ''interesting'' however, since the (par,time) points seem to be arranged into two lines, implying that about half of the runs were somehow ''degraded'' and performed way slower.
2024-02-24 04:17:05 +01:00
return TestChainLoad<>{nodes};
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
}
/** (optional) extension point: base configuration of the test ScheduleCtx */
template<class TL>
auto
testSetup (TL& testLoad)
{
return testLoad.setupSchedule(scheduler)
.withJobDeadline(100ms)
.withUpfrontPlanning();
}
template<class CONF>
struct Launcher : CONF
{
template<template<class> class TOOL, typename...ARGS>
auto
perform (ARGS&& ...args)
{
return TOOL<CONF>{}.perform (std::forward<ARGS> (args)...);
}
};
};
namespace bench { ///< Specialised tools to investigate scheduler performance
using std::declval;
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
/**************************************************//**
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
* Specific stress test scheme to determine the
* »breaking point« where the Scheduler starts to slip
*/
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
template<class CONF>
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
class BreakingPoint
: public CONF
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
{
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
using TestLoad = decltype(declval<BreakingPoint>().testLoad());
using TestSetup = decltype(declval<BreakingPoint>().testSetup (declval<TestLoad&>()));
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
struct Res
{
double stressFac{0};
double percentOff{0};
double stdDev{0};
double avgDelta{0};
double avgTime{0};
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
double expTime{0};
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
};
Scheduler-test: fix deficiencies in search control mechanism In binary search, in order to establish the invariant initially, a loop is necessary, since a single step might not be sufficient. Moreover, the ongoing adjustments jeopardise detection of the statistical breaking point condition, by causing a negative delta due to gradually approaching the point of convergence -- leading to an ongoing search in a region beyond the actual breaking point.
2024-02-19 15:58:05 +01:00
double adjustmentFac{1.0};
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
/** prepare the ScheduleCtx for a specifically parametrised test series */
void
configureTest (TestSetup& testSetup, double stressFac)
{
Scheduler-test: investigate volatile in computation-load The `volatile` was used asymmetrically and there was concern that this usage makes the `ComutationalLoad` dependent on concurrency. However, an impact could not be confirmed empirically. Moreover, to simplify this kind of tests, make the `schedDepends` directly configurable in the Stress-Test-Rig.
2024-01-07 23:37:54 +01:00
testSetup.withLoadTimeBase(CONF::LOAD_BASE)
.withBaseExpense (CONF::BASE_EXPENSE)
Scheduler-test: optionally allow to propagate immediately This is just another (obvious) degree of freedom, which could be interesting to explore in stress testing, while probably not of much relevance in practice (if a job is expected to become runable earlier, in can as well be just scheduled earlier). Some experimentation shows that the timing measurements exhibit more fluctuations, but also slightly better times when pressure is low, which is pretty much what I'd expect. When raising pressure, the average times converge towards the same time range as observed with time bound propagation. Note that enabling this variation requires to wire a boolean switch over various layers of abstraction; arguably this is an unnecessary complexity and could be retracted once the »experimentation phase« is over. This completes the preparation of a Scheduler Stress-Test setup.
2024-01-08 22:58:16 +01:00
.withSchedNotify (CONF::SCHED_NOTIFY)
Scheduler-test: investigate volatile in computation-load The `volatile` was used asymmetrically and there was concern that this usage makes the `ComutationalLoad` dependent on concurrency. However, an impact could not be confirmed empirically. Moreover, to simplify this kind of tests, make the `schedDepends` directly configurable in the Stress-Test-Rig.
2024-01-07 23:37:54 +01:00
.withSchedDepends(CONF::SCHED_DEPENDS)
Scheduler-test: run breaking-point search with gradual adaptation Integrating the changed logic into the StressTest-rig, with bugfixes
2024-02-18 23:02:32 +01:00
.withInstrumentation(CONF::INSTRUMENTATION) // side-effect: clear existing statistics
Scheduler-test: fix deficiencies in search control mechanism In binary search, in order to establish the invariant initially, a loop is necessary, since a single step might not be sufficient. Moreover, the ongoing adjustments jeopardise detection of the statistical breaking point condition, by causing a negative delta due to gradually approaching the point of convergence -- leading to an ongoing search in a region beyond the actual breaking point.
2024-02-19 15:58:05 +01:00
.withAdaptedSchedule(stressFac, CONF::CONCURRENCY, adjustmentFac);
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
}
/** perform a repetition of test runs and compute statistics */
Res
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
runProbes (TestSetup& testSetup, double stressFac)
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
{
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
auto sqr = [](auto n){ return n*n; };
Res res;
auto& [sf,pf,sdev,avgD,avgT,expT] = res;
Scheduler-test: define criterion for breaking point This statistical criterion defines when to count observed Scheduler performance as loosing control. The test is comprised of three observations, which all must be confirmed: - an individual run counts as accidentally failed when the execution slips away by more than 2ms with respect to the defined overall schedule. When more than 55% of all observed runs are considered as failed, the first condition is met - moreover, the observed standard derivation must also surpass the same limit of > 2ms, which indicates that the Scheduling mechanism is under substantial strain (on average, the slip is ~ 200µs) - the third condition is that the ''averaged delta'' has surpassed 4ms, which is 2 times the basic failure indicator. These conditions are based on watching the Scheduler in operation; typically all three conditions slip away by large margin after a very narrow yet critical increase in the stress level. Using three conditions together should improve robustness; often the problems to keep up with the schedule build up over some parameter range, yet the actual decision should be based on complete loss of control.
2024-01-03 21:02:23 +01:00
sf = stressFac;
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
std::array<double, CONF::REPETITIONS> runTime;
for (uint i=0; i<CONF::REPETITIONS; ++i)
{
runTime[i] = testSetup.launch_and_wait() / 1000;
avgT += runTime[i];
Scheduler-test: introduce a form-factor to account for empiric adaptation Relying on the new instrumentation facility, the actually effective concurrency and cumulative run time of the test jobs can be established. These can now be cast into a form-factor to represent actual excess expenses in relation to the theoretical model. By allowing to adjust the adapted schedule by this form factor, it can be made to reflect more closely the actual empiric load, hopefully leading to a more realistic effect of the stress-factor and thus results better suited to conclude on generic behaviour.
2024-02-18 18:01:21 +01:00
testSetup.adaptEmpirically (stressFac, CONF::CONCURRENCY);
Scheduler-test: more precise accounting for expected concurrency It turns out to be not correct using all the divergence in concurrency as a form factor, since it is quite common that not all cores can be active at every level, given the structural constraints as dictated by the load graph. On the other hand, if the empirical work (non wait-time) concurrency systematically differs from the simple model used for establishing the schedule, then this should indeed be considered a form factor and deduced from the effective stress factor, since it is not a reserve available for speed-up The solution entertained here is to derive an effective compounded sum of weights from the calculation used to build the schedule. This compounded weight sum is typically lower than the plain sum of all node weights, which is precisely due to the theoretical amount of expense reduction assumed in the schedule generation. So this gives us a handle at the theoretically expected expense and through the plain weight sum, we may draw conclusion about the effective concurrency expected in this schedule. Taking only this part as base for the empirical deviations yields search results very close to stressFactor ~1 -- implying that the test setup now observes what was intended to observe...
2024-02-19 17:36:46 +01:00
this->adjustmentFac = 1 / (testSetup.getStressFac() / stressFac);
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
}
Scheduler-test: introduce a form-factor to account for empiric adaptation Relying on the new instrumentation facility, the actually effective concurrency and cumulative run time of the test jobs can be established. These can now be cast into a form-factor to represent actual excess expenses in relation to the theoretical model. By allowing to adjust the adapted schedule by this form factor, it can be made to reflect more closely the actual empiric load, hopefully leading to a more realistic effect of the stress-factor and thus results better suited to conclude on generic behaviour.
2024-02-18 18:01:21 +01:00
expT = testSetup.getExpectedEndTime() / 1000;
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
avgT /= CONF::REPETITIONS;
Scheduler-test: optionally allow to propagate immediately This is just another (obvious) degree of freedom, which could be interesting to explore in stress testing, while probably not of much relevance in practice (if a job is expected to become runable earlier, in can as well be just scheduled earlier). Some experimentation shows that the timing measurements exhibit more fluctuations, but also slightly better times when pressure is low, which is pretty much what I'd expect. When raising pressure, the average times converge towards the same time range as observed with time bound propagation. Note that enabling this variation requires to wire a boolean switch over various layers of abstraction; arguably this is an unnecessary complexity and could be retracted once the »experimentation phase« is over. This completes the preparation of a Scheduler Stress-Test setup.
2024-01-08 22:58:16 +01:00
avgD = (avgT-expT); // can be < 0
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
for (uint i=0; i<CONF::REPETITIONS; ++i)
{
sdev += sqr (runTime[i] - avgT);
Scheduler-test: optionally allow to propagate immediately This is just another (obvious) degree of freedom, which could be interesting to explore in stress testing, while probably not of much relevance in practice (if a job is expected to become runable earlier, in can as well be just scheduled earlier). Some experimentation shows that the timing measurements exhibit more fluctuations, but also slightly better times when pressure is low, which is pretty much what I'd expect. When raising pressure, the average times converge towards the same time range as observed with time bound propagation. Note that enabling this variation requires to wire a boolean switch over various layers of abstraction; arguably this is an unnecessary complexity and could be retracted once the »experimentation phase« is over. This completes the preparation of a Scheduler Stress-Test setup.
2024-01-08 22:58:16 +01:00
double delta = (runTime[i] - expT);
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
bool fail = (delta > CONF::FAIL_LIMIT);
if (fail)
++ pf;
showRun(i, delta, runTime[i], runTime[i] > avgT, fail);
}
Scheduler-test: binary search working - Result found in typically 6-7 steps; - running 20 instead of 30 samples seems sufficient Breaking point in this example at stress-Factor 0.47 with run-time 39ms
2024-01-03 23:53:44 +01:00
pf /= CONF::REPETITIONS;
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
sdev = sqrt (sdev/CONF::REPETITIONS);
showStep(res);
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
return res;
}
/** criterion to decide if this test series constitutes a slipped schedule */
bool
decideBreakPoint (Res& res)
{
Scheduler-test: define criterion for breaking point This statistical criterion defines when to count observed Scheduler performance as loosing control. The test is comprised of three observations, which all must be confirmed: - an individual run counts as accidentally failed when the execution slips away by more than 2ms with respect to the defined overall schedule. When more than 55% of all observed runs are considered as failed, the first condition is met - moreover, the observed standard derivation must also surpass the same limit of > 2ms, which indicates that the Scheduling mechanism is under substantial strain (on average, the slip is ~ 200µs) - the third condition is that the ''averaged delta'' has surpassed 4ms, which is 2 times the basic failure indicator. These conditions are based on watching the Scheduler in operation; typically all three conditions slip away by large margin after a very narrow yet critical increase in the stress level. Using three conditions together should improve robustness; often the problems to keep up with the schedule build up over some parameter range, yet the actual decision should be based on complete loss of control.
2024-01-03 21:02:23 +01:00
return res.percentOff > CONF::TRIGGER_FAIL
and res.stdDev > CONF::TRIGGER_SDEV
and res.avgDelta > CONF::TRIGGER_DELTA;
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
}
/**
* invoke a binary search to produce a sequence of test series
* with the goal to narrow down the stressFact where the Schedule slips away.
*/
template<class FUN>
Res
Scheduler-test: simplify binary search implementation While the idea with capturing observation values is nice, it definitively does not belong into a library impl of the search algorithm, because this is usage specific and grossly complicates the invocation. Rather, observation data can be captured by side-effect from the probe-λ holding the actual measurement run.
2024-01-04 01:32:11 +01:00
conductBinarySearch (FUN&& runTestCase, vector<Res> const& results)
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
{
Scheduler-test: extract search algo into lib
2024-01-04 02:03:05 +01:00
double breakPoint = lib::binarySearch_upper (forward<FUN> (runTestCase)
, 0.0, CONF::UPPER_STRESS
, CONF::EPSILON);
Scheduler-test: binary search over continuous domain - textbook implementation - capture results from visited points - average results form the last three points to damp statistic fluctuations
2024-01-03 22:48:49 +01:00
uint s = results.size();
ENSURE (s >= 2);
Res res;
auto& [sf,pf,sdev,avgD,avgT,expT] = res;
// average data over the last three steps investigated for smoothing
uint points = min (results.size(), 3u);
for (uint i=results.size()-points; i<results.size(); ++i)
{
Scheduler-test: simplify binary search implementation While the idea with capturing observation values is nice, it definitively does not belong into a library impl of the search algorithm, because this is usage specific and grossly complicates the invocation. Rather, observation data can be captured by side-effect from the probe-λ holding the actual measurement run.
2024-01-04 01:32:11 +01:00
Res const& resx = results[i];
Scheduler-test: binary search over continuous domain - textbook implementation - capture results from visited points - average results form the last three points to damp statistic fluctuations
2024-01-03 22:48:49 +01:00
pf += resx.percentOff;
sdev += resx.stdDev;
avgD += resx.avgDelta;
avgT += resx.avgTime;
expT += resx.expTime;
}
pf /= points;
sdev /= points;
avgD /= points;
avgT /= points;
expT /= points;
Scheduler-test: simplify binary search implementation While the idea with capturing observation values is nice, it definitively does not belong into a library impl of the search algorithm, because this is usage specific and grossly complicates the invocation. Rather, observation data can be captured by side-effect from the probe-λ holding the actual measurement run.
2024-01-04 01:32:11 +01:00
sf = breakPoint;
Scheduler-test: binary search over continuous domain - textbook implementation - capture results from visited points - average results form the last three points to damp statistic fluctuations
2024-01-03 22:48:49 +01:00
return res;
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
}
Scheduler-test: define criterion for breaking point This statistical criterion defines when to count observed Scheduler performance as loosing control. The test is comprised of three observations, which all must be confirmed: - an individual run counts as accidentally failed when the execution slips away by more than 2ms with respect to the defined overall schedule. When more than 55% of all observed runs are considered as failed, the first condition is met - moreover, the observed standard derivation must also surpass the same limit of > 2ms, which indicates that the Scheduling mechanism is under substantial strain (on average, the slip is ~ 200µs) - the third condition is that the ''averaged delta'' has surpassed 4ms, which is 2 times the basic failure indicator. These conditions are based on watching the Scheduler in operation; typically all three conditions slip away by large margin after a very narrow yet critical increase in the stress level. Using three conditions together should improve robustness; often the problems to keep up with the schedule build up over some parameter range, yet the actual decision should be based on complete loss of control.
2024-01-03 21:02:23 +01:00
Scheduler-test: simplify binary search implementation While the idea with capturing observation values is nice, it definitively does not belong into a library impl of the search algorithm, because this is usage specific and grossly complicates the invocation. Rather, observation data can be captured by side-effect from the probe-λ holding the actual measurement run.
2024-01-04 01:32:11 +01:00
_Fmt fmtRun_ {"....·%-2d: Δ=%4.1f t=%4.1f %s %s"}; // i % Δ % t % t>avg? % fail?
Scheduler-test: more precise accounting for expected concurrency It turns out to be not correct using all the divergence in concurrency as a form factor, since it is quite common that not all cores can be active at every level, given the structural constraints as dictated by the load graph. On the other hand, if the empirical work (non wait-time) concurrency systematically differs from the simple model used for establishing the schedule, then this should indeed be considered a form factor and deduced from the effective stress factor, since it is not a reserve available for speed-up The solution entertained here is to derive an effective compounded sum of weights from the calculation used to build the schedule. This compounded weight sum is typically lower than the plain sum of all node weights, which is precisely due to the theoretical amount of expense reduction assumed in the schedule generation. So this gives us a handle at the theoretically expected expense and through the plain weight sum, we may draw conclusion about the effective concurrency expected in this schedule. Taking only this part as base for the empirical deviations yields search results very close to stressFactor ~1 -- implying that the test setup now observes what was intended to observe...
2024-02-19 17:36:46 +01:00
_Fmt fmtStep_{ "%4.2f| : ∅Δ=%4.1f±%-4.2f ∅t=%4.1f %s %%%-3.0f -- expect:%4.1fms"};// stress % ∅Δ % σ % ∅t % fail % pecentOff % t-expect
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
_Fmt fmtResSDv_{"%9s= %5.2f ±%4.2f%s"};
Scheduler-test: define criterion for breaking point This statistical criterion defines when to count observed Scheduler performance as loosing control. The test is comprised of three observations, which all must be confirmed: - an individual run counts as accidentally failed when the execution slips away by more than 2ms with respect to the defined overall schedule. When more than 55% of all observed runs are considered as failed, the first condition is met - moreover, the observed standard derivation must also surpass the same limit of > 2ms, which indicates that the Scheduling mechanism is under substantial strain (on average, the slip is ~ 200µs) - the third condition is that the ''averaged delta'' has surpassed 4ms, which is 2 times the basic failure indicator. These conditions are based on watching the Scheduler in operation; typically all three conditions slip away by large margin after a very narrow yet critical increase in the stress level. Using three conditions together should improve robustness; often the problems to keep up with the schedule build up over some parameter range, yet the actual decision should be based on complete loss of control.
2024-01-03 21:02:23 +01:00
_Fmt fmtResVal_{"%9s: %5.2f%s"};
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
void
showRun(uint i, double delta, double t, bool over, bool fail)
{
if (CONF::showRuns)
cout << fmtRun_ % i % delta % t % (over? "+":"-") % (fail? "●":"○")
<< endl;
}
void
showStep(Res& res)
{
if (CONF::showStep)
Scheduler-test: simplify binary search implementation While the idea with capturing observation values is nice, it definitively does not belong into a library impl of the search algorithm, because this is usage specific and grossly complicates the invocation. Rather, observation data can be captured by side-effect from the probe-λ holding the actual measurement run.
2024-01-04 01:32:11 +01:00
cout << fmtStep_ % res.stressFac % res.avgDelta % res.stdDev % res.avgTime
% (decideBreakPoint(res)? "—◆—":"—◇—")
Scheduler-test: more precise accounting for expected concurrency It turns out to be not correct using all the divergence in concurrency as a form factor, since it is quite common that not all cores can be active at every level, given the structural constraints as dictated by the load graph. On the other hand, if the empirical work (non wait-time) concurrency systematically differs from the simple model used for establishing the schedule, then this should indeed be considered a form factor and deduced from the effective stress factor, since it is not a reserve available for speed-up The solution entertained here is to derive an effective compounded sum of weights from the calculation used to build the schedule. This compounded weight sum is typically lower than the plain sum of all node weights, which is precisely due to the theoretical amount of expense reduction assumed in the schedule generation. So this gives us a handle at the theoretically expected expense and through the plain weight sum, we may draw conclusion about the effective concurrency expected in this schedule. Taking only this part as base for the empirical deviations yields search results very close to stressFactor ~1 -- implying that the test setup now observes what was intended to observe...
2024-02-19 17:36:46 +01:00
% (100*res.percentOff) % res.expTime
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
<< endl;
}
void
showRes(Res& res)
{
if (CONF::showRes)
{
cout << fmtResVal_ % "stresFac" % res.stressFac % "" <<endl;
cout << fmtResVal_ % "fail" %(res.percentOff * 100) % '%' <<endl;
Scheduler-test: binary search working - Result found in typically 6-7 steps; - running 20 instead of 30 samples seems sufficient Breaking point in this example at stress-Factor 0.47 with run-time 39ms
2024-01-03 23:53:44 +01:00
cout << fmtResSDv_ % "delta" % res.avgDelta % res.stdDev % "ms"<<endl;
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
cout << fmtResVal_ % "runTime" % res.avgTime % "ms"<<endl;
cout << fmtResVal_ % "expected" % res.expTime % "ms"<<endl;
}
}
void
Scheduler-test: binary search working - Result found in typically 6-7 steps; - running 20 instead of 30 samples seems sufficient Breaking point in this example at stress-Factor 0.47 with run-time 39ms
2024-01-03 23:53:44 +01:00
showRef(TestLoad& testLoad)
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
{
if (CONF::showRef)
cout << fmtResVal_ % "refTime"
% (testLoad.calcRuntimeReference(CONF::LOAD_BASE) /1000)
% "ms" << endl;
}
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
public:
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
/**
* Launch a measurement sequence to determine the »breaking point«
* for the configured test load and parametrisation of the Scheduler.
* @return a tuple `[stress-factor, ∅delta, ∅run-time]`
*/
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
auto
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
perform()
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
{
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
TRANSIENTLY(work::Config::COMPUTATION_CAPACITY) = CONF::CONCURRENCY;
TestLoad testLoad = CONF::testLoad().buildTopology();
TestSetup testSetup = CONF::testSetup (testLoad);
Scheduler-test: simplify binary search implementation While the idea with capturing observation values is nice, it definitively does not belong into a library impl of the search algorithm, because this is usage specific and grossly complicates the invocation. Rather, observation data can be captured by side-effect from the probe-λ holding the actual measurement run.
2024-01-04 01:32:11 +01:00
vector<Res> observations;
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
auto performEvaluation = [&](double stressFac)
{
configureTest (testSetup, stressFac);
Scheduler-test: incorporate statistics computation adapt the code written yesterday explicitly for the test case into the new framework for performing a stress-test run. Notable difference: times converted to millisecond immediately
2024-01-03 16:27:07 +01:00
auto res = runProbes (testSetup, stressFac);
Scheduler-test: simplify binary search implementation While the idea with capturing observation values is nice, it definitively does not belong into a library impl of the search algorithm, because this is usage specific and grossly complicates the invocation. Rather, observation data can be captured by side-effect from the probe-λ holding the actual measurement run.
2024-01-04 01:32:11 +01:00
observations.push_back (res);
return decideBreakPoint(res);
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
};
Scheduler-test: simplify binary search implementation While the idea with capturing observation values is nice, it definitively does not belong into a library impl of the search algorithm, because this is usage specific and grossly complicates the invocation. Rather, observation data can be captured by side-effect from the probe-λ holding the actual measurement run.
2024-01-04 01:32:11 +01:00
Res res = conductBinarySearch (move(performEvaluation), observations);
Scheduler-test: binary search working - Result found in typically 6-7 steps; - running 20 instead of 30 samples seems sufficient Breaking point in this example at stress-Factor 0.47 with run-time 39ms
2024-01-03 23:53:44 +01:00
showRes (res);
showRef (testLoad);
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
return make_tuple (res.stressFac, res.avgDelta, res.avgTime);
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
}
};
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
/**************************************************//**
* Specific test scheme to perform a Scheduler setup
* over a given control parameter range to determine
* correlations
*/
template<class CONF>
class ParameterRange
: public CONF
{
Scheduler-test: simple implementation of range coverage - fill the range randomly with probe points - use the node count as independent parameter - measurement method *works as intended* - results indeed show a linear relationship Results are ''interesting'' however, since the (par,time) points seem to be arranged into two lines, implying that about half of the runs were somehow ''degraded'' and performed way slower.
2024-02-24 04:17:05 +01:00
using TestLoad = decltype(declval<ParameterRange>().testLoad(1));
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
using TestSetup = decltype(declval<ParameterRange>().testSetup (declval<TestLoad&>()));
Scheduler-test: simple implementation of range coverage - fill the range randomly with probe points - use the node count as independent parameter - measurement method *works as intended* - results indeed show a linear relationship Results are ''interesting'' however, since the (par,time) points seem to be arranged into two lines, implying that about half of the runs were somehow ''degraded'' and performed way slower.
2024-02-24 04:17:05 +01:00
template<typename PAR>
using Point = std::pair<PAR, double>;
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
Scheduler-test: simple implementation of range coverage - fill the range randomly with probe points - use the node count as independent parameter - measurement method *works as intended* - results indeed show a linear relationship Results are ''interesting'' however, since the (par,time) points seem to be arranged into two lines, implying that about half of the runs were somehow ''degraded'' and performed way slower.
2024-02-24 04:17:05 +01:00
template<typename PAR>
void
runTest (Point<PAR>& point)
{
PAR param = point.first;
double stressFac = 1.0;
TestLoad testLoad = CONF::testLoad(param).buildTopology();
TestSetup testSetup = CONF::testSetup (testLoad)
.withLoadTimeBase(CONF::LOAD_BASE)
.withBaseExpense (CONF::BASE_EXPENSE)
.withSchedNotify (CONF::SCHED_NOTIFY)
.withSchedDepends(CONF::SCHED_DEPENDS)
Scheduler-test: attempt to find a viable Scheduler setup for this measurement - better use a Test-Chain-Load without any dependencies - schedule all at once - employ instrumentation - use the inner »overall time« as dependent result variable The timing results now show an almost perfect linear dependency. Also the inner overall time seems to omit the setup and tear-down time. But other observed values (notably the avgConcurrency) do not line up
2024-03-07 23:29:36 +01:00
.withAdaptedSchedule(stressFac, CONF::CONCURRENCY)
.withInstrumentation();
double testMillis = testSetup.launch_and_wait() / 1000;
auto stat = testSetup.getInvocationStatistic();
point.second = stat.coveredTime / 1000;
cout << "x="<<point.first<<"\t y="<<point.second<<"\t e2e="<<testMillis<<"\t conc:"<<stat.avgConcurrency<<" ∅t="<<stat.activeTime/stat.activationCnt<<" ("<<stat.activationCnt<<")"<<endl;
Scheduler-test: simple implementation of range coverage - fill the range randomly with probe points - use the node count as independent parameter - measurement method *works as intended* - results indeed show a linear relationship Results are ''interesting'' however, since the (par,time) points seem to be arranged into two lines, implying that about half of the runs were somehow ''degraded'' and performed way slower.
2024-02-24 04:17:05 +01:00
}
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
public:
/**
* Launch a measurement sequence running the Scheduler with a
* varying parameter value to investigate (x,y) correlations.
* @return ////TODO a tuple `[stress-factor, ∅delta, ∅run-time]`
*/
Scheduler-test: simple implementation of range coverage - fill the range randomly with probe points - use the node count as independent parameter - measurement method *works as intended* - results indeed show a linear relationship Results are ''interesting'' however, since the (par,time) points seem to be arranged into two lines, implying that about half of the runs were somehow ''degraded'' and performed way slower.
2024-02-24 04:17:05 +01:00
template<typename PAR>
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
auto
Scheduler-test: simple implementation of range coverage - fill the range randomly with probe points - use the node count as independent parameter - measurement method *works as intended* - results indeed show a linear relationship Results are ''interesting'' however, since the (par,time) points seem to be arranged into two lines, implying that about half of the runs were somehow ''degraded'' and performed way slower.
2024-02-24 04:17:05 +01:00
perform (PAR lower, PAR upper)
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
{
TRANSIENTLY(work::Config::COMPUTATION_CAPACITY) = CONF::CONCURRENCY;
Scheduler-test: simple implementation of range coverage - fill the range randomly with probe points - use the node count as independent parameter - measurement method *works as intended* - results indeed show a linear relationship Results are ''interesting'' however, since the (par,time) points seem to be arranged into two lines, implying that about half of the runs were somehow ''degraded'' and performed way slower.
2024-02-24 04:17:05 +01:00
PAR dist = upper - lower;
uint cnt = CONF::REPETITIONS;
vector<Point<PAR>> results(cnt);
PAR minP{upper}, maxP{lower};
for (uint i=0; i<cnt; ++i)
{
auto random = double(rand())/RAND_MAX;
PAR pos = lower + PAR(floor (random*dist + 0.5));
results[i].first = pos;
minP = min (pos, minP);
maxP = max (pos, maxP);
}
// ensure the bounds participate in test
if (maxP < upper) results[cnt-2].first = upper;
if (minP > lower) results[cnt-1].first = lower;
for (auto& point : results)
runTest (point);
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
Scheduler-test: simple implementation of range coverage - fill the range randomly with probe points - use the node count as independent parameter - measurement method *works as intended* - results indeed show a linear relationship Results are ''interesting'' however, since the (par,time) points seem to be arranged into two lines, implying that about half of the runs were somehow ''degraded'' and performed way slower.
2024-02-24 04:17:05 +01:00
return results;
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
}
};
//
}// namespace bench
}}}// namespace vault::gear::test
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
#endif /*VAULT_GEAR_TEST_STRESS_TEST_RIG_H*/
Reference in a new issue Copy permalink