ichthyo/LUMIERA.clone
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
LUMIERA.clone/tests/vault/gear/scheduler-stress-test.cpp

458 lines
19 KiB
C++
Raw Normal View History

Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
/*
SchedulerStress(Test) - verify scheduler performance characteristics
Copyright (C) Lumiera.org
2023, Hermann Vosseler <Ichthyostega@web.de>
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of
the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
* *****************************************************/
/** @file scheduler-usage-test.cpp
** unit test \ref SchedulerStress_test
*/
#include "lib/test/run.hpp"
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
#include "test-chain-load.hpp"
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
#include "stress-test-rig.hpp"
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
#include "vault/gear/scheduler.hpp"
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
#include "lib/time/timevalue.hpp"
Scheduler-test: document and verify weight adapted timing The helper developed thus far produces a sequence of weight factors per level, which could then be multiplied with an actual delay base time to produce a concrete schedule. These calculations, while simple, are difficult to understand; recommended to use the values tabulated in this test together with a `graphviz` rendering of the node graph (🠲 `printTopologyDOT()`)
2023-12-31 21:59:16 +01:00
#include "lib/format-string.hpp"
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
#include "lib/format-cout.hpp"
#include "lib/test/diagnostic-output.hpp"//////////////////////////TODO work in distress
//#include "lib/format-string.hpp"
Scheduler-test: experiment with varying stress levels
2024-01-02 15:45:40 +01:00
#include "lib/test/transiently.hpp"
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
//#include "lib/test/microbenchmark.hpp"
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
//#include "lib/util.hpp"
//#include <utility>
Scheduler-test: watch statistics with increasing stress - repeated invocations of the same test setup for statistics - the usual nasty 64-node graph with massive fork out - limit concurrency to 4 cores - tabulate data to look for clues regarding a trigger criteria Hypothesis: The Scheduler slips off schedule when all of the following three criteria are met: - more than 55% glitches with Δ > 2ms - σ > 2ms - ∅Δ > 4ms
2024-01-02 18:44:20 +01:00
//#include <vector>
#include <array>
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
using test::Test;
//using std::move;
//using util::isSameObject;
namespace vault{
namespace gear {
namespace test {
// using lib::time::FrameRate;
// using lib::time::Offset;
// using lib::time::Time;
Scheduler-test: document and verify weight adapted timing The helper developed thus far produces a sequence of weight factors per level, which could then be multiplied with an actual delay base time to produce a concrete schedule. These calculations, while simple, are difficult to understand; recommended to use the values tabulated in this test together with a `graphviz` rendering of the node graph (🠲 `printTopologyDOT()`)
2023-12-31 21:59:16 +01:00
using util::_Fmt;
Scheduler-test: watch statistics with increasing stress - repeated invocations of the same test setup for statistics - the usual nasty 64-node graph with massive fork out - limit concurrency to 4 cores - tabulate data to look for clues regarding a trigger criteria Hypothesis: The Scheduler slips off schedule when all of the following three criteria are met: - more than 55% glitches with Δ > 2ms - σ > 2ms - ∅Δ > 4ms
2024-01-02 18:44:20 +01:00
// using std::vector;
using std::array;
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
namespace { // Test definitions and setup...
}
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
/***************************************************************************//**
* @test Investigate and verify non-functional characteristics of the Scheduler.
* @see SchedulerActivity_test
* @see SchedulerInvocation_test
* @see SchedulerCommutator_test
Scheduler-test: optionally allow to propagate immediately This is just another (obvious) degree of freedom, which could be interesting to explore in stress testing, while probably not of much relevance in practice (if a job is expected to become runable earlier, in can as well be just scheduled earlier). Some experimentation shows that the timing measurements exhibit more fluctuations, but also slightly better times when pressure is low, which is pretty much what I'd expect. When raising pressure, the average times converge towards the same time range as observed with time bound propagation. Note that enabling this variation requires to wire a boolean switch over various layers of abstraction; arguably this is an unnecessary complexity and could be retracted once the »experimentation phase« is over. This completes the preparation of a Scheduler Stress-Test setup.
2024-01-08 22:58:16 +01:00
* @see stress-test-rig.hpp
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
*/
class SchedulerStress_test : public Test
{
virtual void
run (Arg)
{
Scheduler-test: draft calculation of level-weight based schedule ...the idea is to use the sum of node weights per level to create a schedule, which more closely reflects the distribution of actual computation time. Hopefully such a schedule can then be squeezed or stretched by a time factor to find out a ''breaking point'', at which the Scheduler is no longer able to keep up.
2023-12-28 23:56:52 +01:00
//smokeTest();
Scheduler-test: setup in Stress-Test-Rig ...use the scheme established thus far
2024-01-10 20:39:20 +01:00
// setup_systematicSchedule();
Scheduler-test: run breaking-point search with gradual adaptation Integrating the changed logic into the StressTest-rig, with bugfixes
2024-02-18 23:02:32 +01:00
search_breaking_point();
// verify_instrumentation();
Scheduler-test: integrate instrumentation as optional feature ...can be activated on the Test-Chain-Load ...add a test case to validate its operation
2024-02-15 00:43:03 +01:00
// investigateWorkProcessing();
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
walkingDeadline();
}
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
/** @test TODO demonstrate sustained operation under load
Scheduler-test: draft calculation of level-weight based schedule ...the idea is to use the sum of node weights per level to create a schedule, which more closely reflects the distribution of actual computation time. Hopefully such a schedule can then be squeezed or stretched by a time factor to find out a ''breaking point'', at which the Scheduler is no longer able to keep up.
2023-12-28 23:56:52 +01:00
* - TODO this is a placeholder and works now, but need a better example
* - it should not produce so much overload, rather some stretch of steady-state processing
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
* @todo WIP 12/23 🔁 define ⟶ implement
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
*/
void
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
smokeTest()
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
{
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
MARK_TEST_FUN
Scheduler-test: press harder with long and massive graph ...observing multiple failures, which seem to be interconnected - the test-setup with the planning chunk pre-roll is insufficient - basically it is not possible to perform further concurrent planning, without getting behind the actual schedule; at least in the setup with DUMP print statements (which slowdown everything) - muliple chained re-entrant calls into the planning function can result - the **ASSERTION in the Allocator** was triggered again - the log+stacktrace indicate that there **is still a Gap** in the logic to protect the allocations via Grooming-Token
2023-12-21 23:26:01 +01:00
TestChainLoad testLoad{512};
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
testLoad.configureShape_chain_loadBursts()
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
.buildTopology()
Scheduler-test: press harder with long and massive graph ...observing multiple failures, which seem to be interconnected - the test-setup with the planning chunk pre-roll is insufficient - basically it is not possible to perform further concurrent planning, without getting behind the actual schedule; at least in the setup with DUMP print statements (which slowdown everything) - muliple chained re-entrant calls into the planning function can result - the **ASSERTION in the Allocator** was triggered again - the log+stacktrace indicate that there **is still a Gap** in the logic to protect the allocations via Grooming-Token
2023-12-21 23:26:01 +01:00
// .printTopologyDOT()
;
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
auto stats = testLoad.computeGraphStatistics();
cout << _Fmt{"Test-Load: Nodes: %d Levels: %d ∅Node/Level: %3.1f Forks: %d Joins: %d"}
% stats.nodes
% stats.levels
% stats.indicators[STAT_NODE].pL
% stats.indicators[STAT_FORK].cnt
% stats.indicators[STAT_JOIN].cnt
<< endl;
// while building the calculation-plan graph
// node hashes were computed, observing dependencies
size_t expectedHash = testLoad.getHash();
// some jobs/nodes are marked with a weight-step
// these can be instructed to spend some CPU time
auto LOAD_BASE = 500us;
testLoad.performGraphSynchronously(LOAD_BASE);
CHECK (testLoad.getHash() == expectedHash);
double referenceTime = testLoad.calcRuntimeReference(LOAD_BASE);
cout << "refTime(singleThr): "<<referenceTime/1000<<"ms"<<endl;
// Perform through Scheduler----------
BlockFlowAlloc bFlow;
EngineObserver watch;
Scheduler scheduler{bFlow, watch};
double performanceTime =
testLoad.setupSchedule(scheduler)
.withLoadTimeBase(LOAD_BASE)
Scheduler-test: adapt tests to changed logic at entrance - now there can not be any direct dispatch anymore when entering events - thus there is no decision logic at entrance anymore - rather the work-function implementation moved down into Layer-2 - so add a unit-test like coverage there (integration in SchedulerService_test)
2023-12-26 15:00:35 +01:00
.withJobDeadline(150ms)
Scheduler-test: investigate planning failure - fix mistake in schdule time for planning chunks (must use start, not end of chunk) - allow to configure the heuristics for pre-roll (time reserved for planning a node)
2023-12-23 19:33:55 +01:00
.withPlanningStep(200us)
Scheduler-test: press harder with long and massive graph ...observing multiple failures, which seem to be interconnected - the test-setup with the planning chunk pre-roll is insufficient - basically it is not possible to perform further concurrent planning, without getting behind the actual schedule; at least in the setup with DUMP print statements (which slowdown everything) - muliple chained re-entrant calls into the planning function can result - the **ASSERTION in the Allocator** was triggered again - the log+stacktrace indicate that there **is still a Gap** in the logic to protect the allocations via Grooming-Token
2023-12-21 23:26:01 +01:00
.withChunkSize(20)
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
.launch_and_wait();
cout << "runTime(Scheduler): "<<performanceTime/1000<<"ms"<<endl;
// invocation through Scheduler has reproduced all node hashes
CHECK (testLoad.getHash() == expectedHash);
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
}
Scheduler-test: optionally allow to propagate immediately This is just another (obvious) degree of freedom, which could be interesting to explore in stress testing, while probably not of much relevance in practice (if a job is expected to become runable earlier, in can as well be just scheduled earlier). Some experimentation shows that the timing measurements exhibit more fluctuations, but also slightly better times when pressure is low, which is pretty much what I'd expect. When raising pressure, the average times converge towards the same time range as observed with time bound propagation. Note that enabling this variation requires to wire a boolean switch over various layers of abstraction; arguably this is an unnecessary complexity and could be retracted once the »experimentation phase« is over. This completes the preparation of a Scheduler Stress-Test setup.
2024-01-08 22:58:16 +01:00
/** @test build a scheme to adapt the schedule to expected runtime.
* - as in many other tests, use the massively forking load pattern
* - demonstrate how TestChainLoad computes an idealised level expense
* - verify how schedule times are derived from this expense sequence
* @todo WIP 12/23 ✔ define ⟶ ✔ implement
Scheduler-test: draft calculation of level-weight based schedule ...the idea is to use the sum of node weights per level to create a schedule, which more closely reflects the distribution of actual computation time. Hopefully such a schedule can then be squeezed or stretched by a time factor to find out a ''breaking point'', at which the Scheduler is no longer able to keep up.
2023-12-28 23:56:52 +01:00
*/
void
setup_systematicSchedule()
{
TestChainLoad testLoad{64};
testLoad.configureShape_chain_loadBursts()
Scheduler-test: build configurable measurement setup Elaborate the draft to include all the elements used directly in the test case thus far; the goal is to introduce some structuring and leave room for flexible confguration, while implementing the actual binary search as library function over Lambdas. My expectation is to write a series of individual test instances with varying parameters; while it seems possible to add further performance test variations into that scheme later on.
2024-01-02 23:51:47 +01:00
.buildTopology()
Scheduler-test: verify adapted schedule with stress-factor - schedule can now be adapted to concurrency and expected distribution of runtimes - additional stress factor to press the schedule (1.0 is nominal speed) - observed run-time now without Scheduler start-up and pre-roll - document and verify computed numbers
2024-01-01 23:59:02 +01:00
// .printTopologyDOT()
// .printTopologyStatistics()
Scheduler-test: draft calculation of level-weight based schedule ...the idea is to use the sum of node weights per level to create a schedule, which more closely reflects the distribution of actual computation time. Hopefully such a schedule can then be squeezed or stretched by a time factor to find out a ''breaking point'', at which the Scheduler is no longer able to keep up.
2023-12-28 23:56:52 +01:00
;
auto LOAD_BASE = 500us;
ComputationalLoad cpuLoad;
cpuLoad.timeBase = LOAD_BASE;
cpuLoad.calibrate();
double micros = cpuLoad.invoke();
CHECK (micros < 550);
CHECK (micros > 450);
Scheduler-test: change test setup to use a schedule table ...up to now, we've relied on a regular schedule governed solely by the progression of node levels, with a fixed level speed defaulting to 1ms per level. But in preparation of stress tesging, we need a schedule adapted to the expected distribution of computation times, otherwise we'll not be able to factor out the actual computation graph connectivity. The goal is to establish a distinctive **breaking point** when the scheduler is unable to cope with the provided schedule.
2024-01-01 20:06:53 +01:00
// build a schedule sequence based on
// summing up weight factors, with example concurrency ≔ 4
uint concurrency = 4;
Scheduler-test: verify adapted schedule with stress-factor - schedule can now be adapted to concurrency and expected distribution of runtimes - additional stress factor to press the schedule (1.0 is nominal speed) - observed run-time now without Scheduler start-up and pre-roll - document and verify computed numbers
2024-01-01 23:59:02 +01:00
auto stepFactors = testLoad.levelScheduleSequence(concurrency).effuse();
CHECK (stepFactors.size() == 1+testLoad.topLevel());
CHECK (stepFactors.size() == 27);
Scheduler-test: document and verify weight adapted timing The helper developed thus far produces a sequence of weight factors per level, which could then be multiplied with an actual delay base time to produce a concrete schedule. These calculations, while simple, are difficult to understand; recommended to use the values tabulated in this test together with a `graphviz` rendering of the node graph (🠲 `printTopologyDOT()`)
2023-12-31 21:59:16 +01:00
Scheduler-test: change test setup to use a schedule table ...up to now, we've relied on a regular schedule governed solely by the progression of node levels, with a fixed level speed defaulting to 1ms per level. But in preparation of stress tesging, we need a schedule adapted to the expected distribution of computation times, otherwise we'll not be able to factor out the actual computation graph connectivity. The goal is to establish a distinctive **breaking point** when the scheduler is unable to cope with the provided schedule.
2024-01-01 20:06:53 +01:00
// Build-Performance-test-setup--------
BlockFlowAlloc bFlow;
EngineObserver watch;
Scheduler scheduler{bFlow, watch};
Scheduler-test: document and verify weight adapted timing The helper developed thus far produces a sequence of weight factors per level, which could then be multiplied with an actual delay base time to produce a concrete schedule. These calculations, while simple, are difficult to understand; recommended to use the values tabulated in this test together with a `graphviz` rendering of the node graph (🠲 `printTopologyDOT()`)
2023-12-31 21:59:16 +01:00
Scheduler-test: change test setup to use a schedule table ...up to now, we've relied on a regular schedule governed solely by the progression of node levels, with a fixed level speed defaulting to 1ms per level. But in preparation of stress tesging, we need a schedule adapted to the expected distribution of computation times, otherwise we'll not be able to factor out the actual computation graph connectivity. The goal is to establish a distinctive **breaking point** when the scheduler is unable to cope with the provided schedule.
2024-01-01 20:06:53 +01:00
auto testSetup =
testLoad.setupSchedule(scheduler)
.withLoadTimeBase(LOAD_BASE)
Scheduler-test: verify adapted schedule with stress-factor - schedule can now be adapted to concurrency and expected distribution of runtimes - additional stress factor to press the schedule (1.0 is nominal speed) - observed run-time now without Scheduler start-up and pre-roll - document and verify computed numbers
2024-01-01 23:59:02 +01:00
.withJobDeadline(50ms)
Scheduler-test: change test setup to use a schedule table ...up to now, we've relied on a regular schedule governed solely by the progression of node levels, with a fixed level speed defaulting to 1ms per level. But in preparation of stress tesging, we need a schedule adapted to the expected distribution of computation times, otherwise we'll not be able to factor out the actual computation graph connectivity. The goal is to establish a distinctive **breaking point** when the scheduler is unable to cope with the provided schedule.
2024-01-01 20:06:53 +01:00
.withUpfrontPlanning();
Scheduler-test: document and verify weight adapted timing The helper developed thus far produces a sequence of weight factors per level, which could then be multiplied with an actual delay base time to produce a concrete schedule. These calculations, while simple, are difficult to understand; recommended to use the values tabulated in this test together with a `graphviz` rendering of the node graph (🠲 `printTopologyDOT()`)
2023-12-31 21:59:16 +01:00
Scheduler-test: change test setup to use a schedule table ...up to now, we've relied on a regular schedule governed solely by the progression of node levels, with a fixed level speed defaulting to 1ms per level. But in preparation of stress tesging, we need a schedule adapted to the expected distribution of computation times, otherwise we'll not be able to factor out the actual computation graph connectivity. The goal is to establish a distinctive **breaking point** when the scheduler is unable to cope with the provided schedule.
2024-01-01 20:06:53 +01:00
auto schedule = testSetup.getScheduleSeq().effuse();
CHECK (schedule.size() == testLoad.topLevel() + 2);
CHECK (schedule[ 0] == _uTicks(0ms));
CHECK (schedule[ 1] == _uTicks(1ms));
Scheduler-test: verify adapted schedule with stress-factor - schedule can now be adapted to concurrency and expected distribution of runtimes - additional stress factor to press the schedule (1.0 is nominal speed) - observed run-time now without Scheduler start-up and pre-roll - document and verify computed numbers
2024-01-01 23:59:02 +01:00
CHECK (schedule[ 2] == _uTicks(2ms));
// ....
CHECK (schedule[25] == _uTicks(25ms));
Scheduler-test: change test setup to use a schedule table ...up to now, we've relied on a regular schedule governed solely by the progression of node levels, with a fixed level speed defaulting to 1ms per level. But in preparation of stress tesging, we need a schedule adapted to the expected distribution of computation times, otherwise we'll not be able to factor out the actual computation graph connectivity. The goal is to establish a distinctive **breaking point** when the scheduler is unable to cope with the provided schedule.
2024-01-01 20:06:53 +01:00
CHECK (schedule[26] == _uTicks(26ms));
CHECK (schedule[27] == _uTicks(27ms));
Scheduler-test: document and verify weight adapted timing The helper developed thus far produces a sequence of weight factors per level, which could then be multiplied with an actual delay base time to produce a concrete schedule. These calculations, while simple, are difficult to understand; recommended to use the values tabulated in this test together with a `graphviz` rendering of the node graph (🠲 `printTopologyDOT()`)
2023-12-31 21:59:16 +01:00
Scheduler-test: verify adapted schedule with stress-factor - schedule can now be adapted to concurrency and expected distribution of runtimes - additional stress factor to press the schedule (1.0 is nominal speed) - observed run-time now without Scheduler start-up and pre-roll - document and verify computed numbers
2024-01-01 23:59:02 +01:00
// Adapted Schedule----------
Scheduler-test: build adapted schedule ...based on the adapted time-factor sequence implemented yesterday in TestChainLoad itself - in this case, the TimeBase from the computation load is used as level speed - this »base beat« is then modulated by the timing factor sequence - working in an additional stress factor to press the schedule uniformly - actual start time will be added as offset once the actual test commences
2024-01-01 22:42:51 +01:00
double stressFac = 1.0;
testSetup.withAdaptedSchedule (stressFac, concurrency);
schedule = testSetup.getScheduleSeq().effuse();
CHECK (schedule.size() == testLoad.topLevel() + 2);
Scheduler-test: verify adapted schedule with stress-factor - schedule can now be adapted to concurrency and expected distribution of runtimes - additional stress factor to press the schedule (1.0 is nominal speed) - observed run-time now without Scheduler start-up and pre-roll - document and verify computed numbers
2024-01-01 23:59:02 +01:00
CHECK (schedule[ 0] == _uTicks(0ms));
CHECK (schedule[ 1] == _uTicks(0ms));
// verify the numbers in detail....
_Fmt stepFmt{"lev:%-2d stepFac:%-6.3f schedule:%6.3f"};
auto stepStr = [&](uint i){ return string{stepFmt % i % stepFactors[i>0?i-1:0] % (_raw(schedule[i])/1000.0)}; };
CHECK (stepStr( 0) == "lev:0 stepFac:0.000 schedule: 0.000"_expect);
CHECK (stepStr( 1) == "lev:1 stepFac:0.000 schedule: 0.000"_expect);
CHECK (stepStr( 2) == "lev:2 stepFac:0.000 schedule: 0.000"_expect);
CHECK (stepStr( 3) == "lev:3 stepFac:2.000 schedule: 1.000"_expect);
CHECK (stepStr( 4) == "lev:4 stepFac:2.000 schedule: 1.000"_expect);
CHECK (stepStr( 5) == "lev:5 stepFac:2.000 schedule: 1.000"_expect);
CHECK (stepStr( 6) == "lev:6 stepFac:2.000 schedule: 1.000"_expect);
CHECK (stepStr( 7) == "lev:7 stepFac:3.000 schedule: 1.500"_expect);
CHECK (stepStr( 8) == "lev:8 stepFac:5.000 schedule: 2.500"_expect);
CHECK (stepStr( 9) == "lev:9 stepFac:7.000 schedule: 3.500"_expect);
CHECK (stepStr(10) == "lev:10 stepFac:8.000 schedule: 4.000"_expect);
CHECK (stepStr(11) == "lev:11 stepFac:8.000 schedule: 4.000"_expect);
CHECK (stepStr(12) == "lev:12 stepFac:8.000 schedule: 4.000"_expect);
CHECK (stepStr(13) == "lev:13 stepFac:9.000 schedule: 4.500"_expect);
CHECK (stepStr(14) == "lev:14 stepFac:10.000 schedule: 5.000"_expect);
CHECK (stepStr(15) == "lev:15 stepFac:12.000 schedule: 6.000"_expect);
CHECK (stepStr(16) == "lev:16 stepFac:12.000 schedule: 6.000"_expect);
CHECK (stepStr(17) == "lev:17 stepFac:13.000 schedule: 6.500"_expect);
CHECK (stepStr(18) == "lev:18 stepFac:16.000 schedule: 8.000"_expect);
CHECK (stepStr(19) == "lev:19 stepFac:16.000 schedule: 8.000"_expect);
CHECK (stepStr(20) == "lev:20 stepFac:20.000 schedule:10.000"_expect);
CHECK (stepStr(21) == "lev:21 stepFac:22.500 schedule:11.250"_expect);
CHECK (stepStr(22) == "lev:22 stepFac:24.167 schedule:12.083"_expect);
CHECK (stepStr(23) == "lev:23 stepFac:26.167 schedule:13.083"_expect);
CHECK (stepStr(24) == "lev:24 stepFac:28.167 schedule:14.083"_expect);
CHECK (stepStr(25) == "lev:25 stepFac:30.867 schedule:15.433"_expect);
CHECK (stepStr(26) == "lev:26 stepFac:31.867 schedule:15.933"_expect);
CHECK (stepStr(27) == "lev:27 stepFac:32.867 schedule:16.433"_expect);
// Adapted Schedule with lower stress level and higher concurrency....
stressFac = 0.3;
concurrency = 6;
stepFactors = testLoad.levelScheduleSequence(concurrency).effuse();
testSetup.withAdaptedSchedule (stressFac, concurrency);
schedule = testSetup.getScheduleSeq().effuse();
CHECK (stepStr( 0) == "lev:0 stepFac:0.000 schedule: 0.000"_expect);
CHECK (stepStr( 1) == "lev:1 stepFac:0.000 schedule: 0.000"_expect);
CHECK (stepStr( 2) == "lev:2 stepFac:0.000 schedule: 0.000"_expect);
CHECK (stepStr( 3) == "lev:3 stepFac:2.000 schedule: 3.333"_expect);
CHECK (stepStr( 4) == "lev:4 stepFac:2.000 schedule: 3.333"_expect);
CHECK (stepStr( 5) == "lev:5 stepFac:2.000 schedule: 3.333"_expect);
CHECK (stepStr( 6) == "lev:6 stepFac:2.000 schedule: 3.333"_expect);
CHECK (stepStr( 7) == "lev:7 stepFac:3.000 schedule: 5.000"_expect);
CHECK (stepStr( 8) == "lev:8 stepFac:5.000 schedule: 8.333"_expect);
CHECK (stepStr( 9) == "lev:9 stepFac:7.000 schedule:11.666"_expect);
CHECK (stepStr(10) == "lev:10 stepFac:8.000 schedule:13.333"_expect);
CHECK (stepStr(11) == "lev:11 stepFac:8.000 schedule:13.333"_expect);
CHECK (stepStr(12) == "lev:12 stepFac:8.000 schedule:13.333"_expect);
CHECK (stepStr(13) == "lev:13 stepFac:9.000 schedule:15.000"_expect);
CHECK (stepStr(14) == "lev:14 stepFac:10.000 schedule:16.666"_expect);
CHECK (stepStr(15) == "lev:15 stepFac:12.000 schedule:20.000"_expect);
CHECK (stepStr(16) == "lev:16 stepFac:12.000 schedule:20.000"_expect);
CHECK (stepStr(17) == "lev:17 stepFac:13.000 schedule:21.666"_expect);
CHECK (stepStr(18) == "lev:18 stepFac:16.000 schedule:26.666"_expect);
CHECK (stepStr(19) == "lev:19 stepFac:16.000 schedule:26.666"_expect);
CHECK (stepStr(20) == "lev:20 stepFac:18.000 schedule:30.000"_expect); // note: here the higher concurrency allows to process all 5 concurrent nodes at once
CHECK (stepStr(21) == "lev:21 stepFac:20.500 schedule:34.166"_expect);
CHECK (stepStr(22) == "lev:22 stepFac:22.167 schedule:36.944"_expect);
CHECK (stepStr(23) == "lev:23 stepFac:23.167 schedule:38.611"_expect);
CHECK (stepStr(24) == "lev:24 stepFac:24.167 schedule:40.277"_expect);
CHECK (stepStr(25) == "lev:25 stepFac:25.967 schedule:43.277"_expect);
CHECK (stepStr(26) == "lev:26 stepFac:26.967 schedule:44.944"_expect);
CHECK (stepStr(27) == "lev:27 stepFac:27.967 schedule:46.611"_expect);
Scheduler-test: experiment with varying stress levels
2024-01-02 15:45:40 +01:00
// perform a Test with this low stress level (0.3)
double runTime = testSetup.launch_and_wait();
double expected = testSetup.getExpectedEndTime();
CHECK (fabs (runTime-expected) < 5000);
} // Scheduler should able to follow the expected schedule
Scheduler-test: document new instrumentation facility with simple test ...turns out rather challenging to come up with any test case, that is both meaningful, simple to setup and understand, yet still produces somewhat stable values. `IncidenceCount` seems most valuable for investigation and direct inspection of results
2024-02-17 21:55:21 +01:00
/** @test verify capability for instrumentation of job invocations
* @see IncidenceCount_test
* @todo WIP 2/24 ✔ define ⟶ ✔ implement
Scheduler-test: integrate instrumentation as optional feature ...can be activated on the Test-Chain-Load ...add a test case to validate its operation
2024-02-15 00:43:03 +01:00
*/
void
verify_instrumentation()
{
Scheduler-test: document new instrumentation facility with simple test ...turns out rather challenging to come up with any test case, that is both meaningful, simple to setup and understand, yet still produces somewhat stable values. `IncidenceCount` seems most valuable for investigation and direct inspection of results
2024-02-17 21:55:21 +01:00
const size_t NODES = 20;
const size_t CORES = work::Config::COMPUTATION_CAPACITY;
auto LOAD_BASE = 5ms;
TestChainLoad testLoad{NODES};
Scheduler-test: integrate instrumentation as optional feature ...can be activated on the Test-Chain-Load ...add a test case to validate its operation
2024-02-15 00:43:03 +01:00
BlockFlowAlloc bFlow;
EngineObserver watch;
Scheduler scheduler{bFlow, watch};
auto testSetup =
Scheduler-test: document new instrumentation facility with simple test ...turns out rather challenging to come up with any test case, that is both meaningful, simple to setup and understand, yet still produces somewhat stable values. `IncidenceCount` seems most valuable for investigation and direct inspection of results
2024-02-17 21:55:21 +01:00
testLoad.setWeight(1)
.setupSchedule(scheduler)
.withLoadTimeBase(LOAD_BASE)
.withJobDeadline(50ms)
.withInstrumentation() // activate an instrumentation bracket around each job invocation
;
Scheduler-test: integrate instrumentation as optional feature ...can be activated on the Test-Chain-Load ...add a test case to validate its operation
2024-02-15 00:43:03 +01:00
double runTime = testSetup.launch_and_wait();
Scheduler-test: document new instrumentation facility with simple test ...turns out rather challenging to come up with any test case, that is both meaningful, simple to setup and understand, yet still produces somewhat stable values. `IncidenceCount` seems most valuable for investigation and direct inspection of results
2024-02-17 21:55:21 +01:00
auto stat = testSetup.getInvocationStatistic(); // retrieve observed invocation statistics
CHECK (runTime < stat.activeTime);
CHECK (isLimited (4900, stat.activeTime/NODES, 8000)); // should be close to 5000
CHECK (stat.coveredTime < runTime);
CHECK (NODES == stat.activationCnt); // each node activated once
CHECK (isLimited (CORES/2, stat.avgConcurrency, CORES)); // should ideally come close to hardware concurrency
CHECK (0 == stat.timeAtConc(0));
CHECK (0 == stat.timeAtConc(CORES+1));
CHECK (runTime/2 < stat.timeAtConc(CORES-1)+stat.timeAtConc(CORES));
Scheduler-test: introduce a form-factor to account for empiric adaptation Relying on the new instrumentation facility, the actually effective concurrency and cumulative run time of the test jobs can be established. These can now be cast into a form-factor to represent actual excess expenses in relation to the theoretical model. By allowing to adjust the adapted schedule by this form factor, it can be made to reflect more closely the actual empiric load, hopefully leading to a more realistic effect of the stress-factor and thus results better suited to conclude on generic behaviour.
2024-02-18 18:01:21 +01:00
} // should ideally spend most of the time at highest concurrency levels
Scheduler-test: integrate instrumentation as optional feature ...can be activated on the Test-Chain-Load ...add a test case to validate its operation
2024-02-15 00:43:03 +01:00
Scheduler-test: experiment with varying stress levels
2024-01-02 15:45:40 +01:00
/** @test TODO determine the breaking point towards scheduler overload
Scheduler-test: optionally allow to propagate immediately This is just another (obvious) degree of freedom, which could be interesting to explore in stress testing, while probably not of much relevance in practice (if a job is expected to become runable earlier, in can as well be just scheduled earlier). Some experimentation shows that the timing measurements exhibit more fluctuations, but also slightly better times when pressure is low, which is pretty much what I'd expect. When raising pressure, the average times converge towards the same time range as observed with time bound propagation. Note that enabling this variation requires to wire a boolean switch over various layers of abstraction; arguably this is an unnecessary complexity and could be retracted once the »experimentation phase« is over. This completes the preparation of a Scheduler Stress-Test setup.
2024-01-08 22:58:16 +01:00
* - use the integrated StressRig
* - demonstrate how parameters can be tweaked
* - perform a run, leading to a binary search for the breaking point
* @note on my machine, I observe stress factors close below 0.5, due to the fact
* that the ComputationalLoad typically takes 2 times as long in concurrent
* usage compared to its calibration, which is done in a tight loop. This
* is strange and may well be due to some peculiarity of my system. Which
* also implies that this test's behaviour might be difficult to verify,
* other than by qualitative interpretation of the log output on STDOUT.
* @see stress-test-rig.hpp
* @todo WIP 1/24 ✔ define ⟶ ✔ implement
Scheduler-test: experiment with varying stress levels
2024-01-02 15:45:40 +01:00
*/
void
search_breaking_point()
{
MARK_TEST_FUN
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
struct Setup : StressRig
{
usec LOAD_BASE = 500us;
uint CONCURRENCY = 4;
Scheduler-test: binary search working - Result found in typically 6-7 steps; - running 20 instead of 30 samples seems sufficient Breaking point in this example at stress-Factor 0.47 with run-time 39ms
2024-01-03 23:53:44 +01:00
bool showRuns = true;
Scheduler-test: draft a structure to formalise these investigations - the goal is to run a binary search - the search condition should be factored out - thus some kind of framework or DSL is required, to separate the technicalities of the measurement from the specifics of the actual test case.
2024-01-02 21:46:44 +01:00
auto testLoad() { return TestChainLoad<>{64}.configureShape_chain_loadBursts(); }
};
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
auto [stress,delta,time] = StressRig::with<Setup>()
.perform<bench::BreakingPoint>();
Scheduler-test: more precise accounting for expected concurrency It turns out to be not correct using all the divergence in concurrency as a form factor, since it is quite common that not all cores can be active at every level, given the structural constraints as dictated by the load graph. On the other hand, if the empirical work (non wait-time) concurrency systematically differs from the simple model used for establishing the schedule, then this should indeed be considered a form factor and deduced from the effective stress factor, since it is not a reserve available for speed-up The solution entertained here is to derive an effective compounded sum of weights from the calculation used to build the schedule. This compounded weight sum is typically lower than the plain sum of all node weights, which is precisely due to the theoretical amount of expense reduction assumed in the schedule generation. So this gives us a handle at the theoretically expected expense and through the plain weight sum, we may draw conclusion about the effective concurrency expected in this schedule. Taking only this part as base for the empirical deviations yields search results very close to stressFactor ~1 -- implying that the test setup now observes what was intended to observe...
2024-02-19 17:36:46 +01:00
CHECK (delta > 2.5);
CHECK (1.15 > stress and stress > 0.9);
Scheduler-test: draft calculation of level-weight based schedule ...the idea is to use the sum of node weights per level to create a schedule, which more closely reflects the distribution of actual computation time. Hopefully such a schedule can then be squeezed or stretched by a time factor to find out a ''breaking point'', at which the Scheduler is no longer able to keep up.
2023-12-28 23:56:52 +01:00
}
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
/** @test TODO
Scheduler-test: experiment with varying stress levels
2024-01-02 15:45:40 +01:00
* @todo WIP 1/24 🔁 define ⟶ implement
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
*/
void
Scheduler-test: setup in Stress-Test-Rig ...use the scheme established thus far
2024-01-10 20:39:20 +01:00
investigateWorkProcessing()
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
{
Scheduler-test: setup in Stress-Test-Rig ...use the scheme established thus far
2024-01-10 20:39:20 +01:00
MARK_TEST_FUN
Scheduler-test: continue investigation - combine base expense with stress factor After an extended break due to "real life issues".... Pick up the investigation, with the goal to ascertain a valid definition and understanding of all test parameters. A first step is to establish a baseline ''without using a computational load''; this might be some kind of base overhead of the scheduler. However -- the way the test scaffolding was built, it is difficult to create a feedback loop for the statistical test setup with binary search, since it is not really clear how the single control parameter of the test algorithm, the so called "stress factor", shall be interpreted and how it can be combined with a base load. An extended series of tests, while watching the observed value patterns qualitatively, seems to corroborate the former results, indicating that the base expense in my test setup (using a debug build) is at ~200µs / Node / core. Yet the difficulty to interpret this result and arrive at a logical and generic model prevents me from translating this into a measurement scheme, which can be executed independently from a specific test setup and hardware
2024-02-11 03:53:42 +01:00
TestChainLoad<8> testLoad{256};
Scheduler-test: fix Segfault in test setup ...as it turned out, this segfault was caused by flaws in the ScheduleCtx used for generate the test-schedule; especially when all node-spreads are set to zero and thus all jobs are scheduled immediately at t=0, there was a loophole in the logic to set the dependencies for the final »wake-up« job. When running such a schedule in the Stress-Test-Bench, the next measurement run could be started due to a premature wake-up job, thereby overrunning the previous test-run, which could be still in the middle of computations. So this was not a bug in the Scheduler itself, yet something to take care of later when programming the actual Job-Planning and schedule generation.
2024-01-11 22:03:36 +01:00
testLoad.seedingRule(testLoad.rule().probability(0.6).minVal(2))
.pruningRule(testLoad.rule().probability(0.44))
.setSeed(55)
.buildTopology()
Scheduler-test: investigate behaviour of a load for stress testing The goal is to devise a load more akin to the expected real-world processing patterns, and then to increase the density to establish a breaking point. Preliminary investigations focus on establishing the properties of this load and to ensure the actual computation load behaves as expected. Using the third Graph pattern prepared thus far, which produces short chains of length=2, yet immediately spread out to maximum concurrency. This leads to 5.8 Nodes / Level on average.
2024-01-23 18:29:09 +01:00
// .printTopologyDOT()
// .printTopologyStatistics()
Scheduler-test: fix Segfault in test setup ...as it turned out, this segfault was caused by flaws in the ScheduleCtx used for generate the test-schedule; especially when all node-spreads are set to zero and thus all jobs are scheduled immediately at t=0, there was a loophole in the logic to set the dependencies for the final »wake-up« job. When running such a schedule in the Stress-Test-Bench, the next measurement run could be started due to a premature wake-up job, thereby overrunning the previous test-run, which could be still in the middle of computations. So this was not a bug in the Scheduler itself, yet something to take care of later when programming the actual Job-Planning and schedule generation.
2024-01-11 22:03:36 +01:00
;
// ////////////////////////////////////////////////////////WIP : Run test directly for investigation of SEGFAULT....
// BlockFlowAlloc bFlow;
// EngineObserver watch;
// Scheduler scheduler{bFlow, watch};
Scheduler-test: investigate behaviour of a load for stress testing The goal is to devise a load more akin to the expected real-world processing patterns, and then to increase the density to establish a breaking point. Preliminary investigations focus on establishing the properties of this load and to ensure the actual computation load behaves as expected. Using the third Graph pattern prepared thus far, which produces short chains of length=2, yet immediately spread out to maximum concurrency. This leads to 5.8 Nodes / Level on average.
2024-01-23 18:29:09 +01:00
auto LOAD_BASE = 500us;
Scheduler-test: fix Segfault in test setup ...as it turned out, this segfault was caused by flaws in the ScheduleCtx used for generate the test-schedule; especially when all node-spreads are set to zero and thus all jobs are scheduled immediately at t=0, there was a loophole in the logic to set the dependencies for the final »wake-up« job. When running such a schedule in the Stress-Test-Bench, the next measurement run could be started due to a premature wake-up job, thereby overrunning the previous test-run, which could be still in the middle of computations. So this was not a bug in the Scheduler itself, yet something to take care of later when programming the actual Job-Planning and schedule generation.
2024-01-11 22:03:36 +01:00
// auto stressFac = 1.0;
// auto concurrency = 8;
//
Scheduler-test: investigate behaviour of a load for stress testing The goal is to devise a load more akin to the expected real-world processing patterns, and then to increase the density to establish a breaking point. Preliminary investigations focus on establishing the properties of this load and to ensure the actual computation load behaves as expected. Using the third Graph pattern prepared thus far, which produces short chains of length=2, yet immediately spread out to maximum concurrency. This leads to 5.8 Nodes / Level on average.
2024-01-23 18:29:09 +01:00
ComputationalLoad cpuLoad;
cpuLoad.timeBase = LOAD_BASE;
cpuLoad.calibrate();
Scheduler-test: fix Segfault in test setup ...as it turned out, this segfault was caused by flaws in the ScheduleCtx used for generate the test-schedule; especially when all node-spreads are set to zero and thus all jobs are scheduled immediately at t=0, there was a loophole in the logic to set the dependencies for the final »wake-up« job. When running such a schedule in the Stress-Test-Bench, the next measurement run could be started due to a premature wake-up job, thereby overrunning the previous test-run, which could be still in the middle of computations. So this was not a bug in the Scheduler itself, yet something to take care of later when programming the actual Job-Planning and schedule generation.
2024-01-11 22:03:36 +01:00
//
Scheduler-test: investigate behaviour of a load for stress testing The goal is to devise a load more akin to the expected real-world processing patterns, and then to increase the density to establish a breaking point. Preliminary investigations focus on establishing the properties of this load and to ensure the actual computation load behaves as expected. Using the third Graph pattern prepared thus far, which produces short chains of length=2, yet immediately spread out to maximum concurrency. This leads to 5.8 Nodes / Level on average.
2024-01-23 18:29:09 +01:00
double loadMicros = cpuLoad.invoke();
Scheduler-test: fix Segfault in test setup ...as it turned out, this segfault was caused by flaws in the ScheduleCtx used for generate the test-schedule; especially when all node-spreads are set to zero and thus all jobs are scheduled immediately at t=0, there was a loophole in the logic to set the dependencies for the final »wake-up« job. When running such a schedule in the Stress-Test-Bench, the next measurement run could be started due to a premature wake-up job, thereby overrunning the previous test-run, which could be still in the middle of computations. So this was not a bug in the Scheduler itself, yet something to take care of later when programming the actual Job-Planning and schedule generation.
2024-01-11 22:03:36 +01:00
// double refTime = testLoad.calcRuntimeReference(LOAD_BASE);
Scheduler-test: investigate behaviour of a load for stress testing The goal is to devise a load more akin to the expected real-world processing patterns, and then to increase the density to establish a breaking point. Preliminary investigations focus on establishing the properties of this load and to ensure the actual computation load behaves as expected. Using the third Graph pattern prepared thus far, which produces short chains of length=2, yet immediately spread out to maximum concurrency. This leads to 5.8 Nodes / Level on average.
2024-01-23 18:29:09 +01:00
SHOW_EXPR(loadMicros)
Scheduler-test: fix Segfault in test setup ...as it turned out, this segfault was caused by flaws in the ScheduleCtx used for generate the test-schedule; especially when all node-spreads are set to zero and thus all jobs are scheduled immediately at t=0, there was a loophole in the logic to set the dependencies for the final »wake-up« job. When running such a schedule in the Stress-Test-Bench, the next measurement run could be started due to a premature wake-up job, thereby overrunning the previous test-run, which could be still in the middle of computations. So this was not a bug in the Scheduler itself, yet something to take care of later when programming the actual Job-Planning and schedule generation.
2024-01-11 22:03:36 +01:00
//
// auto testSetup =
// testLoad.setupSchedule(scheduler)
// .withLoadTimeBase(LOAD_BASE)
// .withJobDeadline(50ms)
// .withUpfrontPlanning()
// .withAdaptedSchedule (stressFac, concurrency);
// double runTime = testSetup.launch_and_wait();
// double expected = testSetup.getExpectedEndTime();
//SHOW_EXPR(runTime)
//SHOW_EXPR(expected)
//SHOW_EXPR(refTime)
Scheduler-test: setup in Stress-Test-Rig ...use the scheme established thus far
2024-01-10 20:39:20 +01:00
struct Setup : StressRig
{
usec LOAD_BASE = 500us;
Scheduler-test: investigate behaviour of a load for stress testing The goal is to devise a load more akin to the expected real-world processing patterns, and then to increase the density to establish a breaking point. Preliminary investigations focus on establishing the properties of this load and to ensure the actual computation load behaves as expected. Using the third Graph pattern prepared thus far, which produces short chains of length=2, yet immediately spread out to maximum concurrency. This leads to 5.8 Nodes / Level on average.
2024-01-23 18:29:09 +01:00
usec BASE_EXPENSE = 200us;
Scheduler-test: continue investigation - combine base expense with stress factor After an extended break due to "real life issues".... Pick up the investigation, with the goal to ascertain a valid definition and understanding of all test parameters. A first step is to establish a baseline ''without using a computational load''; this might be some kind of base overhead of the scheduler. However -- the way the test scaffolding was built, it is difficult to create a feedback loop for the statistical test setup with binary search, since it is not really clear how the single control parameter of the test algorithm, the so called "stress factor", shall be interpreted and how it can be combined with a base load. An extended series of tests, while watching the observed value patterns qualitatively, seems to corroborate the former results, indicating that the base expense in my test setup (using a debug build) is at ~200µs / Node / core. Yet the difficulty to interpret this result and arrive at a logical and generic model prevents me from translating this into a measurement scheme, which can be executed independently from a specific test setup and hardware
2024-02-11 03:53:42 +01:00
double UPPER_STRESS = 12;
//
Scheduler-test: investigation... ...and reflection about goals, methods of measurement and possible interpretation
2024-02-11 17:38:20 +01:00
double FAIL_LIMIT = 1.0; //0.7;
double TRIGGER_SDEV = 1.0; //0.25;
double TRIGGER_DELTA = 2.0; //0.5;
Scheduler-test: setup in Stress-Test-Rig ...use the scheme established thus far
2024-01-10 20:39:20 +01:00
// uint CONCURRENCY = 4;
// bool SCHED_DEPENDS = true;
bool showRuns = true;
auto
testLoad()
{
Scheduler-test: continue investigation - combine base expense with stress factor After an extended break due to "real life issues".... Pick up the investigation, with the goal to ascertain a valid definition and understanding of all test parameters. A first step is to establish a baseline ''without using a computational load''; this might be some kind of base overhead of the scheduler. However -- the way the test scaffolding was built, it is difficult to create a feedback loop for the statistical test setup with binary search, since it is not really clear how the single control parameter of the test algorithm, the so called "stress factor", shall be interpreted and how it can be combined with a base load. An extended series of tests, while watching the observed value patterns qualitatively, seems to corroborate the former results, indicating that the base expense in my test setup (using a debug build) is at ~200µs / Node / core. Yet the difficulty to interpret this result and arrive at a logical and generic model prevents me from translating this into a measurement scheme, which can be executed independently from a specific test setup and hardware
2024-02-11 03:53:42 +01:00
TestChainLoad<8> testLoad{256};
Scheduler-test: setup in Stress-Test-Rig ...use the scheme established thus far
2024-01-10 20:39:20 +01:00
testLoad.seedingRule(testLoad.rule().probability(0.6).minVal(2))
.pruningRule(testLoad.rule().probability(0.44))
Scheduler-test: investigation... ...and reflection about goals, methods of measurement and possible interpretation
2024-02-11 17:38:20 +01:00
.weightRule(testLoad.value(1))
Scheduler-test: setup in Stress-Test-Rig ...use the scheme established thus far
2024-01-10 20:39:20 +01:00
.setSeed(55);
return testLoad;
}
};
Scheduler-test: consider using a complementary measurement method With the latest improvements, the »breaking point search« works as expected and yields meaningful data; however — it seems to be well suited rather for specific setups, which involve an extended graph with massive dependencies, because only such a setup produces a clearly defined ''breaking point.'' Thus I'm considering to complement this research by another measurement setup to establish a linear regression model of the Scheduler expense. To allow integration of this different setup into the existing stress-test-rig, some rearrangements of the builder notation are necessary; especially we need to pass the type name of the actual tool, and it seems indicated to reorder the source code to provide the config base class `StressRig` at the top, followed by a long (and very technical) implementation namespace.
2024-02-23 03:04:24 +01:00
auto [stress,delta,time] = StressRig::with<Setup>()
.perform<bench::BreakingPoint>();
Scheduler-test: setup in Stress-Test-Rig ...use the scheme established thus far
2024-01-10 20:39:20 +01:00
SHOW_EXPR(stress)
SHOW_EXPR(delta)
SHOW_EXPR(time)
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
}
/** @test TODO
Scheduler-test: experiment with varying stress levels
2024-01-02 15:45:40 +01:00
* @todo WIP 1/24 🔁 define ⟶ implement
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
*/
void
Scheduler-test: planning for stress-tests
2023-12-19 21:06:23 +01:00
walkingDeadline()
Scheduler: plan for integration identified three distinct tasks - build the external API - establish component integration - performance testing
2023-10-19 22:04:30 +02:00
{
}
};
/** Register this test class... */
LAUNCHER (SchedulerStress_test, "unit engine");
}}} // namespace vault::gear::test
Reference in a new issue Copy permalink