...using the same method for sake of uniformity
Also move the permissions helpers to the file.hpp support functions
and setup a separate unit test for these
Inspired by https://stackoverflow.com/a/58454949
Verified behaviour of fs::create_directory
--> it returns true only if it ''indeed could create'' a new directory
--> it returns false if the directory exists already
--> it throws when some other obstacle shows up
As an aside: the Header include/limits.h could be cleaned up,
and it is used solely from C++ code, thus could be typed, namespaced etc.
Since this is a much more complicated topic,
for now I decided to establish two instances through global variables:
* a sequence seeded with a fixed starting value
* another sequence seeded from a true entropy source
What we actually need however is some kind of execution framework
to define points of random-seeding and to capture seed values for
reproducible tests.
Relying on random numbers for verification and measurements is known to be problematic.
At some point we are bound to control the seed values -- and in the actual
application usage we want to record sequence seeding in the event log.
Some initial thoughts regarding this intricate topic.
* a low-ceremony drop-in replacement for rand() is required
* we want the ability to pick-up and control each and every usage eventually
* however, some usages explicitly require true randomness
* the ability to use separate streams of random-number generation is desirable
Yesterday I decided to include some facilities I have written in 2022
for the Yoshimi-Testsuite. The intention is to use these as-is, and just
to adapt them stylistically to the Lumiera code base.
However — at least some basic documentation in the form of
very basic unit-tests can be considered »acceptance criteria«
- reformat in Lumieara-GNU style
- use the Lumiera exceptions
- use Lumiera format-string frontend
- use lib/util
NOTE: I am the original author of the code introduced here,
and thus I can re-license it under GPL 2+
Initially the model was that of a single graph starting
with one seed node and joining all chains into a single exit node.
This however is not well suited to simulate realistic calculations,
and thus the ability for injecting additional seeds and to randomly
sever some chains was added -- which overthrows the assumption of
a single exit node at the end, where the final hash can be retrieved.
The topology generation used to pick up all open ends, in order to
join them explicitly into a reserved last node; in the light of the
above changes, this seems like an superfluous complexity, and adds
a lot of redundant checks to the code, since the main body of the
algorithm, in its current form, already does all the necessary
bound checks. It suffices thus to just terminate the processing
when the complete node space is visited and wired.
Unfortunately this requires to fix basically all node hashes
and a lot of the statistics values of the test; yet overall
the generated graphs are much more logical; so this change
is deemed worth the effort.
Allow easily to generate a Chain-Load with all nodes unconnected,
yet each node on a separate level.
Fix a deficiency in the graph generation, which caused spurious
connections to be added at the last node, since the prune rule
was not checked
...the previous setup produced a single linear chain
instead of a set of unconnected nodes.
With this, the behaviour is more like expected,
but concurrency is still too low
- better use a Test-Chain-Load without any dependencies
- schedule all at once
- employ instrumentation
- use the inner »overall time« as dependent result variable
The timing results now show an almost perfect linear dependency.
Also the inner overall time seems to omit the setup and tear-down time.
But other observed values (notably the avgConcurrency) do not line up
- fill the range randomly with probe points
- use the node count as independent parameter
- measurement method *works as intended*
- results indeed show a linear relationship
Results are ''interesting'' however, since the (par,time) points
seem to be arranged into two lines, implying that about half
of the runs were somehow ''degraded'' and performed way slower.
With the latest improvements, the »breaking point search« works as expected
and yields meaningful data; however — it seems to be well suited rather
for specific setups, which involve an extended graph with massive dependencies,
because only such a setup produces a clearly defined ''breaking point.''
Thus I'm considering to complement this research by another measurement setup
to establish a linear regression model of the Scheduler expense.
To allow integration of this different setup into the existing stress-test-rig,
some rearrangements of the builder notation are necessary; especially we need
to pass the type name of the actual tool, and it seems indicated to
reorder the source code to provide the config base class `StressRig`
at the top, followed by a long (and very technical) implementation
namespace.
It turns out to be not correct using all the divergence in concurrency
as a form factor, since it is quite common that not all cores can be active
at every level, given the structural constraints as dictated by the load graph.
On the other hand, if the empirical work (non wait-time) concurrency
systematically differs from the simple model used for establishing the schedule,
then this should indeed be considered a form factor and deduced from
the effective stress factor, since it is not a reserve available for speed-up
The solution entertained here is to derive an effective compounded sum
of weights from the calculation used to build the schedule. This compounded
weight sum is typically lower than the plain sum of all node weights, which
is precisely due to the theoretical amount of expense reduction assumed
in the schedule generation. So this gives us a handle at the theoretically
expected expense and through the plain weight sum, we may draw conclusion
about the effective concurrency expected in this schedule.
Taking only this part as base for the empirical deviations yields search results
very close to stressFactor ~1 -- implying that the test setup now
observes what was intended to observe...
In binary search, in order to establish the invariant initially,
a loop is necessary, since a single step might not be sufficient.
Moreover, the ongoing adjustments jeopardise detection of the
statistical breaking point condition, by causing a negative delta
due to gradually approaching the point of convergence -- leading
to an ongoing search in a region beyond the actual breaking point.
Various misconceptions identified in the feedback path of the test algorithm.
- statistics are cumulative, which must be incorporated by norming on time base
- average concurrency includes idle times, which is besides the point within this
test setup, since additional wait-phases are injected when reducing stress
Relying on the new instrumentation facility, the actually effective
concurrency and cumulative run time of the test jobs can be established.
These can now be cast into a form-factor to represent actual excess expenses
in relation to the theoretical model.
By allowing to adjust the adapted schedule by this form factor,
it can be made to reflect more closely the actual empiric load,
hopefully leading to a more realistic effect of the stress-factor
and thus results better suited to conclude on generic behaviour.
...turns out rather challenging to come up with any test case,
that is both meaningful, simple to setup and understand, yet still
produces somewhat stable values. `IncidenceCount` seems most valuable
for investigation and direct inspection of results
Various experiments to watch Scheduler behaviour under extended load.
Notably the example committed here makes the Scheduler run for 1.2 sec
and process 800 jobs with 10ms each, thereby putting the system into
100% load on all CPUs
- supplement the pre-dimensioning for data capture; without that,
sporadic memory corruption indeed happens (as expected, since
concurrent re-allocation of the vector with an entry for each
thread is not threadsafe, and this test shows much contention)
- add a top-level logging for better diagnostics of errors
emanating from the test run
Basically users are free to place the measurement calls to their liking.
This implies that bracketed measurement intervals can be defined overlapping
even within a single thread, thereby accounting the overlapping time interval
several times. However, for the time spent per thread, only actual thread
activity should be counted, disregarding overlaps. Thus introduce a
new aggregate, ''active time'', which is the sum of all thread times.
As an aside, do not need explicit randomness for the simple two-thread
test case — timings are random anyway...
+ bugfix for out-of-bounds access
...since we've established already an integration over the event timeline,
it is just one simple further step to determine the concurrency level
on each individual segment of the timeline. Based on this attribution
- the averaged concurrenty within the observation range can be computed as weighted mean
- moreover we can account for the precise cumulated time spent at each concurrency level
...using a simplistic allocation of next-slot based on initialisation
of a thread_local storage. This implies that this helper can not be
reset or reused, and that there can not be multiple or long-lived instances.
Keep-it-simple for now...
...to sort out the interpretation of measurement results,
the actual duration and concurrency of ComputationLoad invocations
should be recorded, allowing to draw conclusions regarding the
Scheduler's performance as opposed to further system and thread
management effects due to concurrent operation under pressure.
After an extended break due to "real life issues"....
Pick up the investigation, with the goal to ascertain a valid definition
and understanding of all test parameters. A first step is to establish
a baseline ''without using a computational load''; this might be some kind
of base overhead of the scheduler.
However -- the way the test scaffolding was built, it is difficult to
create a feedback loop for the statistical test setup with binary search,
since it is not really clear how the single control parameter of the test algorithm,
the so called "stress factor", shall be interpreted and how it can be
combined with a base load.
An extended series of tests, while watching the observed value patterns qualitatively,
seems to corroborate the former results, indicating that the base expense
in my test setup (using a debug build) is at ~200µs / Node / core.
Yet the difficulty to interpret this result and arrive at a logical and generic model
prevents me from translating this into a measurement scheme, which can
be executed independently from a specific test setup and hardware
The goal is to devise a load more akin to the expected real-world processing patterns,
and then to increase the density to establish a breaking point.
Preliminary investigations focus on establishing the properties of this load
and to ensure the actual computation load behaves as expected.
Using the third Graph pattern prepared thus far, which produces
short chains of length=2, yet immediately spread out to maximum concurrency.
This leads to 5.8 Nodes / Level on average.
...as it turned out, this segfault was caused by flaws in the ScheduleCtx
used for generate the test-schedule; especially when all node-spreads are set
to zero and thus all jobs are scheduled immediately at t=0, there was a loophole
in the logic to set the dependencies for the final »wake-up« job.
When running such a schedule in the Stress-Test-Bench, the next measurement run
could be started due to a premature wake-up job, thereby overrunning the previous
test-run, which could be still in the middle of computations.
So this was not a bug in the Scheduler itself, yet something to take care of
later when programming the actual Job-Planning and schedule generation.
Search processing pattern for massive parallel test.
The goal is to get all cores into active processing most of the time,
thus we need a graph with low dependency management overhead, which is
also consistently wide horizontally to have several jobs in working state
all of the time. The investigation aims at finding out about systematic
overheads in such a setup.
This is just another (obvious) degree of freedom, which could be
interesting to explore in stress testing, while probably not of much
relevance in practice (if a job is expected to become runable earlier,
in can as well be just scheduled earlier).
Some experimentation shows that the timing measurements exhibit more
fluctuations, but also slightly better times when pressure is low, which
is pretty much what I'd expect. When raising pressure, the average
times converge towards the same time range as observed with time bound
propagation.
Note that enabling this variation requires to wire a boolean switch
over various layers of abstraction; arguably this is an unnecessary
complexity and could be retracted once the »experimentation phase«
is over.
This completes the preparation of a Scheduler Stress-Test setup.
The `volatile` was used asymmetrically and there was concern that
this usage makes the `ComutationalLoad` dependent on concurrency.
However, an impact could not be confirmed empirically.
Moreover, to simplify this kind of tests, make the `schedDepends`
directly configurable in the Stress-Test-Rig.
...watching those dumps on the example Graph with excessive dependencies
made blatantly clear that we're dispatching a lot of unnecessary jobs,
since the actual continuation is /always/ triggered by the dependency-NOTIFY.
Before the rework of NOTIFY-Handling, this was rather obscured, but now,
since the NOTIFY trigger itself is also dispatched by the Scheduler,
it ''must be this job'' which actually continues the calculation, since
the main job ''can not pass the gate'' before the dependency notification
arrives.
Thus I've now added a variation to the test setup where all these duplicate
jobs are simply omitted. And, as expected, the computation runs faster
and with less signs of contention. Together with the other additional
parameter (the base expense) we might now actually be able to narrow down
on the observation of a ''expense socket'', which can then be
attributed to something like an ''inherent scheduler overhead''
...actually difficult to integrate into the existing scheme,
which is entirely level-based. Can only be added to the individual Jobs,
not to the planning and completion-jobs — which actually shouldn't be a problem,
since it is beneficial to dispatch the planning runs earlier
The next goal is to determine basic performance characteristics
of the Scheduler implementation written thus far;
to help with these investigations some added flexibility seems expedient
- the ability to define a per-job base expense
- added flexibility regarding the scheduling of dependencies
This changeset introduces configuration options
While the idea with capturing observation values is nice,
it definitively does not belong into a library impl of the
search algorithm, because this is usage specific and grossly
complicates the invocation.
Rather, observation data can be captured by side-effect
from the probe-λ holding the actual measurement run.
This statistical criterion defines when to count observed Scheduler performance
as loosing control. The test is comprised of three observations, which
all must be confirmed:
- an individual run counts as accidentally failed when the execution slips
away by more than 2ms with respect to the defined overall schedule.
When more than 55% of all observed runs are considered as failed,
the first condition is met
- moreover, the observed standard derivation must also surpass the
same limit of > 2ms, which indicates that the Scheduling mechanism
is under substantial strain (on average, the slip is ~ 200µs)
- the third condition is that the ''averaged delta'' has surpassed
4ms, which is 2 times the basic failure indicator.
These conditions are based on watching the Scheduler in operation;
typically all three conditions slip away by large margin after a
very narrow yet critical increase in the stress level.
Using three conditions together should improve robustness; often
the problems to keep up with the schedule build up over some parameter
range, yet the actual decision should be based on complete loss of control.
adapt the code written yesterday explicitly for the test case
into the new framework for performing a stress-test run.
Notable difference: times converted to millisecond immediately
Elaborate the draft to include all the elements used directly in the test case thus far;
the goal is to introduce some structuring and leave room for flexible confguration,
while implementing the actual binary search as library function over Lambdas.
My expectation is to write a series of individual test instances with varying parameters;
while it seems possible to add further performance test variations into that scheme later on.
- the goal is to run a binary search
- the search condition should be factored out
- thus some kind of framework or DSL is required,
to separate the technicalities of the measurement
from the specifics of the actual test case.
- repeated invocations of the same test setup for statistics
- the usual nasty 64-node graph with massive fork out
- limit concurrency to 4 cores
- tabulate data to look for clues regarding a trigger criteria
Hypothesis: The Scheduler slips off schedule when all of the
following three criteria are met:
- more than 55% glitches with Δ > 2ms
- σ > 2ms
- ∅Δ > 4ms
- schedule can now be adapted to concurrency and expected distribution of runtimes
- additional stress factor to press the schedule (1.0 is nominal speed)
- observed run-time now without Scheduler start-up and pre-roll
- document and verify computed numbers
...based on the adapted time-factor sequence
implemented yesterday in TestChainLoad itself
- in this case, the TimeBase from the computation load is used as level speed
- this »base beat« is then modulated by the timing factor sequence
- working in an additional stress factor to press the schedule uniformly
- actual start time will be added as offset once the actual test commences
...up to now, we've relied on a regular schedule governed solely
by the progression of node levels, with a fixed level speed
defaulting to 1ms per level.
But in preparation of stress tesging, we need a schedule adapted
to the expected distribution of computation times, otherwise
we'll not be able to factor out the actual computation graph
connectivity. The goal is to establish a distinctive
**breaking point** when the scheduler is unable to cope with
the provided schedule.
