The rework from yesterday turned out to be effective ... unfortunately
a bit to much: since now late follow-up notifications take precedence,
a single worker tends to process the complete chain depth-first, because
the first chain will be followed and processed, even before the worker
was able to post the tasks for the other branches. Thus this single
worker is the only one to get a chance to proceed.
After some consideration, I am now leaning towards a fundamental change,
instead of just fixing some unfavourable behaviour pattern: while the
language semantics remains the same, the scheduler should no longer
directly dispatch into the next chain **from λ-post**. That is, whenever
a POST / NOTIFY is issued from the Activity-chain, the scheduler goes
through prioritisation.
This has further ramifications: we do not need a self-inhibition mechanism
any more (since now NOTIFY picks up the schedule time of the target).
With these changes, processing seems to proceed more smoothly,
albeit still with lots of contention on the Grooming token,
at least in the example structure tested here.
While the recent refactoring...
206c67cc
...was a step into the right direction, it pushed too hard,
overlooking the requirement to protect the scheduler contents
and thus all of the Activity-chains against concurrent modification.
Moreover, the recent solution still seems not quite orthogonal.
Thus the handling of notifications was thoroughly reworked:
- the explicit "double-dispatch" was removed, since actual usage
of the language indicates that we only need notifications to
Gate (and Hook), but not to any other conceivable Activity.
- thus it seems unnecessary to turn "notification" into some kind
of secondary work mode. Rather, it is folded as special case
into the regular dispatch.
This leads to new processing rules:
- a POST goes into λ-post (obviously... that's its meaning)
- a NOTIFY now passes its *target* into λ-post
- λ-post invokes ''dispatch''
- and **dispatching a Gate now implies to notify the Gate**
This greatly simplifies the »state machine« in the Activity-Language,
but also incurs some limitations (which seems adequate, since it is
now clear that we do not ''schedule'' or ''dispatch'' arbitrary
Activities — rather we'll do this only with POST and NOTIFY,
and all further processing happens by passing activation
along the chain, without involving the Scheduler)
use a feature of the Activity-Language prepared for this purpose:
self-Inhibition of the Chain. This prevents a prerequisite-NOTIFY
to trigger a complete chain of available tasks, before these tasks
have actually reached their nominal scheduling time.
This has the effect to align the computations much more strictly
with the defined schedule
The main (test) thread is kept in a blocking wait until the
planned schedule is completed. If however the schedule overruns,
the wake-up job could just be triggered prematurely.
This can easily be prevented by adding a dependency from the last
computation job to the wake-up job. If the computation somehow
flounders, the SAFETY_TIMEOUT (5s) will eventually raise
an exception to let the test fail cleanly (shutting down
the Scheduler automatically)
...it seems impossible to solve this conundrum other than by
opening a path to override a contextual deadline setting from
within the core Activity-Language logic.
This will be used in two cases
- when processing a explicitly coded POST (using deadline from the POST)
- after successfully opening a Gate by NOTIFY (using deadline from Gate)
All other cases can now supply Time::NEVER, thereby indicating that
the processing layer shall use contextual information (intersection
of the time intervals)
...this is an interesting test failure, which highlights inconsistencies
with handling of deadlines when processing follow-up from NOTIFY-triggers
There was also some fuzziness related to the ''meaning'' of λ-post,
leading to at least one superfluous POST invocation for each propagation;
fixing this does not solve the problem yet removes unnecessary overhead
and lock-contention
...playing around with the graph for the Scheduler integration test
...single threaded run time seemed to behave irregular
...but in fact it is very close to what can be expected
based on an ''averaged node weight''
Fortunately its very simple to add that into the existing node statistics
Basically this is all done and settled already: this is the `usageExample()`
from `TestChainLoadTest`. However, the focus is slightly different here:
We want a demonstration that the Scheduler can work flawlessly through
a massive load. Thus the plan is to use much more challenging parameters,
and then lean back and watch what happens....
...which turns out to be due to the DUMP-Statements,
which seem to create quite some contention on their own.
Test cases with very tight schedule will slip away then;
without print statement everything is GREEN now
this bug was there since the first draft, yet was covered
by another bug with the start-up logic.
And this latter one was fixed recently...
fa8622805
As a result, even when the COMPUTATION_CAPACITY is set to 0
still a single worker boots up (which should not be the case)
Solution: we do not need to "safeguard" against rounding errors,
since this is an internal implementation function, it is assumed
that the caller knows about its limitations...
* added benchmark over synchronous execution as point of reference
* verified running times and execution pattern
* Scheduler **behaves as expected** for this example
- Generally speaking, the calibration uses current baseline settings;
- There are now two different load generation methods, thus both must be calibrated
- Performance contains some socked and non-linear effects, thus calibration
should be done close to the work point, which can be achieved by incremental
calibration until the error is < 5%
Interestingly, longer time-base values run slightly faster than predicted,
which is consistent with the expectation (socket cost). And using a larger
memory block increases time values, which is also plausible, since
cache effects will be diminishing
..initial gauging is a tricky subject,
since existing computer's performance spans a wide scale
Allowing
- pre calibration -98% .. +190%
- single run ±20%
- benchmark ±5%
...which can be deliberately attached (or not attached) to the
individual node invocation functor, allowing to study the effect
of actual load vs. zero-load and worker contention
Some test-runs performed excitingly smooth,
but in one case the processing was was drastically delayed,
due to heavy contention. The relevance of this incident is not clear yet,
since this test run uses a rather atypical load with very short actual work jobs.
Anyway, the dump-logs are documented with this commit.
Within Chain-Load, the infrastructure to add this crucial feature
is minimal: each node gets a `weight` parameter, which is assigned
using another RandomDraw-Rule (by default `weight==0`).
The actual computation load will be developed as a separate component
and tied in from the node calculation job functor.
...during development of the Chain-Load, it became clear that we'll often
need a collection of small trees rather than one huge graph. Thus a rule
for pruning nodes and finishing graphs was added. This has the consequence
that there might now be several exit nodes scattered all over the graph;
we still want one single global hash value to verify computations,
thus those exit hashes must now be picked up from the nodes and
combined into a single value.
All existing hash values hard coded into tests must be updated
...with this change, processing is ''ahead of schedule'' from the beginning,
which has the nice side effect that the problematic contention situation
with these very short computation jobs can not arise, and most of the schedule
is processed by a single worker.
Processing pattern is now pretty much as expected
This is a trick to get much better scheduling and timing guesses.
Instead of targeting a specific level, rather a fixed number of nodes
is processed in each chunk, yet still always processing complete levels.
The final level number to expect can be retrieved from the chain-load graph.
With this refactoring, we can now schedule a wake-up job precisely
after the expected completion of the last level
Scheduling a wake-up job behind the end of the planned schedule did the trick.
Sometimes there is ''strong contention'' immediately after full provision of the WorkForce,
but this seems to be as expected, since the »Jobs« currently used have no
actually relevant run time on their own. It is even more surprising that
the Capacity-control logic is able to cope with this situation in a matter
of just some milliseconds, bringing the average Lag at ~ 300µs
Invent a special JobFunctor...
- can be created / bound from a λ
- self-manages its storage on the heap
- can be invoked once, then discards itself
Intention is to pass such one-time actions to the Scheduler
to cause some ad-hoc transitions tied to curren circumstances;
a notable example will be the callback after load-test completion.
In the first draft version, a blocked Gate was handled by
»polling« the Gate regularly by scheduling a re-invocation
repeatedly into the future (by a stepping defined through
ExecutionCtx::getWaitDelay()).
Yet the further development of the Activity-Language indicates
that the ''Notification mechanism'' is sufficient to handle all
foreseeable aspects of dependency management. Consequently this
''Gate poling is no longer necessary,'' since on Notification
the Gate is automatically checked and the activation impulse
is immediately passed on; thus the re-scheduled check would
never get an opportunity actually to trigger the Gate; such
an active polling would only be necessary if the count down
latch in the Gate is changed by "external forces".
Moreover, the first Scheduler integration tests with TestChainLoad
indicate that the rescheduled polling can create a considerable
additional load when longer dependency chains miss one early
prerequisite, and this additional load (albeit processed
comparatively fast by the Scheduler) will be shifted along
needlessly for quite some time, until all of the activities
from the failed chain have passed their deadline. And what
is even more concerning, these useless checks have a tendency
to miss-focus the capacity management, as it seems there is
much work to do in a near horizon, which in fact may not be
the case altogether.
Thus the Gate implementation is now *changed to just SKIP*
when blocked. This helped to drastically improve the behaviour
of the Scheduler immediately after start-up -- further observation
indicated another adjustment: the first Tick-duty-cycle is now
shortened, because (after the additional "noise" from gate-rescheduling
was removed), the newly scaled-up work capacity has the tendency
to focus in the time horizon directly behind the first jobs added
to the timeline, which typically is now the first »Tick«.
ð¡ this leads to a recommendation, to arrange the first job-planning
chunk in such a way that the first actual work jobs appear in the area
between 5ms and 10ms after triggering the Scheduler start-up.Scheduler¡
Introducing a fixed pre-delay on each new Calc-Streem seemed like an obvious remedy,
yet on closer investigation it turned out that the start-up logic as such was contradictory,
which was only uncovered by some rather special schedule patterns.
After fixing the logic deficiencies, Scheduler starts up as intended
and the probabilistic capacity-control seems to work as designed.
Thus no need to introduce an artificial delay at begin, even while
this implies that typically the first round of job-planning will be
performed synchronous, in the invoking thread (which may be surprising,
but is completely within the limits of the architecture; we do not
employ specifically configured threads and planning should be done
in short chunks, thus the first chunk can well be done by the caller)
The first complete integration test with Chain-Load
highlighted some difficulties with the overall load regulation:
- it works well in the standard case (but is possibly to eager to scale up)
- the scale-up sometimes needs several cycles to get "off the ground"
- when the first job is dispatched immediately instead of going
through the queue, the scheduler fails to boot up
... so this (finally) is the missing cornerstone
... traverse the calculation graph and generate render jobs
... provide a chunk-wise pre-planning of the next batch
... use a future to block the (test) thread until completed
- test setup without actual scheduler
- wire the callbacks such to verify
+ all nodes are touched
+ levels are processed to completion
+ the planning chunk stops at the expected level
+ all node dependencies are properly reported through the callbacks
- decided to abstract the scheduler invocations as λ
- so this functor contains the bare loop logic
Investigation regarding hash-framework:
It turns out that boost::hash uses a different hash_combine,
than what we have extracted/duplicated in lib/hash-value.hpp
(either this was a mistake, or boost::hash did use this weaker
function at that time and supplied a dedicated 64bit implementation later)
Anyway, should use boost::hash for the time being
maybe also fix the duplicated impl in lib/hash-value.hpp
- use a ''special encoding'' to marshal the specific coordinates for this test setup
- use a fixed Frame-Grid to represent the ''time level''
- invoke hash calculation through a specialised JobFunctor subclass
The number of nodes was just defined as template argument
to get a cheap implementation through std::array...
But actually this number of nodes is ''not a characteristics of the type;''
we'd end up with a distinct JobFunctor type for each different test size,
which is plain nonsensical. Usage analysis reveals, now that the implementation
is ''basically complete,'' that all of the topology generation and statistic
calculation code does not integrate deeply with the node storage, but
rather just iterates over all nodes and uses the ''first'' and ''last'' node.
This can actually be achieved very easy with a heap-allocated plain array,
relying on the magic of lib::IterExplorer for all iteration and transformation.
- use a dedicated context "dropped off" the TestChainLoad instance
- encode the node-idx into the InvocationInstanceID
- build an invocation- and a planning-job-functor
- let planning progress over an lib::UninitialisedStorage array
- plant the ActivityTerm instances into that array as Scheduling progresses
Introduced as remedy for a long standing sloppiness:
Using a `char[]` together with `reinterpret_cast` in storage management helpers
bears danger of placing objects with wrong alignment; moreover, there are increasing
risks that modern code optimisers miss the ''backdoor access'' and might apply too
aggressive rewritings.
With C++17, there is a standard conformant way to express such a usage scheme.
* `lib::UninitialisedStorage` can now be used in a situation (e.g. as in `ExtentFamily`)
where a complete block of storage is allocated once and then subsequently used
to plant objects one by one
* moreover, I went over the code base and adapted the most relevant usages of
''placement-new into buffer'' to also include the `std::launder()` marker
Since Chain-Load shall be used for performance testing of the scheduler,
we need a catalogue of realistic load patterns. This extended effort
started with some parameter configurations and developed various graph
shapes with different degree of connectivity and concurrency, ranging
from a stable sequence of very short chains to large and excessively
interconnected dependency networks.
Through introduction of a ''pruning rule'', it is possible
to create exit nodes in the middle of the graph. With increased
intensity of pruning, it is possible to ''choke off'' the generation
and terminate the graph; in such a case a new seed node is injected
automatically. By combination with seed rules, an equilibrium of
graph start and graph termination can be achieved.
Following this path, it should be possible to produce a pattern,
which is random but overall stable and well suited to simulate
a realistic processing load.
However, finding proper parameters turns out quite hard in practice,
since the behaviour is essentially contingent and most combinations
either lead to uninteresting trivial small graph chunks, or to
large, interconnected and exponentially expanding networks
... seeding happens at random points in the middle of the chain
... when combined with reduction, the resulting processing pattern
resembles the real processing pattern of media calcualtions
... special rule to generate a fixed expansion on each seed
... consecutive reductions join everything back into one chain
... can counterbalance expansions and reductions
...as it turns out, the solution embraced first was the cleanest way
to handle dynamic configuration of parameters; just it did not work
at that time, due to the reference binding problem in the Lambdas.
Meanwhile, the latter has been resolved by relying on the LazyInit
mechanism. Thus it is now possible to abandon the manipulation by
side effect and rather require the dynamic rule to return a
''pristine instance''.
With these adjustments, it is now possible to install a rule
which expands only for some kinds of nodes; this is used here
to crate a starting point for a **reduction rule** to kick in.
- present the weight centres relative to overall level count
- detect sub-graphs and add statistics per subgraph
- include an evaluation for ''all nodes''
- include number of levels and subgraphs
- iterate over all nodes and classify them
- group per level
- book in per level statistics into the Indicator records
- close global averages
...just coded, not yet tested...
The graph will be used to generate a computational load
for testing the Scheduler; thus we need to compute some
statistical indicators to characterise this load.
As starting point sum counts and averages will be aggregated,
accounting for particular characterisation of nodes per level.
It seams indicated to verify the generated connectivity
and the hash calculation and recalculation explicitly
at least for one example topology; choosing a topology
comprised of several sub-graphs, to also verify the
propagation of seed values to further start-nodes.
In order to avoid addressing nodes directly by index number,
those sub-graphs can be processed by ''grouping of nodes'';
all parts are congruent because topology is determined by
the node hashes and thus a regular pattern can be exploited.
To allow for easy processing of groups, I have developed a
simplistic grouping device within the IterExplorer framework.
- with the new pruning option, start-Nodes can now be anywhere
- introduce predicates to detect start-Nodes and exit-Nodes
- ensure each new seed node gets the global seed on graph construction
- provide functionality to re-propagate a seed and clear hashes
- provide functionality to recalculate the hashes over the graph
up to now, random values were completely determined by the
Node's hash, leading to completely symmetrical topology.
This is fine, but sometimes additional randomness is desirable,
while still keeping everything deterministic; the obvious solution
is to make the results optionally dependent on the invocation order,
which is simply to achieve with an additional state field. After some
tinkering, I decided to use the most simplistic solution, which is
just a multiplication with the state.
this is only a minor rearrangement in the Algorithm,
but allows to re-boot computation should node connectivity
go to zero. With current capabilities, this could not happen,
but I'm considering to add a »pruning« parameter to create the
possibility to generate multiple shorter chains instead of one
complete chain -- which more closely emulates reality for
Scheduler load patterns.
...so this was yet another digression, caused by the desire
somehow to salvage this problematic component design. Using a
DSL token fluently, while internally maintaining a complex and
totally open function based configuration is a bit of a stretch.
For context: I've engaged into writing a `LazyInit` helper component,
to resolve the inner contradiction between DSL use of `RandomDraw`
(implying value semantics) and the design of a processing pipeline,
which quite naturally leads to binding by reference into the enclosing
implementation.
In most cases, this change (to lazy on-demand initialisation) should be
transparent for the complete implementation code in `RandomDraw` -- with
one notable exception: when configuring an elaborate pipeline, especially
with dynamic changes of the probability profile during the simulation run,
then then obviously there is the desire to use the existing processing
pipeline from the reconfiguration function (in fact it would be quite
hard to explain why and where this should be avoided). `LazyInit` breaks
this usage scenario, since -- at the time the reconfiguration runs --
now the object is not initialised at all, but holds a »Trojan« functor,
which will trigger initialisation eventually.
After some headaches and grievances (why am I engaging into such an
elaborate solution for such an accidental and marginal topic...),
unfortunately it occurred to me that even this problem can be fixed,
with yet some further "minimal" adjustments to the scheme: the LazyInit
mechanism ''just needs to ensure'' that the init-functor ''sees the
same environment as in eager init'' -- that is, it must clear out the
»Trojan« first, and it ''could apply any previous pending init function''
fist. That is, with just a minimal change, we possibly build a chain
of init functors now, and apply them in given order, so each one
sees the state the previous one created -- as if this was just
direct eager object manipulation...
...this is a more realistic demo example, which mimics
some of the patterns present in RandomDraw. The test also
uses lambdas linking to the actual storage location, so that
the invocation would crash on a copy; LazyInit was invented
to safeguard against this, while still allowing leeway
during the initialisation phase in a DSL.
...oh my.
This is getting messy. I am way into danger territory now....
I've made a nifty cool design with automatically adapted functors;
yet at the end of the day, this does not bode well with a DSL usage,
where objects appear to be simple values from a users point of view.
- Helper function to find out of two objects are located
"close to each other" -- which can be used as heuristics
to distinguish heap vs. stack storage
- further investigation shows that libstdc++ applies the
small-object optimisation for functor up to »two slots«
in size -- but only if the copy-ctor is trivial. Thus
a lambda capturing a shared_ptr by value will *always*
be maintained in heap storage (and LazyInit must be
redesigned accordingly)...
- the verify_inlineStorage() unit test will now trigger
if some implementation does not apply small-object optimisation
under these minimal assumptions
...which is crucial for the solution pursued at the moment;
std::function is known to apply a small-object optimisation,
yet unfortunately there are no guarantees by the C++ standard
(it is only mandated that std::function handles a bare function
pointer without overhead)
Other people have investigated that behaviour already,
indicating that at least one additional »slot« of data
can be handled with embedded storage in all known implementations
(while libstdc++ seemingly imposes the strongest limitations)
https://stackoverflow.com/a/77202545/444796
This experiment in the unit-test shows that for my setup
(libstdc++ and GCC-8) only a lambda capturing a single pointer
is handled entirely embedded into the std::function; already
a lambda capturing a shared-ptr leads to overflow into heap
the RandomDraw rules developed last days are meant to be used
with user-provided λ-adapters; employing these in a context
of a DSL runs danger of producing dangling references.
Attempting to resolve this fundamental problem through
late-initialisation, and then locking the component into
a fixed memory location prior to actual usage. Driven by
the goal of a self-contained component, some advanced
trickery is required -- which again indicates better
to write a library component with adequate test coverage.
RandomDraw as a library component was extracted and (grossly) augmented
to cut down the complexity exposed to the user of TestChainLoad.
To control the generated topology, random-selected parameters
must be configured, defining a probability profile; while
this can be achieved with simple math, getting it correct
turned out surprisingly difficult.
...now using the reworked partial-application helper...
...bind to *this and then recursively re-invoke the adaptation process
...need also to copy-capture the previously existing mapping-function
first test seems to work now
Investigation in test setup reveals that the intended solution
for dynamic configuration of the RandomDraw can not possibly work.
The reason is: the processing function binds back into the object instance.
This implies that RandomDraw must be *non-copyable*.
So we have to go full circle.
We need a way to pass the current instance to the configuration function.
And the most obvious and clear way would be to pass it as function argument.
Which however requires to *partially apply* this function.
So -- again -- we have to resort to one of the functor utilities
written several years ago; and while doing so, we must modernise
these tools further, to support perfect forwarding and binding
of reference arguments.
- strive at complete branch coverage for the mapping function
- decide that the neutral value can deliberately lie outside
the value range, in which case the probability setting
controls the number of _value_ result incidents vs
neutral value result incidents.
- introduce a third path to define this case clearly
- implement the range setting Builder-API functions
- absorb boundrary and illegal cases
For sake of simplicity, since this whole exercise is a byproduct,
the mapping calculations are done in doubles. To get even distribution
of values and a good randomisation, it is thus necessary to break
down the size_t hash value in a first step (size_t can be 64bit
and random numbers would be subject to rounding errors otherwise)
The choice of this quantiser is tricky; it must be a power of two
to guarantee even distribution, and if chosen to close to the grid
of the result values, with lower probabilities we'd fail to cover
some of the possible result values. If chosen to large, then
of course we'd run danger of producing correlated numbers on
consecutive picks.
Attempting to use 4 bits of headroom above the log-2 of the
required value range. For example, 10-step values would use
a quantiser of 128, which looks like a good compromise.
The following tests will show how good this choice holds up.
The first step was to allow setting a minimum value,
which in theory could also be negative (at no point is the
code actually limited to unsigned values; this is rather
the default in practice).
But reconsidering this extensions, then you'd also want
the "neutral value" to be handled properly. Within context,
this means that the *probability* controls when values other
than the neutral value are produced; especially with p = 1.0
the neutral value shall not be produced at all
...since the Policy class now defines the function signature,
we can no longer assume that "input" is size_t. Rather, all
invocations must rely on the generic adaptaion scheme.
Getting this correct turns out rather tricky again;
best to rely on a generic function-composition.
Indeed I programmed such a helper several years ago,
with the caveat that at that time we used C++03 and
could not perfect-forward arguments. Today this problem
can be solved much more succinct using generic Lambdas.
to define this as a generic library component,
any reference to the actual data source moust be extracted
from the body of the implementation and supplied later
at usage site. In the actual case at hand the source
for randomness would be the node hash, and that is
absolutely an internal implementation detail.
The idea is to use some source of randomness to pick a
limited parameter value with controllable probability.
While the core of the implementation is nothing more
than some simple numeric adjustments, these turn out
to be rather intricate and obscure; the desire to
package these technicalities into a component
however necessitates to make invocations
at usage site self explanatory.
This might seem totally overblown -- but already the development
of this prototype showed me time and again, that it is warranted.
Because it is damn hard to get the probabilities and the mappings
to fixed output values correct.
After in-depth analysis, I decided completely to abandon the
initially chosen approach with the Cap helper, where the user
just specifies an upper and lower bound. While this seems
compellingly simple at start, it directly lures into writing
hard-to-understand code tied to the implementation logic.
With the changed approach, most code should get along rather with
auto myRule = Draw().probabilty(0.6).maxVal(4);
...which is obviously a thousand times more legible than
any kind of tricky modulus expressions with shifted bounds.
While the Cap-Helper introduced yesterday was already a step in the
right direction, I had considerable difficulties picking the correct
parameters for the upper/lower bounds and the divisor for random generation
so as to match an intended probability profile. Since this tool shall be
used for load testing, an easier to handle notation will both help
with focusing on the main tasks and later to document the test cases.
Thus engaging (again) into the DSL building game...
...start with putting the topology generator to work
- turns out it is still challenging to write the ctrl-rules
- and one example tree looked odd in the visualisation
- which (on investigation) indicated unsound behaviour
...this is basically harmless, but involves an integer wrap-around
in a variable not used under this conditions (toReduce), but also
a rather accidental and no very logical round-up of the topology.
With this fix, the code branch here is no longer overloaded with two
distinct concerns, which I consider an improvement
by default, a linear chain without any forking is generated,
and the result hash is computed by hash-chaining from the seed.
Verify proper connections and validate computed hash
..as can be expected, had do chase down some quite hairy problems,
especially since consumption of the fixed amount of nodes is not
directly linked to the ''beat'' of the main loop and thus boundary
conditions and exhausted storage can happen basically anywhere.
Used a simple expansion rule and got a nod graph,
which looks coherent in DOT visualisation.
writing a control-value rule for topology generation typically
involves some modulus and then arthmetic operations to map
only part of the value range to the expected output range.
These calculations are generic, noisy and error-prone.
Thus introduce a helper type, which allows the client just
to mark up the target range of the provided value to map and
transform to the actually expected result range, including some
slight margin to absorb rounding errors. Moreover, all calculations
done in double, to avoid the perils of unsigned-wrap-around.
...these were developed driven by the immediate need
to visualise ''random generated computation patterns''
for ''Scheduler load testing.''
The abstraction level of this DSL is low
and structures closely match some clauses of the DOT language;
this approach may not yet be adequate to generate more complex
graph structures and was extracted as a starting point
for further refinements....
With all the preceding DSL work, this turns out to be surprisingly easy;
the only minor twist is the grouping of nodes into (time)levels,
which can be achieved with a "lagging" update from the loop body
Note: next step will be to extract the DSL helpers into a Library header
...using a pre-established example as starting point
It seems that building up this kind of generator code
from a set of free functions in a secluded namespace
is the way most suitable to the nature of the C++ language
..the idea is to generate a Graphviz-DOT diagram description
by traversing the internal data structures of TestChainLoad.
- refreshed my Graphviz knowledge
- work out a diagram scheme that can be easily generated
- explore ways to structure code generation as a DSL to keep it legible
...introduce statistical control functions (based on hash)
...add processing stage for current set of nodes
...process forking, reduction and injection of new nodes
- use a specialised class, layered on top of std::array
- use additional storage to mark filling degree
- check/fail on link owerflow directly there
We still use fixed size inline storage for the node links,
yet adding this comparatively small overhead in storage helps
getting the code simpler and adding links is now constant-complexity
A »Node« represents one junction point in the dependency graph,
knows his predecessors and successors and carries out one step
of the chained hash calculation.
...refine the handling of FrameRates close to the definition bounds
...implement the actual rule to scale allocator capacity on announcement
...hook up into the seedCalcStream() with a default of +25FPS
+ test coverage
...whenever a new CalcStream is seeded, it would be prudent
not only to step up the WorkForce (which is already implemented),
but also to provide a hint to the BlockFlow allocator regarding
the expected calculation density.
Such a hint would allow to set a more ample »epoch« spacing,
thereby avoiding to drive the allocator into overload first.
The allocator will cope anyway and re-balance in a matter of
about 2 seconds, but avoiding this kind of control oscillations
altogether will lead to better performance at calculation start.
The test case "scheduleRenderJob()" -- while deliberately operated
quite artificially with a disabled WorkForce (so the test can check
the contents in the queue and then progress manually -- led to discovery
of an open gap in the logic: in the (rare) case that a new task is
added ''from the outside'' without acquiring the Grooming-Token, then
the new task could sit in the entrace queue, in worst case for 50ms,
until the next Scheduler-»Tick« routinely sweeps this queue. Under
normal conditions however, each dispatch of another activity will
also sweep the entrance queue, yet if there happens to be no other
task right now, a new task could be stuck.
Thinking through this problem also helped to amend some aspects
of Grooming-Token handling and clarified the role of the API-functions.
For now, the `EngineObserver` is defined as an empty shell,
outfitted with a low-level binary message dispatch API.
Messages are keyed by a Symbol, which allows evolution of private message types.
Routing and Addressing is governed by an opaque size_t hash.
The `EngineEvent` data base class provides »4 Slots« of inline binary storage;
concrete subclasses shall define the mapping of actual data into this space
and provide a convenience constructor for events.
For use by the Scheduler, a `WorkTiming`-Event is defined based on this scheme;
this allows to implement the λ-work and λ-done of the Scheduler-`ExecutionCtx`.
These hooks will be invoked at begin and end of any render calculations.
...especially to prevent a deadline way too far into the future,
since this would provoke the BlockFlow (epoch based) memory manager
to run out of space.
Just based on gut feeling, I am now imposing a limit of 20seconds,
which, given current parametrisation, with a minimum spacing of 6.6ms
and 500 Activities per Block would at maximum require 360 MiB for
the Activities, or 3000 Blocks. With *that much* blocks, the
linear search would degrade horribly anyway...
WorkForce scales down automatically after 2 seconds when
workers fall idle; thus we need to step up automatically
with each new task.
Later we'll also add some capacity management to both the
LoadController and the Job-Planning, but for now this rather
crude approach should suffice.
NOTE: most of the cases in SchedulerService_test verify parts
of the component integration and thus need to bypass this
automatism, because the test code wants to invoke the
work-Function directly (without any interference
from running workers)
While building increasingly complex integration tests for the Scheduler,
it turns out helpful to be able to manipulate the "full concurreency"
as used by Scheduler, WorkForce and LoadController.
In the current test, I am facing a problem that new entries from the
threadsafe entrance queue are not propagated to the priority queue
soon enough; partly this is due to functionality still to be added
(scaling up when new tasks are passed in) -- but this will further
complicate the test setup.
The invocation structure is effectively determined by the
Activity-chain builder from the Activity-Language; but, taking
into account the complexity of the Scheduler code developed thus far,
it seems prudent to encapsulate the topic of "Activities" altogether
and expose only a convenience builder-API towards the Job-Planning
With the previous change, we allways have an execution scope now,
which (among other things) defines a time-window (start,deadline).
However, the entrance point to an Activity-chain, the POST-Activity
also defines a time window, which is now combined with this scope
by maximum / minimum constraining.
The problem with passing the deadline was just a blatant symptom
that something with the overall design was not quite right, leading
to mix-up of interfaces and implementation functions, and more and more
detail parameters spreading throughout the call chains.
The turning point was to realise the two conceptual levels
crossing and interconnected within the »Scheduler-Service«
- the Activity-Language describes the patterns of processing
- the Scheduler components handle time-bound events
So by turning the (previously private) queue entry into an
ActivationEvent, the design could be balanced.
This record becomes the common agens within the Scheduler,
and builds upon / layers on top of the common agens of the
Language, which is the Activity record.
This is the first step to address the conceptual problems identified yesterday,
and works largely as a drop-in replacement. Instead of just retrieving
the Activity*, now the Queue entry itself is exposed to the rest of the
scheduler implementation, augmented with implicit conversion, allowing
all of the tests to remain unaltered (and legible, without boilerplate)
the attempt to integrate additional deadline and significance parameters
unveils a design problem due to the layering of contexts
- the Activity-Language attempts to abstract away the ''Scheduler mechanics''
- but this implementation logic now needs to pass additional parameters
- and notably there is the possibility of direct re-scheduling from within
the Activity-Dispatch
The symptom of this problem is that it's no longer possible
to implement the ExecutionCtx.post() function in the real Scheduler-context
...it is clear that there must be a way to flush the scheduler queues
an thereby silently drop any obsoleted or irrelevant entries. This topic
turns out to be somewhat involved, as it requires to consider the
deadline (due to the memory management, which is based on deadlines).
Furthermore there is a relation to yet another challenging conceptual
requirement, which is the support for other operation modes beyond
just time-bound rendering; these concerns make it desirable to
expand the internal representation of entries in the queue.
Concerns regarding performance are postponed deliberately,
until we can demonstrate the Scheduler-Service running under
regular operational conditions.
This is the first kind of integration,
albeit still with a synthetic load.
- placed two excessive load peaks in the scheduling timeline
- verified load behaviour
- verified timings
- verified that the scheduler shuts down automatically when done
- sample distance to scheduler head whenever a worker asks for work
- moving average with N = worker-pool size and damp-factor 2
- multiply with the current concurrency fraction
- An important step towards a complete »Scheduler Service«
- Correct timing pattern could be verified in detail by tracing
- Spurred some further concept and design work regarding Load-control
- draft the duty cycle »tick«
- investigate corner cases of state updates and allocation managment
- implement start and forcible stop of the scheduler service
Obviously the better choice and a perfect fit for our requirements;
while the system-clock may jump and even move backwards on time service
adjustments, the steady clock just counts the ticks since last boot.
In libStdC++ both are implemented as int64_t and use nanoseconds resolution
- Ensure the grooming-token (lock) is reliably dropped
- also explicitly drop it prior to trageted sleeps
- properly signal when not able to acquire the token before dispatch
- amend tests broken by changes since yesterday
Notably the work-function is now completely covered, by adding
this last test, and the detailed investigations yesterday
ultimately unveiled nothing of concern; the times sum up.
Further reflection regarding the overall concept led me
to a surprising solution for the problem with priority classes.
...especially for the case »outgoing to sleep«
- reorganise switch-case to avoid falling through
- properly handle the tendedNext() predicate also in boundrary cases
- structure the decision logic clearer
- cover the new behaviour in test
Remark: when the queue falls empty, the scheduler now sends each
worker once into a targted re-shuffling delay, to ensure the
sleep-cycles are statistically evenly spaced
...there seemed to be an anomaly of 50...100µs
==> conclusion: this is due to the instrumentation code
- it largely caused by the EventLog, which was never meant
to be used in performance-critical code, and does hefty
heap allocations and string processing.
- moreover, there clearly is a cache-effect, adding a Factor 2
whenever some time passed since the last EventLog call
==> can be considered just an artifact of the test setup and
will have no impact on the scheduler
remark: this commit adds a lot of instrumentation code
To cover the visible behaviour of the work-Function,
we have to check an amalgam of timing delays and time differences.
This kind of test tends to be problematic, since timings are always
random and also machine dependent, and thus we need to produce pronounced effects
...to make that abundantly clear: we do not aim at precision timing,
rather the goal is to redistribute capacity currently not usable...
Basically we're telling the worker "nothing to do right now, sorry,
but check back in <timespan> because I may need you then"
Workers asking for the next task are classified as belonging
to some fraction of the free capacity, based on the distance
to the closest next Activity known to the scheduler
...to bring it more in line with all the other calls dealing with Activity*
...allows also to harmonise the ActivityLang::dispatchChain()
...and to compose the calls in Scheduler directly
NOTE: there is a twist: our string-formatting helper did not render
custom string conversions for objects passed as pointer. This was a
long standing problem, caused by ambiguous templates overloads;
now I've attempted to solve it one level more down, in util::StringConv.
This solution may turn out brittle, since we need to exclude any direct
string conversion, most notably the ones for C-Strings (const char*)
In case this solution turns out unsustainable, please feel free
to revert this API change, and return to passing Activity& in λ-post,
because in the end this is cosmetics.
- organise by principles rather than implementing a mechanism
- keep the first version simple yet flexible
- conduct empiric research under synthetic load
Basic scheme:
- tend for next
- classify free capacity
- scattered targeted wait
The signature for the »post« operation includes the ExecutionCtx itself,
which is obviously redundant, given that this operation is ''part of this context.''
However, for mock-implementation of the ExecutionCtx for unit testing,
the form of the implementation was deliberately kept unspecified, allowing
to use functor objects, which can be instrumented later. Yet a functor
stored as member has typically no access to the "this"-ptr...
The approach to provide the ExecutionCtx seems to work out well;
after some investigation I found a solution how to code a generic
signature-check for "any kind of function-like member"...
(the trick is to pass a pointer or member-pointer, which happens
to be syntactically the same and can be handled with our existing
function signature helper after some minor tweaks)
The Activity-Language can be defined by abstracting away
some crucial implementation functionality as part of an generic
»ExecutionCtx«, which in the end will be provided by the Scheduler.
But how actually?
We want to avoid unnecessary indirections, and ideally we also want
a concise formulation in-code. Here I'm exploring the idea to let the
scheduler itself provide the ExecutionCtx-operations as member functions,
employing some kind of "compile-time duck-typing"
This seems to work, but breaks the poor-man's preliminary "Concept" check...
Notably I wanted an entirely static and direct binding
to the internals of the Scheduler, which can be completely inlined.
The chosen solution also has the benefit of making the back-reference
to the Scheduler explicitly visible to the reader. This is relevant,
since the Config-Subobject is *copied* into each Worker instance.
The »Scheduler Service« will be assembled
from the components developed during the last months
- Layer-1
- Layer-2
- Activity-Language
- Block-Flow
- Work-Force
* the implementation logic of the Scheduler is essentially complete now
* all functionality necessary for the worker-function has been demonstrated
As next step, the »Scheduler Service« can be assembled from the two
Implementation Layers, the Activity-Language and the `BlockFlow` allocator
This should then be verified by a multi-threaded integration test...
This central operation sits at a crossroad and is used
- from external clients to fed new work to the Scheduler
- from Workers to engage into execution of the next Activity
- recursively from the execution of an Activity-chain
From these requirements the semantics of behaviour can be derived
regarding the GroomingToken and the result values, which indicate
when follow-up work should be processed
Ensure the GroomingToken mechanism indeed creates an
exclusive section protected against concurrent corruption:
Use a without / with-protection test and verify
the results are exact vs. grossly broken
T thread holding the »Grooming Token" is permitted to
manipulate scheduler internals and thus also to define new
activities; this logic is implemented as an Atomic lock,
based on the current thread's ID.
Notably both Layers are conceived as functionality providers;
only at Scheduler top-Level will functionality be combined with
external dependencies to create the actual service.
At first sight, this seems confusing; there is a time window,
there is sometimes a `when` parameter, and mostly a `now` parameter
is passed through the activation chain.
However, taking the operational semantics into account, the existing
definitions seem to be (mostly) adequate already: The scheduler is
assumed to activate a chain only ''when'' the defined start time is reached.
As follow-up to the rework of thread-handling, likewise also
the implementation base for locking was switched over from direct
usage of POSIX primitives to the portable wrappers available in
the C++ standard library. All usages have been reviewed and
modernised to prefer λ-functions where possible.
With this series of changes, the old threadpool implementation
and a lot of further low-level support facilities are not used
any more and can be dismantled. Due to the integration efforts
spurred by the »Playback Vertical Slice«, several questions of
architecture could be decided over the last months. The design
of the Scheduler and Engine turned out different than previously
anticipated; notably the Scheduler now covers a wider array of
functionality, including some asynchronous messaging. This has
ramifications for the organisation of work tasks and threads,
and leads to a more deterministic memory management. Resource
management will be done on a higher level, partially superseding
some of the concepts from the early phase of the Lumiera project.
This is Step-2 : change the API towards application
Notably all invocation variants to support member functions
or a reference to bool flags are retracted, since today a
λ-binding directly at usage site tends to be more readable.
The function names are harmonised with the C++ standard and
emergency shutdown in the Subsystem-Runner is rationalised.
The old thread-wrapper test is repurposed to demonstrate
the effectiveness of monitor based locking.
After the fundamental switch from POSIX to the C++14 wrappers
the existing implementation of the Monitor can now be drastically condensed,
removing several layers of indirection. Moreover, all signatures
shall be changed to blend in with the names and patterns established
by the C++ standard.
This is Step-1 : consolidate the Implementation.
(to ensure correctness, the existing API towards application code was retained)
While not directly related to the thread handling framework,
it seems indicated to clean-up this part of the application alongside.
For »everyday« locking concerns, an Object Monitor abstraction was built
several years ago and together with the thread-wrapper, both at that time
based on direct usage of POSIX. This changeset does a mere literal
replacement of the POSIX calls with the corresponding C++ wrappers
on the lowest level. The resulting code is needlessly indirect, yet
at API-level this change is totally a drop-in replacment.
The WorkForce (passive worker pool) has been coded just recently,
and -- in anticipation of this refactoring -- directly against std::thread
instead of using the old framework.
...the switch is straight-forward, using the default case
...add the ability to decorate the thread-IDs with a running counter
This solution is basically equivalent to the version implemented directly,
but uses the lifecycle-Hooks available through `ThreadHookable`
to structure the code and separate the concerns better.
This largely completes the switch to the new thread-wrapper..
**the old implementation is not referenced anymore**
...likewise using an detached »autonomous« Thread.
In this case however it is simpler to embed the complete use
of GtkLumiera into a lambda function, which ends with invoking
the terminationSignal functor. This way, the order of creation,
running the GTK-Loop and destroying the GUI is hard coded.
This, and the GUI thread prompted an further round of
design extensions and rework of the thread-wrapper.
Especially there is now support for self-managed threads,
which can be launched and operate completely detached from the
context used to start them. This resolves an occasional SEGFAULT
at shutdown. An alternative (admittedly much simpler) solution
would have been to create a fixed context in a static global
variable and to attach a regular thread wrapper from there,
managed through unique_ptr.
It seems obvious that the new solution is preferable,
since all the tricky technicalities are encapsulated now.
Add a complete demonstration for a setup akin to what we use
for the Session thread: a threaded component which manages itself
but also exposes an external interface, which is opened/closed alongside
...extract and improve the tuple-rewriting function
...improve instance tracking test dummy objects
...complete test coverage and verify proper memory handling
After quite some detours, with this take I'm finally able to
provide a stringent design to embody all the variants of thread start
encountered in practice in the Lumiera code base.
Especially the *self-managed* thread is now represented as a special-case
of a lifecycle-hook, and can be embodied into a builder front-end,
able to work with any client-provided thread-wrapper subclass.
to cover the identified use-cases a wide variety of functors
must be accepted and adapted appropriately. A special twist arises
from the fact that the complete thread-wrapper component stack works
without RTTI; a derived class can not access the thread-wrapper internals
while the policy component to handle those hooks can not directly downcast
to some derived user provided class. But obviously at usage site it
can be expected to access both realms from such a callback.
The solution is to detect the argument type of the given functor
and to build a two step path for a safe static cast.
after some further mulling over the design, it became clear that
a rather loose coupling to the actual usage scenario is preferrable.
Thus, instead of devising a fixed scheme how to reflect the thread state,
rather the usage can directly hook into some points in the thread lifecycle.
So this policy can be reduced to provide additional storage for functon objects.
...after resolving the fundamental design problems,
a policy mix-in can be defined now for a thread that deletes
its own wrapper at the end of the thread-function.
Such a setup would allow for »fire-and-forget« threads, but with
wrapper and ensuring safe allocations. The prominent use case
for such a setup would be the GUI-Thread.
So this finally solves the fundamental problem regarding a race on
initialisation of the thread-wrapper; it does *not* solve the same problem
for classes deriving from thread-wrapper, which renders this design questionable
altogether -- but this is another story.
In the end, this initialisation-race is rooted in the very nature of starting a thread;
it seems there are the two design alternatives:
- expose the thread-creation directly to user code (offloading the responsibility)
- offer building blocks which are inherently dangerous
this is a mere rearrangement of code (+lots of comments),
but helps to structure the overall construction better.
ThreadWrapper::launchThread() now does the actual work to build
the active std::thread object and assign it to the thread handle,
while buildLauncher is defined in the context of the constructors
and deals with wiring the functors and decaying/copying of arguments.
If we package all arguments together into a single tuple,
even including the member-function reference and the this-ptr
for the invokeThreadFunction(), which is the actual thread-functor,
then we can rely on std::make_from_tuple<T>(tuple), which implements
precisely the same hand-over via a std::index_sequence, as used by the
explicitly coded solution -- getting rid of some highly technical boilerplate
Concept study of the intended solution successful.
Can now transparently embed any conceivable functor
and an arbitrary argument sequence into a launcher-λ
Materialising into a std::tuple<decay_t<TYPES...>> did the trick.
Considering a solution to shift the actual launch of the new thread
from the initialiser list into the ctor body, to circumvent the possible
"undefined behaviour". This would also be prerequisite for defining
a self-managed variant of the thread-wrapper.
Alternative / Plan.B would be to abandon the idea of a self-contained
"thread" building block, instead relying on precise setup in the usage
context -- however, not willing to yield yet, since that would be exactly
what I wanted to avoid: having technicalities of thread start, argument
handover and failure detection intermingled with the business code.
On a close look, the wrapper design as pursued here
turns out to be prone to insidious data race problems.
This was true also for the existing solution, but becomes
more clear due to the precise definitions from the C++ standard.
This is a confusing situation, because these races typically do not
materialise in practice; due to the latency of the OS scheduler the
new thread starts invoking user code at least 100µs after the Wrapper
object is fully constructed (typically more like 500µs, which is a lot)
The standard case (lib::Thread) in its current form is correct, but borderline
to undefined behaviour, and any initialisation of members in a derived class
would be off limits (the thread-wrapper should not be used as baseclass,
rather as member)
...while reworking the application code, it became clear that
actually there are two further quite distinct variants of usage.
And while these could be implemented with some trickery based on
the Thread-wrapper defined thus far, it seems prudent better to
establish a safely confined explicit setup for these cases:
- a fire-and-forget-thread, which manages its own memory autonomously
- a thread with explicit lifecycle, with detectable not-running state
...hitting a roadblock however:
The existing Threads in the Lumiera application were effectively "detached"
But the C++14 framework requires us to keep the Thread-object alive
FamilyMember::allocateNextMember() was actually a post-increment,
so (different than with TypedCounter) here no correction is necessary
As an asside, WorkForce_test is sometimes unstable immediately after a build.
Seemingly a headstart of 50µs is not enough to compensate for scheduler leeway
The existing TypedCounter_test was excessively clever and convoluted,
yet failed to test the critical elements systematically. Indeed, two
bugs were hidden in synchronisation and instance access.
- build a new concurrent test from scratch, now using the threadBenchmark
function for the actual concurrent execution and just invoked a
random selected access to the counter repeatedly from a large number
of threads.
- rework the TypedContext and counter to use Atomics where applicable;
measurements indicate however that this has only negligible impact
on the amortised invocation times, which are around 60ns for single-threaded
access, yet can increase by factor 100 due to contention.
...these were already written envisionaging he new API,
so it's more or less a drop-in replacement.
- cant use vector anymore, since thread objects are move-only
- use ScopedCollection instead, which also has the benefit of
allocating the requires space up-front. Allow to deduce the
type parameter of the placed elements
... which became apparent after switching to the new Thread-wrapper implementation
... the reason is a bug in the Thread-Monitor (which will also be reworked soon)
While seemingly subtle, this is a ''deep change.''
Up to now, the project attempted to maintain two mutually disjoint
systems of error reporting: C-style error flags and C++ exceptions.
Most notably, an attempt was made to keep both error states synced.
During the recent integration efforts, this increasingly turned out
as an obstacle and source for insidious problems (like deadlocks).
As a resolve, hereby the relation of both systems is **clarified**:
* C-style error flags shall only be set and used by C code henceforth
* C++ exceptions can (optionally) be thrown by retrieving the C-style error code
* but the opposite is now ''discontinued'' : Exceptions ''do not set'' the error flag anymore
- the deadlock was caused by leaking error state through the C-style lumiera_error
- but the reason for the deadlock lies in the »convenience shortcut«
in the Object-Monitor scope guard for entering a wait state immediately.
This function undermines the unlocking-guarantee, when an exception
emanates from within the wait() function itself.
...this function was also ported to the new wrapper,
and can be verified now in a much more succinct way.
''This completes porting of the thread-wrapper''
Since the decision was taken to retain support for this special feature,
and even extend it to allow passing values, the additional functionality
should be documented in the test. Doing so also highlighted subtle problems
with argument binding.
Now the ThreadWrapper_test offers both
- a really simple usage example
- a comprehensive test to verify that actually the
thread-function is invoked the expected number of times
and that this invocations must have been parallelised
- it is not directly possible to provide a variadic join(args...),
due to overload resolution ambiguities
- as a remedy, simplify the invocation of stringify() for the typical cases,
and provide some frequently used shortcuts
