The Lumiera »Reference Platform« is now upgraded to Debian/Buster, which provides GCC-14 and Clang-20.
Thus the compiler support for C++20 language features seems solid enough, and C++23,
while still in ''experimental stage'' can be seen as a complement and addendum.
This changeset
* upgrades the compile switches for the build system
* provides all the necessary adjustments to keep the code base compilable
Notable changes:
* λ-capture by value now requires explicit qualification how to handle `this`
* comparison operators are now handled transparently by the core language,
largely obsoleting boost::operators. This change incurs several changes
to implicit handling rules and causes lots of ambiguities — which typically
pinpoint some long standing design issues, especially related to MObjects
and the ''time entities''. Most tweaks done here can be ''considered preliminary''
* unfortunately the upgraded standard ''fails'' to handle **tuple-like** entities
in a satisfactory way — rather an ''exposition-only'' concept is introduced,
which applies solely to some containers from the STL, thereby breaking some
very crucial code in the render entities, which was built upon the notion of
''tuple-like'' entities and the ''tuple protocol''. The solution is to
abandon the STL in this respect and **provide an alternative implementation**
of the `apply` function and related elements.
Initially we assumed that »handling time« is largely a matter of computation.
''Time is just a value'' and can be treated with integral arithmetic, some
modulus computations and pre-defined constants.
This turned out to be a mistake. Anything related to time is intricate,
and it is essential to distinguish different meanings of "times"
- time values related to an internal computation framework have
implementation-defined meaning and should be ''marked as opaque''
- temporal data can be ''mapped to a grid scale'' — an essential step
for media processing, which however incurs information loss
- externally relevant time specifications are represented symbolically,
by translation into a ''Time Code''
Drawing from these insights, a framework for time handling has been established;
building in part on the low-level function style base implementation.
Exposing this base implementation as a C-library however is considered
dangerous, as it may lure into ''ad hoc'' computations, which are a major
source of inconsistencies and notorious defects in many media applications.
Indeed — this change set is kind of sad.
Because I still admire the design of the GAVL library,
and would love to use it for processing of raw video.
However, up to now, we never got to the point of actually
doing so. For the future, I am not sure if there remains
room to rely on lib-GAVL, since FFmpeg roughly covers
a similar ground (and a lot beyond that). And providing
a plug-in for FFmpeg is unavoidable, practically speaking.
So I still retain the nominal dependency on lib-GAVL
in the Build system (since it is still packaged in Debian).
But it is pointless to rely on this library just for an
external type-def `gavl_time_t`. We owe much to this
inspiration, but it can be expected that we'll wrap
these raw time-values into a dedicated marker type
soon, and we certainly won't be exposing any C-style
interface for time calculations in future, since
we do not want anyone to side-step the Lumiera
time handling framework in favour of working
„just with plain numbers“
NOTE: lib-GAVL hompage has moved to Github:
https://github.com/bplaum/gavl
- remove obsolete configuration settings
- walk through all settings according to the documentation
https://www.doxygen.nl/manual/config.html
- now try to use the new feature to rely on Clang for C++ parsing
- walk through the doxygen-warnings.txt and fix some obvious misspellings
and structural problems in the documentation comments.
With Debian-Trixie, we are now using Doxygen 1.9.8 —
which produces massively better results in various fine points.
However, there are still problems with automatic cross links,
especially from implementation to the corresponding test classes.
- conversion from pointer to bool now counts as ''narrowing conversion''
- constructor names must not include template arguments (enforced with C++20)
- better use std::array for some dummy test code
Several further warnings are due to known obsoleted or questionable constructs
and were left as-is (e.g. for ScopedHolder) or just commented for later referral
This picks up the efforts towards a »Test Ontology« from end November:
d80966c1f
The `TestRandOntology` is intended as a playground to gradually find out
how to maintain bindings processing functionality provided by a specific Library
and thus related to a ''Domain Ontology''
Remark: generating symbolic specs might seem like a mere test exercise, yet is in fact
quite crucial, since the node-identity is based on such a spec, which must be ''semantically correct,''
otherwise caching and especially cache invalidation will be broken.
Yesss .... in Lumiera naming and cache invalidation are linked directly ;-)
As follow-up from the preceding refactorings,
it is now possible to drastically simplify several type signatures.
Generally speaking, iterator pipelines can now pass-through the result type,
and thus it is no longer necessary to handle this result type explicitly
In the case of `IterStateWrapper`, the result type parameter was retained,
but moved to the second position and defaulted; sometimes it can be relevant
to force a specific type; this is especially useful when defining an
`iterator` and a `const_iterator` based on the same »state-core«
* Lumiera source code always was copyrighted by individual contributors
* there is no entity "Lumiera.org" which holds any copyrights
* Lumiera source code is provided under the GPL Version 2+
== Explanations ==
Lumiera as a whole is distributed under Copyleft, GNU General Public License Version 2 or above.
For this to become legally effective, the ''File COPYING in the root directory is sufficient.''
The licensing header in each file is not strictly necessary, yet considered good practice;
attaching a licence notice increases the likeliness that this information is retained
in case someone extracts individual code files. However, it is not by the presence of some
text, that legally binding licensing terms become effective; rather the fact matters that a
given piece of code was provably copyrighted and published under a license. Even reformatting
the code, renaming some variables or deleting parts of the code will not alter this legal
situation, but rather creates a derivative work, which is likewise covered by the GPL!
The most relevant information in the file header is the notice regarding the
time of the first individual copyright claim. By virtue of this initial copyright,
the first author is entitled to choose the terms of licensing. All further
modifications are permitted and covered by the License. The specific wording
or format of the copyright header is not legally relevant, as long as the
intention to publish under the GPL remains clear. The extended wording was
based on a recommendation by the FSF. It can be shortened, because the full terms
of the license are provided alongside the distribution, in the file COPYING.
⚠ __This is a problematic decision__
It temporarily **breaks compatibility with 32bit** until this issue is resolved.
== Explanation ==
Lumiera relies on a mix of the Standard library and Lib-Boost for calculation of hash values.
Before C++11, the Standard did not support and hashtable implementation; meanwhile, we
got several hash based containers in the STL and a framework for hashes,
which unfortunately is incomplete and cumbersome to use.
The C++ Committee has spend endless discussions and was not able to settle
on a convincing solution without major drawbacks regarding one aspect or the other.
This situation is problematic, since Lumiera relies heavily on the technique
of building stable systematic identifiers based on chained hash values.
It is thus essential to use a strong, reliable and portable hash function.
But unfortunately...
* the standard-fallback solution is known to be weak.
* Lib-Boost automatically uses stronger implementations for 64bit systems
* this implies that Hash-Values **are non-portable**
As the Lumiera project currently has no developer time to expend on such a
difficult and deep topic of fundamental research, today I decided to go down
the path of least resistance and **effectively abandon any system
that can not compile and use the 64bit `hash_combine` implementation.
This changeset extracts code from Lib-Boost 1.67 and adds a static assertion
to **break compilation** on non-64bit-platforms (whatever this means)
Last summer, I already identified a problmatic aspect
which could cause the Scheduler to fall idle without further notice:
5b62438eb4
Basically this situation should raise a **Scheduler-Emergency**,
but the only location where it can be easily detected is way down
in the implementation and has currently no clean way of signalling.
Moreover, how to handle a Scheduler-Emergency is likewise an open
question, an will in turn require even more cross-cutting notifications
and trigger actions somewhere at Render-Engine top-level.
By marking the location where this problem could be detected,
inadvertently I broke the SchedulerCommutator_test, which of course
must execute precisely this logic and check for the proper result.
Yet the problem as such is tricky and possibly far-reaching;
notably when processing long-running render jobs will reliably trigger
this situation — unless we establish additional dedicated control-logic
especially to cope with long-running jobs (opened #1382 for this topic)
__Bottom line__: we are far from addressing any of these issues right now,
and thus I reduced that failure to a warning message, so that at least
`SchedulerCommutator_test` passes again (it's not actually a defect there)
We use the memory address to detect reference to ''the same language object.''
While primarily a testing tool, this predicate is also used in the
core application at places, especially to prevent self-assignment
and to handle custom allocations.
It turns out that actually we need two flavours for convenient usage
- `isSameObject` uses strict comparison of address and accepts only references
- `isSameAdr` can also accept pointers and even void*, but will dereference pointers
This leads to some further improvements of helper utilities related to memory addresses...
...to the base-class of all tests
* `seedRand()` shall be invoked by every test using randomisation
* it will draw a new seed for the implicit default-PRNG
* it will document this seed value
* but when a seed was given via cmdline, it will inject that instead
* `makeRandGen()` will create a new dedicated generator instance,
attached (by seeding) to the current default-PRNG
It is not clear yet how to pass the actual `SeedNucleus`, which
for obvious reasons must be maintained by the `test::Suite`
the `BreakingPoint` tool conducts a binary search to find the ''stress factor''
where a given schedule breaks. There are some known deviations related to the
measurement setup, which unfortunately impact the interpretation of the
''stress factor'' scale. Earlier, an attempt was made, to watch those factors
empirically and work a ''form factor'' into the ''effective stress factor''
used to guide this measurement method.
Closer investigation with extended and elastic load patters now revealed
a strong tendency of the Scheduler to scale down the work resources when not
fully loaded. This may be mistaken by the above mentioned adjustments as a sign
of a structural limiation of the possible concurrency.
Thus, as a mitigation, those adjustments are now only performed at the
beginning of the measurement series, and also only when the stress factor
is high (implying that the scheduler is actually overloaded and thus has
no incentive for scaling down).
These observations indicate that the »Breaking Point« search must be taken
with a grain of salt: Especially when the test load does ''not'' contain
a high degree of inter dependencies, it will be ''stretched elastically''
rather than outright broken. And under such circumstances, this measurement
actually gauges the Scheduler's ability to comply to an established
load and computation goal.
...well — more of a logical contradiction, not so much a bug.
The underlying problematic situation arises when meanwhile the
Extent storage has been expanded, and especially the active slots
are in »wrapped state«. In this case, the newly allocated extents
must be rotated in, which invalidates existing index numbers.
This problem was amended by exploting a chaching mechanism, allowing
to re-attach and validate an index position still stored in an old
iterator; especially this can happen when attempting to attach a
follow-up dependency onto a job planned earlier, but not yet scheduled.
The problem here was an assertion failure, which was triggered with a
high probability; the fix for the problem detailed above used the yield()
function, while it actually was only interested in retrieving the
Extent's address to probe if the extent matches an known storage location.
The solution is to provide a dedicated function for this check, which
can then skip the sanity check (because in this case we do not want
to use the Extent, and thus can touch obsoleted/inactive Extents
without problem)
In the end, I decided that it ''is to early to decide anything'' in this respect...
The actual situation encountered is a **Catch-22**:
* in its current form, the »Tick« handler detects compulsory jobs beyond deadline
* since such a Job ''must not be touched anymore,'' there is no way scheduling can proceed
* so this would constitute a ''Scheduler Emergency''
All fine — just the »Tick« handler ''itself is a compulsory job'' — and being a job, it can well be driven beyond its deadline. In fact this situation was encountered as part of stress testing.
Several mitigations or real solutions are conceivable, but in the end,
too little is known yet regarding the integration of the scheduler within the Engine
Thus I'll marked the problematic location and opened #1362
Investigate the behaviour over a wider range of job loads,
job count and worker pool sizes. Seemingly the processing
can not fully utilise the available worker pool capacity.
By inspection of trace-dumps, one impeding mechanism could
be identified: the »stickiness« of the contention mitigation.
Whenever a worker encounters repeated contention, it steps up
and adds more and more wait cycles to remove pressure from the
schedule coordination. As such this is fine and prevents further
degradation of performance by repeated atomic synchronisation.
However, this throttling was kept up needlessly after further
successful work-pulls. Since job times of several milliseconds
can be expected on average in media processing, such a long
retention would spread a performance degradation over a duration
of several frames. Thus, the scheme for step-down was changed
to decrease the throttling by a power series rather than just
documenting the level.
Whenever a class defines a single-arg templated constructor,
there is danger to shadow the auto-generated copy operations,
leading to insidious failures.
Some months ago, I did the ''obvious'' and added a tiny helper,
allowing to mask out the dangerous case when the ''single argument''
is actually the class itself (meaning, it is a copy invocation and
not meant to go through this templated ctor...
As this already turned out as tremendously helpful, I now extended
this helper to also cover cases where the problematic constructor
accepts variadic arguments, which is quite common with builder-helpers
Relying on random numbers for verification and measurements is known to be problematic.
At some point we are bound to control the seed values -- and in the actual
application usage we want to record sequence seeding in the event log.
Some initial thoughts regarding this intricate topic.
* a low-ceremony drop-in replacement for rand() is required
* we want the ability to pick-up and control each and every usage eventually
* however, some usages explicitly require true randomness
* the ability to use separate streams of random-number generation is desirable
- better use a Test-Chain-Load without any dependencies
- schedule all at once
- employ instrumentation
- use the inner »overall time« as dependent result variable
The timing results now show an almost perfect linear dependency.
Also the inner overall time seems to omit the setup and tear-down time.
But other observed values (notably the avgConcurrency) do not line up
This is just another (obvious) degree of freedom, which could be
interesting to explore in stress testing, while probably not of much
relevance in practice (if a job is expected to become runable earlier,
in can as well be just scheduled earlier).
Some experimentation shows that the timing measurements exhibit more
fluctuations, but also slightly better times when pressure is low, which
is pretty much what I'd expect. When raising pressure, the average
times converge towards the same time range as observed with time bound
propagation.
Note that enabling this variation requires to wire a boolean switch
over various layers of abstraction; arguably this is an unnecessary
complexity and could be retracted once the »experimentation phase«
is over.
This completes the preparation of a Scheduler Stress-Test setup.
...watching those dumps on the example Graph with excessive dependencies
made blatantly clear that we're dispatching a lot of unnecessary jobs,
since the actual continuation is /always/ triggered by the dependency-NOTIFY.
Before the rework of NOTIFY-Handling, this was rather obscured, but now,
since the NOTIFY trigger itself is also dispatched by the Scheduler,
it ''must be this job'' which actually continues the calculation, since
the main job ''can not pass the gate'' before the dependency notification
arrives.
Thus I've now added a variation to the test setup where all these duplicate
jobs are simply omitted. And, as expected, the computation runs faster
and with less signs of contention. Together with the other additional
parameter (the base expense) we might now actually be able to narrow down
on the observation of a ''expense socket'', which can then be
attributed to something like an ''inherent scheduler overhead''
In-depth investigation and reasoning highlighted another problem,
which could lead to memory corruption in rare cases; in the end
I found a solution by caching the ''address'' of the current Epoch
and re-validating this address on each Epoch-overflow.
After some difficulties getting any reliable measurement for a Release-build,
it turned out that this solution even ''improves performance by 22%''
Remark-1: the static blockFlow::Config prevents simple measurements by
just recompiling one translation unit; it is necessary to build the
relevant parts of Vault-layer with optimisation to get reliable numbers
Remark-2: performing a full non-DEBUG build highlighted two missing
header-inclusions to allow for the necessary template specialisations.
...discovered by during investigation of latest Scheduler failures.
The root of the problems is that block overflow can potentially trigger
expansion of the allocation pool. Under some circumstances, this on-the fly
allocation requires a rotation of index slots, thereby invalidating
existing iterators.
While such behaviour is not uncommon with storage data structures (see std::vector),
in this case it turns out problematic because due to performance considerations,
a usage pattern emerged which exploits re-using existing storage »Slots« with known
deadline. This optimisation seems to have significant leverage on the
planning jobs, which happen to allocated and arrange a whole strike of
Activities with similar deadlines.
One of these problem situations can easily be fixed, since it is triggered
through the iterator itself, using a delegate function to request a storage expansion,
at which point the iterator is able to re-link and fix its internal index.
This solution also has no tangible performance implications in optimised code.
Unfortunately there remains one obscure corner case where such an pool expansion
could also have invalidated other iterators, which are then used later to
attach dependency relations; even a partial fix for that problem seems
to cause considerable performance cost of about -14% in optimised code.
This amounts to a rather massive refactoring, prompted by the enduring problems
observed when pressing the scheduler. All the various glitches and (fixed) crashes
are related to the way how planning-jobs enter the schedule items,
which is also closely tied to the difficulties getting the locking
for planning-jobs correct.
The solution pursued hereby is to reorder the main avenues into the
scheduler implementation. There is now a streamlined main entrance,
which **always** enqueues only, allowing to omit most checks and
coordination. On the other hand, the complete coordination and dispatch
of the work capacity is now shifted down into the SchedulerCommutator,
thereby linking all coordination and access control close together
into a single implementation facility.
If this works out as intended
- several repeated checks on the Grooming-Token could be omitted (performance)
- the planning-job would no longer be able to loose / drop the Token,
thereby running enforcedly single-threaded (as was the original intention)
- since all planning effectively originates from planning-jobs, this
would allow to omit many safety barriers and complexities at the
scheduler entrance avenue, since now all entries just go into the queue.
WIP: tests pass compiler, but must be adapted / reworked
...whenever the planning falls behind schedule, it can happen that
the planner-worker immediately dispatches its own jobs; while the calculation
is broken anyway in this situation, especially this call scheme leads to
dropping the Grooming-Token prior to the calculation dispatched directly.
Since the dependency relation can only be established after creating
both predecessor and successor schedules, the corresponding allocation
of the NOTIFY-Activity is not protected against concurrent access,
which probably leads to the assertion failure due to corruption of
the allocator's internal data structures...
...causing the system to freeze due to excess memory allocation.
Fortunately it turned out this was not an error in the Scheduler core
or memory manager, but rather a sloppiness in the test scaffolding.
However, this incident highlights that the memory manager lacks some
sanity checks to prevent outright nonsensical allocation requests.
Moreover it became clear again that the allocation happens ''already before''
entering the Scheduler — and thus the existing sanity check comes too late.
Now I've used the same reasoning also for additional checks in the allocator,
limiting the Epoch increment to 3000 and the total memory allocation to 8GiB
Talking of Gibitbytes...
indeed we could use a shorthand notation for that purpose...
The scheduler implementation uses a randomised redistribution of
work capacity, taking into account the current ''scale'' of next pending event.
While this works surprisingly well overall, sometimes, in very tight and dense scheules
the workers seem to be spread somewhat too arbitrarily. Thus, if the scheduler
is working through a zone with several events as close as 1ms, often it takes
up to 3ms for another worker to show up.
With this change, the scattering range in the ''near zone'' (50µs ... 5ms)
is made dynamic, and now flexibly depends on current head time.
The closer the next event, the more tightly focussed will be the
capacity redistribution, if capacity becomes available just some 100µs
ahead of next demand, it is no longer „sent away“, but rather relocated
by roughly the same distance behind the next event.
while my basic assessment is still that contention will not play a significant
role given the expected real world usage scenario — when testing with
tighter schedule and rather short jobs (500µs), some phases of massive contention
can be observed, leading to significant slow-down of the test.
The major problem seems to be that extended phases of contention will
effectively cause several workers to remain in an active spinning-loop for
multiple microseconds, while also permanently reading the atomic lock.
Thus an adaptive scheme is introduced: after some repeated contention events,
workers now throttle down by themselves, with polling delays increased
with exponential stepping up to 2ms. This turns out to be surprisingly
effective and completely removes any observed delays in the test setup.
...turns out to be a secondary problem (but must be fixed non the less).
Since the planning-job no longer drops the token now, the workers
have to wait; since they are waiting actively and contending on the token,
a significant slowdown can happen.
Sometimes the planning job gets behind its own scheduler and thus
enters dispatch, in which case it drops the GoomingToken, causing
an Assertion failure on return.
The **actual problem** however is the slowdown due to active spinning
Turns out that we need to implemented fine grained and explicit handling logic
to ensure that Activity planning only ever happens protected by the Grooming-Token.
This is in accordance to the original design, which dictates that all management tasks
must be done in »management mode«, which can only be entered by a single thread at a time.
The underlying assumption is that the effort for management work is dwarfed in comparison
to any media calculation work.
However, in
5c6354882d
...I discovered an insidious border condition, an in an attempt to fix it,
I broke that fundamental assumpton. The problem arises from the fact that we
do want to expose a *public API* of the Scheduler. Even while this is only used
to ''seed'' a calculation stream, because any further planning- and management work
will be performed by the workers themselves (this is a design decision, we do not
employ a "scheduler thread")
Anyway, since the Scheduler API ''is'' public, ''someone from the outside'' could
invoke those functions, and — unaware of any Scheduler internals — will
automatically acquire the Grooming-Token, yet never release it,
leading to deadlock.
So we need a dedicated solution, which is hereby implemented as a
scoped guard: in the standard case, the caller is a management-job and
thus already holds the token (and nothing must be done). But in the
rare case of an »outsider«, this guard now ''transparently'' acquires
the token (possibly with a blocking wait) and ''drops it when leaving scope''
In the course of the last refactorings, a slight change in processing
order was introduced, which turned out to improve parallelisation considerably.
- Some further implementation logic can be relegated into the ActivationEvent
- the handling of start times now also incldues a check for sake of symmetry
- document the semantics change: λ-post no longer dispatches directly
...this feature seems to be no longer necessary now;
leaving the actual implementation in-code for the time being,
but removed it from the public access API.
The rework from yesterday turned out to be effective ... unfortunately
a bit to much: since now late follow-up notifications take precedence,
a single worker tends to process the complete chain depth-first, because
the first chain will be followed and processed, even before the worker
was able to post the tasks for the other branches. Thus this single
worker is the only one to get a chance to proceed.
After some consideration, I am now leaning towards a fundamental change,
instead of just fixing some unfavourable behaviour pattern: while the
language semantics remains the same, the scheduler should no longer
directly dispatch into the next chain **from λ-post**. That is, whenever
a POST / NOTIFY is issued from the Activity-chain, the scheduler goes
through prioritisation.
This has further ramifications: we do not need a self-inhibition mechanism
any more (since now NOTIFY picks up the schedule time of the target).
With these changes, processing seems to proceed more smoothly,
albeit still with lots of contention on the Grooming token,
at least in the example structure tested here.
While the recent refactoring...
206c67cc
...was a step into the right direction, it pushed too hard,
overlooking the requirement to protect the scheduler contents
and thus all of the Activity-chains against concurrent modification.
Moreover, the recent solution still seems not quite orthogonal.
Thus the handling of notifications was thoroughly reworked:
- the explicit "double-dispatch" was removed, since actual usage
of the language indicates that we only need notifications to
Gate (and Hook), but not to any other conceivable Activity.
- thus it seems unnecessary to turn "notification" into some kind
of secondary work mode. Rather, it is folded as special case
into the regular dispatch.
This leads to new processing rules:
- a POST goes into λ-post (obviously... that's its meaning)
- a NOTIFY now passes its *target* into λ-post
- λ-post invokes ''dispatch''
- and **dispatching a Gate now implies to notify the Gate**
This greatly simplifies the »state machine« in the Activity-Language,
but also incurs some limitations (which seems adequate, since it is
now clear that we do not ''schedule'' or ''dispatch'' arbitrary
Activities — rather we'll do this only with POST and NOTIFY,
and all further processing happens by passing activation
along the chain, without involving the Scheduler)
use a feature of the Activity-Language prepared for this purpose:
self-Inhibition of the Chain. This prevents a prerequisite-NOTIFY
to trigger a complete chain of available tasks, before these tasks
have actually reached their nominal scheduling time.
This has the effect to align the computations much more strictly
with the defined schedule
The main (test) thread is kept in a blocking wait until the
planned schedule is completed. If however the schedule overruns,
the wake-up job could just be triggered prematurely.
This can easily be prevented by adding a dependency from the last
computation job to the wake-up job. If the computation somehow
flounders, the SAFETY_TIMEOUT (5s) will eventually raise
an exception to let the test fail cleanly (shutting down
the Scheduler automatically)
...it seems impossible to solve this conundrum other than by
opening a path to override a contextual deadline setting from
within the core Activity-Language logic.
This will be used in two cases
- when processing a explicitly coded POST (using deadline from the POST)
- after successfully opening a Gate by NOTIFY (using deadline from Gate)
All other cases can now supply Time::NEVER, thereby indicating that
the processing layer shall use contextual information (intersection
of the time intervals)
...this is an interesting test failure, which highlights inconsistencies
with handling of deadlines when processing follow-up from NOTIFY-triggers
There was also some fuzziness related to the ''meaning'' of λ-post,
leading to at least one superfluous POST invocation for each propagation;
fixing this does not solve the problem yet removes unnecessary overhead
and lock-contention
this bug was there since the first draft, yet was covered
by another bug with the start-up logic.
And this latter one was fixed recently...
fa8622805
As a result, even when the COMPUTATION_CAPACITY is set to 0
still a single worker boots up (which should not be the case)
Solution: we do not need to "safeguard" against rounding errors,
since this is an internal implementation function, it is assumed
that the caller knows about its limitations...
This partially reverts commit 72f11549e6.
"Chain-Load: Scheduler instrumentation for observation"
Hint: revert this changeset to re-introduce the print statements for diagnostic
This is a trick to get much better scheduling and timing guesses.
Instead of targeting a specific level, rather a fixed number of nodes
is processed in each chunk, yet still always processing complete levels.
The final level number to expect can be retrieved from the chain-load graph.
With this refactoring, we can now schedule a wake-up job precisely
after the expected completion of the last level
Invent a special JobFunctor...
- can be created / bound from a λ
- self-manages its storage on the heap
- can be invoked once, then discards itself
Intention is to pass such one-time actions to the Scheduler
to cause some ad-hoc transitions tied to curren circumstances;
a notable example will be the callback after load-test completion.
In the first draft version, a blocked Gate was handled by
»polling« the Gate regularly by scheduling a re-invocation
repeatedly into the future (by a stepping defined through
ExecutionCtx::getWaitDelay()).
Yet the further development of the Activity-Language indicates
that the ''Notification mechanism'' is sufficient to handle all
foreseeable aspects of dependency management. Consequently this
''Gate poling is no longer necessary,'' since on Notification
the Gate is automatically checked and the activation impulse
is immediately passed on; thus the re-scheduled check would
never get an opportunity actually to trigger the Gate; such
an active polling would only be necessary if the count down
latch in the Gate is changed by "external forces".
Moreover, the first Scheduler integration tests with TestChainLoad
indicate that the rescheduled polling can create a considerable
additional load when longer dependency chains miss one early
prerequisite, and this additional load (albeit processed
comparatively fast by the Scheduler) will be shifted along
needlessly for quite some time, until all of the activities
from the failed chain have passed their deadline. And what
is even more concerning, these useless checks have a tendency
to miss-focus the capacity management, as it seems there is
much work to do in a near horizon, which in fact may not be
the case altogether.
Thus the Gate implementation is now *changed to just SKIP*
when blocked. This helped to drastically improve the behaviour
of the Scheduler immediately after start-up -- further observation
indicated another adjustment: the first Tick-duty-cycle is now
shortened, because (after the additional "noise" from gate-rescheduling
was removed), the newly scaled-up work capacity has the tendency
to focus in the time horizon directly behind the first jobs added
to the timeline, which typically is now the first »Tick«.
ð¡ this leads to a recommendation, to arrange the first job-planning
chunk in such a way that the first actual work jobs appear in the area
between 5ms and 10ms after triggering the Scheduler start-up.Scheduler¡
Introducing a fixed pre-delay on each new Calc-Streem seemed like an obvious remedy,
yet on closer investigation it turned out that the start-up logic as such was contradictory,
which was only uncovered by some rather special schedule patterns.
After fixing the logic deficiencies, Scheduler starts up as intended
and the probabilistic capacity-control seems to work as designed.
Thus no need to introduce an artificial delay at begin, even while
this implies that typically the first round of job-planning will be
performed synchronous, in the invoking thread (which may be surprising,
but is completely within the limits of the architecture; we do not
employ specifically configured threads and planning should be done
in short chunks, thus the first chunk can well be done by the caller)
The first complete integration test with Chain-Load
highlighted some difficulties with the overall load regulation:
- it works well in the standard case (but is possibly to eager to scale up)
- the scale-up sometimes needs several cycles to get "off the ground"
- when the first job is dispatched immediately instead of going
through the queue, the scheduler fails to boot up
- prime diagnostics with the first time invocation
- print timings relative to this first invocation
- DUMP output to watch the crucial scheduling operations
... so this (finally) is the missing cornerstone
... traverse the calculation graph and generate render jobs
... provide a chunk-wise pre-planning of the next batch
... use a future to block the (test) thread until completed
