While the recent refactoring...
206c67cc
...was a step into the right direction, it pushed too hard,
overlooking the requirement to protect the scheduler contents
and thus all of the Activity-chains against concurrent modification.
Moreover, the recent solution still seems not quite orthogonal.
Thus the handling of notifications was thoroughly reworked:
- the explicit "double-dispatch" was removed, since actual usage
of the language indicates that we only need notifications to
Gate (and Hook), but not to any other conceivable Activity.
- thus it seems unnecessary to turn "notification" into some kind
of secondary work mode. Rather, it is folded as special case
into the regular dispatch.
This leads to new processing rules:
- a POST goes into λ-post (obviously... that's its meaning)
- a NOTIFY now passes its *target* into λ-post
- λ-post invokes ''dispatch''
- and **dispatching a Gate now implies to notify the Gate**
This greatly simplifies the »state machine« in the Activity-Language,
but also incurs some limitations (which seems adequate, since it is
now clear that we do not ''schedule'' or ''dispatch'' arbitrary
Activities — rather we'll do this only with POST and NOTIFY,
and all further processing happens by passing activation
along the chain, without involving the Scheduler)
...it seems impossible to solve this conundrum other than by
opening a path to override a contextual deadline setting from
within the core Activity-Language logic.
This will be used in two cases
- when processing a explicitly coded POST (using deadline from the POST)
- after successfully opening a Gate by NOTIFY (using deadline from Gate)
All other cases can now supply Time::NEVER, thereby indicating that
the processing layer shall use contextual information (intersection
of the time intervals)
Invent a special JobFunctor...
- can be created / bound from a λ
- self-manages its storage on the heap
- can be invoked once, then discards itself
Intention is to pass such one-time actions to the Scheduler
to cause some ad-hoc transitions tied to curren circumstances;
a notable example will be the callback after load-test completion.
In the first draft version, a blocked Gate was handled by
»polling« the Gate regularly by scheduling a re-invocation
repeatedly into the future (by a stepping defined through
ExecutionCtx::getWaitDelay()).
Yet the further development of the Activity-Language indicates
that the ''Notification mechanism'' is sufficient to handle all
foreseeable aspects of dependency management. Consequently this
''Gate poling is no longer necessary,'' since on Notification
the Gate is automatically checked and the activation impulse
is immediately passed on; thus the re-scheduled check would
never get an opportunity actually to trigger the Gate; such
an active polling would only be necessary if the count down
latch in the Gate is changed by "external forces".
Moreover, the first Scheduler integration tests with TestChainLoad
indicate that the rescheduled polling can create a considerable
additional load when longer dependency chains miss one early
prerequisite, and this additional load (albeit processed
comparatively fast by the Scheduler) will be shifted along
needlessly for quite some time, until all of the activities
from the failed chain have passed their deadline. And what
is even more concerning, these useless checks have a tendency
to miss-focus the capacity management, as it seems there is
much work to do in a near horizon, which in fact may not be
the case altogether.
Thus the Gate implementation is now *changed to just SKIP*
when blocked. This helped to drastically improve the behaviour
of the Scheduler immediately after start-up -- further observation
indicated another adjustment: the first Tick-duty-cycle is now
shortened, because (after the additional "noise" from gate-rescheduling
was removed), the newly scaled-up work capacity has the tendency
to focus in the time horizon directly behind the first jobs added
to the timeline, which typically is now the first »Tick«.
ð¡ this leads to a recommendation, to arrange the first job-planning
chunk in such a way that the first actual work jobs appear in the area
between 5ms and 10ms after triggering the Scheduler start-up.Scheduler¡
...there seemed to be an anomaly of 50...100µs
==> conclusion: this is due to the instrumentation code
- it largely caused by the EventLog, which was never meant
to be used in performance-critical code, and does hefty
heap allocations and string processing.
- moreover, there clearly is a cache-effect, adding a Factor 2
whenever some time passed since the last EventLog call
==> can be considered just an artifact of the test setup and
will have no impact on the scheduler
remark: this commit adds a lot of instrumentation code
...to bring it more in line with all the other calls dealing with Activity*
...allows also to harmonise the ActivityLang::dispatchChain()
...and to compose the calls in Scheduler directly
NOTE: there is a twist: our string-formatting helper did not render
custom string conversions for objects passed as pointer. This was a
long standing problem, caused by ambiguous templates overloads;
now I've attempted to solve it one level more down, in util::StringConv.
This solution may turn out brittle, since we need to exclude any direct
string conversion, most notably the ones for C-Strings (const char*)
In case this solution turns out unsustainable, please feel free
to revert this API change, and return to passing Activity& in λ-post,
because in the end this is cosmetics.
The Activity-Language can be defined by abstracting away
some crucial implementation functionality as part of an generic
»ExecutionCtx«, which in the end will be provided by the Scheduler.
But how actually?
We want to avoid unnecessary indirections, and ideally we also want
a concise formulation in-code. Here I'm exploring the idea to let the
scheduler itself provide the ExecutionCtx-operations as member functions,
employing some kind of "compile-time duck-typing"
This seems to work, but breaks the poor-man's preliminary "Concept" check...
Up to now, the DiagnosticFun mock in ActivityDetector only
created an EventLog entry on invocation and was able to retunr
a canned result value. Yet for the job invocation scenario test,
it would be desirable to hook-in a λ with a fake implementation
into the ExecutionContext. As a further convenience, the
return value is now default initialised, instead of being
marked as uninitialised until invocation of "returning(val)"
...seems to work, but not really happy with the test setup,
since in real usage the post()-calls would dispatch, while here,
using the ActivityDetector, these calls just log invoation,
and thus the activation is not passed on
...regarding the kind of activity (the verb),
and also for some special case access of payload data;
deliberately asserting the correct verb, but no mandatory check,
since this whole Activity-Language is conceived as cohesive
and essentially sealed (not meant to be extended)
...to show in test that indeed the actual time is retrieved on each activation,
we can assign a λ -- which is rigged to increase the time on each access
It is not sufficient just to pass this "current time" as parameter
into the ActivityLang::dispatchChain(), since some Activities within
this chain will essentially be long-running (think rendering); thus
we need a real callback from within the chain. The obvious solution
is to make this part of the Execution Context, which is an abstraction
of the scheduler environment anyway
Solved by special treatment of a notification, which happens
to decrement the latch to zero: in this case, the chain is
dispatched, but also the Gate is locked permanently to block
any further activations scheduled or forwareded otherwise
TODO: while correct as implemented, the handling of the
notification seems questionable, since re-scheduling the chain immediately
may lead to multiple invocations of the chain, since it might have been "spinned"
and thus re-scheduled already, and we have no way to find out about that
...attempt to get this intricate state machine sorted out
Notification turned out quite tricky, since it may emanate
from a concurrently executed phase and we try to avoid having
to protect the gate directly with a lock; rather we re-dispatch
the notification through the queue, which indirectly also ensures
that the worker de-queuing the NOTIFY-Activity operates in
management mode (single threaded, holding the GroomingToken)
Decision how to handle a failed Gate-check
- spin forward (re-scheduler) by some time amount
- this spin-offset parameter is retrieved from the Execution Context
- thus it will be some kind of engine parameter
With these determinations and the framework for the Execution Context
it is now possible to code up the logic for Gate check, which in turn
can then be verified by the watchGate diagnostics
due to technical limitations this requires to wire the adaptor
as replacement for the subject Activity, so that it can capture
and log the activation, and then pass it on to its watched subject
requires to supplement EventLog matching primitives
to pick and verify a specific positional argument.
Moreover, it is more or less arbitrary which job invocation parameters
are unpacked and exposed for verification; we'll have to see what is
actually required for writing tests...
doing so would contradict the fundamental architecture,
all kinds of failures and timeouts need to be handled within
Scheduler-Layer-2 rather.
Jobs are never aborted, nor do they need to know if and when they are invoked
Testcase (detect function invocation) passes now as expected
Some Library / Framework changes
- rename event-log-test.cpp
- allow the ExpectString also to work with concatenated expectation strings
Remark: there was a warning in the comment in event-log.hpp,
pointing out that negative assertions are shallow.
However, after the rework in 9/2018 (commit: d923138d1)
...this should no longer be true, since we perform proper backtracking,
leading to an exhaustive search.
ActivityMatch inherits privately from the EventMatch object,
and is thus able to delegate relevant matching queries, but
also to provide high-level special matchers.
This new design resolves the ambiguity regarding function arguments.
Moreover, we can now record the current sequence-Number as *attribute*
in the respective log record (this is the benefit of using structured
log entries instead of just a textual log), thereby avoiding the various
pitfalls with explicit bracketing sequence-number log entries
bottom line: this reworked design seems to be a better fit,
even while technically the implementation with the wrapped matcher
is somewhat ugly...
The EventLog seems to provide all the building blocks, but we need
some higher level special matchers (and maybe we also want to hide
some of the basic EventLog matchers). A soulution might be to wrap
the EventMatcher and delegate all follow-up builder calls.
This seems adequate, since the EventLog-Matcher is basically used as black box,
building up more elaborate matchers from the provided basic matchers...
Spent some time again to understand how EventLog matching works.
My feelings towards this piece of code are always the same: it is
somewhat too "tricky", but I am not aware of any other technique
to get this degree of elaborate chained matching on structured records,
short of building a dedicated matching engine from scratch.
The other alternative would be to use a flat textual log (instead of
the structured log records from EventLog), but then we'd have to
generate quite intricate regular expressions from the builder,
and I'm really doubtful it would be easier and clearer....
...turns out this is entirely generic and not tied to the context
within ActivityDetector, where it was first introduced to build a
mock functor to log all invocations.
Basically this meta-function generates a new instantiation of the
template X, using the variadic argument pack from template U<ARGS...>
...for coverage of the Activity-Language,
various invocations of unspecific functions must be verified,
with the additional twist that the implementation avoids indirections
and is thus hard to rig for tests.
Solution-Idea: provide a λ-mock to log any invocation into the
Event-Log helper, which was created some years ago to trace GUI communication...
...continue to proceed test-driven
...scheduler internals turn out to be intricate and cohesive,
and thus the only hope is to adhere to strict testing discipline
