It seams indicated to verify the generated connectivity
and the hash calculation and recalculation explicitly
at least for one example topology; choosing a topology
comprised of several sub-graphs, to also verify the
propagation of seed values to further start-nodes.
In order to avoid addressing nodes directly by index number,
those sub-graphs can be processed by ''grouping of nodes'';
all parts are congruent because topology is determined by
the node hashes and thus a regular pattern can be exploited.
To allow for easy processing of groups, I have developed a
simplistic grouping device within the IterExplorer framework.
- with the new pruning option, start-Nodes can now be anywhere
- introduce predicates to detect start-Nodes and exit-Nodes
- ensure each new seed node gets the global seed on graph construction
- provide functionality to re-propagate a seed and clear hashes
- provide functionality to recalculate the hashes over the graph
up to now, random values were completely determined by the
Node's hash, leading to completely symmetrical topology.
This is fine, but sometimes additional randomness is desirable,
while still keeping everything deterministic; the obvious solution
is to make the results optionally dependent on the invocation order,
which is simply to achieve with an additional state field. After some
tinkering, I decided to use the most simplistic solution, which is
just a multiplication with the state.
this is only a minor rearrangement in the Algorithm,
but allows to re-boot computation should node connectivity
go to zero. With current capabilities, this could not happen,
but I'm considering to add a »pruning« parameter to create the
possibility to generate multiple shorter chains instead of one
complete chain -- which more closely emulates reality for
Scheduler load patterns.
...so this was yet another digression, caused by the desire
somehow to salvage this problematic component design. Using a
DSL token fluently, while internally maintaining a complex and
totally open function based configuration is a bit of a stretch.
...this is a more realistic demo example, which mimics
some of the patterns present in RandomDraw. The test also
uses lambdas linking to the actual storage location, so that
the invocation would crash on a copy; LazyInit was invented
to safeguard against this, while still allowing leeway
during the initialisation phase in a DSL.
- Helper function to find out of two objects are located
"close to each other" -- which can be used as heuristics
to distinguish heap vs. stack storage
- further investigation shows that libstdc++ applies the
small-object optimisation for functor up to »two slots«
in size -- but only if the copy-ctor is trivial. Thus
a lambda capturing a shared_ptr by value will *always*
be maintained in heap storage (and LazyInit must be
redesigned accordingly)...
- the verify_inlineStorage() unit test will now trigger
if some implementation does not apply small-object optimisation
under these minimal assumptions
...which is crucial for the solution pursued at the moment;
std::function is known to apply a small-object optimisation,
yet unfortunately there are no guarantees by the C++ standard
(it is only mandated that std::function handles a bare function
pointer without overhead)
Other people have investigated that behaviour already,
indicating that at least one additional »slot« of data
can be handled with embedded storage in all known implementations
(while libstdc++ seemingly imposes the strongest limitations)
https://stackoverflow.com/a/77202545/444796
This experiment in the unit-test shows that for my setup
(libstdc++ and GCC-8) only a lambda capturing a single pointer
is handled entirely embedded into the std::function; already
a lambda capturing a shared-ptr leads to overflow into heap
the RandomDraw rules developed last days are meant to be used
with user-provided λ-adapters; employing these in a context
of a DSL runs danger of producing dangling references.
Attempting to resolve this fundamental problem through
late-initialisation, and then locking the component into
a fixed memory location prior to actual usage. Driven by
the goal of a self-contained component, some advanced
trickery is required -- which again indicates better
to write a library component with adequate test coverage.
RandomDraw as a library component was extracted and (grossly) augmented
to cut down the complexity exposed to the user of TestChainLoad.
To control the generated topology, random-selected parameters
must be configured, defining a probability profile; while
this can be achieved with simple math, getting it correct
turned out surprisingly difficult.
...now using the reworked partial-application helper...
...bind to *this and then recursively re-invoke the adaptation process
...need also to copy-capture the previously existing mapping-function
first test seems to work now
Investigation in test setup reveals that the intended solution
for dynamic configuration of the RandomDraw can not possibly work.
The reason is: the processing function binds back into the object instance.
This implies that RandomDraw must be *non-copyable*.
So we have to go full circle.
We need a way to pass the current instance to the configuration function.
And the most obvious and clear way would be to pass it as function argument.
Which however requires to *partially apply* this function.
So -- again -- we have to resort to one of the functor utilities
written several years ago; and while doing so, we must modernise
these tools further, to support perfect forwarding and binding
of reference arguments.
- strive at complete branch coverage for the mapping function
- decide that the neutral value can deliberately lie outside
the value range, in which case the probability setting
controls the number of _value_ result incidents vs
neutral value result incidents.
- introduce a third path to define this case clearly
- implement the range setting Builder-API functions
- absorb boundrary and illegal cases
For sake of simplicity, since this whole exercise is a byproduct,
the mapping calculations are done in doubles. To get even distribution
of values and a good randomisation, it is thus necessary to break
down the size_t hash value in a first step (size_t can be 64bit
and random numbers would be subject to rounding errors otherwise)
The choice of this quantiser is tricky; it must be a power of two
to guarantee even distribution, and if chosen to close to the grid
of the result values, with lower probabilities we'd fail to cover
some of the possible result values. If chosen to large, then
of course we'd run danger of producing correlated numbers on
consecutive picks.
Attempting to use 4 bits of headroom above the log-2 of the
required value range. For example, 10-step values would use
a quantiser of 128, which looks like a good compromise.
The following tests will show how good this choice holds up.
The first step was to allow setting a minimum value,
which in theory could also be negative (at no point is the
code actually limited to unsigned values; this is rather
the default in practice).
But reconsidering this extensions, then you'd also want
the "neutral value" to be handled properly. Within context,
this means that the *probability* controls when values other
than the neutral value are produced; especially with p = 1.0
the neutral value shall not be produced at all
...since the Policy class now defines the function signature,
we can no longer assume that "input" is size_t. Rather, all
invocations must rely on the generic adaptaion scheme.
Getting this correct turns out rather tricky again;
best to rely on a generic function-composition.
Indeed I programmed such a helper several years ago,
with the caveat that at that time we used C++03 and
could not perfect-forward arguments. Today this problem
can be solved much more succinct using generic Lambdas.
to define this as a generic library component,
any reference to the actual data source moust be extracted
from the body of the implementation and supplied later
at usage site. In the actual case at hand the source
for randomness would be the node hash, and that is
absolutely an internal implementation detail.
The idea is to use some source of randomness to pick a
limited parameter value with controllable probability.
While the core of the implementation is nothing more
than some simple numeric adjustments, these turn out
to be rather intricate and obscure; the desire to
package these technicalities into a component
however necessitates to make invocations
at usage site self explanatory.
...start with putting the topology generator to work
- turns out it is still challenging to write the ctrl-rules
- and one example tree looked odd in the visualisation
- which (on investigation) indicated unsound behaviour
...this is basically harmless, but involves an integer wrap-around
in a variable not used under this conditions (toReduce), but also
a rather accidental and no very logical round-up of the topology.
With this fix, the code branch here is no longer overloaded with two
distinct concerns, which I consider an improvement
by default, a linear chain without any forking is generated,
and the result hash is computed by hash-chaining from the seed.
Verify proper connections and validate computed hash
..as can be expected, had do chase down some quite hairy problems,
especially since consumption of the fixed amount of nodes is not
directly linked to the ''beat'' of the main loop and thus boundary
conditions and exhausted storage can happen basically anywhere.
Used a simple expansion rule and got a nod graph,
which looks coherent in DOT visualisation.
writing a control-value rule for topology generation typically
involves some modulus and then arthmetic operations to map
only part of the value range to the expected output range.
These calculations are generic, noisy and error-prone.
Thus introduce a helper type, which allows the client just
to mark up the target range of the provided value to map and
transform to the actually expected result range, including some
slight margin to absorb rounding errors. Moreover, all calculations
done in double, to avoid the perils of unsigned-wrap-around.
...these were developed driven by the immediate need
to visualise ''random generated computation patterns''
for ''Scheduler load testing.''
The abstraction level of this DSL is low
and structures closely match some clauses of the DOT language;
this approach may not yet be adequate to generate more complex
graph structures and was extracted as a starting point
for further refinements....
With all the preceding DSL work, this turns out to be surprisingly easy;
the only minor twist is the grouping of nodes into (time)levels,
which can be achieved with a "lagging" update from the loop body
Note: next step will be to extract the DSL helpers into a Library header
...using a pre-established example as starting point
It seems that building up this kind of generator code
from a set of free functions in a secluded namespace
is the way most suitable to the nature of the C++ language
..the idea is to generate a Graphviz-DOT diagram description
by traversing the internal data structures of TestChainLoad.
- refreshed my Graphviz knowledge
- work out a diagram scheme that can be easily generated
- explore ways to structure code generation as a DSL to keep it legible
...introduce statistical control functions (based on hash)
...add processing stage for current set of nodes
...process forking, reduction and injection of new nodes
- use a specialised class, layered on top of std::array
- use additional storage to mark filling degree
- check/fail on link owerflow directly there
We still use fixed size inline storage for the node links,
yet adding this comparatively small overhead in storage helps
getting the code simpler and adding links is now constant-complexity
A »Node« represents one junction point in the dependency graph,
knows his predecessors and successors and carries out one step
of the chained hash calculation.
...refine the handling of FrameRates close to the definition bounds
...implement the actual rule to scale allocator capacity on announcement
...hook up into the seedCalcStream() with a default of +25FPS
+ test coverage
...whenever a new CalcStream is seeded, it would be prudent
not only to step up the WorkForce (which is already implemented),
but also to provide a hint to the BlockFlow allocator regarding
the expected calculation density.
Such a hint would allow to set a more ample »epoch« spacing,
thereby avoiding to drive the allocator into overload first.
The allocator will cope anyway and re-balance in a matter of
about 2 seconds, but avoiding this kind of control oscillations
altogether will lead to better performance at calculation start.
The test case "scheduleRenderJob()" -- while deliberately operated
quite artificially with a disabled WorkForce (so the test can check
the contents in the queue and then progress manually -- led to discovery
of an open gap in the logic: in the (rare) case that a new task is
added ''from the outside'' without acquiring the Grooming-Token, then
the new task could sit in the entrace queue, in worst case for 50ms,
until the next Scheduler-»Tick« routinely sweeps this queue. Under
normal conditions however, each dispatch of another activity will
also sweep the entrance queue, yet if there happens to be no other
task right now, a new task could be stuck.
Thinking through this problem also helped to amend some aspects
of Grooming-Token handling and clarified the role of the API-functions.
Use a simple destructor-trick to set up a concise notation
for temporarily manipulating a value for testing.
The manipulation will automatically be undone
when leaving scope
...especially to prevent a deadline way too far into the future,
since this would provoke the BlockFlow (epoch based) memory manager
to run out of space.
Just based on gut feeling, I am now imposing a limit of 20seconds,
which, given current parametrisation, with a minimum spacing of 6.6ms
and 500 Activities per Block would at maximum require 360 MiB for
the Activities, or 3000 Blocks. With *that much* blocks, the
linear search would degrade horribly anyway...
WorkForce scales down automatically after 2 seconds when
workers fall idle; thus we need to step up automatically
with each new task.
Later we'll also add some capacity management to both the
LoadController and the Job-Planning, but for now this rather
crude approach should suffice.
NOTE: most of the cases in SchedulerService_test verify parts
of the component integration and thus need to bypass this
automatism, because the test code wants to invoke the
work-Function directly (without any interference
from running workers)
While building increasingly complex integration tests for the Scheduler,
it turns out helpful to be able to manipulate the "full concurreency"
as used by Scheduler, WorkForce and LoadController.
In the current test, I am facing a problem that new entries from the
threadsafe entrance queue are not propagated to the priority queue
soon enough; partly this is due to functionality still to be added
(scaling up when new tasks are passed in) -- but this will further
complicate the test setup.
The invocation structure is effectively determined by the
Activity-chain builder from the Activity-Language; but, taking
into account the complexity of the Scheduler code developed thus far,
it seems prudent to encapsulate the topic of "Activities" altogether
and expose only a convenience builder-API towards the Job-Planning
The problem with passing the deadline was just a blatant symptom
that something with the overall design was not quite right, leading
to mix-up of interfaces and implementation functions, and more and more
detail parameters spreading throughout the call chains.
The turning point was to realise the two conceptual levels
crossing and interconnected within the »Scheduler-Service«
- the Activity-Language describes the patterns of processing
- the Scheduler components handle time-bound events
So by turning the (previously private) queue entry into an
ActivationEvent, the design could be balanced.
This record becomes the common agens within the Scheduler,
and builds upon / layers on top of the common agens of the
Language, which is the Activity record.
the attempt to integrate additional deadline and significance parameters
unveils a design problem due to the layering of contexts
- the Activity-Language attempts to abstract away the ''Scheduler mechanics''
- but this implementation logic now needs to pass additional parameters
- and notably there is the possibility of direct re-scheduling from within
the Activity-Dispatch
The symptom of this problem is that it's no longer possible
to implement the ExecutionCtx.post() function in the real Scheduler-context
...it is clear that there must be a way to flush the scheduler queues
an thereby silently drop any obsoleted or irrelevant entries. This topic
turns out to be somewhat involved, as it requires to consider the
deadline (due to the memory management, which is based on deadlines).
Furthermore there is a relation to yet another challenging conceptual
requirement, which is the support for other operation modes beyond
just time-bound rendering; these concerns make it desirable to
expand the internal representation of entries in the queue.
Concerns regarding performance are postponed deliberately,
until we can demonstrate the Scheduler-Service running under
regular operational conditions.
This is the first kind of integration,
albeit still with a synthetic load.
- placed two excessive load peaks in the scheduling timeline
- verified load behaviour
- verified timings
- verified that the scheduler shuts down automatically when done
- sample distance to scheduler head whenever a worker asks for work
- moving average with N = worker-pool size and damp-factor 2
- multiply with the current concurrency fraction
as an aside, the header lib/test/microbenchmark.hpp
turns out to be prolific for this kind of investigation.
However, it is somewhat obnoxious that the »test subject«
must expose the signature <size_t(size_t)>.
Thus, with some metaprogramming magic, an generic adaptor
can be built to accept a range of typical alternatives,
and even the quite obvious signature void(void).
Since all these will be wrapped directly into a lambda,
the optimiser will remove these adaptations altogether.
- An important step towards a complete »Scheduler Service«
- Correct timing pattern could be verified in detail by tracing
- Spurred some further concept and design work regarding Load-control
- draft the duty cycle »tick«
- investigate corner cases of state updates and allocation managment
- implement start and forcible stop of the scheduler service
Obviously the better choice and a perfect fit for our requirements;
while the system-clock may jump and even move backwards on time service
adjustments, the steady clock just counts the ticks since last boot.
In libStdC++ both are implemented as int64_t and use nanoseconds resolution
- Ensure the grooming-token (lock) is reliably dropped
- also explicitly drop it prior to trageted sleeps
- properly signal when not able to acquire the token before dispatch
- amend tests broken by changes since yesterday
Notably the work-function is now completely covered, by adding
this last test, and the detailed investigations yesterday
ultimately unveiled nothing of concern; the times sum up.
Further reflection regarding the overall concept led me
to a surprising solution for the problem with priority classes.
...especially for the case »outgoing to sleep«
- reorganise switch-case to avoid falling through
- properly handle the tendedNext() predicate also in boundrary cases
- structure the decision logic clearer
- cover the new behaviour in test
Remark: when the queue falls empty, the scheduler now sends each
worker once into a targted re-shuffling delay, to ensure the
sleep-cycles are statistically evenly spaced
...there seemed to be an anomaly of 50...100µs
==> conclusion: this is due to the instrumentation code
- it largely caused by the EventLog, which was never meant
to be used in performance-critical code, and does hefty
heap allocations and string processing.
- moreover, there clearly is a cache-effect, adding a Factor 2
whenever some time passed since the last EventLog call
==> can be considered just an artifact of the test setup and
will have no impact on the scheduler
remark: this commit adds a lot of instrumentation code
To cover the visible behaviour of the work-Function,
we have to check an amalgam of timing delays and time differences.
This kind of test tends to be problematic, since timings are always
random and also machine dependent, and thus we need to produce pronounced effects
...to make that abundantly clear: we do not aim at precision timing,
rather the goal is to redistribute capacity currently not usable...
Basically we're telling the worker "nothing to do right now, sorry,
but check back in <timespan> because I may need you then"
Workers asking for the next task are classified as belonging
to some fraction of the free capacity, based on the distance
to the closest next Activity known to the scheduler
...to bring it more in line with all the other calls dealing with Activity*
...allows also to harmonise the ActivityLang::dispatchChain()
...and to compose the calls in Scheduler directly
NOTE: there is a twist: our string-formatting helper did not render
custom string conversions for objects passed as pointer. This was a
long standing problem, caused by ambiguous templates overloads;
now I've attempted to solve it one level more down, in util::StringConv.
This solution may turn out brittle, since we need to exclude any direct
string conversion, most notably the ones for C-Strings (const char*)
In case this solution turns out unsustainable, please feel free
to revert this API change, and return to passing Activity& in λ-post,
because in the end this is cosmetics.
- organise by principles rather than implementing a mechanism
- keep the first version simple yet flexible
- conduct empiric research under synthetic load
Basic scheme:
- tend for next
- classify free capacity
- scattered targeted wait
The Activity-Language can be defined by abstracting away
some crucial implementation functionality as part of an generic
»ExecutionCtx«, which in the end will be provided by the Scheduler.
But how actually?
We want to avoid unnecessary indirections, and ideally we also want
a concise formulation in-code. Here I'm exploring the idea to let the
scheduler itself provide the ExecutionCtx-operations as member functions,
employing some kind of "compile-time duck-typing"
This seems to work, but breaks the poor-man's preliminary "Concept" check...
The »Scheduler Service« will be assembled
from the components developed during the last months
- Layer-1
- Layer-2
- Activity-Language
- Block-Flow
- Work-Force
* the implementation logic of the Scheduler is essentially complete now
* all functionality necessary for the worker-function has been demonstrated
As next step, the »Scheduler Service« can be assembled from the two
Implementation Layers, the Activity-Language and the `BlockFlow` allocator
This should then be verified by a multi-threaded integration test...
This central operation sits at a crossroad and is used
- from external clients to fed new work to the Scheduler
- from Workers to engage into execution of the next Activity
- recursively from the execution of an Activity-chain
From these requirements the semantics of behaviour can be derived
regarding the GroomingToken and the result values, which indicate
when follow-up work should be processed
Ensure the GroomingToken mechanism indeed creates an
exclusive section protected against concurrent corruption:
Use a without / with-protection test and verify
the results are exact vs. grossly broken
T thread holding the »Grooming Token" is permitted to
manipulate scheduler internals and thus also to define new
activities; this logic is implemented as an Atomic lock,
based on the current thread's ID.
Notably both Layers are conceived as functionality providers;
only at Scheduler top-Level will functionality be combined with
external dependencies to create the actual service.
At first sight, this seems confusing; there is a time window,
there is sometimes a `when` parameter, and mostly a `now` parameter
is passed through the activation chain.
However, taking the operational semantics into account, the existing
definitions seem to be (mostly) adequate already: The scheduler is
assumed to activate a chain only ''when'' the defined start time is reached.
As follow-up to the rework of thread-handling, likewise also
the implementation base for locking was switched over from direct
usage of POSIX primitives to the portable wrappers available in
the C++ standard library. All usages have been reviewed and
modernised to prefer λ-functions where possible.
With this series of changes, the old threadpool implementation
and a lot of further low-level support facilities are not used
any more and can be dismantled. Due to the integration efforts
spurred by the »Playback Vertical Slice«, several questions of
architecture could be decided over the last months. The design
of the Scheduler and Engine turned out different than previously
anticipated; notably the Scheduler now covers a wider array of
functionality, including some asynchronous messaging. This has
ramifications for the organisation of work tasks and threads,
and leads to a more deterministic memory management. Resource
management will be done on a higher level, partially superseding
some of the concepts from the early phase of the Lumiera project.
This is Step-2 : change the API towards application
Notably all invocation variants to support member functions
or a reference to bool flags are retracted, since today a
λ-binding directly at usage site tends to be more readable.
The function names are harmonised with the C++ standard and
emergency shutdown in the Subsystem-Runner is rationalised.
The old thread-wrapper test is repurposed to demonstrate
the effectiveness of monitor based locking.
After the fundamental switch from POSIX to the C++14 wrappers
the existing implementation of the Monitor can now be drastically condensed,
removing several layers of indirection. Moreover, all signatures
shall be changed to blend in with the names and patterns established
by the C++ standard.
This is Step-1 : consolidate the Implementation.
(to ensure correctness, the existing API towards application code was retained)
While not directly related to the thread handling framework,
it seems indicated to clean-up this part of the application alongside.
For »everyday« locking concerns, an Object Monitor abstraction was built
several years ago and together with the thread-wrapper, both at that time
based on direct usage of POSIX. This changeset does a mere literal
replacement of the POSIX calls with the corresponding C++ wrappers
on the lowest level. The resulting code is needlessly indirect, yet
at API-level this change is totally a drop-in replacment.
The WorkForce (passive worker pool) has been coded just recently,
and -- in anticipation of this refactoring -- directly against std::thread
instead of using the old framework.
...the switch is straight-forward, using the default case
...add the ability to decorate the thread-IDs with a running counter
This solution is basically equivalent to the version implemented directly,
but uses the lifecycle-Hooks available through `ThreadHookable`
to structure the code and separate the concerns better.
This largely completes the switch to the new thread-wrapper..
**the old implementation is not referenced anymore**
This, and the GUI thread prompted an further round of
design extensions and rework of the thread-wrapper.
Especially there is now support for self-managed threads,
which can be launched and operate completely detached from the
context used to start them. This resolves an occasional SEGFAULT
at shutdown. An alternative (admittedly much simpler) solution
would have been to create a fixed context in a static global
variable and to attach a regular thread wrapper from there,
managed through unique_ptr.
It seems obvious that the new solution is preferable,
since all the tricky technicalities are encapsulated now.
Add a complete demonstration for a setup akin to what we use
for the Session thread: a threaded component which manages itself
but also exposes an external interface, which is opened/closed alongside
...extract and improve the tuple-rewriting function
...improve instance tracking test dummy objects
...complete test coverage and verify proper memory handling
After quite some detours, with this take I'm finally able to
provide a stringent design to embody all the variants of thread start
encountered in practice in the Lumiera code base.
Especially the *self-managed* thread is now represented as a special-case
of a lifecycle-hook, and can be embodied into a builder front-end,
able to work with any client-provided thread-wrapper subclass.
to cover the identified use-cases a wide variety of functors
must be accepted and adapted appropriately. A special twist arises
from the fact that the complete thread-wrapper component stack works
without RTTI; a derived class can not access the thread-wrapper internals
while the policy component to handle those hooks can not directly downcast
to some derived user provided class. But obviously at usage site it
can be expected to access both realms from such a callback.
The solution is to detect the argument type of the given functor
and to build a two step path for a safe static cast.
...after resolving the fundamental design problems,
a policy mix-in can be defined now for a thread that deletes
its own wrapper at the end of the thread-function.
Such a setup would allow for »fire-and-forget« threads, but with
wrapper and ensuring safe allocations. The prominent use case
for such a setup would be the GUI-Thread.
Concept study of the intended solution successful.
Can now transparently embed any conceivable functor
and an arbitrary argument sequence into a launcher-λ
Materialising into a std::tuple<decay_t<TYPES...>> did the trick.
Considering a solution to shift the actual launch of the new thread
from the initialiser list into the ctor body, to circumvent the possible
"undefined behaviour". This would also be prerequisite for defining
a self-managed variant of the thread-wrapper.
Alternative / Plan.B would be to abandon the idea of a self-contained
"thread" building block, instead relying on precise setup in the usage
context -- however, not willing to yield yet, since that would be exactly
what I wanted to avoid: having technicalities of thread start, argument
handover and failure detection intermingled with the business code.
On a close look, the wrapper design as pursued here
turns out to be prone to insidious data race problems.
This was true also for the existing solution, but becomes
more clear due to the precise definitions from the C++ standard.
This is a confusing situation, because these races typically do not
materialise in practice; due to the latency of the OS scheduler the
new thread starts invoking user code at least 100µs after the Wrapper
object is fully constructed (typically more like 500µs, which is a lot)
The standard case (lib::Thread) in its current form is correct, but borderline
to undefined behaviour, and any initialisation of members in a derived class
would be off limits (the thread-wrapper should not be used as baseclass,
rather as member)
...while reworking the application code, it became clear that
actually there are two further quite distinct variants of usage.
And while these could be implemented with some trickery based on
the Thread-wrapper defined thus far, it seems prudent better to
establish a safely confined explicit setup for these cases:
- a fire-and-forget-thread, which manages its own memory autonomously
- a thread with explicit lifecycle, with detectable not-running state
FamilyMember::allocateNextMember() was actually a post-increment,
so (different than with TypedCounter) here no correction is necessary
As an asside, WorkForce_test is sometimes unstable immediately after a build.
Seemingly a headstart of 50µs is not enough to compensate for scheduler leeway
Set ulimit -v setting to 8 GiB (setting is given in kbyte)
Otherwise it is not possible to start 100 Threads.
This is surprising, because the actual memory usage of the tests in question
are minuscule and also TOP does not show any significant memory peak when running the test.
The existing TypedCounter_test was excessively clever and convoluted,
yet failed to test the critical elements systematically. Indeed, two
bugs were hidden in synchronisation and instance access.
- build a new concurrent test from scratch, now using the threadBenchmark
function for the actual concurrent execution and just invoked a
random selected access to the counter repeatedly from a large number
of threads.
- rework the TypedContext and counter to use Atomics where applicable;
measurements indicate however that this has only negligible impact
on the amortised invocation times, which are around 60ns for single-threaded
access, yet can increase by factor 100 due to contention.
...these were already written envisionaging he new API,
so it's more or less a drop-in replacement.
- cant use vector anymore, since thread objects are move-only
- use ScopedCollection instead, which also has the benefit of
allocating the requires space up-front. Allow to deduce the
type parameter of the placed elements
... which became apparent after switching to the new Thread-wrapper implementation
... the reason is a bug in the Thread-Monitor (which will also be reworked soon)
While seemingly subtle, this is a ''deep change.''
Up to now, the project attempted to maintain two mutually disjoint
systems of error reporting: C-style error flags and C++ exceptions.
Most notably, an attempt was made to keep both error states synced.
During the recent integration efforts, this increasingly turned out
as an obstacle and source for insidious problems (like deadlocks).
As a resolve, hereby the relation of both systems is **clarified**:
* C-style error flags shall only be set and used by C code henceforth
* C++ exceptions can (optionally) be thrown by retrieving the C-style error code
* but the opposite is now ''discontinued'' : Exceptions ''do not set'' the error flag anymore
- the deadlock was caused by leaking error state through the C-style lumiera_error
- but the reason for the deadlock lies in the »convenience shortcut«
in the Object-Monitor scope guard for entering a wait state immediately.
This function undermines the unlocking-guarantee, when an exception
emanates from within the wait() function itself.
...this function was also ported to the new wrapper,
and can be verified now in a much more succinct way.
''This completes porting of the thread-wrapper''
Since the decision was taken to retain support for this special feature,
and even extend it to allow passing values, the additional functionality
should be documented in the test. Doing so also highlighted subtle problems
with argument binding.
Now the ThreadWrapper_test offers both
- a really simple usage example
- a comprehensive test to verify that actually the
thread-function is invoked the expected number of times
and that this invocations must have been parallelised
- it is not directly possible to provide a variadic join(args...),
due to overload resolution ambiguities
- as a remedy, simplify the invocation of stringify() for the typical cases,
and provide some frequently used shortcuts
A common usage pattern is to derive from lib::Thread
and then implement the actual thread function as a member function
of this special-Thread-object (possibly also involving other data members)
Provide a simplified invocation for this special case,
also generating the thread-id automatically from the arguments
after all this groundwork, implementing the invocation,
capturing and hand-over of results is simple, and the
thread-wrapper classes became fairly understandable.
This relieves the Thread policy from a lot of technicalities,
while also creating a generally useful tool: the ability to invoke
/anything callable/ (thanks to std::invoke) in a fail-safe way and
transform the exception into an Either type
on second thought, the ability to transport an exception still seems
worthwhile, and can be achieved by some rearrangements in the design.
As preparation, reorganise the design of the Either-wrapper (lib::Result)
- relocate some code into a dedicated translation unit to reduce #includes
- actually set the thread-ID (the old implementation had only a TODO at that point)
While it would be straight forward from an implementation POV
to just expose both variants on the API (as the C++ standard does),
it seems prudent to enforce the distinction, and to highlight the
auto-detaching behaviour as the preferred standard case.
Creating worker threads just for one computation and joining the results
seemed like a good idea 30 years ago; today we prefer Futures or asynchronous
messaging to achieve similar results in a robust and performant way.
ThreadJoinable can come in handy however for writing unit tests, were
the controlling master thread has to wait prior to perform verification.
So the old design seems well advised in this respect and will be retained
- cut the ties to the old POSIX-based custom threadpool framework
- remove operations deemed no longer necessary
- sync() obsoleted by the new SyncBarrier
- support anything std::invoke supports
...which is the technique used in the existing Threadpool framwork.
As expected, such a solution is significantly slower than the new
atomics-based implementation. Yet how much slower is still striking.
Timing measurements in concurrent usage situation.
Observed delay is in the order of magnitude of known scheduling leeway;
assuming thus no relevant overhead related to implementation technique
Over time, a collection of microbenchmark helper functions was
extracted from occasional use -- including a variant to perform
parallelised microbenchmarks. While not used beyond sporadic experiments yet,
this framework seems a perfect fit for measuring the SyncBarrier performance.
There is only one catch:
- it uses the old Threadpool + POSIX thread support
- these require the Threadpool service to be started...
- which in turn prohibits using them for libary tests
And last but not least: this setup already requires a barrier.
==> switch the existing microbenchmark setup to c++17 threads preliminarily
(until the thread-wrapper has been reworked).
==> also introduce the new SyncBarrier here immediately
==> use this as a validation test of the setup + SyncBarrier
Using the same building blocks, this operation can be generalised even more,
leading to a much cleaner implementation (also with better type deduction).
The feature actually used here, namely summing up all values,
can then be provided as a convenience shortcut, filling in std::plus
as a default reduction operator.
...first used as part of the test harness;
seemingly this is a generic and generally useful shortcut,
similar to algorithm::reduce (or some kind of fold-left operation)
Intended as replacement for the Mutex/ConditionVar based barrier
built into the exiting Lumiera thread handling framework and used
to ensure safe hand-over of a bound functor into the starting new
thread. The standard requires a comparable guarantee for the C++17
concurrency framework, expressed as a "synchronizes_with" assertion
along the lines of the Atomics framework.
While in most cases dedicated synchronisation is thus not required
anymore when swtiching to C++17, some special extended use cases
remain to be addressed, where the complete initialisation of
further support framework must be ensured.
With C++20 this would be easy to achieve with a std::latch, so we
need a simple workaround for the time being. After consideration of
the typical use case, I am aiming at a middle ground in terms of
performance, by using a yield-wait until satisfying the latch condition.
The investigation for #1279 leads to the following conclusions
- the features and the design of our custom thread-wrapper
almost entirely matches the design chosen meanwhile by the C++ committee
- the implementation provided by the standard library however uses
modern techniques (especially Atomics) and is more precisely worked out
than our custom implementation was.
- we do not need an *active* threadpool with work-assignment,
rather we'll use *active* workers and a *passive* pool,
which was easy to implement based on C++17 features
==> decision to drop our POSIX based custom implementation
and to retrofit the Thread-wrapper as a drop-in replacement
+++ start this refactoring by moving code into the Library
+++ create a copy of the Threadwrapper-code to build and test
the refactorings while the application itself still uses
existing code, until the transition is complete
While in principle it would be possible (and desirable)
to control worker behaviour exclusively through the Work-Functor's return code,
in practice we must concede that Exceptions can always happen from situations
beyond our control. And while it is necessary for the WorkForce-dtor to
join and block (we can not just pull away the resources from running threads),
the same destructor (when called out of order) must somehow be able
at least to ask the running threads to terminate.
Especially for unit tests this becomes an obnoxious problem -- otherwise
each test failure would cause the test runner to hang.
Thus adding an emergency halt, and also improve setup for tests
with a convenience function to inject a work-function-λ
No new functionality, and implementation works as expected.
This test case covers an especially tricky setup, where a calculation
shall be triggered from an external event, while ensuring that the actual
processing can start only after also the regular time-bound scheduling
has taken place (this might be used to prevent an unexpectedly early
external signal to cause writing into an output buffer before the
defined window of data delivery)
...based on the new ability in the ActivityDetector, we can now assign
a custom λ, which deflects back the ctx.post() call into the ActivityLang
instance used for this test case.
While the previously seen behaviour was correct, it was not the call sequence
expected in the real implementation; with this change, on the main-chain
activation the post() now immediately dispatches the notification, which in turn
dispatches the rest of the chain, so that the JobFunctor is indeed
called in this second test case as expected
Up to now, the DiagnosticFun mock in ActivityDetector only
created an EventLog entry on invocation and was able to retunr
a canned result value. Yet for the job invocation scenario test,
it would be desirable to hook-in a λ with a fake implementation
into the ExecutionContext. As a further convenience, the
return value is now default initialised, instead of being
marked as uninitialised until invocation of "returning(val)"
...seems to work, but not really happy with the test setup,
since in real usage the post()-calls would dispatch, while here,
using the ActivityDetector, these calls just log invoation,
and thus the activation is not passed on
...regarding the kind of activity (the verb),
and also for some special case access of payload data;
deliberately asserting the correct verb, but no mandatory check,
since this whole Activity-Language is conceived as cohesive
and essentially sealed (not meant to be extended)
...to show in test that indeed the actual time is retrieved on each activation,
we can assign a λ -- which is rigged to increase the time on each access
It is not sufficient just to pass this "current time" as parameter
into the ActivityLang::dispatchChain(), since some Activities within
this chain will essentially be long-running (think rendering); thus
we need a real callback from within the chain. The obvious solution
is to make this part of the Execution Context, which is an abstraction
of the scheduler environment anyway
...turns out there is still a lot of leeway in the possible implementation,
and seemingly it is too early to decide which case to consider the default.
Thus I'll proceed with the drafted preliminary solution...
- on primary-chain, an inhibited Gate dispatches itself into future for re-check
- on Notification, activation happens if and only if this very notification opens the Gate
- provide a specifically wired requireDirectActivation() to allow enforcing a minimal start time
...assembled from parts already implemented
TODO
- need a way to access the »current scheduler time«
- need builder extension points to connect notifications
...this completes the basic setup
- Term builder mechanism working properly
- Memory allocator behaves sane
- the simple default wiring allows to invoke a Job
Solved by special treatment of a notification, which happens
to decrement the latch to zero: in this case, the chain is
dispatched, but also the Gate is locked permanently to block
any further activations scheduled or forwareded otherwise
TODO: while correct as implemented, the handling of the
notification seems questionable, since re-scheduling the chain immediately
may lead to multiple invocations of the chain, since it might have been "spinned"
and thus re-scheduled already, and we have no way to find out about that
...can not take a shortcut here, since the timing information
embedded into the POST-Activity must somehow be transported
to the Scheduler; key point to note is that the chain will
be performed in »management mode« (single threaded)
...attempt to get this intricate state machine sorted out
Notification turned out quite tricky, since it may emanate
from a concurrently executed phase and we try to avoid having
to protect the gate directly with a lock; rather we re-dispatch
the notification through the queue, which indirectly also ensures
that the worker de-queuing the NOTIFY-Activity operates in
management mode (single threaded, holding the GroomingToken)
Decision how to handle a failed Gate-check
- spin forward (re-scheduler) by some time amount
- this spin-offset parameter is retrieved from the Execution Context
- thus it will be some kind of engine parameter
With these determinations and the framework for the Execution Context
it is now possible to code up the logic for Gate check, which in turn
can then be verified by the watchGate diagnostics
due to technical limitations this requires to wire the adaptor
as replacement for the subject Activity, so that it can capture
and log the activation, and then pass it on to its watched subject
requires to supplement EventLog matching primitives
to pick and verify a specific positional argument.
Moreover, it is more or less arbitrary which job invocation parameters
are unpacked and exposed for verification; we'll have to see what is
actually required for writing tests...
doing so would contradict the fundamental architecture,
all kinds of failures and timeouts need to be handled within
Scheduler-Layer-2 rather.
Jobs are never aborted, nor do they need to know if and when they are invoked
Testcase (detect function invocation) passes now as expected
Some Library / Framework changes
- rename event-log-test.cpp
- allow the ExpectString also to work with concatenated expectation strings
Remark: there was a warning in the comment in event-log.hpp,
pointing out that negative assertions are shallow.
However, after the rework in 9/2018 (commit: d923138d1)
...this should no longer be true, since we perform proper backtracking,
leading to an exhaustive search.
ActivityMatch inherits privately from the EventMatch object,
and is thus able to delegate relevant matching queries, but
also to provide high-level special matchers.
This new design resolves the ambiguity regarding function arguments.
Moreover, we can now record the current sequence-Number as *attribute*
in the respective log record (this is the benefit of using structured
log entries instead of just a textual log), thereby avoiding the various
pitfalls with explicit bracketing sequence-number log entries
bottom line: this reworked design seems to be a better fit,
even while technically the implementation with the wrapped matcher
is somewhat ugly...
The EventLog seems to provide all the building blocks, but we need
some higher level special matchers (and maybe we also want to hide
some of the basic EventLog matchers). A soulution might be to wrap
the EventMatcher and delegate all follow-up builder calls.
This seems adequate, since the EventLog-Matcher is basically used as black box,
building up more elaborate matchers from the provided basic matchers...
Spent some time again to understand how EventLog matching works.
My feelings towards this piece of code are always the same: it is
somewhat too "tricky", but I am not aware of any other technique
to get this degree of elaborate chained matching on structured records,
short of building a dedicated matching engine from scratch.
The other alternative would be to use a flat textual log (instead of
the structured log records from EventLog), but then we'd have to
generate quite intricate regular expressions from the builder,
and I'm really doubtful it would be easier and clearer....
...turns out this is entirely generic and not tied to the context
within ActivityDetector, where it was first introduced to build a
mock functor to log all invocations.
Basically this meta-function generates a new instantiation of the
template X, using the variadic argument pack from template U<ARGS...>
...for coverage of the Activity-Language,
various invocations of unspecific functions must be verified,
with the additional twist that the implementation avoids indirections
and is thus hard to rig for tests.
Solution-Idea: provide a λ-mock to log any invocation into the
Event-Log helper, which was created some years ago to trace GUI communication...
Further extensive testing with parameter variations,
using the test setup in `BlockFlow_test::storageFlow()`
- Tweaks to improve convergence under extreme overload;
sudden load peaks are now accomodated typically < 5 sec
- Make the test definition parametric, to simplify variations
- Extract the generic microbenchmark helper function
- Documentation
There seems to be a ''sweet spot'' for somewhat larger Epoch sizes around 500 slots.
At least in the test setup used here, which works with a load of 200 Frames / sec,
which is significantly over the typical value of 50fps (video + audio) for simple playback.
The optimisation of averaged allocation times can not be much improved **below 30ns**.
Overall, this can be considered a good result,
since this allocation scheme does way more than just allocate memory,
it also provides a means to track dependencies and lifecycle.
__For context__:
- we should strive at processing one frame in ~ 10ms
- for 10 Activity records per Frame, we currently use < 0.5 µs for
memory and dependency management in the scheduler
- this leaves enough room for the further administrative efforts
(priority queue, job planning, buffer management)
BUT -> +50% runtime in -O3 (+20ns)
Investigation seems to indicate
- that the increased (+1 Epochs, 10 -> 11) moving average
caused the Algo to perform worse (strong effect)
- that the Optimiser has problems with boost::rational, which however
yields only a minute effect (+5ns), and only on the critical path
The access via Meyers Singleton has no adverse effect,
rather the new setup gives a tiny benefit (46ns -> 37ns).
Surprisingly, the increased pre-allocation has no observable effect.
On the long run, there will be a central Render Engine parametrisation;
some parameters can even be expected to be dynamic; thus prepare the
BlockFlow allocator to fit in with this expectation
For comparison: use individual managment by refcount.
This supports the conclusion that BlockFlow is more than just a
custom allocator; it also supports a non-trivial lifetime management,
and this comes at a cost.
Playing around with various load patterns uncovers further weak spots
in the regulation mechanism. As a remedy, introduce a stronger feed-back
and especially set the target load factor from 100% -> 90%
to add some headroom to absorb intermittent load peaks
Presumably ''much more observation and fine-tuning'' will be necessary
under real-world load conditions (⟹ Ticket #1318 for later)
- BUG: must prevent the Epoch size to become excessive low
- Problem: feedback signal should not be overly aggressive
Fine-Tuning:
- Dose for Overflow-compensation is delicate
- Moving average and Overflow should be balanced
- ideally the compensatory actions should be one order of magnitude
slower than the characteristic regulation time
Improvement: perform Moving-Average calculations in doubles
...leading to PATHETICALLY bad timing comparison
...it seems clear that the Epoch-Step went to zero
(which was neither anticipated, nor protected against)
However, even individual heap allocations fare surprisingly well
under full optimisation; just they don't solve our problem with
tracking dependencies; the most simplest solution that would
also fulfil this requirement would be using shared_ptr
..as a heuristic to regulate optimal Epoch duration;
when Epochs are discarded, the effective fill factor can be used
to guess an Epoch duration time, which would (in hindsight)
lead to perfect usage of storage space
..using a simplistic implementation for now: scale down the
Epoch-stepping by 0.9 to increase capacity accordingly.
This is done on each separate overflow event, and will be
counterbalanced by the observation of Epoch fill ratio
performed later on clean-up of completed Epochs
further implementation makes clear that the AllocationHandle,
which is the primary usage front-end, has to rely both on
services of the underlying ExtentFamily allocator, as well
as on the BlockFlow itself for managing the Epoch spacing.
- fix a bug in IterExplorer: when iterating a »state core« directly,
the helper CoreYield passed the detected type through ValueTypeBindings.
This is logically wrong, because we never want to pick up some typedefs,
rather we always want to use the type directly returned from CORE::yield()
Here the iterator returns an Epoch&, which itself is again iterable
(it inherits from std::array<Activity, N>). However, it is clear
that we must not descent into such a "flatMap" style recursive expansion
- draft a simple scheme how to regulate Epoch lengths dynamically
- add diagnostics to pinpoint a given Activity and find out into which
Epoch it has been allocated; used to cover the allocator behaviour
- add preliminary deadline-check (directly instead of using the Activity)
- with this shortcut, now able to implement discarding obsoleted Epochs
- Iteration and use of the underlying `ExtentFamily` is also settled by now
💡 ''Implementation concept for the allocation scheme complete and validated''
...with the preceding IterableDecorator refactoring,
the navigation and access to the storage extents can now be
organised into a clear progression
Allocator::iterator -> EpochIter -> Epoch&
Convenience management and support functions can then be
pushed down into Epoch, while iteration control can be done
high-level in BlockFlow, based on the helpers in Epoch
..this is the most simple case, where no Epochs are opened yet
..add diagnostics to inspect alloc count and deadlines
..add accessors for the first/last underlying Extent
...continue to proceed test-driven
...scheduler internals turn out to be intricate and cohesive,
and thus the only hope is to adhere to strict testing discipline
Library: add "obvious" utility to the IterExplorer, allowing to
materialise all contents of the Pipeline into a container
...use this to take a snapshot of all currently active Extent addresses
- use a checksum to prove that ctor / dtor of "content" is not invoked
- let the usage of active extents "wrap around" so that the mem block is re-used
- verify that the same data is still there
The low-level allocator is basically implemented now,
but we still need to check thoroughly that the tricky
wrap-around and expansion logic behaves sane...
(see #1311)
Iteration should just yield an Reference to an Extent,
thereby hiding all details of the actual raw storage (char[]).
This can be achieved by usind a wrapper type around a pointer
into the managing vector; from this pointer we may convert
into a vector::iterator with the trick described here
https://stackoverflow.com/a/37101607/444796
Furthermore, continued planning of the Activity-Language,
basically clarified the complete usage scenario for now;
seems all implementable right away without further difficulties
- the idea is to use slot-0 in each extent for administrative metadata
- to that end, a specialised GATE-Activity is placed into slot-0
- decision to use the next-pointer for managing the next free slot
- thus we need the help of the underlying ExtentFamily for navigating Extents
Decision to refrain from any attempt to "fix" excessive memory usage,
caused by Epochs still blocked by pending IO operations. Rather, we
assume the engine uses sane parametrisation (possibly with dynamic adjustment)
Yet still there will be some safety limit, but when exceeding this limit,
the allocator will just throw, thereby killing the playback/render process
- decision to favour small memory footprint
- rather use several Activity records to express invocation
- design Activity record as »POD with constructor«
- conceptually, Activity is polymorphic, but on implementation
level, this is "folded down" into union-based data storage,
layering accessor functions on top
- decision how to handle the Extent storage (by forced-cast)
- decision to place the administrative record directly into the Extent
TODO not clear yet how to handle the implicit limitation for future deadlines
using a simple yet performant data structure.
Not clear yet if this approach is sustainable
- assuming that no value initialisation happens for POD payload
- performance trade-off growth when in wrapped-state vs using a list
The second design from 2017, based on a pipeline builder,
is now renamed `TreeExplorer` ⟼ `IterExplorer` and uses
the memorable entrance point `lib::explore(<seq>)`
✔
after completing the recent clean-up and refactoring work,
the monad based framework for recursive tree expansion
can be abandoned and retracted.
This approach from functional programming leads to code,
which is ''cool to write'' yet ''hard to understand.''
A second design attempt was based on the pipeline and decorator pattern
and integrates the monadic expansion as a special case, used here to
discover the prerequisites for a render job. This turned out to be
more effective and prolific and became standard for several exploring
and backtracking algorithms in Lumiera.
An extended series of refactoring and partial rewrites resulted
in a new definition of the `Dispatcher` interface and completes
the buildup of a Job-Planning pipeline, including the ability
to discover prerequisites and compute scheduling deadlines.
At this point, I am about to ''switch to the topic'' of the `Scheduler`,
''postponing'' the completion of the `RenderDrive` until the related
questions regarding memory management and Scheduler interface are settled.
- allow to configure the expected job runtime in the test spec
- remove link to EngineConfig and hard-wire the engine latency for now
... extended integration testing reveals two further bugs ;-)
... document deadline calculation
This finishes the last series of refactorings; the basic concept
remains the same, but in the initial version we arranged the expander
function in the pipeline to maintain a Tuple (parent, child) for the
JobTickets. Unfortunately this turned out to be insufficient, since
JobTicket is effectively const and responsible for a complete Sement,
so there is no room to memorise a Deadline for the parent dependency.
This leads to the better idea to link the JobPlanning aggregators
themselves by parent-child references, which is possible since the
whole dependency chain actually sits in the stack embedded into the
Expander (in the pipeline)
...in the hope that the Optimiser is able to elide those references entirely,
when (as is here the case) they point into another field of a larger object compound
...as a preparation for solving a logical problem with the Planning-Pipeline;
it can not quite work as intended just by passing down the pair of
current ticket and dependent ticket, since we have to calculate a chained
calculation of job deadlines, leading up to the root ticket for a frame.
My solution idea is to create the JobPlanning earlier in the pipeline,
already *before* the expansion of prerequisites, and rather to integrate
the representation of the dependency relation direcly into JobPlanning
...using hard coded values instead of observation of actual runtimes,
but at least the calculation scheme (now relocated from TimeAnchor to JobPlanning)
should be a reasonable starting point.
TODO: test fails...
The initial implementation effort for Player and Job-Planning
has been reviewed and largely reworked, and some parts are now
obsoleted by the reworked alternative and can be disabled.
The basic idea will be retained though: JobPlanning is a
data aggregator and performs the final step of creating a Job
- had to fix a logical inconsistency in the underlying Expander implementation
in TreeExplorer: the source-pipeline was pulled in advance on expansion,
in order to "consume" the expanded element immediately; now we retain
this element (actually inaccessible) until all of the immediate
children are consumed; thus the (visible) state of the PipeFrameTick
stays at the frame number corresponding to the top-level frame Job,
while possibly expanding a complete tree of flexible prerequisites
This test now gives a nice visualisation of the interconnected states
in the Job-Planning pipeline. This can be quite complex, yet I still think
that this semi-functional approach with a stateful pipeline and expand functors
is the cleanest way to handle this while encapsulating all details
- fix a bug in the MockDispatcher, when duplicating the ExitNodes.
A vector-ctor with curly braces will be interpreted as std::initializer_list
- add visualisation of the contents appearing at the end of the pipeline
*** something still broken here, increments don't happen as expected
`steam/engine/mock-dispatcher.hpp |cpp` now integrates this
''complete mock setup for render jobs and frame dispatching.''
The exising `DummyJob` has been slightly adapted and renamed
to `MockJob` and is tightly integrated with the other mocks.
The implementation of a `MockDispatcher` necessitated to change
the use of `MockJobTicket`. The initial attempts used a complete
mock implementation, but this approach turned out not to be viable.
Instead — based on the ideas developed for the mock setup —
now the prospective real implementation of `JobTicket` is available
and will be used by the mock setup too. Instead of a synthetic spec,
now a setup of recursively connected `ExitNode`(s) is used; the latter
seems to develop into some kind of Facade for the render node network.
Based on this mock setup, we can now demonstrate the (mostly) complete
Job-Planning pipeline, starting from a segmentation up to render jobs,
and verify proper connectivity and job invocation.
✔
- has to be prepared / supported by the RenderEnvironmentClosure
- actual translation happens when building the Dispatcher-Pipeline
- implementation delegate through
virtual size_t Dispatcher::resolveModelPort (ModelPort)
...ouch this was insidious: the STL implementation for list does not
return a pointer to the element just allocated, but rather retrieves
and dereferences the back() / front() iterator after returning from emplace_back|front()
...which in case of re-entrant allocations is something wildly different
than the initial allocation. Thus a *cheap* and dirty placeholder implementation
just using a STL container is not possible, and we need at least
to code up likewise cheesy placeholder implementation by hand.
- separate allocation and ctor all
- use an inline buffer in the STL container
- explicitly handle ctor failures to discard allocation
- NOT THREADSAFE and likely WASTFUL in terms of performance
==> MockSupport_test now back to GREEN after complete refactoring
The existing implementation of the Player from 2012~2015 inclduded
an additional differentiation by media channel (for multichannel media)
and would build a separate CalcStream for each channel.
The in-depth analysis conducted for the ongoing »Vertical Slice« effort
revealed that this differentiation is besides the point and would never
be materialised: Since -- by definition -- all media processing has
to be done by the engine, also the generation of the final output format
including any channel multiplexing will happen in render nodes.
The only exception would be when only a single channel of multichannel
media is extracted -- yet this case would then translate into a
dedicated ModelPort.
Based on this reasoning, a lot of complexity (and some contradictions)
within the JobTicket implementation can be removed -- together with
some further leftovers of the fist attempt to build JobTickets always
from a Mock specification (we now use construction by the Segment,
based on an ExitNode, which is the expected actual implementation
for production setup)
...by defining a new scheme for access to custom allocators
...and then passing a reference to such an accessor into the
JobTicket ctor, thereby allowing the ticket istelf recursively
to place further JobTicket instances into the allocation space
--> success, test passes (finally)
Up to now, a draft/mock implementation was used, relying on a »spec tuple«,
which was fabricated by MockJobTicket. But with the introduction of
NodeGraphAttachment, the MockSequence now generates a nested ExitNode structure,
and thus the JobTicket will be created through the "real" ctor, and
no longer via MockJobTicket.
Thus it is possible to skip this whole interspersed »spec tuple«,
since ExitNode *is* already this aggregated / abstracted Spec
PROBLEM: can not implement Spec-generation, since
- we must use a λ for internal allocation of JobTickets
- but recursive type inference is not possible
Will thus need to abandon the Spec-Tuple and relocate this
traversal-and-generation code into JobTicket itself
Use another unit-test (FixtureSegment_test) to guide and cover
the transition from the existing fake-implementation to the
actual implementation, where the JobTicket will be generated
on-demand, from a NodeGraphAttachment
It turns out that the real (not mocked) implementation of JobTicket creation
is already required now for this planned (mock)Dispatcher setup;
moreover, this real implementation turns out to be almost identical
to the mock implementation written recently -- just nested structure
of prerequiste JobTickets need to be changed into a similar structur
of ExitNodes
-- as an aside: rearrange various tests to be more in-line
with the envisioned architecture of playback and engine
...this opens up yet another difficult question and a host of new problems
- how are prerequisites detected or arranged by the Builder
- how are prerequisites represented?
- what is an ExitNode in terms of implementation? A subclass of ProcNode?
- how will the actual implementation of JobTicket creation (on-demand) work?
- how to adapt the Mock implementation, while retaining the Specification
for Segments and prerequisites?
...it turns out that we actually do not need to wrap TreeExplorer
on the builder types, because basically there is only a single active
builder type, and the complete processing pipeline can be assembled
in a single terminal function.
The type rebinding problem can thus be solved just by a simple
marker struct, which inherits from a template parameter
...hard to tackle...
The idea is to wrap the TreeExplorer builder, so that our specific
builder functions can delegated to the (inherited) generic builder functions
and would just need to supply some cleverly bound lambdas. However,
resulting types are recursive, which does not play nice with type inference,
and working around that problem leads to capturing a self reference,
which at time of invocation is already invalidated (due to moving the
whole pipeline into the final storage)
...which leads to the next daunting problems:
- we need some mocked ModelPort and DataSink placeholders
- we need a way how to inherit from a partial TreeExplorer pipeline
...introduced in preparation for building the Dispatcher pipeline,
which at its core means to iterate over a sequence of frame positions;
thus we need a way to stop rendering at a predetermined point...
several years ago, it seemed like a good idea to incorporate
the link between nominal time and wall-clock time into a dedicated
anchor point, which also regulates the continued frame planning.
But it turned out that such a design mixes up several concepts
and introduces confusion regarding the meaning of "real time"
- latency can not be reasonably defined for a whole planning chunk
- skipping or sliding due to missed deadlines can not reasonably handled
within such an abstract entity; it must be handled rather at the
level of a playback process
- linking the frame grid generation directly to a planning chunk
undercuts the possible abstraction of a planning pipeline
...which is build a »Job planning pipeline« step by step
in a test setup, and then factor that out as RenderDrive,
to supersede the existing CalcPlanContinuation and get
rid of the Monads this way...
Challenges
- there is a inconsistency with channel usage
- need to establish a way how to transport the output-Sink into the JobFunctor
- need a way to propagate the current frame number to the next planning chunk
The prototypical setup of data structures and test support components
is largely complete by now — with the exception of the `MockDispatcher`,
which will be completed while moving to the next steps pertaining the
setup of a frame dispatch pipeline.
* the existing `DummyJob` was augmented to allow verification of
association between Job and `JobTicket`
* the existing implementation of `JobTicket` was verified and augmented
to allow coverage of the whole usage cycle
* a `MockJobTicket` was implemented on top, which can be generated
from a symbolical test specification (rather than from the real
Fixture data structure)
* a complete `MockSegmentation` was developed, allowing to establish
all the aforementioned data structures without an actual backing
Render Engine. Moreover, `MockSegmentation` can be generated
from the aforementioned symbolic test specification.
* as part of this work, an algorithm to split an existing Segmentation
and to splice in new segments was developed and verified
Last testcase: add deeply nested Prerequisites.
Turns out that the allocator must be able to handle
re-entrant allocations, which std::deque can not fulfil.
Thus using std::list here for the Mock implementation.
In the end, the real allocations will be done by our custom
allocator (AllocationCluster), which can be arranged easily
to support re-entrant allocation calls (since the whole point
is to just place those objects into a pre-allocated large block
and only de-allocate them later in one sway. Thus the allocator
does not need to wait for the object constructor to finish, which
trivially allows for re-entrant calls)
...which uncovers further deeply nested problems,
especially when referring to non-copyable types.
Thus need to construct a common type that can be used
both to refer to the source elements and the expanded elements,
and use this common type as result type and also attempt to
produce better diagnostic messages on type mismatch....
...the improved const correctness on STL iterators uncovered another
latent problem with out diagnositc format helper, which provide
consistently rounded float and double output, but failed to take
CV-qualifiaction into account
This is a subtle and far reaching fix, which hopefully removes
a roadblock regarding a Dispatcher pipeline: Our type rebinding
template used to pick up nested type definitions, especially
'value_type' and 'reference' from iterators and containers,
took an overly simplistic approach, which was then fixed
at various places driven by individual problems.
Now:
- value_type is conceptually the "thing" exposed by the iterator
- and pointers are treated as simple values, and no longer linked
to their pointee type; rather we handle the twist regarding
STL const_iterator direcly (it defines a non const value_type,
which is sensible from the STL point of view, but breaks our
generic iterator wrapping mechanism)
To complete the mock setup, the next step would be to extend the GenNode-based spec langage
to allow defining prerequisite Mock-JobTickets. Setting this up seems rather straight forward --
however, defining a simple testcase to cover this extension runs into surprisingly tricky problems..
- for one, the singleValIterator from Itertools has serious difficulties handling references
- but even more surprising, it seems impossible to make the "prerequisites iterator"
fit into the Tree-Explorer framework (which I intend to use as replacement
for the monadic approach)
after some extended analysis of generic types and template instances,
it seems that not TreeExplorer as such is the primary problem, but rather
there is a conceptual mismatch somewhere deep down in Itertools or Iter-Adapter
By reasoning and analysis I conclude that the differentiation into
multiple channels is likely misplaced in JobTicket; it belongs ratther
into the Segment and should provide a suitable JobTicket for each ModelPort
Handling of prerequisites also needs to be reshaped entirely after
switching to a pipeline builder for the Job-planning pipeline; as
preliminary access point, just add an iterator over the immediate
prerequisites, thereby shifting the exploration mechanism entirely
out of the JobTicket implementation
Testcase: A simple Sementation with a single and bounded Segment
As aside, figured out how to unpack an iterator such as to
tie a fixed number of references through a structural binding:
auto const& [s1,s2,s3] = seqTuple<3> (mockSegs.eachSeg());
...now able to build a mock segmentation which issues dummy jobs,
and is wired such as to verify the right job is invoked for each segment.
And this allows to build and verify the Dispatcher,
without being able to invoke actual render jobs yet.
- only the parts actually touched by the algo will be re-allocated
- when a segment is split, the clone copies carry on all data
Library: add function to check for a bare address (without type info)
...this is something I should have done since YEARS, really...
Whenever working with symbolically represented data, tests
typically involve checking *hundreds* of expected results,
and thus it can be really hard to find out where the
failure actually happens; it is better for readability
to have the expected result string immediately in the
test code; now this expected result can be marked
with a user-defined literal, and then on mismatch
the expected and the real value will be printed.
There are 12 distinct cases regarding the orientation of two intervals;
The Segmentation::splitSplice() operation shall insert a new Segment
and adjust / truncate / expand / split / delete existing segments
such as to retain the *Invariant* (seamless segmentation covering
the complete time axis)
- how to pass-in a specification given as GenNode
- now this might be translated into a MockJobTicket allocated in the MockSegmentation
Unimplemented: actually build the Segment with suitable start/end time
right now we're lacking a complete working implementation of render node invocation,
and thus the Dispatcher implementation can only be verified with the help
of mocked jobs. However, at least a preliminary implementation of tagging the
invocation instance is available, and thus we're able to verify that
a given job instance indeed belongs to and is "backed" by a specific JobTicket.
This is prerequisite for building up a (likewise mocked) Fixture datastructure,
and this in turn was meant to form the basis for attacking an actual Scheduler
implementation, followed by a real render node invocation.
- can now create a Job from JobTicket::NIL
- on invocation this Job will to nothing
Only when the first real output backend is implemented,
we can decide if this simplistic implementation is enough,
or if an empty output must be explicitly generated...
* using a simplified preliminary implementation of hash chaining (see #1293)
* simplistic implementation of hashing for time values (half-rotation)
* for now just hashing the time into the upper part of the LUID
Maybe we can even live with that implementation for some time,
depending on how important uniform distribution of hash values is
for proper usage of the frame cache.
Needless to say, various further fine points need more consideration,
especially questions of portability (32bit anyone?). Moreover, since
frame times are typically quantised, the search space for the hashed
time values is drastically reduced; conceivably we should rather
research and implement a good hash function for 128bit and then combine
all information into a single hash key....
...using the MockJobTicket setup as point of reference,
since the actual invocation of render nodes will only be drafted
later in this "Vertical Slice" integration effort...
- introduce a JobTicket::NOP (null-object pattern)
- assuming that the function splitSplice() will retain complete coverage allways
Remark:
`Fixture::getPlaylistForRender()` is a leftover from the very early implementation drafts.
This function was more or less based on the way Cinelerra works; it is clear by now
that Lumiera can not possibly work this way, given that we'll build a low-level model
and dispatch precompiled render jobs....
The Fixture and the low-level model backbone deserve a distinct namespace on their own.
Since it's built by the Builder from the Session contents, and also used by the frame dispatch,
we can expect dependence on some types from Steam-Layer, and thus this namespace
needs to reside in Steam-Layer rather, while the actual low-level Model
might become part of Vault-Layer, creating a hierarchy of data structures.
(Remark: likely also the session related namespaces will need a reorganisation)
The idea is to escape a "design deadlock" by using a test-driven prototype
implementation of the data structure to back a further development
of the Dispatcher and Scheduler implementation, which then can be used
to gradually elaborate and switch over to an actual implementation
data structure
...requires a first attempt towards defining a `JobTiket`.
This turns out quite tricky, due to using those `LinkedElements`
(intrusive single linked list), which requires all added records
actually to live elsewhere. Since we want to use a custom allocator
later (the `AllocationCluster`), this boils down to allocating those
records only when about to construct the `JobTicket` itself.
What makes matters even worse: at the moment we use a separate spec
per Media channel (maybe these specs can be collapsed later non).
And thus we need to pass a collection -- or better an iterator
with raw specs, which in turn must reveal yet another nested
sequence for the prerequisite `JobTickets`.
Anyhow, now we're able at least to create an empty `JobTicket`,
backed by a dummy `JobFunctor`....
Looks like we'll actually retain and use this low-level solution
in cases where we just can not afford heap allocations but need
to keep polymorphic objects close to one another in memory.
Since single linked lists are filled by prepending, it is rather
common to need the reversed order of elements for traversal,
which can be achieved in linear time.
And while we're here, we can modernise the templated emplacement functions
- build the reworked Job-planning pipeline more or less from scratch
- back that with mocked `Dispatcher` and `JobTicket`
- then transfer this into a `RenderDrive`, which can be tested as well
- could continue then to a `CalcStream` integration test....
- decision: the Monad-style iteration framework will be abandoned
- the job-planning will be recast in terms of the iter-tree-explorer
- job-planning and frame dispatch will be disentangled
- the Scheduler will deliberately offer a high-level interface
- on this high-level, Scheduler will support dependency management
- the low-level implementation of the Scheduler will be based on Activity verbs
This finishes a long lasting effort to rework the top-level of the Lumiera GTK UI,
to adapt to GTK-3 and the new asynchronous message based architecture.
Special credits and thanks to
* Joel Holdsworth
* Stefan Kangas
Without their relentless foundational work, the Lumiera UI could
never be where it is now. Even if some code was rewritten and several
parts of the old GTK-2 implementation are now obsolete, numerous ideas
solutions and inspirations were drawn from those early contributions
and live on as part of the reworked GUI.
Note: changing behaviour of TimeSpan to possibly flip start and end,
and also to use Offset as Offset and then re-orient,
since this seems the least surprising behaviour.
These changes carry over into changed default and limiting
on ZoomWindow constructor and various mutators, and most
notably shifting the time span always into allowed domain.
...the implementation was way too naive; in some cases we could go
into an infinite loop. In the end, using Newton approximation was not
necessary (and thus there is no loop anymore), but it helped me get
at a much better solution with very small error margin on average case.
All these corner cases are obviously "academic" to some degree,
but it turns out there is no clear-cut point where you'd be able
just so set a limit and be sure that fractional integer arithmetic
works flawless in all cases.
Thus the choice is
- give up (fractional) integers and work with floats and have to
deal with error accumulation
- or do something as chosen here, namely add a boundary zone, where
fractional integer arithmetic can be kept under control, while admitting
small errors, and in turn get the absolutely precise integers in all
everyday standard cases
The value used previously was too conservative, and prevented ZommWindow
from zooming out to the complete Time domain. This was due to missing the
Time::SCALE denominator, which increaded the limit by factor 1e6
In fact the code is able to handle even this extremely reduced limit,
but doing so seems over the top, since now detox() kicks in on several
calculations, leading to rather coarse grained errors.
Thus I decided to use a compromise: lower the limit only by factor 1000;
with typical screen pixel widths, we can reach the full time domain,
while most scaling and zoom calculations can be performed precisely,
without detox() kicking in. Obviously this change requires adjusting
a lot of the test case expectations, since we can now zoom out maximally.
As it turns out, the calculation path initially choosen for the mutateScale(Rat)
was needlessly indirect, and also duplicated several of the safeguards,
meanwhile implemented way better in conformWindowToMetric(Rat)
Thus, instead of relatively re-scaling the window, now we just
limit the given zoomFactor and pass it to conformWindowToMetric()
There is a built-in limitation, which now is even
lowered to 100000 pixels horizontally.
With the techniques introduced in this changeset, it seems possible
to support more -- yet this would be a case of unnecessary genricity;
handling such large numbers will drive more computations into the
danger zone, and doing so incurs cost in terms of testing and debugging.
Placing that into context, contemporary displays are not even 4K on
average, and it does not look likely even for cinema display to go
way beyond 8k -- so yes, I want display hardware with 100000 pixels!!
The key takeaway of this changeset:
- can calculate px = trunc(zoomFactor * duration) step wise,
even when the direct calculation would lead to wrap-around
- can safely adjust and fix the zoomFactor using Newton approximation
...even zooming out to span the complete time domain (~19000 years).
But only under the condition that the display window is sufficiently
large in terms of pixels, so we can handle the computation without
glitches.
This should not be a relevant limitation in practice, since a window
size of some 100 pixels is enough to handle Duration::MAX. Needless to add
that it's hard to imagine a media timeline of such tremendous size...
building on these Library changes, plus the safe-add function
developed some days ago, it is now possible to mark a large displacement
as `time::Offset`, and apply this to yield any valid time position,
even extreme negative values
...building on these Library changes, plus the safe-add function
developed some days ago, it is now possible to mark a large displacement
as `time::Offset`, and apply this to yield any valid time position,
even extreme negative values
The APIs for time quantisation were drafted in an early stage of the project
and then never followed-up. Especially Grid::gridAlign has no
real-world usage yet, and is only massaged in some tests.
When looking at QuantiserBasics_test, I was puzzled and led astray,
since this function suggests to materialise a continuous time into
a quantised time -- which it doesn't (there is another dedicated
function Quantiser::materialise() to that end); so, without engaging
into the discussion if this function is of any use, I'll hereby
choose a name better reflecting what it does.
This is a deep refactoring to allow to represent the distance
between all valid time points as a time::Offset or time::Duration.
By design this is possible, since Time::MAX was defined as 1/30 of
the maximum value technically representable as int64_t. However,
introducing a different limiter for offsets and durations turns
out difficult, due to the inconsistencies in the exiting hierarchy
of temporal entities. Which in turn seems to stem from the unfortunate
decision to make time entities immutable, see #1261
Since the limiter is hard wired into the `time::TimeValue` constructor,
we are forced to create a "backdoor" of sorts, to pass up values
with different limiting from child classes. This would not be so
much of a problem if calculations weren't forced to go through `TimeVar`,
which does not distinguish between time points and time durations.
This solution rearranges all checks to be performed now by time::Offset,
while time::Duration will only take the absolute value at construction,
based on the fact that there is no valid construction path to yield
a duration which does not go through an offset first.
Later, when we're ready to sort out the implementation base of time values
(see #1258), this design issue should be revisited
- either we'll allow derived classes explicitly to invoke the limiter functions
- or we may be able to have an automatic conversion path from clearly
marked base implementation types, in which case we wouldn't use the
buildRaw_() and _raw() "backdoor" functions any more...
While the calculation was already basically under control, I just was not content
with the achieved numeric precision -- and the fact that the test case in fact
misses the bar, making it difficult do demonstrate that the calculation
is not derailed. I just had the gut feeling that it must be somehow possible
to achieve an absolute error level, not just a relative error level of 1e-6
Thus I reworked this part into a generic helper function, see #1262
The end result is:
* partial failure. we can only ''guarantee'' the relative error margin of 1e-6
* but in most cases not out to the most extreme numbers, the sophisticated
solution achieves much better results way below the absolute error level of 1µ-Tick
Thus with using rational numbers, we have now a solution that is absolutely precise
in the regular case, and gradually introduces errors at the domain bound
but with an guaranteed relative error margin of 1e-6 (== Time::SCALE)
...in a similar vein as done for the product calculation.
In this case, we need to check the dimensions carefully and pick
the best calculation path, but as long as the overall result can
be represented, it should be possible to carry out the calculation
with fractional values, albeit introducing a small error.
As a follow-up, I have now also refactored the re-quantisation
functions, to be usable for general requantisation to another grid,
and I used these to replace the *naive* implementation of the
conversion FSecs -> µ-Grid, which caused a lot of integer-wrap-around
However, while the test now works basically without glitch or wrap,
the window position is still numerically of by 1e-6, which becomes
quite noticeably here due to the large overall span used for the test.
...using a requantisation trick to cancel out some factors in the
product of two rational numbers, allowing to calculate the product
without actual multiplication of (dangerously large) numbers.
with these additional safeguards, the anchorWindowAtPosition()
succeeds without Integer-wrap, but the result is not fully correct
(some further calculation error hidden somewhere??)
- detailed documentation of known problematic behaviour
when working with rational fractions
- demonstrate the heuristic predicate to detect dangerous numbers
- add extensive coverage and microbenchmarks for the integer-logarithm
implementation, based on an example on Stackoverflow. Surprising result:
The std::ilog(double) function is of comparable speed, at least for
GCC-8 on Debian-Buster.
