The idea is to use some source of randomness to pick a
limited parameter value with controllable probability.
While the core of the implementation is nothing more
than some simple numeric adjustments, these turn out
to be rather intricate and obscure; the desire to
package these technicalities into a component
however necessitates to make invocations
at usage site self explanatory.
...these were developed driven by the immediate need
to visualise ''random generated computation patterns''
for ''Scheduler load testing.''
The abstraction level of this DSL is low
and structures closely match some clauses of the DOT language;
this approach may not yet be adequate to generate more complex
graph structures and was extracted as a starting point
for further refinements....
With all the preceding DSL work, this turns out to be surprisingly easy;
the only minor twist is the grouping of nodes into (time)levels,
which can be achieved with a "lagging" update from the loop body
Note: next step will be to extract the DSL helpers into a Library header
...refine the handling of FrameRates close to the definition bounds
...implement the actual rule to scale allocator capacity on announcement
...hook up into the seedCalcStream() with a default of +25FPS
+ test coverage
...whenever a new CalcStream is seeded, it would be prudent
not only to step up the WorkForce (which is already implemented),
but also to provide a hint to the BlockFlow allocator regarding
the expected calculation density.
Such a hint would allow to set a more ample »epoch« spacing,
thereby avoiding to drive the allocator into overload first.
The allocator will cope anyway and re-balance in a matter of
about 2 seconds, but avoiding this kind of control oscillations
altogether will lead to better performance at calculation start.
Use a simple destructor-trick to set up a concise notation
for temporarily manipulating a value for testing.
The manipulation will automatically be undone
when leaving scope
While testing, I repeatedly had SEGFAULT in the new thread-wrapper,
but only when running under debugger. While the language spec guarantees
that exit from the thread handle initialisation synchronizes-with
the start of the new thread, there is no guarantee in the reverse
direction. Here this means that the new thread may not see the
newly initialised thread handle ID at start. Thus I've added
a yield-wait at the very beginning of the new thread function.
Under normal conditions, the startup of a thread takes at least
100 - 500µs and thus I've never seen the problematic behaviour
without debugger. However, adding a yield-wait loop at that point
seems harmless (it typically checks back every 400ns or so).
All real usages of the thread wrapper in the application use
some kind of additional coordination or even a sync barrier
to ensure the thread can pick up all further data before
going into active work.
WARNING: if someone would detach() the thread immediately after
creating it, then this added condition would cause the starting
thread function to hang forever. In our current setup for the
thread wrapper, this is not possible, since the thread handle
is embedded into protected code. The earliest point you could
do that would be in the handle_begin_thread(), which is called
from the thread itself *after* the new check. And moreover,
this would require to write a new variation of the Policy.
While building increasingly complex integration tests for the Scheduler,
it turns out helpful to be able to manipulate the "full concurreency"
as used by Scheduler, WorkForce and LoadController.
In the current test, I am facing a problem that new entries from the
threadsafe entrance queue are not propagated to the priority queue
soon enough; partly this is due to functionality still to be added
(scaling up when new tasks are passed in) -- but this will further
complicate the test setup.
as an aside, the header lib/test/microbenchmark.hpp
turns out to be prolific for this kind of investigation.
However, it is somewhat obnoxious that the »test subject«
must expose the signature <size_t(size_t)>.
Thus, with some metaprogramming magic, an generic adaptor
can be built to accept a range of typical alternatives,
and even the quite obvious signature void(void).
Since all these will be wrapped directly into a lambda,
the optimiser will remove these adaptations altogether.
...there seemed to be an anomaly of 50...100µs
==> conclusion: this is due to the instrumentation code
- it largely caused by the EventLog, which was never meant
to be used in performance-critical code, and does hefty
heap allocations and string processing.
- moreover, there clearly is a cache-effect, adding a Factor 2
whenever some time passed since the last EventLog call
==> can be considered just an artifact of the test setup and
will have no impact on the scheduler
remark: this commit adds a lot of instrumentation code
To cover the visible behaviour of the work-Function,
we have to check an amalgam of timing delays and time differences.
This kind of test tends to be problematic, since timings are always
random and also machine dependent, and thus we need to produce pronounced effects
...to bring it more in line with all the other calls dealing with Activity*
...allows also to harmonise the ActivityLang::dispatchChain()
...and to compose the calls in Scheduler directly
NOTE: there is a twist: our string-formatting helper did not render
custom string conversions for objects passed as pointer. This was a
long standing problem, caused by ambiguous templates overloads;
now I've attempted to solve it one level more down, in util::StringConv.
This solution may turn out brittle, since we need to exclude any direct
string conversion, most notably the ones for C-Strings (const char*)
In case this solution turns out unsustainable, please feel free
to revert this API change, and return to passing Activity& in λ-post,
because in the end this is cosmetics.
The approach to provide the ExecutionCtx seems to work out well;
after some investigation I found a solution how to code a generic
signature-check for "any kind of function-like member"...
(the trick is to pass a pointer or member-pointer, which happens
to be syntactically the same and can be handled with our existing
function signature helper after some minor tweaks)
As follow-up to the rework of thread-handling, likewise also
the implementation base for locking was switched over from direct
usage of POSIX primitives to the portable wrappers available in
the C++ standard library. All usages have been reviewed and
modernised to prefer λ-functions where possible.
With this series of changes, the old threadpool implementation
and a lot of further low-level support facilities are not used
any more and can be dismantled. Due to the integration efforts
spurred by the »Playback Vertical Slice«, several questions of
architecture could be decided over the last months. The design
of the Scheduler and Engine turned out different than previously
anticipated; notably the Scheduler now covers a wider array of
functionality, including some asynchronous messaging. This has
ramifications for the organisation of work tasks and threads,
and leads to a more deterministic memory management. Resource
management will be done on a higher level, partially superseding
some of the concepts from the early phase of the Lumiera project.
This is Step-2 : change the API towards application
Notably all invocation variants to support member functions
or a reference to bool flags are retracted, since today a
λ-binding directly at usage site tends to be more readable.
The function names are harmonised with the C++ standard and
emergency shutdown in the Subsystem-Runner is rationalised.
The old thread-wrapper test is repurposed to demonstrate
the effectiveness of monitor based locking.
After the fundamental switch from POSIX to the C++14 wrappers
the existing implementation of the Monitor can now be drastically condensed,
removing several layers of indirection. Moreover, all signatures
shall be changed to blend in with the names and patterns established
by the C++ standard.
This is Step-1 : consolidate the Implementation.
(to ensure correctness, the existing API towards application code was retained)
While not directly related to the thread handling framework,
it seems indicated to clean-up this part of the application alongside.
For »everyday« locking concerns, an Object Monitor abstraction was built
several years ago and together with the thread-wrapper, both at that time
based on direct usage of POSIX. This changeset does a mere literal
replacement of the POSIX calls with the corresponding C++ wrappers
on the lowest level. The resulting code is needlessly indirect, yet
at API-level this change is totally a drop-in replacment.
The WorkForce (passive worker pool) has been coded just recently,
and -- in anticipation of this refactoring -- directly against std::thread
instead of using the old framework.
...the switch is straight-forward, using the default case
...add the ability to decorate the thread-IDs with a running counter
This solution is basically equivalent to the version implemented directly,
but uses the lifecycle-Hooks available through `ThreadHookable`
to structure the code and separate the concerns better.
This largely completes the switch to the new thread-wrapper..
**the old implementation is not referenced anymore**
Add a complete demonstration for a setup akin to what we use
for the Session thread: a threaded component which manages itself
but also exposes an external interface, which is opened/closed alongside
...extract and improve the tuple-rewriting function
...improve instance tracking test dummy objects
...complete test coverage and verify proper memory handling
After quite some detours, with this take I'm finally able to
provide a stringent design to embody all the variants of thread start
encountered in practice in the Lumiera code base.
Especially the *self-managed* thread is now represented as a special-case
of a lifecycle-hook, and can be embodied into a builder front-end,
able to work with any client-provided thread-wrapper subclass.
extract into helper function to improve legibility.
This code is rather tricky since on invocation the hook is only provided
but not invoked. Rather, to adapt the argument types, it is wrapped
into a λ for adaptation, which then must be again bound *by value*
into yet another λ, since the Launch configuration builder is comprised
of a chain of captured functors, to be invoked later from the body of the
thread-wrapper object; this indirect procedure is necessary to ensure
all members are initialised *before* the new thread starts
to cover the identified use-cases a wide variety of functors
must be accepted and adapted appropriately. A special twist arises
from the fact that the complete thread-wrapper component stack works
without RTTI; a derived class can not access the thread-wrapper internals
while the policy component to handle those hooks can not directly downcast
to some derived user provided class. But obviously at usage site it
can be expected to access both realms from such a callback.
The solution is to detect the argument type of the given functor
and to build a two step path for a safe static cast.
after some further mulling over the design, it became clear that
a rather loose coupling to the actual usage scenario is preferrable.
Thus, instead of devising a fixed scheme how to reflect the thread state,
rather the usage can directly hook into some points in the thread lifecycle.
So this policy can be reduced to provide additional storage for functon objects.
...after resolving the fundamental design problems,
a policy mix-in can be defined now for a thread that deletes
its own wrapper at the end of the thread-function.
Such a setup would allow for »fire-and-forget« threads, but with
wrapper and ensuring safe allocations. The prominent use case
for such a setup would be the GUI-Thread.
So this finally solves the fundamental problem regarding a race on
initialisation of the thread-wrapper; it does *not* solve the same problem
for classes deriving from thread-wrapper, which renders this design questionable
altogether -- but this is another story.
In the end, this initialisation-race is rooted in the very nature of starting a thread;
it seems there are the two design alternatives:
- expose the thread-creation directly to user code (offloading the responsibility)
- offer building blocks which are inherently dangerous
this is a mere rearrangement of code (+lots of comments),
but helps to structure the overall construction better.
ThreadWrapper::launchThread() now does the actual work to build
the active std::thread object and assign it to the thread handle,
while buildLauncher is defined in the context of the constructors
and deals with wiring the functors and decaying/copying of arguments.
If we package all arguments together into a single tuple,
even including the member-function reference and the this-ptr
for the invokeThreadFunction(), which is the actual thread-functor,
then we can rely on std::make_from_tuple<T>(tuple), which implements
precisely the same hand-over via a std::index_sequence, as used by the
explicitly coded solution -- getting rid of some highly technical boilerplate
Concept study of the intended solution successful.
Can now transparently embed any conceivable functor
and an arbitrary argument sequence into a launcher-λ
Materialising into a std::tuple<decay_t<TYPES...>> did the trick.
Considering a solution to shift the actual launch of the new thread
from the initialiser list into the ctor body, to circumvent the possible
"undefined behaviour". This would also be prerequisite for defining
a self-managed variant of the thread-wrapper.
Alternative / Plan.B would be to abandon the idea of a self-contained
"thread" building block, instead relying on precise setup in the usage
context -- however, not willing to yield yet, since that would be exactly
what I wanted to avoid: having technicalities of thread start, argument
handover and failure detection intermingled with the business code.
On a close look, the wrapper design as pursued here
turns out to be prone to insidious data race problems.
This was true also for the existing solution, but becomes
more clear due to the precise definitions from the C++ standard.
This is a confusing situation, because these races typically do not
materialise in practice; due to the latency of the OS scheduler the
new thread starts invoking user code at least 100µs after the Wrapper
object is fully constructed (typically more like 500µs, which is a lot)
The standard case (lib::Thread) in its current form is correct, but borderline
to undefined behaviour, and any initialisation of members in a derived class
would be off limits (the thread-wrapper should not be used as baseclass,
rather as member)
...while reworking the application code, it became clear that
actually there are two further quite distinct variants of usage.
And while these could be implemented with some trickery based on
the Thread-wrapper defined thus far, it seems prudent better to
establish a safely confined explicit setup for these cases:
- a fire-and-forget-thread, which manages its own memory autonomously
- a thread with explicit lifecycle, with detectable not-running state
FamilyMember::allocateNextMember() was actually a post-increment,
so (different than with TypedCounter) here no correction is necessary
As an asside, WorkForce_test is sometimes unstable immediately after a build.
Seemingly a headstart of 50µs is not enough to compensate for scheduler leeway
The existing TypedCounter_test was excessively clever and convoluted,
yet failed to test the critical elements systematically. Indeed, two
bugs were hidden in synchronisation and instance access.
- build a new concurrent test from scratch, now using the threadBenchmark
function for the actual concurrent execution and just invoked a
random selected access to the counter repeatedly from a large number
of threads.
- rework the TypedContext and counter to use Atomics where applicable;
measurements indicate however that this has only negligible impact
on the amortised invocation times, which are around 60ns for single-threaded
access, yet can increase by factor 100 due to contention.
...these were already written envisionaging he new API,
so it's more or less a drop-in replacement.
- cant use vector anymore, since thread objects are move-only
- use ScopedCollection instead, which also has the benefit of
allocating the requires space up-front. Allow to deduce the
type parameter of the placed elements
... which became apparent after switching to the new Thread-wrapper implementation
... the reason is a bug in the Thread-Monitor (which will also be reworked soon)
While seemingly subtle, this is a ''deep change.''
Up to now, the project attempted to maintain two mutually disjoint
systems of error reporting: C-style error flags and C++ exceptions.
Most notably, an attempt was made to keep both error states synced.
During the recent integration efforts, this increasingly turned out
as an obstacle and source for insidious problems (like deadlocks).
As a resolve, hereby the relation of both systems is **clarified**:
* C-style error flags shall only be set and used by C code henceforth
* C++ exceptions can (optionally) be thrown by retrieving the C-style error code
* but the opposite is now ''discontinued'' : Exceptions ''do not set'' the error flag anymore
- the deadlock was caused by leaking error state through the C-style lumiera_error
- but the reason for the deadlock lies in the »convenience shortcut«
in the Object-Monitor scope guard for entering a wait state immediately.
This function undermines the unlocking-guarantee, when an exception
emanates from within the wait() function itself.
...this function was also ported to the new wrapper,
and can be verified now in a much more succinct way.
''This completes porting of the thread-wrapper''
Since the decision was taken to retain support for this special feature,
and even extend it to allow passing values, the additional functionality
should be documented in the test. Doing so also highlighted subtle problems
with argument binding.
A subtle yet important point: arguments will always be copied into the new thread.
This is a (very sensible) limitation introduced by the C++ standard.
To support seamless use, the thread-wrapper now rewrites the argument types
picked up from the invocation, to prevent passing on a reference type,
which typically ensues when invoking with a variable name. Otherwise
confusing error messages would be emitted from deep within the STD library.
As a further consequence, function signatures involving reference arguments
can no longer be bound (which is desirable; a function to be performed
within a separate thread must either rely on value arguments, or deliberately
use std::ref wrappers to pass references, assuming you know what you're doing)
- it is not directly possible to provide a variadic join(args...),
due to overload resolution ambiguities
- as a remedy, simplify the invocation of stringify() for the typical cases,
and provide some frequently used shortcuts
A common usage pattern is to derive from lib::Thread
and then implement the actual thread function as a member function
of this special-Thread-object (possibly also involving other data members)
Provide a simplified invocation for this special case,
also generating the thread-id automatically from the arguments
after all this groundwork, implementing the invocation,
capturing and hand-over of results is simple, and the
thread-wrapper classes became fairly understandable.
This relieves the Thread policy from a lot of technicalities,
while also creating a generally useful tool: the ability to invoke
/anything callable/ (thanks to std::invoke) in a fail-safe way and
transform the exception into an Either type
on second thought, the ability to transport an exception still seems
worthwhile, and can be achieved by some rearrangements in the design.
As preparation, reorganise the design of the Either-wrapper (lib::Result)
VERIFY_ERROR allows to check that an expected except is actually thrown.
The implementation was lazy however;
it just investigated the C-style error flag instead of *really* verifying
that an *lumiera::Exception* with the expected flag was caught.
This discrepancy can be a problem when there is a stray error flag set,
or for some reason the error flag gets cleared before the exception
reaches the top-level catch-block in the test.
- relocate some code into a dedicated translation unit to reduce #includes
- actually set the thread-ID (the old implementation had only a TODO at that point)
While it would be straight forward from an implementation POV
to just expose both variants on the API (as the C++ standard does),
it seems prudent to enforce the distinction, and to highlight the
auto-detaching behaviour as the preferred standard case.
Creating worker threads just for one computation and joining the results
seemed like a good idea 30 years ago; today we prefer Futures or asynchronous
messaging to achieve similar results in a robust and performant way.
ThreadJoinable can come in handy however for writing unit tests, were
the controlling master thread has to wait prior to perform verification.
So the old design seems well advised in this respect and will be retained
- cut the ties to the old POSIX-based custom threadpool framework
- remove operations deemed no longer necessary
- sync() obsoleted by the new SyncBarrier
- support anything std::invoke supports
...which is the technique used in the existing Threadpool framwork.
As expected, such a solution is significantly slower than the new
atomics-based implementation. Yet how much slower is still striking.
Timing measurements in concurrent usage situation.
Observed delay is in the order of magnitude of known scheduling leeway;
assuming thus no relevant overhead related to implementation technique
Over time, a collection of microbenchmark helper functions was
extracted from occasional use -- including a variant to perform
parallelised microbenchmarks. While not used beyond sporadic experiments yet,
this framework seems a perfect fit for measuring the SyncBarrier performance.
There is only one catch:
- it uses the old Threadpool + POSIX thread support
- these require the Threadpool service to be started...
- which in turn prohibits using them for libary tests
And last but not least: this setup already requires a barrier.
==> switch the existing microbenchmark setup to c++17 threads preliminarily
(until the thread-wrapper has been reworked).
==> also introduce the new SyncBarrier here immediately
==> use this as a validation test of the setup + SyncBarrier
Using the same building blocks, this operation can be generalised even more,
leading to a much cleaner implementation (also with better type deduction).
The feature actually used here, namely summing up all values,
can then be provided as a convenience shortcut, filling in std::plus
as a default reduction operator.
...first used as part of the test harness;
seemingly this is a generic and generally useful shortcut,
similar to algorithm::reduce (or some kind of fold-left operation)
Intended as replacement for the Mutex/ConditionVar based barrier
built into the exiting Lumiera thread handling framework and used
to ensure safe hand-over of a bound functor into the starting new
thread. The standard requires a comparable guarantee for the C++17
concurrency framework, expressed as a "synchronizes_with" assertion
along the lines of the Atomics framework.
While in most cases dedicated synchronisation is thus not required
anymore when swtiching to C++17, some special extended use cases
remain to be addressed, where the complete initialisation of
further support framework must be ensured.
With C++20 this would be easy to achieve with a std::latch, so we
need a simple workaround for the time being. After consideration of
the typical use case, I am aiming at a middle ground in terms of
performance, by using a yield-wait until satisfying the latch condition.
The investigation for #1279 leads to the following conclusions
- the features and the design of our custom thread-wrapper
almost entirely matches the design chosen meanwhile by the C++ committee
- the implementation provided by the standard library however uses
modern techniques (especially Atomics) and is more precisely worked out
than our custom implementation was.
- we do not need an *active* threadpool with work-assignment,
rather we'll use *active* workers and a *passive* pool,
which was easy to implement based on C++17 features
==> decision to drop our POSIX based custom implementation
and to retrofit the Thread-wrapper as a drop-in replacement
+++ start this refactoring by moving code into the Library
+++ create a copy of the Threadwrapper-code to build and test
the refactorings while the application itself still uses
existing code, until the transition is complete
Up to now, the DiagnosticFun mock in ActivityDetector only
created an EventLog entry on invocation and was able to retunr
a canned result value. Yet for the job invocation scenario test,
it would be desirable to hook-in a λ with a fake implementation
into the ExecutionContext. As a further convenience, the
return value is now default initialised, instead of being
marked as uninitialised until invocation of "returning(val)"
...turns out that util::toString does not explicitly handle pointers differently,
for very good reasons; this function must always work, always produce a simple and
compact representation, and it must be possible to instantiate the template
and take a function reference (which precludes adding an overload for pointers)
requires to supplement EventLog matching primitives
to pick and verify a specific positional argument.
Moreover, it is more or less arbitrary which job invocation parameters
are unpacked and exposed for verification; we'll have to see what is
actually required for writing tests...
Testcase (detect function invocation) passes now as expected
Some Library / Framework changes
- rename event-log-test.cpp
- allow the ExpectString also to work with concatenated expectation strings
Remark: there was a warning in the comment in event-log.hpp,
pointing out that negative assertions are shallow.
However, after the rework in 9/2018 (commit: d923138d1)
...this should no longer be true, since we perform proper backtracking,
leading to an exhaustive search.
ActivityMatch inherits privately from the EventMatch object,
and is thus able to delegate relevant matching queries, but
also to provide high-level special matchers.
This new design resolves the ambiguity regarding function arguments.
Moreover, we can now record the current sequence-Number as *attribute*
in the respective log record (this is the benefit of using structured
log entries instead of just a textual log), thereby avoiding the various
pitfalls with explicit bracketing sequence-number log entries
bottom line: this reworked design seems to be a better fit,
even while technically the implementation with the wrapped matcher
is somewhat ugly...
The EventLog seems to provide all the building blocks, but we need
some higher level special matchers (and maybe we also want to hide
some of the basic EventLog matchers). A soulution might be to wrap
the EventMatcher and delegate all follow-up builder calls.
This seems adequate, since the EventLog-Matcher is basically used as black box,
building up more elaborate matchers from the provided basic matchers...
Spent some time again to understand how EventLog matching works.
My feelings towards this piece of code are always the same: it is
somewhat too "tricky", but I am not aware of any other technique
to get this degree of elaborate chained matching on structured records,
short of building a dedicated matching engine from scratch.
The other alternative would be to use a flat textual log (instead of
the structured log records from EventLog), but then we'd have to
generate quite intricate regular expressions from the builder,
and I'm really doubtful it would be easier and clearer....
...turns out this is entirely generic and not tied to the context
within ActivityDetector, where it was first introduced to build a
mock functor to log all invocations.
Basically this meta-function generates a new instantiation of the
template X, using the variadic argument pack from template U<ARGS...>
...for coverage of the Activity-Language,
various invocations of unspecific functions must be verified,
with the additional twist that the implementation avoids indirections
and is thus hard to rig for tests.
Solution-Idea: provide a λ-mock to log any invocation into the
Event-Log helper, which was created some years ago to trace GUI communication...
Further extensive testing with parameter variations,
using the test setup in `BlockFlow_test::storageFlow()`
- Tweaks to improve convergence under extreme overload;
sudden load peaks are now accomodated typically < 5 sec
- Make the test definition parametric, to simplify variations
- Extract the generic microbenchmark helper function
- Documentation
- BUG: must prevent the Epoch size to become excessive low
- Problem: feedback signal should not be overly aggressive
Fine-Tuning:
- Dose for Overflow-compensation is delicate
- Moving average and Overflow should be balanced
- ideally the compensatory actions should be one order of magnitude
slower than the characteristic regulation time
Improvement: perform Moving-Average calculations in doubles
- fix a bug in IterExplorer: when iterating a »state core« directly,
the helper CoreYield passed the detected type through ValueTypeBindings.
This is logically wrong, because we never want to pick up some typedefs,
rather we always want to use the type directly returned from CORE::yield()
Here the iterator returns an Epoch&, which itself is again iterable
(it inherits from std::array<Activity, N>). However, it is clear
that we must not descent into such a "flatMap" style recursive expansion
- draft a simple scheme how to regulate Epoch lengths dynamically
- add diagnostics to pinpoint a given Activity and find out into which
Epoch it has been allocated; used to cover the allocator behaviour
Especially for the BlockFlow allocator, sanity checks are elided
for performance reasons; yet, generally speaking, it can be a very bad idea
to "optimise" away sanity checks. Thus an additional adaptor is provided
to layer such checks on top of an existing core; and IterEplorer now
always wires in this additional adaptor, and so the original behaviour
is now restored in this respect (and for the largest part of the code base)
While at first sight just a superficial variation of the existing IterStateWrapper,
it became clear with the evolution of the IterExplorer framework that
this setup represents a distinct concept, and especially lends itself
for complex and cohesive collaboration in a layered pipeline. Which
may, or may not be a good idea, depending on the circumstances.
Now, for the implementation of the scheduler memory allocation scheme,
another twist is added to the picture: we can not effort the sanity checks
on each access, even more so when layering / adapting iterators, where
it is essential that the optimiser can remove all unnecessary warts.
Library: add "obvious" utility to the IterExplorer, allowing to
materialise all contents of the Pipeline into a container
...use this to take a snapshot of all currently active Extent addresses
Using a Storage* within a wrapper as "pos" will work,
but is borderline trickery, since it amounts to subverting
the idea behind IterAdapter (which is to encapsulate a target
pointer with some control-logic in the managing container).
Using the same storage size and implementation overhead,
it is much more straight-forward to package the complete
iteration logic into a »State Core«, which in this case
however maintains a back-link to the ExtentFamily.
Iteration should just yield an Reference to an Extent,
thereby hiding all details of the actual raw storage (char[]).
This can be achieved by usind a wrapper type around a pointer
into the managing vector; from this pointer we may convert
into a vector::iterator with the trick described here
https://stackoverflow.com/a/37101607/444796
Furthermore, continued planning of the Activity-Language,
basically clarified the complete usage scenario for now;
seems all implementable right away without further difficulties
- the idea is to use slot-0 in each extent for administrative metadata
- to that end, a specialised GATE-Activity is placed into slot-0
- decision to use the next-pointer for managing the next free slot
- thus we need the help of the underlying ExtentFamily for navigating Extents
Decision to refrain from any attempt to "fix" excessive memory usage,
caused by Epochs still blocked by pending IO operations. Rather, we
assume the engine uses sane parametrisation (possibly with dynamic adjustment)
Yet still there will be some safety limit, but when exceeding this limit,
the allocator will just throw, thereby killing the playback/render process
The second design from 2017, based on a pipeline builder,
is now renamed `TreeExplorer` ⟼ `IterExplorer` and uses
the memorable entrance point `lib::explore(<seq>)`
✔
after completing the recent clean-up and refactoring work,
the monad based framework for recursive tree expansion
can be abandoned and retracted.
This approach from functional programming leads to code,
which is ''cool to write'' yet ''hard to understand.''
A second design attempt was based on the pipeline and decorator pattern
and integrates the monadic expansion as a special case, used here to
discover the prerequisites for a render job. This turned out to be
more effective and prolific and became standard for several exploring
and backtracking algorithms in Lumiera.
This very deep change (which requires almost complete rebuild)
was prompted by the need to process an object (JobPlanning),
which holds several references and is thus move-only, in the
middle of a complex processing pipeline with child expansion.
If this works out well, a long-standing and obnoxious problem
with transforming iterators would be solved, albeit by incurring
a (presumably small) performance overhead, since now the new
value is no longer *assigned*, but rather the existing payload
is destroyed and a new instance is copy/move constructed into
the inline buffer.
The primary purpose (and widely used in Lumieara) is to have a
Lambda create a new Object, which is then returned by value
and thus immediately moved into this inline buffer, where it
resides for further use (as long as the enclosing pipeline
stays alive). Unless such an object does very elaborate
allocations and registrations behind the scene, the
expense of assigning vs creating should be the same.
- had to fix a logical inconsistency in the underlying Expander implementation
in TreeExplorer: the source-pipeline was pulled in advance on expansion,
in order to "consume" the expanded element immediately; now we retain
this element (actually inaccessible) until all of the immediate
children are consumed; thus the (visible) state of the PipeFrameTick
stays at the frame number corresponding to the top-level frame Job,
while possibly expanding a complete tree of flexible prerequisites
This test now gives a nice visualisation of the interconnected states
in the Job-Planning pipeline. This can be quite complex, yet I still think
that this semi-functional approach with a stateful pipeline and expand functors
is the cleanest way to handle this while encapsulating all details
`steam/engine/mock-dispatcher.hpp |cpp` now integrates this
''complete mock setup for render jobs and frame dispatching.''
The exising `DummyJob` has been slightly adapted and renamed
to `MockJob` and is tightly integrated with the other mocks.
The implementation of a `MockDispatcher` necessitated to change
the use of `MockJobTicket`. The initial attempts used a complete
mock implementation, but this approach turned out not to be viable.
Instead — based on the ideas developed for the mock setup —
now the prospective real implementation of `JobTicket` is available
and will be used by the mock setup too. Instead of a synthetic spec,
now a setup of recursively connected `ExitNode`(s) is used; the latter
seems to develop into some kind of Facade for the render node network.
Based on this mock setup, we can now demonstrate the (mostly) complete
Job-Planning pipeline, starting from a segmentation up to render jobs,
and verify proper connectivity and job invocation.
✔
...ouch this was insidious: the STL implementation for list does not
return a pointer to the element just allocated, but rather retrieves
and dereferences the back() / front() iterator after returning from emplace_back|front()
...which in case of re-entrant allocations is something wildly different
than the initial allocation. Thus a *cheap* and dirty placeholder implementation
just using a STL container is not possible, and we need at least
to code up likewise cheesy placeholder implementation by hand.
- separate allocation and ctor all
- use an inline buffer in the STL container
- explicitly handle ctor failures to discard allocation
- NOT THREADSAFE and likely WASTFUL in terms of performance
==> MockSupport_test now back to GREEN after complete refactoring
...by defining a new scheme for access to custom allocators
...and then passing a reference to such an accessor into the
JobTicket ctor, thereby allowing the ticket istelf recursively
to place further JobTicket instances into the allocation space
--> success, test passes (finally)
...hard to tackle...
The idea is to wrap the TreeExplorer builder, so that our specific
builder functions can delegated to the (inherited) generic builder functions
and would just need to supply some cleverly bound lambdas. However,
resulting types are recursive, which does not play nice with type inference,
and working around that problem leads to capturing a self reference,
which at time of invocation is already invalidated (due to moving the
whole pipeline into the final storage)
...which leads to the next daunting problems:
- we need some mocked ModelPort and DataSink placeholders
- we need a way how to inherit from a partial TreeExplorer pipeline
...introduced in preparation for building the Dispatcher pipeline,
which at its core means to iterate over a sequence of frame positions;
thus we need a way to stop rendering at a predetermined point...
several years ago, it seemed like a good idea to incorporate
the link between nominal time and wall-clock time into a dedicated
anchor point, which also regulates the continued frame planning.
But it turned out that such a design mixes up several concepts
and introduces confusion regarding the meaning of "real time"
- latency can not be reasonably defined for a whole planning chunk
- skipping or sliding due to missed deadlines can not reasonably handled
within such an abstract entity; it must be handled rather at the
level of a playback process
- linking the frame grid generation directly to a planning chunk
undercuts the possible abstraction of a planning pipeline
The prototypical setup of data structures and test support components
is largely complete by now — with the exception of the `MockDispatcher`,
which will be completed while moving to the next steps pertaining the
setup of a frame dispatch pipeline.
* the existing `DummyJob` was augmented to allow verification of
association between Job and `JobTicket`
* the existing implementation of `JobTicket` was verified and augmented
to allow coverage of the whole usage cycle
* a `MockJobTicket` was implemented on top, which can be generated
from a symbolical test specification (rather than from the real
Fixture data structure)
* a complete `MockSegmentation` was developed, allowing to establish
all the aforementioned data structures without an actual backing
Render Engine. Moreover, `MockSegmentation` can be generated
from the aforementioned symbolic test specification.
* as part of this work, an algorithm to split an existing Segmentation
and to splice in new segments was developed and verified
...which uncovers further deeply nested problems,
especially when referring to non-copyable types.
Thus need to construct a common type that can be used
both to refer to the source elements and the expanded elements,
and use this common type as result type and also attempt to
produce better diagnostic messages on type mismatch....
...the improved const correctness on STL iterators uncovered another
latent problem with out diagnositc format helper, which provide
consistently rounded float and double output, but failed to take
CV-qualifiaction into account
This is a subtle and far reaching fix, which hopefully removes
a roadblock regarding a Dispatcher pipeline: Our type rebinding
template used to pick up nested type definitions, especially
'value_type' and 'reference' from iterators and containers,
took an overly simplistic approach, which was then fixed
at various places driven by individual problems.
Now:
- value_type is conceptually the "thing" exposed by the iterator
- and pointers are treated as simple values, and no longer linked
to their pointee type; rather we handle the twist regarding
STL const_iterator direcly (it defines a non const value_type,
which is sensible from the STL point of view, but breaks our
generic iterator wrapping mechanism)
...in an attempt to resolve the deeply nested problems encountered
while building an iterator pipeline for the Dispatcher. It seems
that I was sloppy some years ago and just "bashed them into submission",
thereby mixing up two different meanings of "value_type"
Moreover I seemingly implemented the same helper trait template twice,
so the first step is to switch all usages to meta::TypeBinding
To complete the mock setup, the next step would be to extend the GenNode-based spec langage
to allow defining prerequisite Mock-JobTickets. Setting this up seems rather straight forward --
however, defining a simple testcase to cover this extension runs into surprisingly tricky problems..
- for one, the singleValIterator from Itertools has serious difficulties handling references
- but even more surprising, it seems impossible to make the "prerequisites iterator"
fit into the Tree-Explorer framework (which I intend to use as replacement
for the monadic approach)
after some extended analysis of generic types and template instances,
it seems that not TreeExplorer as such is the primary problem, but rather
there is a conceptual mismatch somewhere deep down in Itertools or Iter-Adapter
By reasoning and analysis I conclude that the differentiation into
multiple channels is likely misplaced in JobTicket; it belongs ratther
into the Segment and should provide a suitable JobTicket for each ModelPort
Handling of prerequisites also needs to be reshaped entirely after
switching to a pipeline builder for the Job-planning pipeline; as
preliminary access point, just add an iterator over the immediate
prerequisites, thereby shifting the exploration mechanism entirely
out of the JobTicket implementation
- only the parts actually touched by the algo will be re-allocated
- when a segment is split, the clone copies carry on all data
Library: add function to check for a bare address (without type info)
This macro has turned out to be quite useful in cases
where a generic setup / algorithm / builder need to be customised
with λ adaptors for binding to local or custom types. It relies
on the metafunctions defined in lib/meta/function.hpp to match
the signature of "anything function-like"; so this seems the
proper place to provide that macro alongside
...this is something I should have done since YEARS, really...
Whenever working with symbolically represented data, tests
typically involve checking *hundreds* of expected results,
and thus it can be really hard to find out where the
failure actually happens; it is better for readability
to have the expected result string immediately in the
test code; now this expected result can be marked
with a user-defined literal, and then on mismatch
the expected and the real value will be printed.
There are 12 distinct cases regarding the orientation of two intervals;
The Segmentation::splitSplice() operation shall insert a new Segment
and adjust / truncate / expand / split / delete existing segments
such as to retain the *Invariant* (seamless segmentation covering
the complete time axis)
- can now create a Job from JobTicket::NIL
- on invocation this Job will to nothing
Only when the first real output backend is implemented,
we can decide if this simplistic implementation is enough,
or if an empty output must be explicitly generated...
* using a simplified preliminary implementation of hash chaining (see #1293)
* simplistic implementation of hashing for time values (half-rotation)
* for now just hashing the time into the upper part of the LUID
Maybe we can even live with that implementation for some time,
depending on how important uniform distribution of hash values is
for proper usage of the frame cache.
Needless to say, various further fine points need more consideration,
especially questions of portability (32bit anyone?). Moreover, since
frame times are typically quantised, the search space for the hashed
time values is drastically reduced; conceivably we should rather
research and implement a good hash function for 128bit and then combine
all information into a single hash key....
...using the MockJobTicket setup as point of reference,
since the actual invocation of render nodes will only be drafted
later in this "Vertical Slice" integration effort...
...requires a first attempt towards defining a `JobTiket`.
This turns out quite tricky, due to using those `LinkedElements`
(intrusive single linked list), which requires all added records
actually to live elsewhere. Since we want to use a custom allocator
later (the `AllocationCluster`), this boils down to allocating those
records only when about to construct the `JobTicket` itself.
What makes matters even worse: at the moment we use a separate spec
per Media channel (maybe these specs can be collapsed later non).
And thus we need to pass a collection -- or better an iterator
with raw specs, which in turn must reveal yet another nested
sequence for the prerequisite `JobTickets`.
Anyhow, now we're able at least to create an empty `JobTicket`,
backed by a dummy `JobFunctor`....
Looks like we'll actually retain and use this low-level solution
in cases where we just can not afford heap allocations but need
to keep polymorphic objects close to one another in memory.
Since single linked lists are filled by prepending, it is rather
common to need the reversed order of elements for traversal,
which can be achieved in linear time.
And while we're here, we can modernise the templated emplacement functions
- decision: the Monad-style iteration framework will be abandoned
- the job-planning will be recast in terms of the iter-tree-explorer
- job-planning and frame dispatch will be disentangled
- the Scheduler will deliberately offer a high-level interface
- on this high-level, Scheduler will support dependency management
- the low-level implementation of the Scheduler will be based on Activity verbs
This finishes a long lasting effort to rework the top-level of the Lumiera GTK UI,
to adapt to GTK-3 and the new asynchronous message based architecture.
Special credits and thanks to
* Joel Holdsworth
* Stefan Kangas
Without their relentless foundational work, the Lumiera UI could
never be where it is now. Even if some code was rewritten and several
parts of the old GTK-2 implementation are now obsolete, numerous ideas
solutions and inspirations were drawn from those early contributions
and live on as part of the reworked GUI.
It is now tied to the start of ZoomWindow::overallSpan(),
thereby defining the (technical) pixel coordinates within the window
and for drawing on the canvas to be always positive. Whenever ZoomWindow
re-calibrates, it's change signal will trigger, causing the
TimelineLayout to perform a new DisplayEvaluationPass,
which in turn prompts all embedded widgets to readjust
their positions accordingly.
Note: changing behaviour of TimeSpan to possibly flip start and end,
and also to use Offset as Offset and then re-orient,
since this seems the least surprising behaviour.
These changes carry over into changed default and limiting
on ZoomWindow constructor and various mutators, and most
notably shifting the time span always into allowed domain.
The value used previously was too conservative, and prevented ZommWindow
from zooming out to the complete Time domain. This was due to missing the
Time::SCALE denominator, which increaded the limit by factor 1e6
In fact the code is able to handle even this extremely reduced limit,
but doing so seems over the top, since now detox() kicks in on several
calculations, leading to rather coarse grained errors.
Thus I decided to use a compromise: lower the limit only by factor 1000;
with typical screen pixel widths, we can reach the full time domain,
while most scaling and zoom calculations can be performed precisely,
without detox() kicking in. Obviously this change requires adjusting
a lot of the test case expectations, since we can now zoom out maximally.
The APIs for time quantisation were drafted in an early stage of the project
and then never followed-up. Especially Grid::gridAlign has no
real-world usage yet, and is only massaged in some tests.
When looking at QuantiserBasics_test, I was puzzled and led astray,
since this function suggests to materialise a continuous time into
a quantised time -- which it doesn't (there is another dedicated
function Quantiser::materialise() to that end); so, without engaging
into the discussion if this function is of any use, I'll hereby
choose a name better reflecting what it does.
This is a deep refactoring to allow to represent the distance
between all valid time points as a time::Offset or time::Duration.
By design this is possible, since Time::MAX was defined as 1/30 of
the maximum value technically representable as int64_t. However,
introducing a different limiter for offsets and durations turns
out difficult, due to the inconsistencies in the exiting hierarchy
of temporal entities. Which in turn seems to stem from the unfortunate
decision to make time entities immutable, see #1261
Since the limiter is hard wired into the `time::TimeValue` constructor,
we are forced to create a "backdoor" of sorts, to pass up values
with different limiting from child classes. This would not be so
much of a problem if calculations weren't forced to go through `TimeVar`,
which does not distinguish between time points and time durations.
This solution rearranges all checks to be performed now by time::Offset,
while time::Duration will only take the absolute value at construction,
based on the fact that there is no valid construction path to yield
a duration which does not go through an offset first.
Later, when we're ready to sort out the implementation base of time values
(see #1258), this design issue should be revisited
- either we'll allow derived classes explicitly to invoke the limiter functions
- or we may be able to have an automatic conversion path from clearly
marked base implementation types, in which case we wouldn't use the
buildRaw_() and _raw() "backdoor" functions any more...
...in a similar vein as done for the product calculation.
In this case, we need to check the dimensions carefully and pick
the best calculation path, but as long as the overall result can
be represented, it should be possible to carry out the calculation
with fractional values, albeit introducing a small error.
As a follow-up, I have now also refactored the re-quantisation
functions, to be usable for general requantisation to another grid,
and I used these to replace the *naive* implementation of the
conversion FSecs -> µ-Grid, which caused a lot of integer-wrap-around
However, while the test now works basically without glitch or wrap,
the window position is still numerically of by 1e-6, which becomes
quite noticeably here due to the large overall span used for the test.
...using a requantisation trick to cancel out some factors in the
product of two rational numbers, allowing to calculate the product
without actual multiplication of (dangerously large) numbers.
with these additional safeguards, the anchorWindowAtPosition()
succeeds without Integer-wrap, but the result is not fully correct
(some further calculation error hidden somewhere??)
- detailed documentation of known problematic behaviour
when working with rational fractions
- demonstrate the heuristic predicate to detect dangerous numbers
- add extensive coverage and microbenchmarks for the integer-logarithm
implementation, based on an example on Stackoverflow. Surprising result:
The std::ilog(double) function is of comparable speed, at least for
GCC-8 on Debian-Buster.
Especially rational numbers with large denominator can be insidious,
since they might cause numeric overflow on seemingly harmless operations,
like adding a small number.
A solution might be to *requantise* the number into a different,
way smaller denominator. Obviously this is a lossy operation;
yet a small and controlled numeric error is always better than
an uncontrolled numeric wrap-around.
Extensive tests with corner cases soon highlighted this problem
inherent to integer calculations with fractional numbers: it is
possible to derail the calculation by numeric overflow with values
not excessively large, but using large numbers as denominator.
This problem is typically triggered by addition and subtraction,
where you'd naively not expect any problems.
Thus changed the approach in the normalisation function, relying
on an explicitly coded test rather, and performing the adjustment
only after conversion back to simple integral micro-tick scale.
Writing this specification unveiled a limitation of our internal
time base implementation, which is a 64bit microsecond grid.
As it turns out, any grid based time representation will always
be not precise enough to handle some relevant time specifications,
which are defined by a divisor. Most notably this affects the precise
display of frame duration in the GUI, and even more relevant,
the sample accurate editing of sound in the timeline.
Thus I decided to perform the internal computation in ZoomWindow
as rational numbers, based on boost::rational
Note: implementation stubbed only, test fails
This ZoomWindow_test highlights again the question about the intended usage
of the Lumiera time entities. In which way do we want to perform time calculations,
and under which circumstances is it adequate to perform arithmetic on
raw time values?
These questions made me think about rather far reaching concerns regarding
subsidiarity and implicit or explicit usage context. Basically I could
reconfirm the design choices taken some years ago -- while I must admit
that the project is headed towards a way larger scale and more loose
coupling of the parts, than I could imagine several years ago, at the
time when the design started...
As a side note: we can not avoid that some knowledge about the time implementation
leaks out from the support lib; time codes themselves are tightly coupled
to the usage scenario within the session and can not be used as means
for implementing UI concerns. And the more generic time frameworks,
like std::chrono (as much as it is desirable to have some integration here)
will not be of any help for most of our specific usage patterns.
The reason is, for film editing we do not have a global time scale,
rather the truth is when the film starts....
implement the first test case: nudge the zoom factor
⟹ scale factor doubled
⟹ visible window reduced to half size
⟹ visible window placed in the middle of the overall range
The solution is to provide a standard implementation in the form of a mix-in,
which directly houses a `ZoomWindow` instance. Moreover, the latter
is deemed a prominent use case for the time::Control, allowing other
components to attach and push changes of the zoom state or register
as listeners to react to state changes.
Actually, the `TimelineLayout`, which hosts all the actual visible
widgets forming the timeline-UI, now integrates this mix-in; and since
`TimelineLayout` is passed to `TimelineController` and used there as
reference-`CanvasHook` for the root track, this implementation of
the `DisplayMetric` interface will ''effectively be used by all
widgets'' attached to the timeline canvas.
According to plan, this was more or less a drop-in replacement.
However, this first integration prototype highlights some design problems
* `ElementBoxWidget` is designed ''constructor-centric''
* but the population by diff messages will supply crucial information later
* and seemingly the size-constraint code is now invoked prior to widget realisation \\
⟹ Assertion Failure
The header "format-cout.hpp" offers a convenience function
to print pretty much any object or data in human readable form.
However, the formatter for pointers used within this framework
switched std::cout into hexadecimal display of numbers and failed
to clean-up this state.
Since the "stickyness" of IOS stream manipulators is generally a problem,
we now provide a RAII helper to capture the previous stream state and
automatically restore it when leaving the scope.
Complete the investigation and turn the solution into a generic
mix-in-template, which can be used in flexible ways to support
this qualifier notation.
Moreover, recapitulate requirements for the ElementBoxWidget
The ClipPresenter can access the CanvasHook wired into its actual ClipDelegate (widget).
And this in turn exposes the DisplayMetric, with the ability to transform
presentation coordinates (pixels) into a model representation (Time)
The actual translation is still hardwired placeholder code,
since it is planned to build an generic component "ZoomWindow"
to provide all the typical zomming and view window translations
found in every timeline editor
- move construct into the buffer
- directly invoke the payload constructor through PlantingHandle
- reconsider type signature and size constraint
- extend the unit test
- document a corner case of c++ "perfect forwarding",
which caused me some grief here
...this extension was spurred by the previeous refactoring.
Since 'emplace' now clearly denotes an operation to move-embed an existing object,
we could as well offer a separate 'create' API, which would take forwarding
arguments as usual and just delegates to the placement-new operation 'create'
already available in the InPlaceBuffer class.
Such would be a convenience shortcut and is not strictly necessary,
since move-construction is typically optimised away; yet it would also
allow to support strictly non-copyable payload types.
This refactoring also highlights a fuzziness in the existing design,
where we just passed the interface type, while being sloppy about the
DEFAULT type. In fact this *is* relevant, since any kind of construction
might fail, necessitating to default-construct a placeholder, since
InPlaceBuffer was intended for zero-overhead usage and thus has in itself
no means to know about the state of its buffer's contents. Thus the
only sane contract is that there is always a valid object emplaced
into the buffer, which in turn forces us to provide a loophole for
class hierarchies with an abstract base class -- in such a case the
user has to provide a fallback type explicitly.
...for the operation on a PlantingHandle, which allows
to implant a sub type instance into the opaque buffer.
* "create" should be used for a constructor invocation
* "emplace" takes an existing object and move-constructs
...in an attempt to clarify why numerous cross links are not generated.
In the end, this attempt was not very successful, yet I could find some breadcrumbs...
- file comments generally seem to have a problem with auto link generation;
only fully qualified names seem to work reliably
- cross links to entities within a namespace do not work,
if the corresponding namespace is not documented in Doxygen
- documentation for entities within anonymous namespaces
must be explicitly enabled. Of course this makes only sense
for detailed documentation (but we do generate detailed
documentation here, including implementation notes)
- and the notorious problem: each file needs a valid @file comment
- the hierarchy of Markdown headings must be consistent within each
documentation section. This entails also to individual documented
entities. Basically, there must be a level-one heading (prefix "#"),
otherwise all headings will just disappear...
- sometimes the doc/devel/doxygen-warnings.txt gives further clues
...by relying on the newly implemented automatic standard binding
Looks like a significant improvement for me, now the actual bindings
only details aspects, which are related to the target, and no longer
such technicalitis like how to place a Child-Mutator into a buffer handle
After this long break during the "Covid Year 2020",
I pick this clean-up task as a means to fresh up my knowledge about the code base
The point to note is, when looking at all the existing diff bindings,
seemingly there is a lot of redundancy on some technical details,
which do not cary much meaining or relevance at the usage site:
- the most prominent case is binding to a collection of DiffMutables hold by smart-ptr
- all these objects expose an object identity (getID() function), which can be used as »Matcher«
- and all these objects can just delegate to the child's buildMutator() function
for entering a recursive mutation.
As it turned out, it is rather easy to extend the existing listener
for structural changes to detect also value assignments. Actually
it seems we'd need both flavours, so be it.
Yeah, C++17, finally!
...not totally sure if we want to go that route.
However, the noise reduction in terms of code size at call site looks compelling
...while the first solution looked as a nice API, abstracting away
the actual collections (and in fact helped me to sport and fix a problem
with type substitution), in the end I prefer a simpler solution.
Since we're now passing in a lambda for transform anyway, it is
completely pointless to create an abstracted iterator type, just
for the sole purpose of dereferencing an unique_ptr.
As it stands now, this is all tightly interwoven implementation code,
and the DisplayFrame is no longer intended to become an important
interface on it's own (this role has been taken by the ViewHook /
ViewHooked types).
Note: as an asside, this solution also highlights, that our
TreeExplorer framework has gradually turned into a generic
pipeline building framework, rendering the "monadic use" just
one usage scenario amongst others. And since C++20 will bring
us a language based framework for building iteration pipelines,
very similar to what we have here, we can expect to retrofit
this framework eventually. For this reason, I now start using
the simple name `lib::explore(IT)` as a synonym.
the reason for the failure, as it turned out,
is that 'noexcept' is part of the function signature since C++17
And, since typically a STL container has const and non-const variants
of the begin() and end() function, the match to a member function pointer
became ambuguous, when probing with a signature without 'noexcept'
However, we deliberately want to support "any STL container like" types,
and this IMHO should include types with a possibly throwing iterator.
The rationale is, sometimes we want to expose some element *generator*
behind a container-like interface.
At this point I did an investigation if we can emulate something
in the way of a Concept -- i.e. rather than checking for the presence
of some functions on the interface, better try to cover the necessary
behaviour, like in a type class.
Unfortunately, while doable, this turns out to become quite technical;
and this highlights why the C++20 concepts are such an important addition
to the language.
So for the time being, we'll amend the existing solution
and look ahead to C++20
as it turns out, "almost" the whole codebase compiles in C++17 mode.
with the exception of two metaprogramming-related problems:
- our "duck detector" for STL containers does not trigger anymore
- the Metafunction to dissect Function sigantures (meta::_Fun) flounders
When drafting the time handling framework some years ago,
I foresaw the possible danger of mixing up numbers relating
to fractional seconds, with other plain numbers intended as
frame counts or as micro ticks. Thus I deliberately picked
an incompatible integer type for FSecs = boost::rational<long>
However, using long is problematic in itself, since its actual
bit length is not fixed, and especially on 32bit platforms long
is quite surprisingly defined to be the same as int.
However, meanwhile, using the new C++ features, I have blocked
pretty much any possible implicit conversion path, requiring
explicit conversions in the relevant ctor invocations. So,
after weighting in the alternatives, FSecs is now defined
as boost::rational<int64_t>.
GCC8 now spots and warns about such mismatches.
And we should take such warnings seriously;
code produced by the newer GCC versions tends to segfault,
especially under -O2 and above, when a return statement is
actually missing, even if the return value is actually not
used at call site.
Here, a functor to unlock the active "guard" is passed into
a macro construct, which basically allows to abstract the
various kinds of "guards", be it mutex, condition variable
or the like.
Seemingly, the intention was to deal with a failure when
unlocking -- however all the real implementations prefer
to kill the whole application without much ado.
Our diff language requires a diff to handle the complete contents of the target.
Through this clean-up hook this is now in fact enforced.
The actual reason for adding this however was that I need to ensure
listeners are triggered
As it turned out, the reason was a missing move-ctor.
The base of the whole DSL-Stack, TreeMutator, is defined MoveOnly,
and this is also the intended use (build an anonymous instance
through the DSL and move it into the work buffer prior to diff application)
However, C++ does *cease to define* a move ctor implicitly,
whenever /one of the "big five" is defined explicitly/.
So Detector4StructuralChanges was the culprit, it defined a dtor,
but failed to define the move ctor explicitly.
So.... well, this did cost me several hours to track down,
yet I still rather do not want to write all those ctors explicitly all the time,
and so I am still in favour of implicitly generated ctors, even if they hurt sometimes.
with the new decorator layer, we suddenly trigger a chain of template instantiation errors.
At first sight, they are almost undecipherable, yet after some experimentation, it becomes clear
that they relate down to the base class (TreeMutator), which is defined MoveOnly
This seems to indicate that, at some point in the call chain, we are
digressing from the move-construction scheme and switch over to copy construction,
which in the end failst (and shall fail).
Inconclusive, to be investigated further
basically the solution was a bit too naive and assumed everything is similar to a vector.
It is not, and this leads to some insidious problems with std::map, which hereby
are resolved by introducing ContainerTraits
All of the existing "simple" tests for the »Diff Framework« are way to much low-level;
they might indeed be elementary, but not introductory and simple to grasp.
We need a very simplistic example to show off the idea of mutation by diff,
and this simple example can then be used to build further usage test cases.
My actual goal for #1206 to have such a very basic usage demonstration and then
to attach a listener to this setup, and verify it is actually triggered.
PS: the name "GenNodeBasic_test" is somewhat pathetic, this test covers a lot
of ground and is anything but "basic". GenNode in fact became a widely used
fundamental data structure within Lumiera, and -- admittedly -- the existing
implementation might be somewhat simplistic, while the whole concept as such
is demanding, and we should accept that as the state of affairs
The population message is just made up, in order to create more interesting structures
in the UI and so to further the development of the timeline display.
For the actual structure I choose to mirror my example drawing in draw/UI-TimelineLayout-1.png
which is also used in the TiddlyWiki, on the #GuiTimelineView tiddler
https://lumiera.org/wiki/renderengine.html#GuiTimelineView
Mostly, std::regexp can be used as a drop-in replacement.
Note: unfortunately ECMA regexps do not support lookbehind assertions.
This lookbehind is necesary here because we want to allow parsing values
from strings with additional content, which means we need explicitly to
exclude mismatches due to invalid syntax.
We can work around that issue like "either line start, or *not* one of these characters.
Alternatively we could consider to make the match more rigid,
i.e we would require the string to conain *only* the timecode spec to be parsed.
The existing implementation created a Buffer-Type based on various traits,
including the constructor and destructor functions for the buffer content.
However, this necessitates calculating the hash_value of a std::function,
which (see #294) is generally not possible to implement.
So with this changeset we now store an additional identity hash value
right into the TypeHandler, based on the target type placed into the buffer
<rant>
the "improved" boost::rational can no longer compute 1/x
quite brilliant
</rant>
well... the reason is again signed vs unsigned int.
FrameRate is based on unsigned int (since a negative frame rate makes no sense).
seemingly, the newer boost libraries added an internal type rational<I>::bool_type
together with an overload for the equality comparison operator.
Unfortunately this now renders a comparison ambiguous with the constant zero (i.e. int{0})
because in our use case we employ rational<uint>.
Workaround is to compare explicitly to a zero of the underlying integer type.
