The second design from 2017, based on a pipeline builder,
is now renamed `TreeExplorer` ⟼ `IterExplorer` and uses
the memorable entrance point `lib::explore(<seq>)`
✔
after completing the recent clean-up and refactoring work,
the monad based framework for recursive tree expansion
can be abandoned and retracted.
This approach from functional programming leads to code,
which is ''cool to write'' yet ''hard to understand.''
A second design attempt was based on the pipeline and decorator pattern
and integrates the monadic expansion as a special case, used here to
discover the prerequisites for a render job. This turned out to be
more effective and prolific and became standard for several exploring
and backtracking algorithms in Lumiera.
...as a preparation for solving a logical problem with the Planning-Pipeline;
it can not quite work as intended just by passing down the pair of
current ticket and dependent ticket, since we have to calculate a chained
calculation of job deadlines, leading up to the root ticket for a frame.
My solution idea is to create the JobPlanning earlier in the pipeline,
already *before* the expansion of prerequisites, and rather to integrate
the representation of the dependency relation direcly into JobPlanning
- had to fix a logical inconsistency in the underlying Expander implementation
in TreeExplorer: the source-pipeline was pulled in advance on expansion,
in order to "consume" the expanded element immediately; now we retain
this element (actually inaccessible) until all of the immediate
children are consumed; thus the (visible) state of the PipeFrameTick
stays at the frame number corresponding to the top-level frame Job,
while possibly expanding a complete tree of flexible prerequisites
This test now gives a nice visualisation of the interconnected states
in the Job-Planning pipeline. This can be quite complex, yet I still think
that this semi-functional approach with a stateful pipeline and expand functors
is the cleanest way to handle this while encapsulating all details
...introduced in preparation for building the Dispatcher pipeline,
which at its core means to iterate over a sequence of frame positions;
thus we need a way to stop rendering at a predetermined point...
...which uncovers further deeply nested problems,
especially when referring to non-copyable types.
Thus need to construct a common type that can be used
both to refer to the source elements and the expanded elements,
and use this common type as result type and also attempt to
produce better diagnostic messages on type mismatch....
...the improved const correctness on STL iterators uncovered another
latent problem with out diagnositc format helper, which provide
consistently rounded float and double output, but failed to take
CV-qualifiaction into account
This is a subtle and far reaching fix, which hopefully removes
a roadblock regarding a Dispatcher pipeline: Our type rebinding
template used to pick up nested type definitions, especially
'value_type' and 'reference' from iterators and containers,
took an overly simplistic approach, which was then fixed
at various places driven by individual problems.
Now:
- value_type is conceptually the "thing" exposed by the iterator
- and pointers are treated as simple values, and no longer linked
to their pointee type; rather we handle the twist regarding
STL const_iterator direcly (it defines a non const value_type,
which is sensible from the STL point of view, but breaks our
generic iterator wrapping mechanism)
- only the parts actually touched by the algo will be re-allocated
- when a segment is split, the clone copies carry on all data
Library: add function to check for a bare address (without type info)
...this is something I should have done since YEARS, really...
Whenever working with symbolically represented data, tests
typically involve checking *hundreds* of expected results,
and thus it can be really hard to find out where the
failure actually happens; it is better for readability
to have the expected result string immediately in the
test code; now this expected result can be marked
with a user-defined literal, and then on mismatch
the expected and the real value will be printed.
There are 12 distinct cases regarding the orientation of two intervals;
The Segmentation::splitSplice() operation shall insert a new Segment
and adjust / truncate / expand / split / delete existing segments
such as to retain the *Invariant* (seamless segmentation covering
the complete time axis)
- how to pass-in a specification given as GenNode
- now this might be translated into a MockJobTicket allocated in the MockSegmentation
Unimplemented: actually build the Segment with suitable start/end time
Looks like we'll actually retain and use this low-level solution
in cases where we just can not afford heap allocations but need
to keep polymorphic objects close to one another in memory.
Since single linked lists are filled by prepending, it is rather
common to need the reversed order of elements for traversal,
which can be achieved in linear time.
And while we're here, we can modernise the templated emplacement functions
- decision: the Monad-style iteration framework will be abandoned
- the job-planning will be recast in terms of the iter-tree-explorer
- job-planning and frame dispatch will be disentangled
- the Scheduler will deliberately offer a high-level interface
- on this high-level, Scheduler will support dependency management
- the low-level implementation of the Scheduler will be based on Activity verbs
...in a similar vein as done for the product calculation.
In this case, we need to check the dimensions carefully and pick
the best calculation path, but as long as the overall result can
be represented, it should be possible to carry out the calculation
with fractional values, albeit introducing a small error.
As a follow-up, I have now also refactored the re-quantisation
functions, to be usable for general requantisation to another grid,
and I used these to replace the *naive* implementation of the
conversion FSecs -> µ-Grid, which caused a lot of integer-wrap-around
However, while the test now works basically without glitch or wrap,
the window position is still numerically of by 1e-6, which becomes
quite noticeably here due to the large overall span used for the test.
- detailed documentation of known problematic behaviour
when working with rational fractions
- demonstrate the heuristic predicate to detect dangerous numbers
- add extensive coverage and microbenchmarks for the integer-logarithm
implementation, based on an example on Stackoverflow. Surprising result:
The std::ilog(double) function is of comparable speed, at least for
GCC-8 on Debian-Buster.
The header "format-cout.hpp" offers a convenience function
to print pretty much any object or data in human readable form.
However, the formatter for pointers used within this framework
switched std::cout into hexadecimal display of numbers and failed
to clean-up this state.
Since the "stickyness" of IOS stream manipulators is generally a problem,
we now provide a RAII helper to capture the previous stream state and
automatically restore it when leaving the scope.
Complete the investigation and turn the solution into a generic
mix-in-template, which can be used in flexible ways to support
this qualifier notation.
Moreover, recapitulate requirements for the ElementBoxWidget
- move construct into the buffer
- directly invoke the payload constructor through PlantingHandle
- reconsider type signature and size constraint
- extend the unit test
- document a corner case of c++ "perfect forwarding",
which caused me some grief here
...this extension was spurred by the previeous refactoring.
Since 'emplace' now clearly denotes an operation to move-embed an existing object,
we could as well offer a separate 'create' API, which would take forwarding
arguments as usual and just delegates to the placement-new operation 'create'
already available in the InPlaceBuffer class.
Such would be a convenience shortcut and is not strictly necessary,
since move-construction is typically optimised away; yet it would also
allow to support strictly non-copyable payload types.
This refactoring also highlights a fuzziness in the existing design,
where we just passed the interface type, while being sloppy about the
DEFAULT type. In fact this *is* relevant, since any kind of construction
might fail, necessitating to default-construct a placeholder, since
InPlaceBuffer was intended for zero-overhead usage and thus has in itself
no means to know about the state of its buffer's contents. Thus the
only sane contract is that there is always a valid object emplaced
into the buffer, which in turn forces us to provide a loophole for
class hierarchies with an abstract base class -- in such a case the
user has to provide a fallback type explicitly.
...for the operation on a PlantingHandle, which allows
to implant a sub type instance into the opaque buffer.
* "create" should be used for a constructor invocation
* "emplace" takes an existing object and move-constructs
...in an attempt to clarify why numerous cross links are not generated.
In the end, this attempt was not very successful, yet I could find some breadcrumbs...
- file comments generally seem to have a problem with auto link generation;
only fully qualified names seem to work reliably
- cross links to entities within a namespace do not work,
if the corresponding namespace is not documented in Doxygen
- documentation for entities within anonymous namespaces
must be explicitly enabled. Of course this makes only sense
for detailed documentation (but we do generate detailed
documentation here, including implementation notes)
- and the notorious problem: each file needs a valid @file comment
- the hierarchy of Markdown headings must be consistent within each
documentation section. This entails also to individual documented
entities. Basically, there must be a level-one heading (prefix "#"),
otherwise all headings will just disappear...
- sometimes the doc/devel/doxygen-warnings.txt gives further clues
...by relying on the newly implemented automatic standard binding
Looks like a significant improvement for me, now the actual bindings
only details aspects, which are related to the target, and no longer
such technicalitis like how to place a Child-Mutator into a buffer handle
After this long break during the "Covid Year 2020",
I pick this clean-up task as a means to fresh up my knowledge about the code base
The point to note is, when looking at all the existing diff bindings,
seemingly there is a lot of redundancy on some technical details,
which do not cary much meaining or relevance at the usage site:
- the most prominent case is binding to a collection of DiffMutables hold by smart-ptr
- all these objects expose an object identity (getID() function), which can be used as »Matcher«
- and all these objects can just delegate to the child's buildMutator() function
for entering a recursive mutation.
As it turned out, it is rather easy to extend the existing listener
for structural changes to detect also value assignments. Actually
it seems we'd need both flavours, so be it.
Yeah, C++17, finally!
...not totally sure if we want to go that route.
However, the noise reduction in terms of code size at call site looks compelling
the reason for the failure, as it turned out,
is that 'noexcept' is part of the function signature since C++17
And, since typically a STL container has const and non-const variants
of the begin() and end() function, the match to a member function pointer
became ambuguous, when probing with a signature without 'noexcept'
However, we deliberately want to support "any STL container like" types,
and this IMHO should include types with a possibly throwing iterator.
The rationale is, sometimes we want to expose some element *generator*
behind a container-like interface.
At this point I did an investigation if we can emulate something
in the way of a Concept -- i.e. rather than checking for the presence
of some functions on the interface, better try to cover the necessary
behaviour, like in a type class.
Unfortunately, while doable, this turns out to become quite technical;
and this highlights why the C++20 concepts are such an important addition
to the language.
So for the time being, we'll amend the existing solution
and look ahead to C++20
as it turns out, "almost" the whole codebase compiles in C++17 mode.
with the exception of two metaprogramming-related problems:
- our "duck detector" for STL containers does not trigger anymore
- the Metafunction to dissect Function sigantures (meta::_Fun) flounders
"%broken" is not broken anymore, but renders a boolean,
and we configured the formatter not to complain on missing values.
Fortunately "%madness" is still broken ;-)
basically the solution was a bit too naive and assumed everything is similar to a vector.
It is not, and this leads to some insidious problems with std::map, which hereby
are resolved by introducing ContainerTraits
All of the existing "simple" tests for the »Diff Framework« are way to much low-level;
they might indeed be elementary, but not introductory and simple to grasp.
We need a very simplistic example to show off the idea of mutation by diff,
and this simple example can then be used to build further usage test cases.
My actual goal for #1206 to have such a very basic usage demonstration and then
to attach a listener to this setup, and verify it is actually triggered.
PS: the name "GenNodeBasic_test" is somewhat pathetic, this test covers a lot
of ground and is anything but "basic". GenNode in fact became a widely used
fundamental data structure within Lumiera, and -- admittedly -- the existing
implementation might be somewhat simplistic, while the whole concept as such
is demanding, and we should accept that as the state of affairs
For context: The »Advice System« was coined a long time ago, in 2010,
based on the vague impression that it might be useful for that kind of application
we are about to build here. And, as can be expected, none of the usage situations
envisioned at that time was brought to bear. Non the less, the facility came in
handy at times, precisely because it is cross-cutting and allows to pass
information without imposing any systematic relationship between the
communication partners.
And now we've got again such a situation.
The global style manager in the UI has to build a virtual CSS path,
which is needed by drawing code somewhere deep down, and we absolutely
do not want to pass a reference to the style manager over 20 recursive calls.
The alternatives would be
(1) to turn the style manager into a public service
(2) to have a static access function somewhere
(3) to use a global variable.
For rationale, (1) would be overblown, because we do not actually request
a service to do work for us, rather we need some global piece of information.
(2) would be equivalent to (1), just more confusing. And (3) is basically
what the Advice system does, with the added benefit of a clear-cut service
access point and a well defined lifecycle.
This changeset adds the ability to check if actual Advice has been published,
which allows us to invoke the (possibly expensive) GTK path building and
style context building code only once.
The existing implementation created a Buffer-Type based on various traits,
including the constructor and destructor functions for the buffer content.
However, this necessitates calculating the hash_value of a std::function,
which (see #294) is generally not possible to implement.
So with this changeset we now store an additional identity hash value
right into the TypeHandler, based on the target type placed into the buffer
the template lib::PolymorphicValue seemingly picked the wrong
implementation strategy for "virtual copy support": In fact it is possible
to use the optimal strategy here, since our interface inherits from CloneSupport,
yet the metaprogramming logic picked the mix-in-adapter (which requires one additional "slot"
of storage plus a dynamic_cast at runtime).
The reason for this malfunction was the fact that we used META_DETECT_FUNCTION
to detect the presence of a clone-support-function. This is not correct, since
it can only detect a function in the *same* class, not an inherited function.
Thus, switching to META_DETECT_FUNCTION_NAME solves this problem
Well, this solution has some downsides, but since I intend to rewrite the
whole virtual copy support (#1197) anyway, I'll deem this acceptable for now
TODO / WIP: still some diagnostics code to clean up, plus a better solution for the EmptyBase
...which, in the end, can even be considered the more logical design choice,
since the "verb visitor" is a more elaborated and sophisiticated Verb-Token,
adding the special twist of embedded storage for variable function arguments
...but bad news on the main issue:
the workaround consumes the tuple and thus is not tenable!
And what is even worse: the textbook implementation of std::apply is
equivalent to our workaround and also consumes the argument tuple
A simple yet weird workaround (and basically equivalent to our helper function)
is to wrap the argument tuple itself into std::forward<Args> -- which has the
effect of exposing RValue references to the forwarding function, thus silencing
the compiler.
I am not happy with this result, since it contradicts the notion of perfect forwarding.
As an asside, the ressearch has sorted out some secondary suspicions..
- it is *not* the Varargs argument pack as such
- it is *not* the VerbToken type as such
The problem clearly is related to exposing tuple elements to a forwarding function.
this is a generalisation of what we use in the diff framework;
typically you'd package the VerbToken into some kind of container,
together with the concrete invocation argument.
However, the specific twist here is that we want *variable arguments*,
depending on the actual operation called on the interpreter interface.
...which leads to a specific twist here; while in the simple version
we still could hope to get away with a simple uniform uint argument,
the situation has changed altogether now. The canvas has turned into
some generic component, since it is instantiated two times, onece for
the time ruler and once for the actual body content. Thus all of the
specifics of the drawing code need to be pushed into a new, dedicated
renderer component. And this more or less forces us to pass all the
actual presentation variations through the invocation arguments of
the visitor.
So we're now off again for a digression, we need a more generalised visitor
effectively we rely in the micro tick timescale promoted by libGAVL,
but it seems indicated to introduce our own constant definition.
And also clarify some comments and tests.
(this changeset does not change any values or functionality)
- most notably the NOBUG logging flags have been renamed now
- but for the configuration, I'll stick to "GUI" for now,
since "Stage" would be bewildering for an occasional user
- in a similar vein, most documentation continues to refer to the GUI
...rather extend the "object builder" DSL notation to allow passing in a given EntryID literally.
Rationale is, we should handle the problem of unique IDs on the level of the domain model.
If we attempt to "fix" this within GenNode, the price would be to make the ETD creation stateful
seems to work surprisingly well...
the diff application poceeds in the GUI up to the point
where the TrackPresenter need to be inserted into a two-fold display context
As it turns out, using the functional-notation form conversion
with *parentheses* will fall back on a C-style (wild, re-interpret) cast
when the target type is *not* a class. As in the case in question here, where
it is a const& to a class. To the contrary, using *curly braces* will always
attempt to go through a constructor, and thus fail as expected, when there is
no conversion path available.
I wasn't aware of that pitfall. I noticed it since the recently introduced
class TimelineGui lacked a conversion operator to BareEntryID const& and just
happily used the TimelineGui object itself and did a reinterpret_cast into BareEntryID
...seems basically sane now.
Just we still need to wrap it one more time into IterableDecorator;
which means the overall scheme how to build and package the whole pipeline
is not correct yet.
Maybe it is not possible to get it packaged all into one single class?
on closer investigation it turned out that the logic of the
first design attempt was broken altogether. It did not properly
support backtracking (which was the reason to start this whole
exercise) and it caused dangling references within the lambda
closure once the produced iterator pipeline was moved out
into the target location.
Reasoning from first principles then indicated that the only sane
way to build such a search evaluation component is to use *two*
closely collaborating layers. The actual filter configuration
and evaluation logic can not reside and work from within the
expander. Rather, it must sit in a layer on top and work in
a conventional, imperative way (with a while loop).
Sometimes, functional programming is *not* the natural way
of doing things, and we should then stop attempting to force
matters against their nature.
...and TADAA ... there we get an insidious bug:
we capture *this by reference into the expansion functor,
and then we move *this away, out from the builder into the target....
Up to now, we had a very simplistic configuration option just
to search for a match, and we had the complete full-blown reconfiguration
builder option, which accepts a functor to work on and reconfigure the
embedded Filter chain.
It occurred to me that in many cases you'd rather want some intermediary
level of flexibility: you want to replace the filter predicate entirely
by some explicitly given functor, yet you don't need the full ability
to re-shape the Filter chain as a whole. In fact the intended use case
for IterChainSearch (which is the EventLog I am about to augment with
backtracking capabilities) will only ever need that intermediate level.
Thus wer're adding this intermediary level of configurability now.
The only twist is that doing so requires us to pass an "arbitrary function like thing"
(captured by universal reference) through a "layer of lambdas". Which means,
we have to capture an "arbitrary thingie" by value.
Fortunately, as I just found out today, C++14 allows something which comes
close to that requirement: the value capture of a lambda is allowe to have
an intialiser. Which means, we can std::forward into the value captured
by the intermediary lambda. I just hope I never need to know or understand
the actual type this captured "value" takes on.... :-)
with the augmented TreeExplorer, we're now able to get rid of the
spurious base layer, and we're able to discard the filter and
continue with the unfiltered sequence starting from current position.
build a special feature into the Explorer component of TreeExplorer,
causing it to "lock into" the current child sequence and discard
all previous sequences from the stack of child explorations
So we have now a reworked version of the internals of TreeExplorer in place.
It should be easier to debug template instantation traces now, since most
of the redundancy on the type parameters could be remove. Moreover, existing
pipelines can now be re-assigned with similarily built pipelines in many cases,
since the concrete type of the functor is now erased.
The price tag for this refactoring is that we have now to perform a call
through a function pointer on each functor invocation (due to the type erasure).
And seemingly the bloat in the debugging information has been increased slightly
(this overhead is removed by stripping the binary)
...step by step switch over to the new usage pattern.
Transformer should be the blueprint for all other functor usages.
The reworked solutions behaves as expected;
we see two functor invocations; the outer functor, which does
the argument adaptation, is allocated in heap memory
...based on a monadic tree expansion: we define a single step,
which takes the current filter configuration and builds the next
filter configuration, based on a stored chain of configuration functions
The actual exhausting depth-first results just by the greedy application pattern,
and uses the stack embedded in the "Explorer" layer of TreeExplorer
..this resolves the most challenging part of the construction work;
we use the static helper functions to infer a type and construct a suitable
processing pipeline and we invoke the same helper to initialise the base class
in the ctor.
Incidentally... we can now drop all the placeholder stubs,
since we now inherit the full iterator and child explorer API.
The test now starts actually to work... we get spam and sausage!
TODO: now actually fill in the expand functor such as to pick the
concrete filter step in the chain from a sequence of preconfigured
filter bindings
...now matters start to get really nasty,
since we have to pick up an infered type from a partially built pipeline
and use it to construct the signature for a functor to bind into the more elaborate complete pipeline
this is a tricky undertaking, since our treeExplore() helper constructs
a complex wrapped type, depending on the actual builder expressions used.
Solution is to use decltype on the result of a helper function,
and let the _DecoratorTraits from TreeExplorer do the necessary type adaptations
...it should have been this way all the time.
Generic code might otherwise be ill guided to assume a conversion
from the Iterator to its value type, while in fact an explicit dereferentiation is necessary
The intention is to augment the iterator based (linear) search
used in EventLog to allow for real backtracking, based on a evaluation tree.
This should be rather staight forward to implement, relying on the
exploreChildren() functionality of TreeExplorer. The trick is to package
the chained search step as a monadic flatMap operation
we did an unnecessary copy of the argument, which was uncovered
by the test case manipulating the state core.
Whew.
Now we have a beautiful new overengineered solution
outift the Filter base class with the most generic form of the Functor
wrapper, and rather wrap each functor argument individually. This allows
then to combine various kinds of functors
...this solution works, but has a shortcoming:
the type of the passed lambdas is effectively pinned to conform
with the signature of the first lambda used initially when building the filter.
Well, this is the standard use case, but it kind of turns all the
tricky warpping and re-binding into a nonsense excercise; in this form
the filter can only be used in the monadic case (value -> bool).
Especially this rules out all the advanced usages, where the filter
collaborates with the internals of the source.
