Now able to remove most complicated comparison operators and most usages of boost::operators...
In most cases it is sufficient just to define one ''spaceship operator'',
and often even that one can be synthesised.
However — we still use boost::operators for arithmetic types,
notably the `lib::time::TimeValue`, which is addable and mutipliable
Only minor rearrangements necessary to make that possible with C++20
And while at this change (which requires a full rebuild of Lumiera)
- simplify the defined comparison operators, as C++20 can infer most variations
- also mark various usages of `const char*` either as Literal or CStr
Remark: regarding copyright, up to now this is entirely my work,
with two major creation steps in 2008 (conception) and
in 2017 (introduction of a symbol table)
This changeset removes various heuristics and marker-traits
by a constraint to tuple_like types. Furthermore, several usages
of `apply` can thereby be generalised to work on any tuple_like.
This generalisation is essential for the passing generic data blocks
via `FeedManifold` into the node invocation
- integrate the concept definition into tuple-helper.hpp
- use it to replace the `is_Structured` traits check
- do not need `enable_if_TupleProtocol` any more
Integrate test coverage of the concept metafunctions
and the generalised get accessor
''This changeset was made at LAC 2025 in Lyon, France''
...to rely on the new formulation and the extended template `WithIdxSeq`
This is in preparation to use this new iteration scheme also from the tuple_like concept
Motivated by the difficulties encountered with `std::apply` —
which basically forced us to define our own alternative with
conceptually more adequate limitations....
...so these are the first attempts towards building a C++20 concept.
The Lumiera »Reference Platform« is now upgraded to Debian/Buster, which provides GCC-14 and Clang-20.
Thus the compiler support for C++20 language features seems solid enough, and C++23,
while still in ''experimental stage'' can be seen as a complement and addendum.
This changeset
* upgrades the compile switches for the build system
* provides all the necessary adjustments to keep the code base compilable
Notable changes:
* λ-capture by value now requires explicit qualification how to handle `this`
* comparison operators are now handled transparently by the core language,
largely obsoleting boost::operators. This change incurs several changes
to implicit handling rules and causes lots of ambiguities — which typically
pinpoint some long standing design issues, especially related to MObjects
and the ''time entities''. Most tweaks done here can be ''considered preliminary''
* unfortunately the upgraded standard ''fails'' to handle **tuple-like** entities
in a satisfactory way — rather an ''exposition-only'' concept is introduced,
which applies solely to some containers from the STL, thereby breaking some
very crucial code in the render entities, which was built upon the notion of
''tuple-like'' entities and the ''tuple protocol''. The solution is to
abandon the STL in this respect and **provide an alternative implementation**
of the `apply` function and related elements.
This resolves an intricate problem related to metaprogramming with
variadic templates and function signatures. Due to exceptional complexity,
a direct solution was blocked for several years, and required a better
organisation of the support code involved; several workarounds were
developed, gradually leading to a transition path, which could now
be completed in an focused clean-up effort over the last week.
Metaprogramming with sequences of types is organised into three layers:
- simple tasks can be solved with the standard facilities of the language,
using pattern match with variadic template specialisations
- the ''type-sequence'' construct `Types<T...>` takes the centre stage
for the explicit definition of collections of types; it can be re-bound
to other variadic templates and supports simple direct manipulation
- for more elaborate and advanced processing tasks, a ''Loki-style type list''
can be obtained from a type-sequence, allowing to perform recursive
list processing task with a technique similar to LISP.
after all the relevant library components do support both kinds of
type sequences transparently, any usages in core code can now be
switched over to the new, variadic type sequences.
A very performance relevant shortcoming of the existing implementation
of partial function closure is that the result is always wrapped into a
std::function, which typically causes a heap allocation when more than
a single pre-bound argument must be stored — which is annoying,
since the underlying Binder provides inline storage and thus
could be handled directly as a value object.
However, returning the Binder directly is also problematic, since
this object is outfitted with several overloaded function call operators,
which defeats most techniques to detect a function signature. Notably,
relevant down-stream metaprogramming code, like the tuple-closure used
in the `NodeBuilder` would break when being confronted directly with
a binder object.
An investigation shows that there is no direct remedy, short of
wrapping the binder into another functor. This can be accomplished
with a helper template, that generates a wrapper; however, this
wrapper builder must be supplied with explicit type information
regarding the function arguments (precisely because this type
signature can not be picked up from the Binder object itself)
This is a rather intricate and technical change, but allows in the end
to switch back all usages to a main implementation patch, which is now
based on `func::BindToArgument` — so this could become the final
implementation core and replace the old `PApply` template eventually...
Largely, these changes are related to allow for ''perfect forwarding''
of the functor and the argument values to be closed; these will be
copied into the ''Binder object'' created by `std::bind`.
Notably the `TupleConstructor` was changed to perfect-forward its »source«
into the specialised `ElmMapper`; this is possible since the latter
already receives a `SRC` template parameter, which can be supplied
with whatever base type the `std::forward` invocation will expose.
In the specialisation relevant here, template `PartiallyInitTuple`,
now an ''universal reference'' is stored and passed to `std::get`,
so that (depending on the input used), either a LValue or an
RValue reference is used for the extracted data elements.
After these changes, all existing usages of `applyFirst()` or `applyLast()`
can be replaced by this modernised implementation back-end, thus obsoleting
the various hard-coded workaround added during the last years.
A lot of repetitive, pre C++11 metaprogramming code can now be removed,
and several helper constructs, which were needed to handle generic
function application and passing a tuple of values to create a binder.
Note however, the highly complex and technical core of this header
still remains intact; which is to create a ''partial closure'' over
some arguments of a function, while keeping the remaining arguments
open as parameters for invocation.
TODO: Even in the remaining code there is a lot of redundancy
and helper construct which are no longer necessary
...because swapping in the new standards-based implementation
leads to compile failures on tests to cover out-of-bounds cases.
Under the (wrong) assumption, that some mistake must be hidden in
the Splice-metafunction, I first provided a complete test coverage;
while the actual problem was right below my nose, and quite obvious...
The old implementation, being based on a case distinction over the argument count,
simply was not able even to notice excess arguments; other the new implementation,
based on variadics and `std::apply`, which is fully generic and thus
passes excess arguments to `std::bind` when a position beyond the actual
argument list is specified to be closed.
The old behaviour was to silently ignore such an out-of-bounds spec,
and this can be reinstated by explicitly capping the prepared tuple
of binders and actual arguments passed to `std::bind`
Another question of course is, if being tolerant here is a good idea.
And beyond that, function-closure.hpp is still terrifyingly complex,
unorganised and use-case driven, to start with....
...can now be formulated in a single function,
based on the apply-to-λ technique invented by David Vandervoorde.
WARNING: the rewritten version of BindToArgument<...>::reduced()
does not compile in the out-of-bounds case, revealing a possibly
long standing defect in the typelist-metafunction Splice
This library header was developed at a time, where C++ had no built-in support
for so called "invokables"; `std::invoke` and `std::apply` were added much later;
So in that early version that was a significant technical hurdle to overcome.
seems like it might be possible to get rid of the TupleApplicator alltogether?
This is one of the most problematic headers, because it is highly complex
and comprises tightly interwoven definitions (in functional programming style),
which in turn are used deep within other features.
What concerns me is that this header is very much tangled
and pushes me (as the author) to my mental limits.
And on top of this comes that this code has to deal with intricate aspects
like perfect forwarding, and proper handling of binder instances and
function argument copying (which basically should be left to `std::bind`)
Fortunately, the changes ''for this specific topic'' are transparent:
Type sequences are not used on the API for function closure and composition,
but only as an internal tool to assemble argument tuples used for either
binding or invocation of the resulting (partially closed) function.
To bootstrap this tricky refactoring, initially a bridge definition
was used, with a variadic argument pack, but delegating to the old
non-variadic type sequence and from there further into LISP style
list processing of types and meta definitions, as pioneered by the
Loki libarary. Luimiera uses this technique since a long time to
perform the complex tasks sometimes required for code generation
and generic function and type adaptation.
with this changeset, a direct variadics based entrance into
type list processing is provided, so that the old definition
is now completely separate and can be removed eventually.
Most of the type-list and type-sequence related eccosystem can be
just switched over, after having added the conversion variants for
the new-style variadic type sequences
Again this was used as opportunity to improve readability of related tests
As expected, these work on the new-style variadic type sequences
equally well than on the old ones (tail-filled with `Nil` markers).
On that occasion, a complete makeover of the huge test case was carried out,
now relying on `ExpectString` instead of printing to STDOUT. This has the
benefit of showing the expectation immediately next to the code to be tested,
and thus makes it much easier to ''actually see'' how these meta-functions
operate on their parameters (which in fact are types in a type list)
- provide complete conversion paths old-style ⟷ new-style
- switch the basic tests to the new variadic sequences
- modernise the code; replace typedefs by `using`
- change some struct-style ''meta-functions'' into
constexpr or compile-time constants
Since I've convinced myself during the last years that this kind
of typelist programming is ''not a workaround'' — it is even
superior to pattern matching on variadics for certain kinds
of tasks — the empty struct defined as `NullType` got into
more widespread use as a marker type in the Lumiera code base.
It seems adequate though to give it a much more evocative name
Attempting to reduce the remaining pre-C++11 workarounds before upgrade to C++20...
As a first step: rename the old type-sequence implementation into `TyOLD`
to make it clearly distinguishable; a new variadic implementation `TySeq`
was already introduced as partial workaround, and the next steps
will be to switch over essential parts of the type-sequence library.
After the leftovers of the first Render-Engine implementation attempt were removed,
only one further usage of `RefArray` remains to be sorted out: the ''Session Element Tracker''.
Luckily, this one did not actually make any use of the abstraction abilities
of the `RefArray` — rather it basically stated that ''the interface is a data structure...''
After considering ''what kind of data'' can be expected to live in this structure,
it became clear that ''this will be a symbolic representation''
And thus the container can be simply switched to a `std::vector`.
This change allows to retain the existing placeholder-implementation unaffected,
while it would be possible to maintain algebraic terms here, in future.
__As an asside__: in order to decide about a suitable replacement in the Session,
I had to consier a first draft regarding the intended usage
and the prospective way of content representation
Investigated this topic again...
* these were initially created before C++11
* at that time, ''non-copyable'' objects were not common place
* but we embraced that concept already, and thus had quite some pain
when attempting to use such objects in STL containers
* with C++11 and ''move semantics'' these problems basically evaporated
* most usages were already upgraded and resolved
* another use case is to handle a state variable, which is based on
an immutable entity (like Time entities); `ItemWrapper` can be used
as a remedy in such a situation
This was a pre-C++11 implementation, and at that time,
I developed the ScopedHolder to allow handling non-copyable objects in STL containers
Meanwhile we have move semantics to achieve the same goal;
and since `ScopedPtrVect` shall be retained, it should be upgraded,
using the copy-and-swap approach
This is a plausible concept, and without obvious replacement
(letting aside `boost::ptr_vector`). It has a small number
of usages, and provides a dedicated API to show the
semantics when used as implementation of an ''Object Manager''
The original implementation used private inheritance from `std::vector`,
which is not really justified here, since we neither use the ''template method''
pattern, nor want to gain access to protected internals.
So this can be replaced with a private member.
...initially, this header was a collection of small helpers made on occasion;
one of them, the `ItemWrapper` used in transforming pipelines came
into widespread use and was much augmented and improved over the years.
many other tiny helpers could be replaced by standard library facilities...
Now looking into largely obsolete library facilities...
Starting from `ScopedHolder`, I found a surprising problem with ''perfect forwarding....''
...which however turned out to be the result of ''sloppy programming'' on my side.
At that time, in 2017, seemingly I was under pressure to define a Session-Command,
which captures a Time-entity as »State Memento«. Which turned out to be impossible,
since the Time entities were conceived as being immutable -- a questionable design
decision (see #1261), which already generated quite some additional complexities.
In the course of this »exercise«, I could again clarify that the implementation
of perfect forwarding works as expected on modern compilers — confusion may arrise
sometimes due to ''copy elision'' (which the compiler is required to perform
since C++17, even when the elided constructor has observable side effects).
And it can be derailed when (as was the case here) a »grab everything« constructor
accidentally ends up generating a copy- or move-constructor for the container class
itself. This is an instance of the problem documented with #963 ...
.... and the best solution is to abide by the rule-of-five (and a half)
and to put that `ReplacableItem` to where it belongs...
During the early stage of the Project, at some point I attempted
to »attack« the topic of Engine and Render Nodes following a ''top down path.''
This effort went into a dead end eventually — due to the total lack
of tangible reference points to relate to. However, the implementation
at that time prompted the development of several supporting facilities,
which remain relevant until today. And it resulted in a ''free wheeling''
compound of implementation structures, which could even be operated
through some highly convoluted unit test.
This piece of implementation code was valuable as starting point for th
»Playback Vertical Slice« in 2024 — resulting in a new design which was
''re-oriented'' towards a new degree of freedom (the »Domain Ontology«)
while handling the configuration and connectivity of Render Nodes in
a rather fixed and finite way. This new approach seems to be much more
successful, as we're now able to build, connect and invoke Render Nodes,
thereby mapping the processing through a functor binding into some
arbitrary, external processing function (which will later be supplied
by a media processing library — and thus be part of some »Domain Ontology«)
The `FixedFrameQuantiser` relied on three functions from the raw-time handling library.
Since this (and NTSC drop-frame) are the only usages, these functions
can be relocated into the implemntation translation unit `lib/time/quantiser.cpp`
On closer inspection, this reveals some room for improvements:
Instead of relying on raw-computation functions written in C,
we could rather revert the dependency and express these computations
in terms of our Time-entities, which are written in C++, are much more
systematic and provide consistency checks and protection against numeric
overflow, all integrated with linear arithmetic and concise notation.
After performing these rearrangements,
most of the functions can be collapsed into ''almost nothing''.
This was taken as opportunity to re-check and improve the remaining
implementation core of the `FixedFrameQuantiser` -- the handling of
extreme corner cases can be much improved, now representing the
"grid-local time" as `Offset`, which doubles the possible value range.
The reworked unit test shows that, with this change, now the limitation
happens prior to quantisation, meaning that we always get a grid-aligned
result, even in the most extreme corner cases.
...extract these functions and the associated test
from the low-level C time handling library and
document them with a dedicated C++ header and test.
''This is unfinished work'' —
the extracted functions provide only the low level computations;
actually, a specialised time quantisation or time code would be required.
------------
Note though,
after extracting these functions, the rest of the plain-C test
can be removed, since equivalent functionality is covered in
much more detail by the tests of the C++ time handling framework.
Notably this allows to get rid of the direct component accessor functions.
------------
__Remark__: the base implementation of many time conversion functions
and especially NTSC drop-frame was provided by Stefan Kangas
See:
6a44134833
While these function may seem superficially plausible,
I more and more come to the conclusion that offering such
function as ''basic building blocks'' is in itself an
ill-guided approach to handling of time entities.
Time is neither „just a number“ — nor does it „contain“ hours, minutes and seconds.
It is possible to ''represent'' it through a **time-code**, which incurs
a quantisation step and implies a reference grid.
Thus Lumiera ''should not offer'' a »basic time handling library«.
Doing so would be just an invitation to bypass proper time handling
and avoid the use of more demanding but also more adequate mental concepts.
So the next step will be to remove functions not deemed adequate, and
better directly inline the respective modulus based computations.
Other functions can be integrated into the respective implementation
translation units for time quantisation and timecode representation.
Indeed — this change set is kind of sad.
Because I still admire the design of the GAVL library,
and would love to use it for processing of raw video.
However, up to now, we never got to the point of actually
doing so. For the future, I am not sure if there remains
room to rely on lib-GAVL, since FFmpeg roughly covers
a similar ground (and a lot beyond that). And providing
a plug-in for FFmpeg is unavoidable, practically speaking.
So I still retain the nominal dependency on lib-GAVL
in the Build system (since it is still packaged in Debian).
But it is pointless to rely on this library just for an
external type-def `gavl_time_t`. We owe much to this
inspiration, but it can be expected that we'll wrap
these raw time-values into a dedicated marker type
soon, and we certainly won't be exposing any C-style
interface for time calculations in future, since
we do not want anyone to side-step the Lumiera
time handling framework in favour of working
„just with plain numbers“
NOTE: lib-GAVL hompage has moved to Github:
https://github.com/bplaum/gavl
This is a first step towards the ability to produce several different output formats...
Refactor the code to separate
- the double buffering
- the actual image generation, which works in RGB
- the conversion routine
Furthermore, replace unsigned char by std::byte
and introduce std::array and structured binding
to avoid many usages of pointers; hopefully this
makes the intention of the code clearer.
Verified and cross-checked the actual converion logic;
in fact this is a conversion to "YUV" as used by MPEG,
which in more precise terms is Y'CrCb with Rec.601 colour space
and a scan range limitation (16...235) on the Luma component.
...to be more compliant to the »Lumiera Forward Iterator« concept.
This can be easily achieved by inheriting from util::RegexSearchIter,
similar to the example in CSV.hpp
Regarding #896, I changed the string rendering to use fs::path::generic_string
where appropriate, which means that we're using the normalised path rendering
Since C++17 we can use the std::filesystem instead (and we ''do use it'' indeed)
- relocate the `/lib/file.hpp` header
- adapt the self-discovery of the executable to using std::filesystem
Furthermore, some recherche regarding XVideo and Video Output
- remove obsolete configuration settings
- walk through all settings according to the documentation
https://www.doxygen.nl/manual/config.html
- now try to use the new feature to rely on Clang for C++ parsing
- walk through the doxygen-warnings.txt and fix some obvious misspellings
and structural problems in the documentation comments.
With Debian-Trixie, we are now using Doxygen 1.9.8 —
which produces massively better results in various fine points.
However, there are still problems with automatic cross links,
especially from implementation to the corresponding test classes.
The Boost-Libraries changed their internal implementation
of the formula to chain hash values.
Fortunately, we had already extracted the existing implementation
from Boost 1.67 and incorporated it in-tree, in the Lumiera support libary.
After switching to that `lib:#️⃣:combine()` function, all the graph
computations related to the Scheduler-test-load can be shown to be identical.
So at the moment, the impact is still limited, but this incident highlights
the importance of a controlled, stable (and ideally also portable) hash implementation.
seems that I've played too much with »undefined behaviour« in this test;
basically we can not assume ''any'' specific placement of local variables
in a stack frame....
In this test, what I wanted to demonstrate is that the overflow-block can reside
just »anywhere«, and that HeteraoData is just a light-weight front-End and accessor.
However, I can just demonstrate that without totally ''undefined behaviour;''
placement-new can be used to force the storage at a specific location (in the UninitialiesdStorage);
continue to access and use that data after leaving the nested scope is still
kind-of borderline, yet demonstrates that the data itself is just residing in a storage block...
- with Debian 12/13, the top-level `/bin`, `/sbin` and `/lib`
are collapsed into `/usr`. Seemingly this has prompted changes
to the way the shell prints some error messages. This broke
the expectation of some test of the test-framework itself.
- SCons always had the policy to ''sanitise'' the invocation environment,
to prevent unintended impact of environment settings to the test subject.
Seemingly this now also leads to `$HOME` not being defined.
Our file handling framework however expects to be able to expand "~"
- An old-style cast in the constructor lib::diff::Record(Mutator const&)
is now translated into a static_cast (≙conversion); and since the appropriate
conversion operator is missing on Mutator, the constructor attempts to
create a temporary, by re-invoking the same constructor ⟹ Stackoverflow ↯
- conversion from pointer to bool now counts as ''narrowing conversion''
- constructor names must not include template arguments (enforced with C++20)
- better use std::array for some dummy test code
Several further warnings are due to known obsoleted or questionable constructs
and were left as-is (e.g. for ScopedHolder) or just commented for later referral
This is an advanced diagnostics added (presumably) with GCC-13
and attempts to protect against an insidious side-effect of ''overload resolution''
Basically C++ (like its ancestor C) is oriented towards direct linkage and adds
the OO-style dynamic dispatch (through virtual functions and a VTable)
only as an extension, which must be requested explicitly.
Thus the resolution of ''overloads'' (as opposed to ''overridden'' virtual functions)
always takes precedence and happens within the directly visible scope,
which can cause the compiler to perform an implicit conversion instead of
invoking a different virtual function, which is defined in a base class.
However, this diagnostics seems to be implemented in an overly zealous way:
The compiler warns at the time of the type instantiation, and even in cases
where it is effectively impossible to encounter this dangerous shadowing situation.
See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109740
This leads to several ill-guided warnings in the Lumiera code base, which unfortunately
can only be addressed by disabling this diagnostics for all usages of some header.
The reason is, we often generate chains of template instantiations driven by type lists,
and in such usage pattern, it is not even possible to bring all the other inherited overloads
into scope (with a using `BASE::func` clause), because such a specification would be ambiguous
and result in a real compile error, because even the interface is generated from a chain of mix-in templates
Future C++ versions will no longer generate default copy operations
once any single one was defined explicitly. So the goal is to kind-of
''enforce the rule of five'' (if you define one, define them all).
However, sometimes one of these special operators must be defined for a different reason,
e.g. because it is defined as protected, yet should not be exposed on the public API.
In such cases, any other copy operation which still is valid in the default form
must be declared explicitly ''as defaulted''
Overall this seems to be quite an improvement --
and it highlights (again) some known instances of questionable design,
which are mostly obsoleted and require clean-up anyway, or (as in the case of the
Placements) indicate »placeholder code« where the actual solution still needs to be worked out
Oh this is an interesting one...
GCC now highlights situations with `-Wpessimizing-move`,
where an overly zealous developer attempts to optimise by `std::move`,
which however prevents the compiler from applying the ''Return Value Optimisation''
The latter is mandatory since C++17, and essentially means that a value object
created within a function and then returned (by value) will actually be created
directly in the target location, possibly eliding a whole chain of
delegating value returns.
Thus: if we write `std::move(value)`, we change the returned type into an RValue reference,
and thereby ''force the compiler'' to invoke a move-ctor....
* need to upgrade our custom packages to current standards
* switch those packages from CDBS to dh
* re-build on Trixie and upgrade the Lumiera DEB-Depot
After these (in detail quite expensive) preparations,
build with Scons and GCC-14 can be started.
Fix some further (basically trivial) compile problems,
uncovered by the improved type checking of modern compilers.
Note: a tremendous amount of warnings (and depreciations) is
also indicated, which will be addressed later....
Based on the building blocks developed thus far,
it was possible to assemble a typical media processing chain
* two source nodes
* one of these passes data through a filter
* a mixer node on top to combine both chains
* time-based automation for processing parameters
As actual computation, hash-chaining on blocks of
reproducible random data was used, allowing to verify
for every data word that expected computations were
carried out, in the expected order.
Using basically the same topology as in the preceding test, which focused on connectivity. However, in this case we retrieve actual processing functions from the »Test-Rand« ontology in order to perform hash-chaining computations on full data blocks. And, in addition, a »Param Agent Node« is used.
While initially intended as introductory test, it meanwhile
focuses on intricate technical details on the level of
basic building blocks, notably the `FeedManifold`
Now I have added a simple end-to-end demonstration example
how a Render Node is built from scratch, leaving out all
technical details and all convenience front-ends like
the `NodeBuilder` — just one dummy port invoked directly.
NodeBase_test demonstrates the building blocks of a Render Node,
and verifies low-level mechanics of those building blocks, which
can be quite technical. At the top of this test however are some
very basic interactions, which serve as an introduction.
__Remark__: renamed the low-level technical dispatch-access
for the parameter-accessors in `TurnoutSystem` to be more obvious,
and added comment (I was confused myself how to use them properly)
This is a crucial feature, discovered only late, while building
an overall integration test: it is quite common for processing functionality
to require both a technical, and an artistic parametrisation. Obviously,
both are configured from quite different sources, and thus we need a way
to pre-configure ''some parameter values,'' while addressing other ones
later by an automation function. Probably there will be further similar
requirements, regarding the combination of automation and fixed
user-provided settings (but I'll leave that for later to settle).
On a technical level, wiring such independent sources of information
can be quite a challenging organisational problem — which however can be
decomposed using ''partial function closure'' (as building a value tuple
can be packaged into a builder function). Thus in the end I was able to
delegate a highly technical problem to an existing generic library function.
* now able to demonstrate close-front, close-back and close-argument
* can also apply the same cases to `std::array`, with input and
output type seamlessly adapted to `std::array`
With these additions, all conceivable cases are basically addressed.
Take this as opportunity to investigate how the existing implementation
transports values into the Binder, where they will be stored as data fields.
Notably the mechanism of the `TupleConstructor` / `ElmMapper` indeed
''essentially requires'' to pass the initialisers ''by-reference'',
because otherwise there would be limitations on possible mappings.
This implies that not much can be done for ''perfect forwarding'' of initialisers,
but at least the `BindToArgument` can be simplified to take the value directly.
...which should ''basically work,'' since `std::array` is ''»tuple-like«'' —
BUT unfortunately it has a quite distinct template signature which does not fit
into the generic scheme of a product type.
Obviously we'd need a partial specialisation, but even re-implementing this
turns out to be damn hard, because there is no way to generate a builder method
with a suitable explicit type signature directly, because such a builder would
need to accept precisely N arguments of same type. This leads to a different
solution approach: we can introduce an ''adapter type'', which will be layered
on top of `std::array` and just expose the proper type signature so that the
existing Implementation can handle the array, relying on the tuple-protocol.
__Note__: this changeset adds a convenient pretty-printer for `std::array`,
based on the same forward-declaration trick employed recently for `lib::Several`.
You need to include 'lib/format-util.hpp' to actually use it.
What emerges here, seems to be a generic helper to handle
partial closure of ''tuple-like'' data records. In any case,
this is highly technical meta-programming code and mandates
extraction into a separate header — simplifying `NodeBuilder`
Likely the most widely used facility, which enters into meta-programming
with type sequences, is our function-signature-detector `_Fun<X>`,
which returns a argument type-sequence.
After adding some bridges for cross-compatibility,
function-arguments are now extracted as a new-style,
''variadic sequence'' without trailing `NullType`.
Doing so required to augment some of the most widely used
sequence-processing helpers to work seamlessly also with the
new-style variadic sequences with a definition variant based
on variadics, which typically will later-on obsolete the original
solution, which at that time needed to be tediously coded as a
series of explicit specialisations for N arguments.
...on top of the parameter-decorating functionality developed thus far.
The idea is to allow in the `NodeBuilder` to supply ''some parameters''
directly, while the remaining parameters will be drawn from automation.
Several years ago, I developed some helpers for partial function closure.
Unfortunately these utils are somewhat limited, and rely on some pre-C++11
constructs, yet seem to be usable for the task at hand, since parameters
are always expected as value objects by definition.
This changeset shows a working proof-of concept for left-closing a
parameter tuple with 5 elements; this turns out to surprisingly difficult
due to the full genericity of the acceptable parameter-aggregates...
seemingly the definition can not be much simplified,
since there is no way around handling several definition flavours
of the processing-functor distinctly.
However, the definitions can be rearranged to be clearer,
the resulting type of the `FeedPrototype` can be deduced from the
builder function, and more stringent assertions can be added
...the idea is to limit the scope of possible changes
and rather directly accept a functor to transform the parameters.
We need then to account for the possible flexibility in processing-functor
arguments, while in fact only two cases must be actually handled.
''This proof-of-concept works in test setup''
It seemed that the integration test will end up as a dull repetition
of already coded stuff, just with more ports and thus more boilerplate;
and so I reconsidered what an actually relevant integration test might encompass
- getting parameters from the invocation
- translating and wiring parameters
- which entails to adapt / partially close a processing function!
Thus — surprise — there is a new feature not yet supported by the `NodeBuilder`,
which would be very likely to be used in many real-world use cases: which is
to adapt the parameter tuple expected by the binding from the library.
Obviously we want this, since many »raw« processing functions will expose a mix
of technical and artistic parameters; and we'd like to ''close'' the technical ones.
Such a feature ''should be implementable,'' based on the already developed
technique with the »cross builder«, which implies to switch the template arguments
from within a builder expression. We already do this very thing for adapting
parameter functor, and thus the main difficulty would be to compose an
adaptor functor to the correct argument of the processing functor...
Which is... (well, it is nasty and technical, yet feasible).
Just wanted to use a helper function to build a source-data node.
However, the resulting node had a corrupted Node-ID spec.
Investigation with the debugger showed that the ID was still valid
while in construction and shows up corrupted after returning from the
helper function.
As it turned out, the reason is related to the de-duplication of ProcID data.
While the de-duplicated strings themselves are ''not'' affected, the corruption
happened by an intermediate instance of ProcID, which was inadvertently created
and bound by-value to the builder-λ. The created Port then picks up a reference
to this temporary, leading to the use-after-free of the string_view obejcts.
Obviously, `ProcID` must not be instantiated other than through the static
front-end `ProcID::describe`. Due to the private constructor, I can not make this
object non-copyable (because then the hash-set would not be allowed to emplace it).
But making it at least move-only will provoke a compiler error whenever binding
to a lambda capture by value, which hopefully helps to pinpoint this
insidious problem in the future...
The scheme to provide preconfigured nodes with random `TestFrame` data
seems to be suitable and easy to extend to further cases; should however
always document the setup through a dedicated case in `NodeDevel_test`
Seems to be straight forward now, based on the implementation
of `TestFrame` manipulation provided by the »Test Rand Ontology«
__Remark__: the next goal is to reproduce the complex Node tree
with operations on TestFrame and then to invoke these and verify results.
...while this is not the main objective of this test case,
and another test will focus on invocation with full-fledged
`TestFrame` buffers and hash computation...
...it is still a nice achievement to see that these simple
algebraic operations used for demonstration can actually be
invoked in the whole connected network :-)
Using a Node network with
* two source nodes
* one of them chained up linearly with a filter node
* then on top a mix node to combine both chains
Can now verify the generated port specs and verify proper connections
at node level and at port level
This was a lot of intricate technical work,
and is now verified in-depth, covering all possible cases.
__We can now__
* build Nodes
* verify in detail correct connectivity
* read Node-IDs and processing specifications
* maintain a symbolic spec for the arguments of a Port
(and beyond that, we can also **invoke nodes**, which remains to be formally verified)
An essential goal still to reach is a verification of the `NodeBuilder`'s products
Relying on the low-level diagnostic facilities pioneered last days,
it should now be possible to define simple and readable connectivity-clauses,
allowing to build some connected nodes and then verify the connections explicitly.
Handling of extended attributes in conjunction with the hash
turns out to be a rather complicated topic, with some tricky fine details.
And, most important, at the moment I am lacking the proper perspective
to address it and find adequate solutions. Luckily, the cache-key is
not required at the moment, ''and so this topic will be postponed''
As a minimum to complete the diagnostics functions, it is sufficient to set
the appropriate flags in the `ProcID` directly -- and to add some convenience wrappers.
...exploiting the ''backdoor access'' bypassing the VTable,
as made possible by a common congruent storage layout.
This is a first proof-of-concept, but also shows that the demo nodes
in NodeMeta_test are wired as expected. What is needed now is to make
this diagnostic access easier to invoke and more bullet-proof, by setting
the proper Attribute bits directly in the `NodeBuilder`
...to create an ''access path for diagnostics'' and further evaluations
while ''bypassing the VTable.''
It is a well-known downside of specifically typed, highly optimisable
template-based code to create a dangerous leverage for generating spurious,
mostly identical virtual function instances added for secondary concerns.
Thus it is a consequence of this design choice, either to forego some diagnostic
and analytical possibilities, or to exploit ''other means'' for retrieving
internal data, which is needed for tangential purposes only. The solution
pursued hereby exploits similar layout of various ''weaving pattern''
template instances to create an ''access backdoor'' for use cases
beyond the primary performance-critical path.
Some additional tests to challenge the parser, which seems to work well.
Without extended analysis into the usage of those node specifications,
it is pointless to expand further on its capabilities. For now, it is
sufficient to have a foundation for hash-computation in place.
__Note__: found a nifty way to give lib::Several an easy toString rendering,
without cranking up the header inclusion load.
This is a nice little goodie: allow to write repeated arguments with the
shorthand notation known from lisp and logic programming. For multi-channel media,
structurally similar wirings for each channel will be quite common....
...at the point where I identified the need to parse nested terms.
The goals are still the same
* write tests to ''verify connectivity'' of nodes generated by the new `NodeBuilder`
* allow for ''extended custom attributes'' in the ProcID
* provide the ability to mark specific parametrisations
* build a Hash-Key to identify a given processing step
__Note Library__: this is the first time `lib::Several` was used to hold a ''const object''.
Some small adjustments in type detection were necessary to make that work.
Access to stored data happens through the `lib::Several` front-end and thus always includes
the const modifier; so casting any const-ness out of the way in the low-level memory management
is not a concern...
This finishes an ''exercise'' in tool design,
which was set off by the requirement to parse the spec-ID of a render node.
While generally within the confines of a helper utility for simple use cases,
the solution became quite succinct and generic, as it allows to handle arbitrary
LL(n) grammars, possibly with recursion.
...which is the reason for this whole excursion into parser business;
we want to accept specification terms with elements from C++ type expressions,
which especially requires to accept complete comma separated lists within
angle brackets or parenthesis, while separating by comma at top level.
The idea is to model ''not as an expression'' but rather as an ''extended quote'',
and to use inverted regular expressions for non-quote-characters as terminal
...evaluating the recursive syntax of a numerical expression!
* so this light-weight parsing support framework indeed allows
to build fully capable LL(x) parsers, when the user knows how
handle syntax clauses and bind the result models
* furthermore, a notation is demonstrated how to arrange the
binding functions so to keep the syntax definition legible
* this involves a shortcut for homogeneous alternatives
The concept was indeed successful, albeit quite difficult to pull off in detail.
It requires a carefully crafted path of Deduction guides and overloads
to effect the switch from std::function to std::function& at the point
where a predeclared syntax clause placeholder is used recursively
In accordance to the plan drafted yesterday, I will try to integrate
this essential capability into the framework established thus far by a trick,
requiring only minimal adjustment, but some work by the user.
Since the parse function is defined as a (unqualified) template argument,
it is possible to emplace either a `std::function`, or a reference thereto.
For this to work, the user is required to pre-define the expected result type,
and, furthermore, must later on assign a fully specified clause, which
also has a model transformation binding attached to yield this predeclared
result type
...several improvements as result from the more elaborate test cases
- spelling out the model types taken as argument can be challenging and tedious,
thus improve the ability to pass a λ-generic.
- furthermore, using structured bindings on a SeqModel can also simplifiy
binding code; this did not work because the compiler picks the wrong strategy
and attempts to bind the structure fields; need to provide explicit speicalisations
to support the »tuple protocol« for SeqModel.
..considered several further helpers, (like auto-joining into a single string),
but in the end did not implement them, due to questionable relevance
The `bindMatch()` as implemented yesterday works only directly on top
of the terminal parsers, which yield a `RegExp`-Matcher. However,
it would be desirable to provide a generic shortcut to always get
some string as result model. A simple fallback is to return
the part of the input-string accepted thus far.
Basically the implementation is already in place;
yet for better error messages we need to find out if the given functor
can handle the model present at this stage. Since generic-λ are not
functions by themselves (but rather templates), we need to ''probe''
with the expected argument and see if instantiation is possible.
⚠ NOTE: still a strange bug related to using the same Syntax several times
Allowing free recursion in grammars is the key enabling feature,
which allows to accept arbitrary complex structures (like numeric expressions).
It is however also the element which makes the task of parsing a challenging endeavour;
after weighting the arguments, I decided ''not to place the focus on advanced usage,''
yet to open a pathway towards representation of such grammars.
Essentially, I consider it acceptable to require some additional work by the user,
if arbitrary recursive grammars are desired; because this design relies on explicitly
given parse functions, we need to introduce some kind of indirection interface,
to allow ''declaring'' a recursive rule first and later to ''supply the definition,''
which obviously then will involve other rules (or itself) recursively.
This leads to a very ''nifty approach'' towards recursion: we require the user
to provide an ''explicit model type'' beforehand, which implies that this is a
simple type, that can be spelled out (no λ) — and so the user is also
''forced to augment the actual rule with a model-binding,'' thereby reducing
the structured return types from the parse into something simple and uniform.
The user ''has to do the hard work,'' but can ''exploit additional knowledge''
related to the specific use case.
All this framework needs to do then is to supply a `std::function`, using the
explicit return type given; everything else will still work as implemented,
since a `std::function` can always stand-in for any arbitrary λ.
This is the very key feature that requires a real parser and can not be handled by regular expressions.
After all the groundwork, it is surprisingly easy provide now;
only coding up all those DSL-variants is tedious. Notably we also
support accepting an ''optional'' bracket, and we support arbitrary
expressions for the ''opening'' and ''closing construct.''
It seemed that using postfix-decorating operators would be a good fit
for the DSL. Exploring this idea further showed however, that such a scheme
is indeed a good fit from the implementation side, but leads to rather confusing
and hard to grasp DSL statements for many non-trivial syntax definition.
The reason is: such a postfix-decorator will by default work on ''everything defined''
up to that point; this is too much in many cases.
The other alternative would be a function-style definition, which has the benefit
to take the sub-clause directly as argument (so the scope is always explicit).
The downside is that argument arrangement is a bit more tricky for the repetition
combinator (there can be mis-matches, since we take the »SPEC« as free-template argument)
And, moreover, with function-style, having more top-level entrance points would
be helpful. Overall, no fundamental roadblock, just more technicalities in the setup
of the DSL functions.
With that re-arrangd structure, an optional combinator could be easily integrated,
and a solution was provided to pick up the parser function from a sub-expression
defined as Syntax object.
Meanwhile, some kind of style scheme has emerged for the DSL:
We're working much with postfix-decorating operators, which
augment or extend the ''whole syntax clauses defined thus far''
In accordance with this scheme, I decided also to treat repeated expression
as a postfix operator (other than initially planned). This means, the actual
body to be repeated is ''the syntax clause defined thus far'', and the
repeat()-operator only details the number of repetitions and an optional delimiter.
After all the preparation, now this panes out quite well:
* use a simple 3-way branch structure
* the model type was already pre-selected by the `_Join` Model selector
* can just pass the result-model elements to a constructor/builder
* incremental extension can be directly mapped to the predecessor model
This is a rather obnoxious limitation of C++ variadics:
the inability to properly match against a mixed sequence with variadics.
The argument pack must always be the last element, which precludes to match
the last or even the penultimate element (which we need here).
After some tinkering, I found a way to recast this as ''rebinding to a remoulded sequence'',
and could package a multitude of related tools into a single helper-template,
which works without any further library dependencies.
🠲 extract into a separate header (`variadic-rebind.hpp`) for ease of use.
So this turned out to be much more challenging than expected,
due to the fact that, with this design, typing information is
only available at compile-time. The key trick was to use a
''double-dispatch'' based on a generic lambda. In the end,
this could be rounded out to be self-contained library helper,
which is even fully copyable and assignable and properly
invokes all payload constructors and destructors.
The flip side is that such a design is obviously very flexible
and direct regarding the parser model-bindings, and it should
be fairly well optimisable, since the structure is entirely
static and without any virtual dispatch.
Proper handling of payload lifecycle was verified using
a tracking test object with checksum.
* the implementation of this ''Sum Type'' got quite technical and complicated;
thus better to be extracted as separate library component
* use this as base for the `AltModel`
* make a usage sketch, invoking only the model interactions required
