...because swapping in the new standards-based implementation
leads to compile failures on tests to cover out-of-bounds cases.
Under the (wrong) assumption, that some mistake must be hidden in
the Splice-metafunction, I first provided a complete test coverage;
while the actual problem was right below my nose, and quite obvious...
The old implementation, being based on a case distinction over the argument count,
simply was not able even to notice excess arguments; other the new implementation,
based on variadics and `std::apply`, which is fully generic and thus
passes excess arguments to `std::bind` when a position beyond the actual
argument list is specified to be closed.
The old behaviour was to silently ignore such an out-of-bounds spec,
and this can be reinstated by explicitly capping the prepared tuple
of binders and actual arguments passed to `std::bind`
Another question of course is, if being tolerant here is a good idea.
And beyond that, function-closure.hpp is still terrifyingly complex,
unorganised and use-case driven, to start with....
...can now be formulated in a single function,
based on the apply-to-λ technique invented by David Vandervoorde.
WARNING: the rewritten version of BindToArgument<...>::reduced()
does not compile in the out-of-bounds case, revealing a possibly
long standing defect in the typelist-metafunction Splice
This library header was developed at a time, where C++ had no built-in support
for so called "invokables"; `std::invoke` and `std::apply` were added much later;
So in that early version that was a significant technical hurdle to overcome.
seems like it might be possible to get rid of the TupleApplicator alltogether?
This is one of the most problematic headers, because it is highly complex
and comprises tightly interwoven definitions (in functional programming style),
which in turn are used deep within other features.
What concerns me is that this header is very much tangled
and pushes me (as the author) to my mental limits.
And on top of this comes that this code has to deal with intricate aspects
like perfect forwarding, and proper handling of binder instances and
function argument copying (which basically should be left to `std::bind`)
Fortunately, the changes ''for this specific topic'' are transparent:
Type sequences are not used on the API for function closure and composition,
but only as an internal tool to assemble argument tuples used for either
binding or invocation of the resulting (partially closed) function.
To bootstrap this tricky refactoring, initially a bridge definition
was used, with a variadic argument pack, but delegating to the old
non-variadic type sequence and from there further into LISP style
list processing of types and meta definitions, as pioneered by the
Loki libarary. Luimiera uses this technique since a long time to
perform the complex tasks sometimes required for code generation
and generic function and type adaptation.
with this changeset, a direct variadics based entrance into
type list processing is provided, so that the old definition
is now completely separate and can be removed eventually.
Most of the type-list and type-sequence related eccosystem can be
just switched over, after having added the conversion variants for
the new-style variadic type sequences
Again this was used as opportunity to improve readability of related tests
As expected, these work on the new-style variadic type sequences
equally well than on the old ones (tail-filled with `Nil` markers).
On that occasion, a complete makeover of the huge test case was carried out,
now relying on `ExpectString` instead of printing to STDOUT. This has the
benefit of showing the expectation immediately next to the code to be tested,
and thus makes it much easier to ''actually see'' how these meta-functions
operate on their parameters (which in fact are types in a type list)
- provide complete conversion paths old-style ⟷ new-style
- switch the basic tests to the new variadic sequences
- modernise the code; replace typedefs by `using`
- change some struct-style ''meta-functions'' into
constexpr or compile-time constants
Since I've convinced myself during the last years that this kind
of typelist programming is ''not a workaround'' — it is even
superior to pattern matching on variadics for certain kinds
of tasks — the empty struct defined as `NullType` got into
more widespread use as a marker type in the Lumiera code base.
It seems adequate though to give it a much more evocative name
Attempting to reduce the remaining pre-C++11 workarounds before upgrade to C++20...
As a first step: rename the old type-sequence implementation into `TyOLD`
to make it clearly distinguishable; a new variadic implementation `TySeq`
was already introduced as partial workaround, and the next steps
will be to switch over essential parts of the type-sequence library.
During the early stage of the Project, at some point I attempted
to »attack« the topic of Engine and Render Nodes following a ''top down path.''
This effort went into a dead end eventually — due to the total lack
of tangible reference points to relate to. However, the implementation
at that time prompted the development of several supporting facilities,
which remain relevant until today. And it resulted in a ''free wheeling''
compound of implementation structures, which could even be operated
through some highly convoluted unit test.
This piece of implementation code was valuable as starting point for th
»Playback Vertical Slice« in 2024 — resulting in a new design which was
''re-oriented'' towards a new degree of freedom (the »Domain Ontology«)
while handling the configuration and connectivity of Render Nodes in
a rather fixed and finite way. This new approach seems to be much more
successful, as we're now able to build, connect and invoke Render Nodes,
thereby mapping the processing through a functor binding into some
arbitrary, external processing function (which will later be supplied
by a media processing library — and thus be part of some »Domain Ontology«)
This is an advanced diagnostics added (presumably) with GCC-13
and attempts to protect against an insidious side-effect of ''overload resolution''
Basically C++ (like its ancestor C) is oriented towards direct linkage and adds
the OO-style dynamic dispatch (through virtual functions and a VTable)
only as an extension, which must be requested explicitly.
Thus the resolution of ''overloads'' (as opposed to ''overridden'' virtual functions)
always takes precedence and happens within the directly visible scope,
which can cause the compiler to perform an implicit conversion instead of
invoking a different virtual function, which is defined in a base class.
However, this diagnostics seems to be implemented in an overly zealous way:
The compiler warns at the time of the type instantiation, and even in cases
where it is effectively impossible to encounter this dangerous shadowing situation.
See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109740
This leads to several ill-guided warnings in the Lumiera code base, which unfortunately
can only be addressed by disabling this diagnostics for all usages of some header.
The reason is, we often generate chains of template instantiations driven by type lists,
and in such usage pattern, it is not even possible to bring all the other inherited overloads
into scope (with a using `BASE::func` clause), because such a specification would be ambiguous
and result in a real compile error, because even the interface is generated from a chain of mix-in templates
This is a crucial feature, discovered only late, while building
an overall integration test: it is quite common for processing functionality
to require both a technical, and an artistic parametrisation. Obviously,
both are configured from quite different sources, and thus we need a way
to pre-configure ''some parameter values,'' while addressing other ones
later by an automation function. Probably there will be further similar
requirements, regarding the combination of automation and fixed
user-provided settings (but I'll leave that for later to settle).
On a technical level, wiring such independent sources of information
can be quite a challenging organisational problem — which however can be
decomposed using ''partial function closure'' (as building a value tuple
can be packaged into a builder function). Thus in the end I was able to
delegate a highly technical problem to an existing generic library function.
* now able to demonstrate close-front, close-back and close-argument
* can also apply the same cases to `std::array`, with input and
output type seamlessly adapted to `std::array`
__Summary__:
* the first part to prepare a binding involves creating a mapped tuple,
with re-ordered elements and some elements replaced by placehoder-markers.
This part **must not use RValue-References** (doing so would be possible
only under very controlled conditions)
* the second part, which transports these mapped-tuple elements into the binder
''could be converted to perfect-forwarding.'' This would require to replace
the `Apply<N>' by a variadic template, delegating to `std::apply` and `std::bind`
With this changeset, I have modernised a lot of typedefs to make them more legible,
and I have introduced perfect-forwarding in the entrance path, up to the point
where the values are passed to `TupleConstructor`.
With these additions, all conceivable cases are basically addressed.
Take this as opportunity to investigate how the existing implementation
transports values into the Binder, where they will be stored as data fields.
Notably the mechanism of the `TupleConstructor` / `ElmMapper` indeed
''essentially requires'' to pass the initialisers ''by-reference'',
because otherwise there would be limitations on possible mappings.
This implies that not much can be done for ''perfect forwarding'' of initialisers,
but at least the `BindToArgument` can be simplified to take the value directly.
...which should ''basically work,'' since `std::array` is ''»tuple-like«'' —
BUT unfortunately it has a quite distinct template signature which does not fit
into the generic scheme of a product type.
Obviously we'd need a partial specialisation, but even re-implementing this
turns out to be damn hard, because there is no way to generate a builder method
with a suitable explicit type signature directly, because such a builder would
need to accept precisely N arguments of same type. This leads to a different
solution approach: we can introduce an ''adapter type'', which will be layered
on top of `std::array` and just expose the proper type signature so that the
existing Implementation can handle the array, relying on the tuple-protocol.
__Note__: this changeset adds a convenient pretty-printer for `std::array`,
based on the same forward-declaration trick employed recently for `lib::Several`.
You need to include 'lib/format-util.hpp' to actually use it.
What emerges here, seems to be a generic helper to handle
partial closure of ''tuple-like'' data records. In any case,
this is highly technical meta-programming code and mandates
extraction into a separate header — simplifying `NodeBuilder`
Likely the most widely used facility, which enters into meta-programming
with type sequences, is our function-signature-detector `_Fun<X>`,
which returns a argument type-sequence.
After adding some bridges for cross-compatibility,
function-arguments are now extracted as a new-style,
''variadic sequence'' without trailing `NullType`.
Doing so required to augment some of the most widely used
sequence-processing helpers to work seamlessly also with the
new-style variadic sequences with a definition variant based
on variadics, which typically will later-on obsolete the original
solution, which at that time needed to be tediously coded as a
series of explicit specialisations for N arguments.
...on top of the parameter-decorating functionality developed thus far.
The idea is to allow in the `NodeBuilder` to supply ''some parameters''
directly, while the remaining parameters will be drawn from automation.
Several years ago, I developed some helpers for partial function closure.
Unfortunately these utils are somewhat limited, and rely on some pre-C++11
constructs, yet seem to be usable for the task at hand, since parameters
are always expected as value objects by definition.
This changeset shows a working proof-of concept for left-closing a
parameter tuple with 5 elements; this turns out to surprisingly difficult
due to the full genericity of the acceptable parameter-aggregates...
After all the preparation, now this panes out quite well:
* use a simple 3-way branch structure
* the model type was already pre-selected by the `_Join` Model selector
* can just pass the result-model elements to a constructor/builder
* incremental extension can be directly mapped to the predecessor model
This is a rather obnoxious limitation of C++ variadics:
the inability to properly match against a mixed sequence with variadics.
The argument pack must always be the last element, which precludes to match
the last or even the penultimate element (which we need here).
After some tinkering, I found a way to recast this as ''rebinding to a remoulded sequence'',
and could package a multitude of related tools into a single helper-template,
which works without any further library dependencies.
🠲 extract into a separate header (`variadic-rebind.hpp`) for ease of use.
Unfortunately, there are some common syntactic structures, which can not easily be dissected by regular expressions alone, since they entail nested subexpressions. While it is possible to get beyond those fundamental limitations with some trickery, doing so remains precisely that, ''trickery.''
After fighting some inner conflicts, since ''I do know how to write a parser'' —
in the end I have brought myself to just do it.
And indeed, as you'd might expect, I have looked into existing library solutions,
and I would not like to have any one of them as part of the project.
* I do not want a ''parser engine'' or ''parser generator''
* I want the directness of recursive-descent, but combined with Regular Expressions as terminal
* I want to see the structure of the used grammar at the definition site of the custom parser function
* I want deep integration of ''model bindings'' into the parse process, i.e. binding-λ
* I do not want to write model-dissecting or pattern-matching code after the parse
* I do not want to expose ''Monads'' as an interface, since they tend to spread unhealthy structure to surrounding code
* I do not want to leak technicalities of the parse mechanics into the using code
* I do not want to impose hard to remember specific conventions onto the user
Thus I've set the following aims:
* The usage should require only a single header include (ideally header-only)
* The entrance point should be a small number of DSL-starter functions
* The parser shall be implemented by recursive-descent, using the parser-combinator technique
* But I want that wrapped into a DSL, to be able to control what is (not) provided or exposed.
* I want a stateful, applicative logic, since parsing, by its very nature, is stateful!
* I want complete compile-time typing, visible to the optimiser, without a virtual »Parser« interface
And last but not least, ''I do not want to create a ticket, since I do not know if those goals can be achieved...''
Now reaping the benefits of the ambitious refactorings done yesterday.
- Only retaining the basic distinction of the four use cases
- all further adaptation now directly based on the »lifted« types
- can even add quite stringent compile-time sanity checks.
Now the refactoring is on-par with the capabilities of the old downstream code,
which, btw, could be retained in compilable (yet not working) state. But the new
traits logic is already more capable and could accept tuples and arrays as well.
Next major topic to address is to provide the foundation for parameter handling.
Tuples and the ''C++ tuple protocol'' build upon variadic arguments
and are thus rather tedious to handle, especially in this situation here,
where the argument can ''sometimes be a tuple...''
Several years ago I made the observation that processing by explicit ''type sequences''
(Loki-style) is much simpler to handle and easier to lift to a generic level of processing.
Thus I'll attempt now to extract the ''iteration and extraction part'' of the logic into a new helper.
`lib::meta::ElmTypes<TUP>` allows to process all ''tuple-like types'' and generic ''type sequences'' uniformely
and enables to use both styles interchangably (btw, it is quite common to ''abuse'' `std::tuple` as a type sequence).
With this helper, we can now
- build a ''type sequence'' from any ''tuple-like'' object (and vice-versa)
- re-bind (i.e. transfer the template parameters to another template)
- apply some wrapper
- create AND / OR evaluations over the types
This changeset is a sketch how to switch the entire implementation of the ''Invocation Adatper''
over to a generic argument usage scheme. This requires the ability to
- detect if some argument is actually a ''structured type''
- investigate components of such a structured type to draw a distinction between »Buffer« and »Parameter«
- ''lift'' the implementation of simple values to work on tuples
- provide a way to ''bridge'' from ''tuple-style'' programming to ''array access''
As a building block, we use a new iteration-over-index construct,
based on an idea discussed in https://stackoverflow.com/q/53522781/444796
The trick is to pass a `std::integer_constant` to a λ-generic
This solution checks only the minimal precondition,
which is that a type supports `std::tuple_size<T>`.
A more complete implementation turns out to be surprisingly complex,
since a direct check likely requires compile-time reflection capabilities
at the level of at least C++23
- `std::tuple_element<i,T>` typically implements limits checks,
which interfere with the detection of empty structured types
- the situation regarding `std::get<i>()` is even more complicated,
since we might have to probe for ADL-based solutions, or member templates
The check for minimal necessary precondition however allows us to
single out std::tuple, std::array and our own structured types in
compilation branching, which suffices to fulfils actual needs.
This is an attempt to rework gradually while keeping the existing code valid.
For the simple reason that the existing code is quite elaborate and difficult to re-orient.
Thus using a ''second branch,'' and sharing the traits template while expanding its capabilities
This was a tough nut to crack, but recalling the actual usage situation was helpful
* the ''constructor type'' must be created / picked-up beforehand
* we are about to build a ''parameter-computation node''
* so this constructor presumably is passed to a type parameter of a specific weaving pattern
* the constructor must be invoked directly to drop-off the new data frame into the local scope
* it is preferable to attach it only in a second step to the existing HeteroData-Chain (residing in `TurnoutSystem`)
What would be ''desirable'' though is to have some additional safeguard in the type system
to prevent attaching the newly constructed block to a chain with a non-fitting layout,
i.e. the case when several constructors or types get mixed up (because without any further
safe-guard this would lead to uncoordinated out-of-bounds memory access)
For sake of completeness, since the `IterExplorer` supports building extended
search- and evaluation patterns, a tuple-zipping adapter can be expected
to handle these extended usages transparently.
While the idea is simple, making this actually happen had several ramifications
and required to introduce additional flexibility within the adaptor-framework
to cope better with those cases were some iterator must return a value, not a ref.
In the end, this could be solved with a bit of metaprogramming based on `std::common_type`
...and indeed, this is all quite nasty stuff — in hindsight, my initial intuition
to shy away from this topic was spot-on....
Basically I am sick of writing for-loops in those cases
where the actual iteration is based on one or several data sources,
and I just need some damn index counter. Nothing against for-loops
in general — they have their valid uses — sometimes a for-loop is KISS
But in these typical cases, an iterator-based solution would be a
one-liner, when also exploiting the structured bindings of C++17
''I must admit that I want this for a loooooong time —''
...but always got intimidated again when thinking through the fine points.
Basically it „should be dead simple“ — as they say
Well — — it ''is'' simple, after getting the nasty aspects of tuple binding
and reference data types out of the way. Yesterday, while writing those
`TestFrame` test cases (which are again an example where you want to iterate
over two word sequences simultaneously and just compare them), I noticed that
last year I learned about the `std::apply`-to-fold-expression trick, and
that this solution pattern could be adapted to construct a tuple directly,
thereby circumventing most of the problems related to ''perfect forwarding''
So now we have a new util function `mapEach` (defined in `tuple-helper.hpp`)
and I have learned how to make this application completely generic.
As a second step, I implemented a proof-of-concept in `IterZip_test`,
which indeed was not really challenging, because the `IterExplorer`
is so very sophisticated by now and handles most cases with transparent
type-driven adaptors. A lot of work went into `IterExplorer` over the years,
and this pays off now.
The solution works as follows:
* apply the `lib::explore()` constructor function to the varargs
* package the resulting `IterExplorer` instantiations into a tuple
* build a »state core« implementation which just lifts out the three
iterator primitives onto this ''product type'' (i.e. the tuple)
* wrap it in yet another `IterExplorer`
* add a transformer function on top to extract a value-tuple for each ''yield'
As expected, works out-of-the-box, with all conceivable variants and wild
mixes of iterators, const, pointers, references, you name it....
PS: I changed the rendering of unsigned types in diagnostic output
to use the short notation, e.g. `uint` instead of `unsigned int`.
This dramatically improves the legibility of verification strings.
I changed the rendering of unsigned types in diagnostic output
to use the short notation, e.g. `uint` instead of `unsigned int`.
This dramatically improves the legibility of verification strings.
Moreover, I took the opportunigy to look through the existing page
with codeing style guides to explicitly write down some conventions
formed over years of usage.
I did not just »make up« those light heartedly, rather these conventions
are the result of a craftsman's ''attentive observation and self-reflection.''
* Lumiera source code always was copyrighted by individual contributors
* there is no entity "Lumiera.org" which holds any copyrights
* Lumiera source code is provided under the GPL Version 2+
== Explanations ==
Lumiera as a whole is distributed under Copyleft, GNU General Public License Version 2 or above.
For this to become legally effective, the ''File COPYING in the root directory is sufficient.''
The licensing header in each file is not strictly necessary, yet considered good practice;
attaching a licence notice increases the likeliness that this information is retained
in case someone extracts individual code files. However, it is not by the presence of some
text, that legally binding licensing terms become effective; rather the fact matters that a
given piece of code was provably copyrighted and published under a license. Even reformatting
the code, renaming some variables or deleting parts of the code will not alter this legal
situation, but rather creates a derivative work, which is likewise covered by the GPL!
The most relevant information in the file header is the notice regarding the
time of the first individual copyright claim. By virtue of this initial copyright,
the first author is entitled to choose the terms of licensing. All further
modifications are permitted and covered by the License. The specific wording
or format of the copyright header is not legally relevant, as long as the
intention to publish under the GPL remains clear. The extended wording was
based on a recommendation by the FSF. It can be shortened, because the full terms
of the license are provided alongside the distribution, in the file COPYING.
We use the memory address to detect reference to ''the same language object.''
While primarily a testing tool, this predicate is also used in the
core application at places, especially to prevent self-assignment
and to handle custom allocations.
It turns out that actually we need two flavours for convenient usage
- `isSameObject` uses strict comparison of address and accepts only references
- `isSameAdr` can also accept pointers and even void*, but will dereference pointers
This leads to some further improvements of helper utilities related to memory addresses...
part of the observed deviation stems form bugs in logging and checksum calculation;
but there seems to be a real problem hidden in the allocator usage of the
new component, since the use-cnt of the handle does not drop to zero
...turns out that we must ensure to pass a plain "object" type
to the standard allocator framework (no const, no references).
Here, ''object in C++ terminology'' means a scalar or record type,
but no functor, no references and no void,
- `forElse` belongs to the metaprogramming utils
- have a CSVLine, which is a string with custom appending mechanism
- this in turn allows CSVData to accept arbitrary sized tuples,
by rendering them into CSVLine
the metafunction `is_basically<X>` performed only an equality match,
while, given it's current usage, it should also include a subtype-interface-match.
This changes especially the `is_StringLike<S>` metafunction,
both on const references and on classes built on top of string
or string_view.
Whenever a class defines a single-arg templated constructor,
there is danger to shadow the auto-generated copy operations,
leading to insidious failures.
Some months ago, I did the ''obvious'' and added a tiny helper,
allowing to mask out the dangerous case when the ''single argument''
is actually the class itself (meaning, it is a copy invocation and
not meant to go through this templated ctor...
As this already turned out as tremendously helpful, I now extended
this helper to also cover cases where the problematic constructor
accepts variadic arguments, which is quite common with builder-helpers
Basically GenNode and the enclosed record were designed to be
immutable — yet some valid usage patterns emerged where gradually
building structure seems adequate — which can be accomodated by
entering a ''mutation mode'' explicitly through the Rec::Mutator.
Over time, a builder usage style came in widespread usage, especially
when building test data structures. There seems to be no deeper reason
preventing the Mutator from being ''moved'' — notwithstanding the fact
that using such a ''movable builder'' can be dangerous, especially
when digressing from the strict »fluent inline builder« usage style
and storing the mutator into a variable.
For populating the data context for a text template instantiation,
we have now a valid case where it seems helpful to partially populate
the context and then move it further down into an implementation
function, which does the bulk of the work.
Sting-view is tricky, since it deliberately does not define a
conversion operator; rather, string has an explicit constructor.
This design was chosen on purpose, since creating a string will
„materialise“ the string-view, which could have severe performance
ramifications when done automatically.
Regarding Lumiera's string-conversion tooling, it seems indicated
thus to add std::string_view explicitly as a known conversion path,
even while this conversion does not happen implicitly.
In the Lumiera code base, a convenient string conversion is used
an many places, and is also ''magically'' integrated into the usual
C++ style output with `<<` operators.
However, there is a ''gotcha'' — in the ''rare cases'' when we
actually want to use the C++ input/output framework to copy stream
data from an input source into an output sink, obviously we do not want
the input source to be »string converted«....
showDecimal -> decimal10 (maximal precision to survive round-trip through decimal representation=
showComplete -> max_decimal10 (enough decimal places to capture each possible distinct floating-point value)
Use these new functions to rewrite the format4csv() helper
...causing the system to freeze due to excess memory allocation.
Fortunately it turned out this was not an error in the Scheduler core
or memory manager, but rather a sloppiness in the test scaffolding.
However, this incident highlights that the memory manager lacks some
sanity checks to prevent outright nonsensical allocation requests.
Moreover it became clear again that the allocation happens ''already before''
entering the Scheduler — and thus the existing sanity check comes too late.
Now I've used the same reasoning also for additional checks in the allocator,
limiting the Epoch increment to 3000 and the total memory allocation to 8GiB
Talking of Gibitbytes...
indeed we could use a shorthand notation for that purpose...
...during development of the Chain-Load, it became clear that we'll often
need a collection of small trees rather than one huge graph. Thus a rule
for pruning nodes and finishing graphs was added. This has the consequence
that there might now be several exit nodes scattered all over the graph;
we still want one single global hash value to verify computations,
thus those exit hashes must now be picked up from the nodes and
combined into a single value.
All existing hash values hard coded into tests must be updated
Introduced as remedy for a long standing sloppiness:
Using a `char[]` together with `reinterpret_cast` in storage management helpers
bears danger of placing objects with wrong alignment; moreover, there are increasing
risks that modern code optimisers miss the ''backdoor access'' and might apply too
aggressive rewritings.
With C++17, there is a standard conformant way to express such a usage scheme.
* `lib::UninitialisedStorage` can now be used in a situation (e.g. as in `ExtentFamily`)
where a complete block of storage is allocated once and then subsequently used
to plant objects one by one
* moreover, I went over the code base and adapted the most relevant usages of
''placement-new into buffer'' to also include the `std::launder()` marker
