Some additional tests to challenge the parser, which seems to work well.
Without extended analysis into the usage of those node specifications,
it is pointless to expand further on its capabilities. For now, it is
sufficient to have a foundation for hash-computation in place.
__Note__: found a nifty way to give lib::Several an easy toString rendering,
without cranking up the header inclusion load.
This finishes an ''exercise'' in tool design,
which was set off by the requirement to parse the spec-ID of a render node.
While generally within the confines of a helper utility for simple use cases,
the solution became quite succinct and generic, as it allows to handle arbitrary
LL(n) grammars, possibly with recursion.
...which is the reason for this whole excursion into parser business;
we want to accept specification terms with elements from C++ type expressions,
which especially requires to accept complete comma separated lists within
angle brackets or parenthesis, while separating by comma at top level.
The idea is to model ''not as an expression'' but rather as an ''extended quote'',
and to use inverted regular expressions for non-quote-characters as terminal
...evaluating the recursive syntax of a numerical expression!
* so this light-weight parsing support framework indeed allows
to build fully capable LL(x) parsers, when the user knows how
handle syntax clauses and bind the result models
* furthermore, a notation is demonstrated how to arrange the
binding functions so to keep the syntax definition legible
* this involves a shortcut for homogeneous alternatives
The concept was indeed successful, albeit quite difficult to pull off in detail.
It requires a carefully crafted path of Deduction guides and overloads
to effect the switch from std::function to std::function& at the point
where a predeclared syntax clause placeholder is used recursively
In accordance to the plan drafted yesterday, I will try to integrate
this essential capability into the framework established thus far by a trick,
requiring only minimal adjustment, but some work by the user.
Since the parse function is defined as a (unqualified) template argument,
it is possible to emplace either a `std::function`, or a reference thereto.
For this to work, the user is required to pre-define the expected result type,
and, furthermore, must later on assign a fully specified clause, which
also has a model transformation binding attached to yield this predeclared
result type
...several improvements as result from the more elaborate test cases
- spelling out the model types taken as argument can be challenging and tedious,
thus improve the ability to pass a λ-generic.
- furthermore, using structured bindings on a SeqModel can also simplifiy
binding code; this did not work because the compiler picks the wrong strategy
and attempts to bind the structure fields; need to provide explicit speicalisations
to support the »tuple protocol« for SeqModel.
..considered several further helpers, (like auto-joining into a single string),
but in the end did not implement them, due to questionable relevance
The `bindMatch()` as implemented yesterday works only directly on top
of the terminal parsers, which yield a `RegExp`-Matcher. However,
it would be desirable to provide a generic shortcut to always get
some string as result model. A simple fallback is to return
the part of the input-string accepted thus far.
Basically the implementation is already in place;
yet for better error messages we need to find out if the given functor
can handle the model present at this stage. Since generic-λ are not
functions by themselves (but rather templates), we need to ''probe''
with the expected argument and see if instantiation is possible.
⚠ NOTE: still a strange bug related to using the same Syntax several times
Allowing free recursion in grammars is the key enabling feature,
which allows to accept arbitrary complex structures (like numeric expressions).
It is however also the element which makes the task of parsing a challenging endeavour;
after weighting the arguments, I decided ''not to place the focus on advanced usage,''
yet to open a pathway towards representation of such grammars.
Essentially, I consider it acceptable to require some additional work by the user,
if arbitrary recursive grammars are desired; because this design relies on explicitly
given parse functions, we need to introduce some kind of indirection interface,
to allow ''declaring'' a recursive rule first and later to ''supply the definition,''
which obviously then will involve other rules (or itself) recursively.
This leads to a very ''nifty approach'' towards recursion: we require the user
to provide an ''explicit model type'' beforehand, which implies that this is a
simple type, that can be spelled out (no λ) — and so the user is also
''forced to augment the actual rule with a model-binding,'' thereby reducing
the structured return types from the parse into something simple and uniform.
The user ''has to do the hard work,'' but can ''exploit additional knowledge''
related to the specific use case.
All this framework needs to do then is to supply a `std::function`, using the
explicit return type given; everything else will still work as implemented,
since a `std::function` can always stand-in for any arbitrary λ.
This is the very key feature that requires a real parser and can not be handled by regular expressions.
After all the groundwork, it is surprisingly easy provide now;
only coding up all those DSL-variants is tedious. Notably we also
support accepting an ''optional'' bracket, and we support arbitrary
expressions for the ''opening'' and ''closing construct.''
It seemed that using postfix-decorating operators would be a good fit
for the DSL. Exploring this idea further showed however, that such a scheme
is indeed a good fit from the implementation side, but leads to rather confusing
and hard to grasp DSL statements for many non-trivial syntax definition.
The reason is: such a postfix-decorator will by default work on ''everything defined''
up to that point; this is too much in many cases.
The other alternative would be a function-style definition, which has the benefit
to take the sub-clause directly as argument (so the scope is always explicit).
The downside is that argument arrangement is a bit more tricky for the repetition
combinator (there can be mis-matches, since we take the »SPEC« as free-template argument)
And, moreover, with function-style, having more top-level entrance points would
be helpful. Overall, no fundamental roadblock, just more technicalities in the setup
of the DSL functions.
With that re-arrangd structure, an optional combinator could be easily integrated,
and a solution was provided to pick up the parser function from a sub-expression
defined as Syntax object.
Meanwhile, some kind of style scheme has emerged for the DSL:
We're working much with postfix-decorating operators, which
augment or extend the ''whole syntax clauses defined thus far''
In accordance with this scheme, I decided also to treat repeated expression
as a postfix operator (other than initially planned). This means, the actual
body to be repeated is ''the syntax clause defined thus far'', and the
repeat()-operator only details the number of repetitions and an optional delimiter.
After all the preparation, now this panes out quite well:
* use a simple 3-way branch structure
* the model type was already pre-selected by the `_Join` Model selector
* can just pass the result-model elements to a constructor/builder
* incremental extension can be directly mapped to the predecessor model
So this turned out to be much more challenging than expected,
due to the fact that, with this design, typing information is
only available at compile-time. The key trick was to use a
''double-dispatch'' based on a generic lambda. In the end,
this could be rounded out to be self-contained library helper,
which is even fully copyable and assignable and properly
invokes all payload constructors and destructors.
The flip side is that such a design is obviously very flexible
and direct regarding the parser model-bindings, and it should
be fairly well optimisable, since the structure is entirely
static and without any virtual dispatch.
Proper handling of payload lifecycle was verified using
a tracking test object with checksum.
* the implementation of this ''Sum Type'' got quite technical and complicated;
thus better to be extracted as separate library component
* use this as base for the `AltModel`
* make a usage sketch, invoking only the model interactions required
After exploring the »nested decorator-chain« implementation variant,
I decided to stick to the solution with the λ-vistor, while attempting
to level and smooth-out the design.
* allow to engage ''any'' «slot» at construction
* reverse the order of type parameters to be ascending (as in `std::tuple`)
* make it fully copyable, movable, assignable and provide a `swap()`,
relying on a variant of the "copy-and-swap"-idiom
* add an ''cross-constructor'' for an extended branch set
To represent the result-model for syntax alternatives,
we need a C++ representation for a ''sum type,'' i.e.
a type that can be one from a fixed set of alternatives.
Obviously the implementation will rely on some kind of Union,
or otherwise employ an opaque buffer and perform a forced cast.
Moreover, to be actually usable, a branch-selector-ID must be
captured and stored alongside, so that code processing the results
can detect which branch of the syntax was chosen.
There seem to be several possible avenues to build and structure
an actual class template to provide this implementation model
* a nested decorator-chain
* using a recursive selector-function with a generic-λ
''all these look quite unattractive, unfortunately....''
Seems like a pragmatic choice, which simplifies most syntax definitions significantly.
In exceptional cases, it is still possible to enforce a situation with `\b` or `\B`
* need to pass the parse end-point in the Eval-Result to allow composed models
* this also prepares for support of generic model-binding-λ
With the help of the model-joining case definitions it is then possible to handle sequence extension.
Deliberately I do not engage into fine grained signature checking, since this would lead to very technical code and moreover this is an implementation feature and we control all invocations (with signatures guaranteed to be correct)
Unfortunately, there are some common syntactic structures, which can not easily be dissected by regular expressions alone, since they entail nested subexpressions. While it is possible to get beyond those fundamental limitations with some trickery, doing so remains precisely that, ''trickery.''
After fighting some inner conflicts, since ''I do know how to write a parser'' —
in the end I have brought myself to just do it.
And indeed, as you'd might expect, I have looked into existing library solutions,
and I would not like to have any one of them as part of the project.
* I do not want a ''parser engine'' or ''parser generator''
* I want the directness of recursive-descent, but combined with Regular Expressions as terminal
* I want to see the structure of the used grammar at the definition site of the custom parser function
* I want deep integration of ''model bindings'' into the parse process, i.e. binding-λ
* I do not want to write model-dissecting or pattern-matching code after the parse
* I do not want to expose ''Monads'' as an interface, since they tend to spread unhealthy structure to surrounding code
* I do not want to leak technicalities of the parse mechanics into the using code
* I do not want to impose hard to remember specific conventions onto the user
Thus I've set the following aims:
* The usage should require only a single header include (ideally header-only)
* The entrance point should be a small number of DSL-starter functions
* The parser shall be implemented by recursive-descent, using the parser-combinator technique
* But I want that wrapped into a DSL, to be able to control what is (not) provided or exposed.
* I want a stateful, applicative logic, since parsing, by its very nature, is stateful!
* I want complete compile-time typing, visible to the optimiser, without a virtual »Parser« interface
And last but not least, ''I do not want to create a ticket, since I do not know if those goals can be achieved...''
Building a correct processing-identification is a complex and challenging task; only some aspects can be targeted and implemented right now, as part of the »Playback Vertical Slice«
* components of the ProcID
* parsing the argument-spec
* dispatch of detail information function to retrieve source ports
The choice to rely on strictly typed functor bindings for the Node operation
bears the danger to produce ''template bloat'' — it would be dangerous to add
further functions to the Port-API naïvely; espeically simple information functions
will likely not depend on the full type information.
A remedy to explore would be to exploit properties marked into the Port's `ProcID`
as key for a dispatcher hashtable; assuming that the `NodeBuilder` will be responsible
for registering the corresponding implementation functions, such a solution could even
be somewhat type-safe, as long as the semantics of the ProcID are maintained correctly.
So this is a design sketch how a `ParamBuildSpec` descriptor could be created,
which in turn would provide the foundation to implement a ''Parameter Weaving Pattern...''
__Note__: since this is an extension for advanced usage, yet relies on a storage layout
defined to allow for extensions like this use case here, the anchor type is now defined
to reside in the `TurnoutSystem` in the form of a ''standard parameter block''.
Those standard invocation parameters are fixed and thus can be hard coded.
Tuples and the ''C++ tuple protocol'' build upon variadic arguments
and are thus rather tedious to handle, especially in this situation here,
where the argument can ''sometimes be a tuple...''
Several years ago I made the observation that processing by explicit ''type sequences''
(Loki-style) is much simpler to handle and easier to lift to a generic level of processing.
Thus I'll attempt now to extract the ''iteration and extraction part'' of the logic into a new helper.
`lib::meta::ElmTypes<TUP>` allows to process all ''tuple-like types'' and generic ''type sequences'' uniformely
and enables to use both styles interchangably (btw, it is quite common to ''abuse'' `std::tuple` as a type sequence).
With this helper, we can now
- build a ''type sequence'' from any ''tuple-like'' object (and vice-versa)
- re-bind (i.e. transfer the template parameters to another template)
- apply some wrapper
- create AND / OR evaluations over the types
This solution checks only the minimal precondition,
which is that a type supports `std::tuple_size<T>`.
A more complete implementation turns out to be surprisingly complex,
since a direct check likely requires compile-time reflection capabilities
at the level of at least C++23
- `std::tuple_element<i,T>` typically implements limits checks,
which interfere with the detection of empty structured types
- the situation regarding `std::get<i>()` is even more complicated,
since we might have to probe for ADL-based solutions, or member templates
The check for minimal necessary precondition however allows us to
single out std::tuple, std::array and our own structured types in
compilation branching, which suffices to fulfils actual needs.
It seemed like we're doomed...
Yet we barely escaped our horrid fate, because the C++ structured bindings happen to look also for get<i> member functions!
Any other solution involving a free function `get<i>(h)` would not work, since the `std::tuple` used as base class would inevitably drag in std::get via ADL
Obviously, the other remedy would be to turn the `StorageFrame` into a member; yet doing so is not desirable, as makes the actual storage layout more obscure (and also more brittle)
Actually it is the implementation of `std::get` from our STL implementation
which causes the problems; our new custom implementation works as intended an
would also be picked by the compiler's overload resolution. But unfortunately,
the bounds checking assertion built into std::tuple_element<I,T> triggers
immediately when instantiated with out-of-bounds argument, which happens
during the preparation of overload resolution, even while the compiler
would pick another implementation in the following routine.
So we're out of luck and need to find a workaround...
Why is our specialisation of `std::get` not picked up by the compiler?
* it must somehow be related to the fact that `std::tuple` itself is a base class of lib::HeteroData
* if we remove this inheritance relation, our specialisation is used by the compiler and works as intended
* however, this strange behaviour can not be reproduced in a simple synthetic setup
It must be some further subtlety which marks the tuple case as preferrable
Seems like low hanging fruit and would especially allow to use
those storage blocks with ''structural bindings''
Providing the necessary specialisations for `std::get` however turns out to be difficult;
the compiler insists on picking the direct tuple specialisation, since std::tuple is a
protected base class; yet still surprising -- I was under the impression
that the direct overload should be the closest match
This basically solves this implementation challenge:
It was possible to construct a ''compile-time type-safe'' overlay,
while using force-casts ''without any metadata'' at runtime.
Obviously this is a dangerous setup, but ''should be resonably safe'' when used within the defined scheme...
* now yields an instance of the full `HeteroData<X,X,Z>` template
* work around problems with std::tuple_element_t for derived classes
Can now default-create and direct-init a front-End data block,
access and modify its elements — and the API looks ok-ish for me
Decision to use the generic case as short-hand for the first data block,
and thus ''hide the more technical Loki-List specialisations''
With that, I'm finally able to write the first test case...
This was a tough nut to crack, but recalling the actual usage situation was helpful
* the ''constructor type'' must be created / picked-up beforehand
* we are about to build a ''parameter-computation node''
* so this constructor presumably is passed to a type parameter of a specific weaving pattern
* the constructor must be invoked directly to drop-off the new data frame into the local scope
* it is preferable to attach it only in a second step to the existing HeteroData-Chain (residing in `TurnoutSystem`)
What would be ''desirable'' though is to have some additional safeguard in the type system
to prevent attaching the newly constructed block to a chain with a non-fitting layout,
i.e. the case when several constructors or types get mixed up (because without any further
safe-guard this would lead to uncoordinated out-of-bounds memory access)
...as part of the rendering process, executed on top of the
low-level-model (Render Node network) as conceived thus far.
Parameter handling could be ''encoded'' into render nodes altogether,
or, at the other extreme, an explicit parameter handling could be specified
as part of the Render Node execution. As both extremes will lead to some
unfavourable consequences, I am aiming at a middle ground: largely, the
''automation computation'' will be encoded and hidden within the network,
implying that this topic remains to be addressed as part of defining
the Builder semantics and implementation. Yet in part the required
processing structure can be foreseen at an abstract level, and thus
the essential primitive operations are specified explicitly as part
of the Render Node definition. Notably the ''standard Weaving Pattern''
will include a ''parameter tuple'' into each `FeedManifold` and require
a binding function, which accepts this tuple as first argument.
Moreover — at implementation level, a library facility must be provided
to support handling of ''arbitrary heterogeneous data values'' embedded
directly into stack frame memory, together with a type-safe compile-time
overlay, which allows the builder to embed specific ''accessor handles''
into functor bindings, even while the actual storage location is not
yet known at that time (obviously, as being located on the stack).
__Note__: a recurring topic is how to return descriptor objects from builder functions; for this purpose, I am adjusting the semantics of `lib/nocopy.hpp` to be more specific...
As follow-up from the preceding refactorings,
it is now possible to drastically simplify several type signatures.
Generally speaking, iterator pipelines can now pass-through the result type,
and thus it is no longer necessary to handle this result type explicitly
In the case of `IterStateWrapper`, the result type parameter was retained,
but moved to the second position and defaulted; sometimes it can be relevant
to force a specific type; this is especially useful when defining an
`iterator` and a `const_iterator` based on the same »state-core«
For sake of completeness, since the `IterExplorer` supports building extended
search- and evaluation patterns, a tuple-zipping adapter can be expected
to handle these extended usages transparently.
While the idea is simple, making this actually happen had several ramifications
and required to introduce additional flexibility within the adaptor-framework
to cope better with those cases were some iterator must return a value, not a ref.
In the end, this could be solved with a bit of metaprogramming based on `std::common_type`
...and indeed, this is all quite nasty stuff — in hindsight, my initial intuition
to shy away from this topic was spot-on....
This involves some quite tricky changes in the way types are composed to form an iterator-pipeline.
Some wrappers are added as adaptors or for additional safety-checks, and to provide a builder-API.
Unfortunately, when building a new `IterExplorer` iterator pipeline from an existing pipeline naively,
composing all those types will add several unecessary intermediary wrapper-layers.
Worse even, the handling of `BaseAdapter` prevents the new tuple-zipping iterator
actually to pass-through any `expandChildren()` call.
These issues are a consequence of using templated types, instead of fixed types with an interface;
we can not just determine if some wrapper is present — unless the wrapper itself ''helps by exposing a tag.''
Even while I must admit that the whole packaging and adaptation machinery of `IterExplorer`
looks dangerously complex already, using dedicated type tags for this single purpose
seems like a tenable soulution.
and yes ... this revealed a **long standing bug**
The `Filter::pullFilter()` invocation in the ctor may produce dangling refs,
whenever an underlying source-iterator generates a reference that points
into the iterator itself.
The reason is: due to the »onion shell« design of the iterator pipeline,
we are bound to move a source iterator into the next layer constructor.
With this minor change, the internal result-tuple may now also hold references,
in case a source iterator exposes a reference (which is in fact the standard case).
Under the right circumstances, source-manipulation through the iterator becomes possible.
Moreover, the optimiser should now be able to elide the result-value tuple in many cases.
and access the iterator internals directly instead.
Obviously this is an advanced and possibly dangerous feature, and only possible
when no additional transformer functions are interspersed; moreover this prompted
a review of some long standing type definitions to more precisely reflect the intention.
Note: most deliberately, the Transformer element in IterExplorer must expose a reference type,
and capture the results into an internal ItemWrapper. This is the only way we can support arbitrary functions.
Indeed the solution worked out yesterday could be extracted and turned generic.
Some in-depth testing is necessary though, and possibly some qualifications to allow pass-through of references...
Moreover, last days I started collecting notes regarding problem solving patterns,
which I tend to use frequently, but which might not be obvious and thus can easily
be forgotten. In fact, I had encountered several cases, where I did invent some
roughly similar solution repeatedly, having forgotten about already settled matters.
Hopefully the habit of collecting notes and hints at a central location serves to remedy
Basically I am sick of writing for-loops in those cases
where the actual iteration is based on one or several data sources,
and I just need some damn index counter. Nothing against for-loops
in general — they have their valid uses — sometimes a for-loop is KISS
But in these typical cases, an iterator-based solution would be a
one-liner, when also exploiting the structured bindings of C++17
''I must admit that I want this for a loooooong time —''
...but always got intimidated again when thinking through the fine points.
Basically it „should be dead simple“ — as they say
Well — — it ''is'' simple, after getting the nasty aspects of tuple binding
and reference data types out of the way. Yesterday, while writing those
`TestFrame` test cases (which are again an example where you want to iterate
over two word sequences simultaneously and just compare them), I noticed that
last year I learned about the `std::apply`-to-fold-expression trick, and
that this solution pattern could be adapted to construct a tuple directly,
thereby circumventing most of the problems related to ''perfect forwarding''
So now we have a new util function `mapEach` (defined in `tuple-helper.hpp`)
and I have learned how to make this application completely generic.
As a second step, I implemented a proof-of-concept in `IterZip_test`,
which indeed was not really challenging, because the `IterExplorer`
is so very sophisticated by now and handles most cases with transparent
type-driven adaptors. A lot of work went into `IterExplorer` over the years,
and this pays off now.
The solution works as follows:
* apply the `lib::explore()` constructor function to the varargs
* package the resulting `IterExplorer` instantiations into a tuple
* build a »state core« implementation which just lifts out the three
iterator primitives onto this ''product type'' (i.e. the tuple)
* wrap it in yet another `IterExplorer`
* add a transformer function on top to extract a value-tuple for each ''yield'
As expected, works out-of-the-box, with all conceivable variants and wild
mixes of iterators, const, pointers, references, you name it....
PS: I changed the rendering of unsigned types in diagnostic output
to use the short notation, e.g. `uint` instead of `unsigned int`.
This dramatically improves the legibility of verification strings.
I changed the rendering of unsigned types in diagnostic output
to use the short notation, e.g. `uint` instead of `unsigned int`.
This dramatically improves the legibility of verification strings.
Moreover, I took the opportunigy to look through the existing page
with codeing style guides to explicitly write down some conventions
formed over years of usage.
I did not just »make up« those light heartedly, rather these conventions
are the result of a craftsman's ''attentive observation and self-reflection.''
* Lumiera source code always was copyrighted by individual contributors
* there is no entity "Lumiera.org" which holds any copyrights
* Lumiera source code is provided under the GPL Version 2+
== Explanations ==
Lumiera as a whole is distributed under Copyleft, GNU General Public License Version 2 or above.
For this to become legally effective, the ''File COPYING in the root directory is sufficient.''
The licensing header in each file is not strictly necessary, yet considered good practice;
attaching a licence notice increases the likeliness that this information is retained
in case someone extracts individual code files. However, it is not by the presence of some
text, that legally binding licensing terms become effective; rather the fact matters that a
given piece of code was provably copyrighted and published under a license. Even reformatting
the code, renaming some variables or deleting parts of the code will not alter this legal
situation, but rather creates a derivative work, which is likewise covered by the GPL!
The most relevant information in the file header is the notice regarding the
time of the first individual copyright claim. By virtue of this initial copyright,
the first author is entitled to choose the terms of licensing. All further
modifications are permitted and covered by the License. The specific wording
or format of the copyright header is not legally relevant, as long as the
intention to publish under the GPL remains clear. The extended wording was
based on a recommendation by the FSF. It can be shortened, because the full terms
of the license are provided alongside the distribution, in the file COPYING.
After augmenting our `lib/random.hpp` abstraction framework to add the necessary flexibility,
a common seeding scheme was ''built into the Test-Runner.''
* all tests relying on some kind of randomness should invoke `seedRand()`
* this draws a seed from the `entropyGen` — which is also documented in the log
* individual tests can now be launched with `--seed` to force a dedicated seed
* moreover, tests should build a coherent structure of linked generators,
especially when running concurrently. The existing tests were adapted accordingly
All usages of `rand()` in the code base were investigated and replaced
by suitable calls to our abstraction framework; the code base is thus
isolated from the actual implementation, simplifying further adaptation.
A deeper investigation revealed that we can show the result of glitches
for each relevant situation, simply by scrutinising the produced distribution.
Even the 64-bit-Variant shows a skewed distribuion, in spite of all numbers
being within definition range.
So the conclusion is: we can expect tilted results, but in many cases
this might not be an issue, if the result range is properly wrapped / clipped.
Notably this is the case if we just want to inject a randomised sleep into a multithreaded test setup
Build a self-contained test case to document these findings.
