This was a lot of intricate technical work,
and is now verified in-depth, covering all possible cases.
__We can now__
* build Nodes
* verify in detail correct connectivity
* read Node-IDs and processing specifications
* maintain a symbolic spec for the arguments of a Port
(and beyond that, we can also **invoke nodes**, which remains to be formally verified)
An essential goal still to reach is a verification of the `NodeBuilder`'s products
Relying on the low-level diagnostic facilities pioneered last days,
it should now be possible to define simple and readable connectivity-clauses,
allowing to build some connected nodes and then verify the connections explicitly.
Handling of extended attributes in conjunction with the hash
turns out to be a rather complicated topic, with some tricky fine details.
And, most important, at the moment I am lacking the proper perspective
to address it and find adequate solutions. Luckily, the cache-key is
not required at the moment, ''and so this topic will be postponed''
As a minimum to complete the diagnostics functions, it is sufficient to set
the appropriate flags in the `ProcID` directly -- and to add some convenience wrappers.
...especially the extended attributes remain somewhat nebulous,
since non of the prospective usages are close to being implemented right now.
It seems, we'll get two distinct sources at construction time of the Node
* additional qualifiers from the Library plug-in
* internal flags or qualifiers provided by the `NodeBuilder`
Another related concern seems to be generation of cache-keys,
which however will ''consume'' the proc-hash generated by the ProcID,
but not change the ID itself; cache-key generation is a tricky subject
and was somewhat overlooked regarding the connection to the `BufferProvider`.
Opened a new ticket #1292 as reminder for this issue.
...exploiting the ''backdoor access'' bypassing the VTable,
as made possible by a common congruent storage layout.
This is a first proof-of-concept, but also shows that the demo nodes
in NodeMeta_test are wired as expected. What is needed now is to make
this diagnostic access easier to invoke and more bullet-proof, by setting
the proper Attribute bits directly in the `NodeBuilder`
...to create an ''access path for diagnostics'' and further evaluations
while ''bypassing the VTable.''
It is a well-known downside of specifically typed, highly optimisable
template-based code to create a dangerous leverage for generating spurious,
mostly identical virtual function instances added for secondary concerns.
Thus it is a consequence of this design choice, either to forego some diagnostic
and analytical possibilities, or to exploit ''other means'' for retrieving
internal data, which is needed for tangential purposes only. The solution
pursued hereby exploits similar layout of various ''weaving pattern''
template instances to create an ''access backdoor'' for use cases
beyond the primary performance-critical path.
Some additional tests to challenge the parser, which seems to work well.
Without extended analysis into the usage of those node specifications,
it is pointless to expand further on its capabilities. For now, it is
sufficient to have a foundation for hash-computation in place.
__Note__: found a nifty way to give lib::Several an easy toString rendering,
without cranking up the header inclusion load.
This is a nice little goodie: allow to write repeated arguments with the
shorthand notation known from lisp and logic programming. For multi-channel media,
structurally similar wirings for each channel will be quite common....
...at the point where I identified the need to parse nested terms.
The goals are still the same
* write tests to ''verify connectivity'' of nodes generated by the new `NodeBuilder`
* allow for ''extended custom attributes'' in the ProcID
* provide the ability to mark specific parametrisations
* build a Hash-Key to identify a given processing step
__Note Library__: this is the first time `lib::Several` was used to hold a ''const object''.
Some small adjustments in type detection were necessary to make that work.
Access to stored data happens through the `lib::Several` front-end and thus always includes
the const modifier; so casting any const-ness out of the way in the low-level memory management
is not a concern...
This finishes an ''exercise'' in tool design,
which was set off by the requirement to parse the spec-ID of a render node.
While generally within the confines of a helper utility for simple use cases,
the solution became quite succinct and generic, as it allows to handle arbitrary
LL(n) grammars, possibly with recursion.
...which is the reason for this whole excursion into parser business;
we want to accept specification terms with elements from C++ type expressions,
which especially requires to accept complete comma separated lists within
angle brackets or parenthesis, while separating by comma at top level.
The idea is to model ''not as an expression'' but rather as an ''extended quote'',
and to use inverted regular expressions for non-quote-characters as terminal
...evaluating the recursive syntax of a numerical expression!
* so this light-weight parsing support framework indeed allows
to build fully capable LL(x) parsers, when the user knows how
handle syntax clauses and bind the result models
* furthermore, a notation is demonstrated how to arrange the
binding functions so to keep the syntax definition legible
* this involves a shortcut for homogeneous alternatives
The concept was indeed successful, albeit quite difficult to pull off in detail.
It requires a carefully crafted path of Deduction guides and overloads
to effect the switch from std::function to std::function& at the point
where a predeclared syntax clause placeholder is used recursively
In accordance to the plan drafted yesterday, I will try to integrate
this essential capability into the framework established thus far by a trick,
requiring only minimal adjustment, but some work by the user.
Since the parse function is defined as a (unqualified) template argument,
it is possible to emplace either a `std::function`, or a reference thereto.
For this to work, the user is required to pre-define the expected result type,
and, furthermore, must later on assign a fully specified clause, which
also has a model transformation binding attached to yield this predeclared
result type
...several improvements as result from the more elaborate test cases
- spelling out the model types taken as argument can be challenging and tedious,
thus improve the ability to pass a λ-generic.
- furthermore, using structured bindings on a SeqModel can also simplifiy
binding code; this did not work because the compiler picks the wrong strategy
and attempts to bind the structure fields; need to provide explicit speicalisations
to support the »tuple protocol« for SeqModel.
..considered several further helpers, (like auto-joining into a single string),
but in the end did not implement them, due to questionable relevance
The `bindMatch()` as implemented yesterday works only directly on top
of the terminal parsers, which yield a `RegExp`-Matcher. However,
it would be desirable to provide a generic shortcut to always get
some string as result model. A simple fallback is to return
the part of the input-string accepted thus far.
Basically the implementation is already in place;
yet for better error messages we need to find out if the given functor
can handle the model present at this stage. Since generic-λ are not
functions by themselves (but rather templates), we need to ''probe''
with the expected argument and see if instantiation is possible.
⚠ NOTE: still a strange bug related to using the same Syntax several times
Allowing free recursion in grammars is the key enabling feature,
which allows to accept arbitrary complex structures (like numeric expressions).
It is however also the element which makes the task of parsing a challenging endeavour;
after weighting the arguments, I decided ''not to place the focus on advanced usage,''
yet to open a pathway towards representation of such grammars.
Essentially, I consider it acceptable to require some additional work by the user,
if arbitrary recursive grammars are desired; because this design relies on explicitly
given parse functions, we need to introduce some kind of indirection interface,
to allow ''declaring'' a recursive rule first and later to ''supply the definition,''
which obviously then will involve other rules (or itself) recursively.
This leads to a very ''nifty approach'' towards recursion: we require the user
to provide an ''explicit model type'' beforehand, which implies that this is a
simple type, that can be spelled out (no λ) — and so the user is also
''forced to augment the actual rule with a model-binding,'' thereby reducing
the structured return types from the parse into something simple and uniform.
The user ''has to do the hard work,'' but can ''exploit additional knowledge''
related to the specific use case.
All this framework needs to do then is to supply a `std::function`, using the
explicit return type given; everything else will still work as implemented,
since a `std::function` can always stand-in for any arbitrary λ.
This is the very key feature that requires a real parser and can not be handled by regular expressions.
After all the groundwork, it is surprisingly easy provide now;
only coding up all those DSL-variants is tedious. Notably we also
support accepting an ''optional'' bracket, and we support arbitrary
expressions for the ''opening'' and ''closing construct.''
It seemed that using postfix-decorating operators would be a good fit
for the DSL. Exploring this idea further showed however, that such a scheme
is indeed a good fit from the implementation side, but leads to rather confusing
and hard to grasp DSL statements for many non-trivial syntax definition.
The reason is: such a postfix-decorator will by default work on ''everything defined''
up to that point; this is too much in many cases.
The other alternative would be a function-style definition, which has the benefit
to take the sub-clause directly as argument (so the scope is always explicit).
The downside is that argument arrangement is a bit more tricky for the repetition
combinator (there can be mis-matches, since we take the »SPEC« as free-template argument)
And, moreover, with function-style, having more top-level entrance points would
be helpful. Overall, no fundamental roadblock, just more technicalities in the setup
of the DSL functions.
With that re-arrangd structure, an optional combinator could be easily integrated,
and a solution was provided to pick up the parser function from a sub-expression
defined as Syntax object.
Meanwhile, some kind of style scheme has emerged for the DSL:
We're working much with postfix-decorating operators, which
augment or extend the ''whole syntax clauses defined thus far''
In accordance with this scheme, I decided also to treat repeated expression
as a postfix operator (other than initially planned). This means, the actual
body to be repeated is ''the syntax clause defined thus far'', and the
repeat()-operator only details the number of repetitions and an optional delimiter.
After all the preparation, now this panes out quite well:
* use a simple 3-way branch structure
* the model type was already pre-selected by the `_Join` Model selector
* can just pass the result-model elements to a constructor/builder
* incremental extension can be directly mapped to the predecessor model
This is a rather obnoxious limitation of C++ variadics:
the inability to properly match against a mixed sequence with variadics.
The argument pack must always be the last element, which precludes to match
the last or even the penultimate element (which we need here).
After some tinkering, I found a way to recast this as ''rebinding to a remoulded sequence'',
and could package a multitude of related tools into a single helper-template,
which works without any further library dependencies.
🠲 extract into a separate header (`variadic-rebind.hpp`) for ease of use.
So this turned out to be much more challenging than expected,
due to the fact that, with this design, typing information is
only available at compile-time. The key trick was to use a
''double-dispatch'' based on a generic lambda. In the end,
this could be rounded out to be self-contained library helper,
which is even fully copyable and assignable and properly
invokes all payload constructors and destructors.
The flip side is that such a design is obviously very flexible
and direct regarding the parser model-bindings, and it should
be fairly well optimisable, since the structure is entirely
static and without any virtual dispatch.
Proper handling of payload lifecycle was verified using
a tracking test object with checksum.
* the implementation of this ''Sum Type'' got quite technical and complicated;
thus better to be extracted as separate library component
* use this as base for the `AltModel`
* make a usage sketch, invoking only the model interactions required
After exploring the »nested decorator-chain« implementation variant,
I decided to stick to the solution with the λ-vistor, while attempting
to level and smooth-out the design.
* allow to engage ''any'' «slot» at construction
* reverse the order of type parameters to be ascending (as in `std::tuple`)
* make it fully copyable, movable, assignable and provide a `swap()`,
relying on a variant of the "copy-and-swap"-idiom
* add an ''cross-constructor'' for an extended branch set
To represent the result-model for syntax alternatives,
we need a C++ representation for a ''sum type,'' i.e.
a type that can be one from a fixed set of alternatives.
Obviously the implementation will rely on some kind of Union,
or otherwise employ an opaque buffer and perform a forced cast.
Moreover, to be actually usable, a branch-selector-ID must be
captured and stored alongside, so that code processing the results
can detect which branch of the syntax was chosen.
There seem to be several possible avenues to build and structure
an actual class template to provide this implementation model
* a nested decorator-chain
* using a recursive selector-function with a generic-λ
''all these look quite unattractive, unfortunately....''
Seems like a pragmatic choice, which simplifies most syntax definitions significantly.
In exceptional cases, it is still possible to enforce a situation with `\b` or `\B`
* need to pass the parse end-point in the Eval-Result to allow composed models
* this also prepares for support of generic model-binding-λ
With the help of the model-joining case definitions it is then possible to handle sequence extension.
Deliberately I do not engage into fine grained signature checking, since this would lead to very technical code and moreover this is an implementation feature and we control all invocations (with signatures guaranteed to be correct)
Unfortunately, there are some common syntactic structures, which can not easily be dissected by regular expressions alone, since they entail nested subexpressions. While it is possible to get beyond those fundamental limitations with some trickery, doing so remains precisely that, ''trickery.''
After fighting some inner conflicts, since ''I do know how to write a parser'' —
in the end I have brought myself to just do it.
And indeed, as you'd might expect, I have looked into existing library solutions,
and I would not like to have any one of them as part of the project.
* I do not want a ''parser engine'' or ''parser generator''
* I want the directness of recursive-descent, but combined with Regular Expressions as terminal
* I want to see the structure of the used grammar at the definition site of the custom parser function
* I want deep integration of ''model bindings'' into the parse process, i.e. binding-λ
* I do not want to write model-dissecting or pattern-matching code after the parse
* I do not want to expose ''Monads'' as an interface, since they tend to spread unhealthy structure to surrounding code
* I do not want to leak technicalities of the parse mechanics into the using code
* I do not want to impose hard to remember specific conventions onto the user
Thus I've set the following aims:
* The usage should require only a single header include (ideally header-only)
* The entrance point should be a small number of DSL-starter functions
* The parser shall be implemented by recursive-descent, using the parser-combinator technique
* But I want that wrapped into a DSL, to be able to control what is (not) provided or exposed.
* I want a stateful, applicative logic, since parsing, by its very nature, is stateful!
* I want complete compile-time typing, visible to the optimiser, without a virtual »Parser« interface
And last but not least, ''I do not want to create a ticket, since I do not know if those goals can be achieved...''
Building a correct processing-identification is a complex and challenging task; only some aspects can be targeted and implemented right now, as part of the »Playback Vertical Slice«
* components of the ProcID
* parsing the argument-spec
* dispatch of detail information function to retrieve source ports
The choice to rely on strictly typed functor bindings for the Node operation
bears the danger to produce ''template bloat'' — it would be dangerous to add
further functions to the Port-API naïvely; espeically simple information functions
will likely not depend on the full type information.
A remedy to explore would be to exploit properties marked into the Port's `ProcID`
as key for a dispatcher hashtable; assuming that the `NodeBuilder` will be responsible
for registering the corresponding implementation functions, such a solution could even
be somewhat type-safe, as long as the semantics of the ProcID are maintained correctly.
* this changeset builds a complex processing network for the first time
* furthermore, some ideas towards verification are spelled out
''verification not implemented''
...which aims at building up increasingly more complex Node Graphs,
to validate that all clauses are defined and connected properly.
Reconsidering the testing plan: initially especially this test was aimed
primarily at driving me through the construction of the Node builder and
connection scheme. Surprisingly enough, already the first test case basically
forced the complete construction, by setting me on tangential routes,
notably the **parameter handling**.
Now I'm returning to this test plan with an already finished construction,
and thus it can be straightened just to give enough coverage to validate
the correctness of this construction...
The namespace `steam::engine::test::ont` will hold some typical definitions
for the fake „media processing library“ — to be used for validating aspects of mapping and binding.
This picks up the efforts towards a »Test Ontology« from end November:
d80966c1f
The `TestRandOntology` is intended as a playground to gradually find out
how to maintain bindings processing functionality provided by a specific Library
and thus related to a ''Domain Ontology''
Remark: generating symbolic specs might seem like a mere test exercise, yet is in fact
quite crucial, since the node-identity is based on such a spec, which must be ''semantically correct,''
otherwise caching and especially cache invalidation will be broken.
Yesss .... in Lumiera naming and cache invalidation are linked directly ;-)
This is a high-level integration test to sum up this development effort
* an advanced refactoring was carried out to introduce a
flexible and fully-typed binding for the ''processing-functor''
* this entailed a complete rework of the `FeedManifold` to integrate
inline storage for a ''parameter tuple'' and input / output ''buffer tuples''
* optional ''parameter functors'' were included into the design at a deep level,
closely related to the binding of the processing-functor
* the chosen design is thus a compromise between ''everything nodes''
and a ''dedicated parameter-handling'' at invocation level
As a proof-of-concept, an scheme to handle extended parameters was devised,
using a special »Param Agent Node« and extension storage blocks in stack memory.
While not immediately necessary, this design exercise proves the overall design
is flexible enough to accommodate future extended needs.
Actually this is now quite easy to implement, as a shortcut on top of generic functionality;
just in this case the param-functor takes a Time value as argument.
So its more a matter of documentation to provide a dedicated hook for this common case.
incidentally, this is also the first test case ever to involve linked nodes,
so it revealed several bugs in the related code, which was not yet tested.
This is a ''move-builder'' and thus represents a tricky and sometimes dangerous setup,
while allowing to switch the type context in the middle of the build process.
It is essential to return a RValue-Reference from all builder calls which
stay on the same builder context.
After fixing those minor (and potentially dangerous) aspects regarding move-references,
the code built yesterday worked as expected!
This is some quite technical and redundant code, which largely maps
the configured elements from the Builder-DSL level down into the delegate
builder functors. For the ''Param Agent Node,'' most of the structure
is already embedded deep into the `ParamWeavingPattern`, by virtue of a
tuple of parameter-functors, which are supplied to the builder-API
as a `ParamBuildSpec` (which in fact is in itself a builder and will be
used on a higher level to fill in suitable parameter-functors)
This changeset is assumed to complete the definition of a builder and
weaving pattern for a ''Param Agent Scheme'' — yet only the tests to be
elaborated next will show the extent to which this is true....
unfortunately the "mechanics" of this builder setup are quite convoluted,
due to constrains with the memory manager, which basically force us to
collect a set of ''builder-λ'', together with summing up all the required storage,
so that the actual allocation of all Ports can be done into one contiguous block
of memory, to be connected to the actual Node.
For the regular `PortBuilder`, we use a helper subclass, the `WeavingBuilder`,
to construct this builderλ. But here, for the setup of an ''Param Agent Node,''
the actual wiring is much simpler and it is not justified to use a delegate builder;
rather we perfrom the complete setup directly in the terminal sub-builder operation,
prior to returning up to the NodeBuilder, which controls the overall build.
Still having some doubts if using a ''weaving-pattern'' is the right approach here,
but if we do, then the steps would be mapped as drafted here. This includes
passing additional parameters, notably the `TurnoutSystem&` to every step.
As it turns out, we need to embed the Param-Functor tuple,
but only for a single use from a »builder« component.
On the other hand, the nested »Slot« classes are deemed dangerous,
since they just seem to invite being bound into some functor, which
would create a dangling reference once the `ParamBuildSpec` is gone.
Thus it's better to do away with this reference and make those accessors
basically static, because this way they ''can'' be embedded into param-access
functors (and I'd expect precisely that to happen in real use)
...intended to be used as a Turnout for a ''Param Agent Node....''
This leads to several problems, since the ''chain-data-block'' was defined to be non-copyable,
which as such is a good idea, since it will be accessed by a force-cast through the TurnoutSystem.
So the question is how to group and arrange the various steps into the general scheme of a Weaving-Pattern...
