Meanwhile, some kind of style scheme has emerged for the DSL:
We're working much with postfix-decorating operators, which
augment or extend the ''whole syntax clauses defined thus far''
In accordance with this scheme, I decided also to treat repeated expression
as a postfix operator (other than initially planned). This means, the actual
body to be repeated is ''the syntax clause defined thus far'', and the
repeat()-operator only details the number of repetitions and an optional delimiter.
After all the preparation, now this panes out quite well:
* use a simple 3-way branch structure
* the model type was already pre-selected by the `_Join` Model selector
* can just pass the result-model elements to a constructor/builder
* incremental extension can be directly mapped to the predecessor model
This is a rather obnoxious limitation of C++ variadics:
the inability to properly match against a mixed sequence with variadics.
The argument pack must always be the last element, which precludes to match
the last or even the penultimate element (which we need here).
After some tinkering, I found a way to recast this as ''rebinding to a remoulded sequence'',
and could package a multitude of related tools into a single helper-template,
which works without any further library dependencies.
🠲 extract into a separate header (`variadic-rebind.hpp`) for ease of use.
So this turned out to be much more challenging than expected,
due to the fact that, with this design, typing information is
only available at compile-time. The key trick was to use a
''double-dispatch'' based on a generic lambda. In the end,
this could be rounded out to be self-contained library helper,
which is even fully copyable and assignable and properly
invokes all payload constructors and destructors.
The flip side is that such a design is obviously very flexible
and direct regarding the parser model-bindings, and it should
be fairly well optimisable, since the structure is entirely
static and without any virtual dispatch.
Proper handling of payload lifecycle was verified using
a tracking test object with checksum.
* the implementation of this ''Sum Type'' got quite technical and complicated;
thus better to be extracted as separate library component
* use this as base for the `AltModel`
* make a usage sketch, invoking only the model interactions required
After exploring the »nested decorator-chain« implementation variant,
I decided to stick to the solution with the λ-vistor, while attempting
to level and smooth-out the design.
* allow to engage ''any'' «slot» at construction
* reverse the order of type parameters to be ascending (as in `std::tuple`)
* make it fully copyable, movable, assignable and provide a `swap()`,
relying on a variant of the "copy-and-swap"-idiom
* add an ''cross-constructor'' for an extended branch set
To represent the result-model for syntax alternatives,
we need a C++ representation for a ''sum type,'' i.e.
a type that can be one from a fixed set of alternatives.
Obviously the implementation will rely on some kind of Union,
or otherwise employ an opaque buffer and perform a forced cast.
Moreover, to be actually usable, a branch-selector-ID must be
captured and stored alongside, so that code processing the results
can detect which branch of the syntax was chosen.
There seem to be several possible avenues to build and structure
an actual class template to provide this implementation model
* a nested decorator-chain
* using a recursive selector-function with a generic-λ
''all these look quite unattractive, unfortunately....''
Seems like a pragmatic choice, which simplifies most syntax definitions significantly.
In exceptional cases, it is still possible to enforce a situation with `\b` or `\B`
* need to pass the parse end-point in the Eval-Result to allow composed models
* this also prepares for support of generic model-binding-λ
With the help of the model-joining case definitions it is then possible to handle sequence extension.
Deliberately I do not engage into fine grained signature checking, since this would lead to very technical code and moreover this is an implementation feature and we control all invocations (with signatures guaranteed to be correct)
Unfortunately, there are some common syntactic structures, which can not easily be dissected by regular expressions alone, since they entail nested subexpressions. While it is possible to get beyond those fundamental limitations with some trickery, doing so remains precisely that, ''trickery.''
After fighting some inner conflicts, since ''I do know how to write a parser'' —
in the end I have brought myself to just do it.
And indeed, as you'd might expect, I have looked into existing library solutions,
and I would not like to have any one of them as part of the project.
* I do not want a ''parser engine'' or ''parser generator''
* I want the directness of recursive-descent, but combined with Regular Expressions as terminal
* I want to see the structure of the used grammar at the definition site of the custom parser function
* I want deep integration of ''model bindings'' into the parse process, i.e. binding-λ
* I do not want to write model-dissecting or pattern-matching code after the parse
* I do not want to expose ''Monads'' as an interface, since they tend to spread unhealthy structure to surrounding code
* I do not want to leak technicalities of the parse mechanics into the using code
* I do not want to impose hard to remember specific conventions onto the user
Thus I've set the following aims:
* The usage should require only a single header include (ideally header-only)
* The entrance point should be a small number of DSL-starter functions
* The parser shall be implemented by recursive-descent, using the parser-combinator technique
* But I want that wrapped into a DSL, to be able to control what is (not) provided or exposed.
* I want a stateful, applicative logic, since parsing, by its very nature, is stateful!
* I want complete compile-time typing, visible to the optimiser, without a virtual »Parser« interface
And last but not least, ''I do not want to create a ticket, since I do not know if those goals can be achieved...''
Building a correct processing-identification is a complex and challenging task; only some aspects can be targeted and implemented right now, as part of the »Playback Vertical Slice«
* components of the ProcID
* parsing the argument-spec
* dispatch of detail information function to retrieve source ports
The choice to rely on strictly typed functor bindings for the Node operation
bears the danger to produce ''template bloat'' — it would be dangerous to add
further functions to the Port-API naïvely; espeically simple information functions
will likely not depend on the full type information.
A remedy to explore would be to exploit properties marked into the Port's `ProcID`
as key for a dispatcher hashtable; assuming that the `NodeBuilder` will be responsible
for registering the corresponding implementation functions, such a solution could even
be somewhat type-safe, as long as the semantics of the ProcID are maintained correctly.
* this changeset builds a complex processing network for the first time
* furthermore, some ideas towards verification are spelled out
''verification not implemented''
...which aims at building up increasingly more complex Node Graphs,
to validate that all clauses are defined and connected properly.
Reconsidering the testing plan: initially especially this test was aimed
primarily at driving me through the construction of the Node builder and
connection scheme. Surprisingly enough, already the first test case basically
forced the complete construction, by setting me on tangential routes,
notably the **parameter handling**.
Now I'm returning to this test plan with an already finished construction,
and thus it can be straightened just to give enough coverage to validate
the correctness of this construction...
The namespace `steam::engine::test::ont` will hold some typical definitions
for the fake „media processing library“ — to be used for validating aspects of mapping and binding.
This picks up the efforts towards a »Test Ontology« from end November:
d80966c1f
The `TestRandOntology` is intended as a playground to gradually find out
how to maintain bindings processing functionality provided by a specific Library
and thus related to a ''Domain Ontology''
Remark: generating symbolic specs might seem like a mere test exercise, yet is in fact
quite crucial, since the node-identity is based on such a spec, which must be ''semantically correct,''
otherwise caching and especially cache invalidation will be broken.
Yesss .... in Lumiera naming and cache invalidation are linked directly ;-)
This is a high-level integration test to sum up this development effort
* an advanced refactoring was carried out to introduce a
flexible and fully-typed binding for the ''processing-functor''
* this entailed a complete rework of the `FeedManifold` to integrate
inline storage for a ''parameter tuple'' and input / output ''buffer tuples''
* optional ''parameter functors'' were included into the design at a deep level,
closely related to the binding of the processing-functor
* the chosen design is thus a compromise between ''everything nodes''
and a ''dedicated parameter-handling'' at invocation level
As a proof-of-concept, an scheme to handle extended parameters was devised,
using a special »Param Agent Node« and extension storage blocks in stack memory.
While not immediately necessary, this design exercise proves the overall design
is flexible enough to accommodate future extended needs.
Actually this is now quite easy to implement, as a shortcut on top of generic functionality;
just in this case the param-functor takes a Time value as argument.
So its more a matter of documentation to provide a dedicated hook for this common case.
incidentally, this is also the first test case ever to involve linked nodes,
so it revealed several bugs in the related code, which was not yet tested.
This is a ''move-builder'' and thus represents a tricky and sometimes dangerous setup,
while allowing to switch the type context in the middle of the build process.
It is essential to return a RValue-Reference from all builder calls which
stay on the same builder context.
After fixing those minor (and potentially dangerous) aspects regarding move-references,
the code built yesterday worked as expected!
This is some quite technical and redundant code, which largely maps
the configured elements from the Builder-DSL level down into the delegate
builder functors. For the ''Param Agent Node,'' most of the structure
is already embedded deep into the `ParamWeavingPattern`, by virtue of a
tuple of parameter-functors, which are supplied to the builder-API
as a `ParamBuildSpec` (which in fact is in itself a builder and will be
used on a higher level to fill in suitable parameter-functors)
This changeset is assumed to complete the definition of a builder and
weaving pattern for a ''Param Agent Scheme'' — yet only the tests to be
elaborated next will show the extent to which this is true....
unfortunately the "mechanics" of this builder setup are quite convoluted,
due to constrains with the memory manager, which basically force us to
collect a set of ''builder-λ'', together with summing up all the required storage,
so that the actual allocation of all Ports can be done into one contiguous block
of memory, to be connected to the actual Node.
For the regular `PortBuilder`, we use a helper subclass, the `WeavingBuilder`,
to construct this builderλ. But here, for the setup of an ''Param Agent Node,''
the actual wiring is much simpler and it is not justified to use a delegate builder;
rather we perfrom the complete setup directly in the terminal sub-builder operation,
prior to returning up to the NodeBuilder, which controls the overall build.
Still having some doubts if using a ''weaving-pattern'' is the right approach here,
but if we do, then the steps would be mapped as drafted here. This includes
passing additional parameters, notably the `TurnoutSystem&` to every step.
As it turns out, we need to embed the Param-Functor tuple,
but only for a single use from a »builder« component.
On the other hand, the nested »Slot« classes are deemed dangerous,
since they just seem to invite being bound into some functor, which
would create a dangling reference once the `ParamBuildSpec` is gone.
Thus it's better to do away with this reference and make those accessors
basically static, because this way they ''can'' be embedded into param-access
functors (and I'd expect precisely that to happen in real use)
...intended to be used as a Turnout for a ''Param Agent Node....''
This leads to several problems, since the ''chain-data-block'' was defined to be non-copyable,
which as such is a good idea, since it will be accessed by a force-cast through the TurnoutSystem.
So the question is how to group and arrange the various steps into the general scheme of a Weaving-Pattern...
In `NodeFeed_test`...
Demonstrate the base mechanism of creating a ''Param Spec'' with a
functor-definition for each parameter. This can then later be used to
invoke those functors and materialise the results into a data tuple,
and this data tuple can be linked into the TurnoutSystem, so that
the parameter values can be accessed type-safe with getter-functors.
Relying basically on the trick to invoke std::apply with a generic variadic Lambda
onto the tuple of functors; within the lambda we can use variadic expansion
to pass the results directly into the builder and so construct the param-tuple in-place.
Oh well.
2024 is almost gone by now.
Had to endure yet another performance of Beethoven's 9th symphony...
This is rather the easy part, building upon the foundation developed with `HeteroData`:
* the `TurnoutSystem` will now accept a `HeteroData`-Accessor
* the `ParamBuldSpec` can thus construct an Accessor-Type for each »slot«
...the more tricky part will be how actually to build, populate and attach
such an extension data slot, placed into the local stack frame...
...which in turn would then allow
* to refer to extended parameters within scope
* to build a Param(Agent)Node, which builds a parameter tuple
by invoking the given parameter-functors
Can now demonstrate in the test
* define several »slots«, each with either value or functor
* apply these functors to a `TurnoutSystem`
So this is a design sketch how a `ParamBuildSpec` descriptor could be created,
which in turn would provide the foundation to implement a ''Parameter Weaving Pattern...''
__Note__: since this is an extension for advanced usage, yet relies on a storage layout
defined to allow for extensions like this use case here, the anchor type is now defined
to reside in the `TurnoutSystem` in the form of a ''standard parameter block''.
Those standard invocation parameters are fixed and thus can be hard coded.
Based on ''theoretical reasoning,'' I draw the conclusion that some advanced usages
of processing parameters can not be satisfied by the simple direct integration of a
parameter-functor...
Thus the concept for an extension point, which relies on a dedicated ''Param (Agent) Node''
and a specifically tailored ''Param Weaving Pattern'' to evaluate several parameter functors
and place the results into an extension data block in the invocation stack frame.
* ...by defining a parameter-functor to »drop off« a given value
* ...also add a static sanity check to reject unsuitable parameter-functor \\
(e.g. for a processing-functor with different or even no parameters)
This required some ''type massaging'' to construct the proper follow-up builder type;
other than that, all components work together as expected.
This can be demonstrated both in a direct setup and using the builder.
While the handling of invocation parameters is now integrated in the node processing,
there is still a gap to close in the Node Builder, which is tricky due to the way
the parameter-functor is now integrated deeply into the setup of the `FeedManifold`;
so the `PortBuilder` is tasked now with implanting a `FeedPrototype` -- which must be
adapted to a ''specific parameter-functor,'' which is only supplied optionally,
as a further build step.
At first this seemed to present a pattern very similar to a ''State Monad'' — and thus
I investigated if encapsulating the build of the prototype into such a State Monad would
simplify the structure of the builder — yet once again, Monads turned out as ''Anti Pattern''
rather: we'd had to ad an extra component, which is superficially generic
but without any tangible relation to patterns of the real world, it would be
rather technical (using lots of composed lambda primitives, which will be condensed
into a single builder function by the compiler / optimiser. But worse still,
this highly complicated setup does not actually solve the problem with N x M
typed contexts — implying that it ''is not actually an abstraction,'' rather just
pretends to be generic.
The benefit of this lengthy design exercise is to understand better the situation
in the builder, which amounts to building up mixed typed context with several
degrees of freedom. It is better to accept this reality and keep it in plain sight,
i.e. let the builder be explicitly typed from end to end and do not try
to package parts of this selection process behind a virtualisation.
**This is a Milestone for the Render Engine integration effort**
After various rounds of prototyping and refactoring,
the Render Node builder and invocation code is now able to
* bind a simple function
* handle arbitrary input / output and parameter types
* invoke a Render Node configured with this function
The ''design exercise'' started yesterday ran into a total rodadblock.
And this is a good thing, as this unveils inconsistencies in our memory handling protocols
* Buffer Provider Protocol
* Output Slot Protocol
The latter exposes a `BuffHandle`, which should be usable from within the Render Node code
like any other regular buffer handle — which especially would require to ''delegate the lifecycle calls...''
So while this topic does not hinder us right now to proceed with a Node invocation in test setup,
it must be addressed before we're able to deliver data into an actual OutputSlot.
Created #1387 to track this topic...
This investigation started out as solving an already solved problem...
I'll continue this as a design exercise non the less.
__Some explanation__: To achieve the goal of invoking a Node end-to-end,
the gap between the `Port` API, the `ProcNode` API and the `RenderInvocation` must be closed.
This leads to questions of API design: ''what core operation should the `ProcNode` API expose?''
* is `ProcNode` just a forwarding / delegating container and becoming redundant?
* or does the API rather move in the direction of an ''Exit Node''?
This leads to the question how the opened `OutputSlot` can be exposed as a `BuffHandle`
to allow to set off the recursive Node invocation. As it turns out, the onerous for this step
lies on the actual `OutputSlot` implementation, since the API and output protocol already requires
to expose a `BuffHandle`. Yet there is no "real" implementation available, just a Mock setup based
on `DiagnosticBufferProvider`, which obviously can just be passed-through.
Which leaves me with mixed feelings. For one it is conveninent to skip this topic for now,
but on the other hand the design of `BufferProvider` does not seem well suited for such an proxying task.
Thus I decided to explore this aspect in the form of a prototyping test....
After this extended excursion to lift the internals of Node invocation
to the use of structured and typed data (notably the invocation parameters),
the »Playback Vertical Slice« continues to push ahead towards the goal of integration.
The existing code has been re-oriented and some aspects of node invocation have been reworked
in a prototyping effort, which (in part though the aforementioned rework)
is meanwhile on a good path to lead to a consolidated final version.
* ✔ building a simple Render Node works now with the revamped code
* 🔁invoking this simple Node ''should be just one step away'' (since all parts are known to work)
* ⌛ the next step would then be to build a Node outfitted with a ''Parameter Functor'', which is the new concept introduced by recent changes
* ⌛ this should then get us at the point to take the hurdle of invoking one of our **Random Test** functions as a Render Node
Remove left-overs from the preceding prototypical implementation,
which is now obliterated by the change to a flexibly configured `FeedManifold`
with structured, typed storage for buffers and for parameter data.
The Render Node invocation sequence, as rearranged and reworked for the »Playback Vertical Slice«, now seems reasonably clear and settled.
Adding extensive documentation to describe the conventions and structures worked out thus far;
moreover, start makeover of old documentation in the !TiddlyWiki to remove concepts obviously obsoleted now...
After the complete makeover of the `FeedManifold` structure,
which among other entails a switch from ''buffer arrays'' to tuples
and the ''introduction of a parameter tuple'', this changeset now
switches the „downstream code“ of the builder and node invocation,
relying on an largely identical invocation API.
The partially finished NodeLink_test now **runs as before**
but on top of a drastically more flexible and open infrastructure.
Quite a feat.
This completes a deep and very challenging series of refactorings
with the goal to introduce support for **Parameters** into the Render invocation code.
A secondary goal was to re-assess the prototype code written thus far
and thereby to establish a standard processing scheme.
With these rearrangements, the `FeedManifold` is poised to act as **central link**
between the Render-Node invocation code and the actual Media-Processing code in a Library Plug-in
Up to this point, the existing code from the Prototype is still compilable, yet broken.
The __next step__ will be to harness the possible simplifications and enable
the actual invocation to work on arbitrary combinations of buffers and parameters,
enabled by the **compile-time use-case classification** now provided by `FeedManifold`
While basically the `FeedPrototype` could be created directly,
passing both the processing- and the parameter-functor, in practice
a two-step configuration can be expected, since the processing-functor
is built by the Library-Plug-in, while the parameter-functor is then
later added as decoration by the builder.
Thus we need the ability to ''collect configuration'' within the Level-2 builder,
which can be achieved by a ''cross-builder'' mechanic, where we create an adapted builder
from the augmented configuration. A similar approach is also used to add
the configuration of the custom allocator.
Added an extensive demo in the test, playing with several instances
to highlight the point where the parameter-functor is actually invoked.
Some further tweaks to the logic to allow using the `FeedPrototype` in the default setup,
where ''nothing shall be done with parameters...''
Provide the basic constructors and a type constructor in FeedManifold,
so that it is possible to install a ''processing functor'' into the prototype
and then drop off a copy into each new `FeedManifold`
With this additions, can now **demonstrate simple usage**
__Remark__: using the `DiagnosticBufferProvider` developed several years ago;
Seems to work well; however, when creating a new instance in the next test case,
we get a hard failure when the previous test case did not discard all buffers.
Not sure what to think about that
* for one, it is good to get an alarm, since actually there should not be any leak
* but on the other hand, `reset()` does imply IMHO „I want a clean slate“
Adding some code thus to clean out memory blocks marked as used.
When a test wants to check that all memory was released, there are tools to do so.
Based on the usage concept developed thus far, we rely on a `FeedPrototype`
to generate the actual `FeedManifold` for each invocation — and this is the extension point
where a ''parameter functor'' can be attached.
Notably, such a parameter functor will be configured from a different part of the builder logic
than the underlying processing function, which is adapted by a Library Plug-in.
Parameters on the other hand will be controlled mostly by configuration within the
Session, because the user chooses to use specific settings, e.g. for an effect.
An important extension to this scheme is **Parameter Automation** — which will be
also attached over the extension point designed here.
Since Parameter can be defined in various flavours, there is some concern that we'll end up
with an excessive number of template instantiations. Thus, we'll explicitly create a »loop hole«
by allowing to define the ''parameter functor'' to be a `std::function`.
This would open a secondary possibility: configuring such a function, but leaving it empty,
which would be a further control switch usable by the builder.
This basically completes the reworked implementation of the `FeedManifold`
An important aspect however is now separated out and still remains to be solved:
''how to configure and invoke a Parameter-Functor?''
This is one remaining tricky detail to be solved.
The underlying difficulty is architectural:
- the processing functor will be supplied by the Media-Lib-Plug-in
- while a functor to set parameters and automation will be added from another context
Yet both have to work together, and both together will determine the effective type of the ''Weaving Pattern''
Thus we'll have to get both functors somehow integrated into the Level-2-Builder,
yet we must be able first to pass this builder instance to the Library-Plug-in and then,
in a second step, another part of the Lumiera Builder logic will have to add the Parameter wiring.
The solution I'm proposing is to exploit the observation that in fact the processing functor
is stored as a kind of »Prototype« within the ''Weaving Pattern'' and will be ''copied'' from there
for each individual Render Node invocation. The reasons for this is, we want the optimiser
to see the full instantiation of the library function and thus get maximum leverage;
thus the code doing the actual call must see the functor or lambda to be able to inline it.
This leads to the idea to ''separate'' this »prototype« from the `FeedManifold`;
the latter thereby becomes mostly agnostic of parameter processing.
However, `FeedManifold` must then accept a copy of the parameter values
as constructor argument and pass it into its internal storage.
This forces yet another reorganisation of the class structure.
Basically the storage modules for `FeedManifold` are now prepared within a configuratiton class,
which actually helps to simplify the metaprogramming definitions and keeps the enclosing namespace clean.
