Meanwhile, some kind of style scheme has emerged for the DSL:
We're working much with postfix-decorating operators, which
augment or extend the ''whole syntax clauses defined thus far''
In accordance with this scheme, I decided also to treat repeated expression
as a postfix operator (other than initially planned). This means, the actual
body to be repeated is ''the syntax clause defined thus far'', and the
repeat()-operator only details the number of repetitions and an optional delimiter.
After all the preparation, now this panes out quite well:
* use a simple 3-way branch structure
* the model type was already pre-selected by the `_Join` Model selector
* can just pass the result-model elements to a constructor/builder
* incremental extension can be directly mapped to the predecessor model
This is a rather obnoxious limitation of C++ variadics:
the inability to properly match against a mixed sequence with variadics.
The argument pack must always be the last element, which precludes to match
the last or even the penultimate element (which we need here).
After some tinkering, I found a way to recast this as ''rebinding to a remoulded sequence'',
and could package a multitude of related tools into a single helper-template,
which works without any further library dependencies.
🠲 extract into a separate header (`variadic-rebind.hpp`) for ease of use.
So this turned out to be much more challenging than expected,
due to the fact that, with this design, typing information is
only available at compile-time. The key trick was to use a
''double-dispatch'' based on a generic lambda. In the end,
this could be rounded out to be self-contained library helper,
which is even fully copyable and assignable and properly
invokes all payload constructors and destructors.
The flip side is that such a design is obviously very flexible
and direct regarding the parser model-bindings, and it should
be fairly well optimisable, since the structure is entirely
static and without any virtual dispatch.
Proper handling of payload lifecycle was verified using
a tracking test object with checksum.
* the implementation of this ''Sum Type'' got quite technical and complicated;
thus better to be extracted as separate library component
* use this as base for the `AltModel`
* make a usage sketch, invoking only the model interactions required
After exploring the »nested decorator-chain« implementation variant,
I decided to stick to the solution with the λ-vistor, while attempting
to level and smooth-out the design.
* allow to engage ''any'' «slot» at construction
* reverse the order of type parameters to be ascending (as in `std::tuple`)
* make it fully copyable, movable, assignable and provide a `swap()`,
relying on a variant of the "copy-and-swap"-idiom
* add an ''cross-constructor'' for an extended branch set
To represent the result-model for syntax alternatives,
we need a C++ representation for a ''sum type,'' i.e.
a type that can be one from a fixed set of alternatives.
Obviously the implementation will rely on some kind of Union,
or otherwise employ an opaque buffer and perform a forced cast.
Moreover, to be actually usable, a branch-selector-ID must be
captured and stored alongside, so that code processing the results
can detect which branch of the syntax was chosen.
There seem to be several possible avenues to build and structure
an actual class template to provide this implementation model
* a nested decorator-chain
* using a recursive selector-function with a generic-λ
''all these look quite unattractive, unfortunately....''
Seems like a pragmatic choice, which simplifies most syntax definitions significantly.
In exceptional cases, it is still possible to enforce a situation with `\b` or `\B`
* need to pass the parse end-point in the Eval-Result to allow composed models
* this also prepares for support of generic model-binding-λ
With the help of the model-joining case definitions it is then possible to handle sequence extension.
Deliberately I do not engage into fine grained signature checking, since this would lead to very technical code and moreover this is an implementation feature and we control all invocations (with signatures guaranteed to be correct)
Unfortunately, there are some common syntactic structures, which can not easily be dissected by regular expressions alone, since they entail nested subexpressions. While it is possible to get beyond those fundamental limitations with some trickery, doing so remains precisely that, ''trickery.''
After fighting some inner conflicts, since ''I do know how to write a parser'' —
in the end I have brought myself to just do it.
And indeed, as you'd might expect, I have looked into existing library solutions,
and I would not like to have any one of them as part of the project.
* I do not want a ''parser engine'' or ''parser generator''
* I want the directness of recursive-descent, but combined with Regular Expressions as terminal
* I want to see the structure of the used grammar at the definition site of the custom parser function
* I want deep integration of ''model bindings'' into the parse process, i.e. binding-λ
* I do not want to write model-dissecting or pattern-matching code after the parse
* I do not want to expose ''Monads'' as an interface, since they tend to spread unhealthy structure to surrounding code
* I do not want to leak technicalities of the parse mechanics into the using code
* I do not want to impose hard to remember specific conventions onto the user
Thus I've set the following aims:
* The usage should require only a single header include (ideally header-only)
* The entrance point should be a small number of DSL-starter functions
* The parser shall be implemented by recursive-descent, using the parser-combinator technique
* But I want that wrapped into a DSL, to be able to control what is (not) provided or exposed.
* I want a stateful, applicative logic, since parsing, by its very nature, is stateful!
* I want complete compile-time typing, visible to the optimiser, without a virtual »Parser« interface
And last but not least, ''I do not want to create a ticket, since I do not know if those goals can be achieved...''
Building a correct processing-identification is a complex and challenging task; only some aspects can be targeted and implemented right now, as part of the »Playback Vertical Slice«
* components of the ProcID
* parsing the argument-spec
* dispatch of detail information function to retrieve source ports
The choice to rely on strictly typed functor bindings for the Node operation
bears the danger to produce ''template bloat'' — it would be dangerous to add
further functions to the Port-API naïvely; espeically simple information functions
will likely not depend on the full type information.
A remedy to explore would be to exploit properties marked into the Port's `ProcID`
as key for a dispatcher hashtable; assuming that the `NodeBuilder` will be responsible
for registering the corresponding implementation functions, such a solution could even
be somewhat type-safe, as long as the semantics of the ProcID are maintained correctly.
* this changeset builds a complex processing network for the first time
* furthermore, some ideas towards verification are spelled out
''verification not implemented''
...which aims at building up increasingly more complex Node Graphs,
to validate that all clauses are defined and connected properly.
Reconsidering the testing plan: initially especially this test was aimed
primarily at driving me through the construction of the Node builder and
connection scheme. Surprisingly enough, already the first test case basically
forced the complete construction, by setting me on tangential routes,
notably the **parameter handling**.
Now I'm returning to this test plan with an already finished construction,
and thus it can be straightened just to give enough coverage to validate
the correctness of this construction...
The namespace `steam::engine::test::ont` will hold some typical definitions
for the fake „media processing library“ — to be used for validating aspects of mapping and binding.
This picks up the efforts towards a »Test Ontology« from end November:
d80966c1f
The `TestRandOntology` is intended as a playground to gradually find out
how to maintain bindings processing functionality provided by a specific Library
and thus related to a ''Domain Ontology''
Remark: generating symbolic specs might seem like a mere test exercise, yet is in fact
quite crucial, since the node-identity is based on such a spec, which must be ''semantically correct,''
otherwise caching and especially cache invalidation will be broken.
Yesss .... in Lumiera naming and cache invalidation are linked directly ;-)
This is a high-level integration test to sum up this development effort
* an advanced refactoring was carried out to introduce a
flexible and fully-typed binding for the ''processing-functor''
* this entailed a complete rework of the `FeedManifold` to integrate
inline storage for a ''parameter tuple'' and input / output ''buffer tuples''
* optional ''parameter functors'' were included into the design at a deep level,
closely related to the binding of the processing-functor
* the chosen design is thus a compromise between ''everything nodes''
and a ''dedicated parameter-handling'' at invocation level
As a proof-of-concept, an scheme to handle extended parameters was devised,
using a special »Param Agent Node« and extension storage blocks in stack memory.
While not immediately necessary, this design exercise proves the overall design
is flexible enough to accommodate future extended needs.
Actually this is now quite easy to implement, as a shortcut on top of generic functionality;
just in this case the param-functor takes a Time value as argument.
So its more a matter of documentation to provide a dedicated hook for this common case.
incidentally, this is also the first test case ever to involve linked nodes,
so it revealed several bugs in the related code, which was not yet tested.
This is a ''move-builder'' and thus represents a tricky and sometimes dangerous setup,
while allowing to switch the type context in the middle of the build process.
It is essential to return a RValue-Reference from all builder calls which
stay on the same builder context.
After fixing those minor (and potentially dangerous) aspects regarding move-references,
the code built yesterday worked as expected!
As it turns out, we need to embed the Param-Functor tuple,
but only for a single use from a »builder« component.
On the other hand, the nested »Slot« classes are deemed dangerous,
since they just seem to invite being bound into some functor, which
would create a dangling reference once the `ParamBuildSpec` is gone.
Thus it's better to do away with this reference and make those accessors
basically static, because this way they ''can'' be embedded into param-access
functors (and I'd expect precisely that to happen in real use)
...intended to be used as a Turnout for a ''Param Agent Node....''
This leads to several problems, since the ''chain-data-block'' was defined to be non-copyable,
which as such is a good idea, since it will be accessed by a force-cast through the TurnoutSystem.
So the question is how to group and arrange the various steps into the general scheme of a Weaving-Pattern...
In `NodeFeed_test`...
Demonstrate the base mechanism of creating a ''Param Spec'' with a
functor-definition for each parameter. This can then later be used to
invoke those functors and materialise the results into a data tuple,
and this data tuple can be linked into the TurnoutSystem, so that
the parameter values can be accessed type-safe with getter-functors.
Relying basically on the trick to invoke std::apply with a generic variadic Lambda
onto the tuple of functors; within the lambda we can use variadic expansion
to pass the results directly into the builder and so construct the param-tuple in-place.
Oh well.
2024 is almost gone by now.
Had to endure yet another performance of Beethoven's 9th symphony...
This is rather the easy part, building upon the foundation developed with `HeteroData`:
* the `TurnoutSystem` will now accept a `HeteroData`-Accessor
* the `ParamBuldSpec` can thus construct an Accessor-Type for each »slot«
...the more tricky part will be how actually to build, populate and attach
such an extension data slot, placed into the local stack frame...
...which in turn would then allow
* to refer to extended parameters within scope
* to build a Param(Agent)Node, which builds a parameter tuple
by invoking the given parameter-functors
Can now demonstrate in the test
* define several »slots«, each with either value or functor
* apply these functors to a `TurnoutSystem`
So this is a design sketch how a `ParamBuildSpec` descriptor could be created,
which in turn would provide the foundation to implement a ''Parameter Weaving Pattern...''
__Note__: since this is an extension for advanced usage, yet relies on a storage layout
defined to allow for extensions like this use case here, the anchor type is now defined
to reside in the `TurnoutSystem` in the form of a ''standard parameter block''.
Those standard invocation parameters are fixed and thus can be hard coded.
Based on ''theoretical reasoning,'' I draw the conclusion that some advanced usages
of processing parameters can not be satisfied by the simple direct integration of a
parameter-functor...
Thus the concept for an extension point, which relies on a dedicated ''Param (Agent) Node''
and a specifically tailored ''Param Weaving Pattern'' to evaluate several parameter functors
and place the results into an extension data block in the invocation stack frame.
* ...by defining a parameter-functor to »drop off« a given value
* ...also add a static sanity check to reject unsuitable parameter-functor \\
(e.g. for a processing-functor with different or even no parameters)
This required some ''type massaging'' to construct the proper follow-up builder type;
other than that, all components work together as expected.
This can be demonstrated both in a direct setup and using the builder.
**This is a Milestone for the Render Engine integration effort**
After various rounds of prototyping and refactoring,
the Render Node builder and invocation code is now able to
* bind a simple function
* handle arbitrary input / output and parameter types
* invoke a Render Node configured with this function
The ''design exercise'' started yesterday ran into a total rodadblock.
And this is a good thing, as this unveils inconsistencies in our memory handling protocols
* Buffer Provider Protocol
* Output Slot Protocol
The latter exposes a `BuffHandle`, which should be usable from within the Render Node code
like any other regular buffer handle — which especially would require to ''delegate the lifecycle calls...''
So while this topic does not hinder us right now to proceed with a Node invocation in test setup,
it must be addressed before we're able to deliver data into an actual OutputSlot.
Created #1387 to track this topic...
This investigation started out as solving an already solved problem...
I'll continue this as a design exercise non the less.
__Some explanation__: To achieve the goal of invoking a Node end-to-end,
the gap between the `Port` API, the `ProcNode` API and the `RenderInvocation` must be closed.
This leads to questions of API design: ''what core operation should the `ProcNode` API expose?''
* is `ProcNode` just a forwarding / delegating container and becoming redundant?
* or does the API rather move in the direction of an ''Exit Node''?
This leads to the question how the opened `OutputSlot` can be exposed as a `BuffHandle`
to allow to set off the recursive Node invocation. As it turns out, the onerous for this step
lies on the actual `OutputSlot` implementation, since the API and output protocol already requires
to expose a `BuffHandle`. Yet there is no "real" implementation available, just a Mock setup based
on `DiagnosticBufferProvider`, which obviously can just be passed-through.
Which leaves me with mixed feelings. For one it is conveninent to skip this topic for now,
but on the other hand the design of `BufferProvider` does not seem well suited for such an proxying task.
Thus I decided to explore this aspect in the form of a prototyping test....
After this extended excursion to lift the internals of Node invocation
to the use of structured and typed data (notably the invocation parameters),
the »Playback Vertical Slice« continues to push ahead towards the goal of integration.
The existing code has been re-oriented and some aspects of node invocation have been reworked
in a prototyping effort, which (in part though the aforementioned rework)
is meanwhile on a good path to lead to a consolidated final version.
* ✔ building a simple Render Node works now with the revamped code
* 🔁invoking this simple Node ''should be just one step away'' (since all parts are known to work)
* ⌛ the next step would then be to build a Node outfitted with a ''Parameter Functor'', which is the new concept introduced by recent changes
* ⌛ this should then get us at the point to take the hurdle of invoking one of our **Random Test** functions as a Render Node
After the complete makeover of the `FeedManifold` structure,
which among other entails a switch from ''buffer arrays'' to tuples
and the ''introduction of a parameter tuple'', this changeset now
switches the „downstream code“ of the builder and node invocation,
relying on an largely identical invocation API.
The partially finished NodeLink_test now **runs as before**
but on top of a drastically more flexible and open infrastructure.
Quite a feat.
This completes a deep and very challenging series of refactorings
with the goal to introduce support for **Parameters** into the Render invocation code.
A secondary goal was to re-assess the prototype code written thus far
and thereby to establish a standard processing scheme.
With these rearrangements, the `FeedManifold` is poised to act as **central link**
between the Render-Node invocation code and the actual Media-Processing code in a Library Plug-in
Up to this point, the existing code from the Prototype is still compilable, yet broken.
The __next step__ will be to harness the possible simplifications and enable
the actual invocation to work on arbitrary combinations of buffers and parameters,
enabled by the **compile-time use-case classification** now provided by `FeedManifold`
While basically the `FeedPrototype` could be created directly,
passing both the processing- and the parameter-functor, in practice
a two-step configuration can be expected, since the processing-functor
is built by the Library-Plug-in, while the parameter-functor is then
later added as decoration by the builder.
Thus we need the ability to ''collect configuration'' within the Level-2 builder,
which can be achieved by a ''cross-builder'' mechanic, where we create an adapted builder
from the augmented configuration. A similar approach is also used to add
the configuration of the custom allocator.
Added an extensive demo in the test, playing with several instances
to highlight the point where the parameter-functor is actually invoked.
Some further tweaks to the logic to allow using the `FeedPrototype` in the default setup,
where ''nothing shall be done with parameters...''
Provide the basic constructors and a type constructor in FeedManifold,
so that it is possible to install a ''processing functor'' into the prototype
and then drop off a copy into each new `FeedManifold`
With this additions, can now **demonstrate simple usage**
__Remark__: using the `DiagnosticBufferProvider` developed several years ago;
Seems to work well; however, when creating a new instance in the next test case,
we get a hard failure when the previous test case did not discard all buffers.
Not sure what to think about that
* for one, it is good to get an alarm, since actually there should not be any leak
* but on the other hand, `reset()` does imply IMHO „I want a clean slate“
Adding some code thus to clean out memory blocks marked as used.
When a test wants to check that all memory was released, there are tools to do so.
This basically completes the reworked implementation of the `FeedManifold`
An important aspect however is now separated out and still remains to be solved:
''how to configure and invoke a Parameter-Functor?''
This is one remaining tricky detail to be solved.
The underlying difficulty is architectural:
- the processing functor will be supplied by the Media-Lib-Plug-in
- while a functor to set parameters and automation will be added from another context
Yet both have to work together, and both together will determine the effective type of the ''Weaving Pattern''
Thus we'll have to get both functors somehow integrated into the Level-2-Builder,
yet we must be able first to pass this builder instance to the Library-Plug-in and then,
in a second step, another part of the Lumiera Builder logic will have to add the Parameter wiring.
The solution I'm proposing is to exploit the observation that in fact the processing functor
is stored as a kind of »Prototype« within the ''Weaving Pattern'' and will be ''copied'' from there
for each individual Render Node invocation. The reasons for this is, we want the optimiser
to see the full instantiation of the library function and thus get maximum leverage;
thus the code doing the actual call must see the functor or lambda to be able to inline it.
This leads to the idea to ''separate'' this »prototype« from the `FeedManifold`;
the latter thereby becomes mostly agnostic of parameter processing.
However, `FeedManifold` must then accept a copy of the parameter values
as constructor argument and pass it into its internal storage.
This forces yet another reorganisation of the class structure.
Basically the storage modules for `FeedManifold` are now prepared within a configuratiton class,
which actually helps to simplify the metaprogramming definitions and keeps the enclosing namespace clean.
Can now invoke the FeedManifold with
- either only one output buffer pointer
- or an input and output buffer pointer
With the new support tooling developed yesterday,
the decision logic is now stright-forward to express
__NOTE__ there is a known problem with type-handler registration in the `BufferProvider`;
basically all functors with the same signature are treated as ''identical type'',
which does not account for the fact that functors may hold captured data:
in the example here the second buffer is created with the constructor arguments
given to the first one, ignoring all further sets of similar arguments
Tuples and the ''C++ tuple protocol'' build upon variadic arguments
and are thus rather tedious to handle, especially in this situation here,
where the argument can ''sometimes be a tuple...''
Several years ago I made the observation that processing by explicit ''type sequences''
(Loki-style) is much simpler to handle and easier to lift to a generic level of processing.
Thus I'll attempt now to extract the ''iteration and extraction part'' of the logic into a new helper.
`lib::meta::ElmTypes<TUP>` allows to process all ''tuple-like types'' and generic ''type sequences'' uniformely
and enables to use both styles interchangably (btw, it is quite common to ''abuse'' `std::tuple` as a type sequence).
With this helper, we can now
- build a ''type sequence'' from any ''tuple-like'' object (and vice-versa)
- re-bind (i.e. transfer the template parameters to another template)
- apply some wrapper
- create AND / OR evaluations over the types
This changeset is a sketch how to switch the entire implementation of the ''Invocation Adatper''
over to a generic argument usage scheme. This requires the ability to
- detect if some argument is actually a ''structured type''
- investigate components of such a structured type to draw a distinction between »Buffer« and »Parameter«
- ''lift'' the implementation of simple values to work on tuples
- provide a way to ''bridge'' from ''tuple-style'' programming to ''array access''
As a building block, we use a new iteration-over-index construct,
based on an idea discussed in https://stackoverflow.com/q/53522781/444796
The trick is to pass a `std::integer_constant` to a λ-generic
This solution checks only the minimal precondition,
which is that a type supports `std::tuple_size<T>`.
A more complete implementation turns out to be surprisingly complex,
since a direct check likely requires compile-time reflection capabilities
at the level of at least C++23
- `std::tuple_element<i,T>` typically implements limits checks,
which interfere with the detection of empty structured types
- the situation regarding `std::get<i>()` is even more complicated,
since we might have to probe for ADL-based solutions, or member templates
The check for minimal necessary precondition however allows us to
single out std::tuple, std::array and our own structured types in
compilation branching, which suffices to fulfils actual needs.
This is a possible extension which frequently comes up again during the design of the Engine.
Basically, the `TypeHandler` in the metadata-descriptor used by the `BufferProvder` could capture
additional context-arguments, which are then later passed to an object instance embedded into the buffer.
Yesterday I attempted to use this feature for a simple demonstration in `NodeBasic_test`,
just to find out that passing additional constructor arguments to the capture fails with
a confusing compilation error message. This failure could be traced down to the function binder;
and what at first sight seemed to be a compiler error, turned out to be a quite logical limitation:
When we »close« some objects of the constructor, but delay the construction itself, we'll have to
store a copy in the constructor-λ. And this implies, that we'll have to change the types
used for instantiation of the compiler, so that the construction-function can be invoked
by passing references from the captured copy of the additional arguments.
When naively passing those forwarded arguments into the std::bind()-call,
the resulting functor will fail at instantiation, when the compiler attempts
to generate the function-call `operator()`
see: https://stackoverflow.com/q/30968573/444796
We have now a roughly complete classification of possible use cases.
The invocation can only produce output, process input to output,
and can optionally also accept parameters.
Moreover, each of these cases can require an arbitrary number of actual arguments.
To support all these drastically different case by a common scheme,
`FeedManifold` now uses a »storage slice« for output, input and parameters,
which can be configured at compile time.
TODO: there is an unresolved bug in the test-helper code for the `DiagnosticHeapBlockProvider`,
which prevents us to embed constructor arguments into a buffer descriptor
This is an attempt to rework gradually while keeping the existing code valid.
For the simple reason that the existing code is quite elaborate and difficult to re-orient.
Thus using a ''second branch,'' and sharing the traits template while expanding its capabilities
What I'm about to do amounts to a massive generalisation, which is tricky.
Instead of having a fixed array-style layout, we want to accept arbitrary and mixed arguments.
Notably, we want to give the ''actual Library Plug-in'' a lot of leeway for binding:
- optionally, the library might want to require **Parameters** (which is the reason for this change)
- moreover, accepting input-buffers shall now be optional, since many generation functions do not need them
- and on top of all this, we want to accept an arbitrary mix of types for each kind.
So conceptually we are switching from C-style arrays to tuples with full type safety
''this going to become quite nasty and technical, I'm afraid...''
Starting from a prototypical implementation,
where each »slot« in the function is directly connected to the corresponding lead / port,
the implementation of the `SimpleWeavingPattern` (as it was called previously) could be
augmented and adapted gradually — and seems well suited to cover most standard cases of ''media processing''
So a name change is mandated, and the code is also extracted and relocated, possibly even
to be combined with the code of the `InvocationAdapter`, thereby hopefully making the implementation more accessible
Generally speaking, ''weaving patterns'' take on the role of the prime extension point regarding `Port` implementation.
It seemed like we're doomed...
Yet we barely escaped our horrid fate, because the C++ structured bindings happen to look also for get<i> member functions!
Any other solution involving a free function `get<i>(h)` would not work, since the `std::tuple` used as base class would inevitably drag in std::get via ADL
Obviously, the other remedy would be to turn the `StorageFrame` into a member; yet doing so is not desirable, as makes the actual storage layout more obscure (and also more brittle)
Actually it is the implementation of `std::get` from our STL implementation
which causes the problems; our new custom implementation works as intended an
would also be picked by the compiler's overload resolution. But unfortunately,
the bounds checking assertion built into std::tuple_element<I,T> triggers
immediately when instantiated with out-of-bounds argument, which happens
during the preparation of overload resolution, even while the compiler
would pick another implementation in the following routine.
So we're out of luck and need to find a workaround...
Why is our specialisation of `std::get` not picked up by the compiler?
* it must somehow be related to the fact that `std::tuple` itself is a base class of lib::HeteroData
* if we remove this inheritance relation, our specialisation is used by the compiler and works as intended
* however, this strange behaviour can not be reproduced in a simple synthetic setup
It must be some further subtlety which marks the tuple case as preferrable
Seems like low hanging fruit and would especially allow to use
those storage blocks with ''structural bindings''
Providing the necessary specialisations for `std::get` however turns out to be difficult;
the compiler insists on picking the direct tuple specialisation, since std::tuple is a
protected base class; yet still surprising -- I was under the impression
that the direct overload should be the closest match
This basically solves this implementation challenge:
It was possible to construct a ''compile-time type-safe'' overlay,
while using force-casts ''without any metadata'' at runtime.
Obviously this is a dangerous setup, but ''should be resonably safe'' when used within the defined scheme...
* now yields an instance of the full `HeteroData<X,X,Z>` template
* work around problems with std::tuple_element_t for derived classes
Can now default-create and direct-init a front-End data block,
access and modify its elements — and the API looks ok-ish for me
Decision to use the generic case as short-hand for the first data block,
and thus ''hide the more technical Loki-List specialisations''
With that, I'm finally able to write the first test case...
This was a tough nut to crack, but recalling the actual usage situation was helpful
* the ''constructor type'' must be created / picked-up beforehand
* we are about to build a ''parameter-computation node''
* so this constructor presumably is passed to a type parameter of a specific weaving pattern
* the constructor must be invoked directly to drop-off the new data frame into the local scope
* it is preferable to attach it only in a second step to the existing HeteroData-Chain (residing in `TurnoutSystem`)
What would be ''desirable'' though is to have some additional safeguard in the type system
to prevent attaching the newly constructed block to a chain with a non-fitting layout,
i.e. the case when several constructors or types get mixed up (because without any further
safe-guard this would lead to uncoordinated out-of-bounds memory access)
...as part of the rendering process, executed on top of the
low-level-model (Render Node network) as conceived thus far.
Parameter handling could be ''encoded'' into render nodes altogether,
or, at the other extreme, an explicit parameter handling could be specified
as part of the Render Node execution. As both extremes will lead to some
unfavourable consequences, I am aiming at a middle ground: largely, the
''automation computation'' will be encoded and hidden within the network,
implying that this topic remains to be addressed as part of defining
the Builder semantics and implementation. Yet in part the required
processing structure can be foreseen at an abstract level, and thus
the essential primitive operations are specified explicitly as part
of the Render Node definition. Notably the ''standard Weaving Pattern''
will include a ''parameter tuple'' into each `FeedManifold` and require
a binding function, which accepts this tuple as first argument.
Moreover — at implementation level, a library facility must be provided
to support handling of ''arbitrary heterogeneous data values'' embedded
directly into stack frame memory, together with a type-safe compile-time
overlay, which allows the builder to embed specific ''accessor handles''
into functor bindings, even while the actual storage location is not
yet known at that time (obviously, as being located on the stack).
__Note__: a recurring topic is how to return descriptor objects from builder functions; for this purpose, I am adjusting the semantics of `lib/nocopy.hpp` to be more specific...
During Render Node invocation, automation parameter data must be maintained.
For the simple standard path, this just implies to store the ''absolute nominal Time''
directly in the invoking stack frame and let some parameter adaptors do the translation.
However, it is conceivable to have much more elaborate translation functions,
and thus we must be prepared to handle an arbitrary number of parameter slots,
where each slot has arbitrary storage requirements.
The conclusion is to start with an intrusive linked list of overflow buckets.
This is an attempt to take aim at the next step,
which is to fill in the missing part for an actual node invocation...
''...still fighting to get ahead, due to complexity of involced concerns...''
This was an extended digression into architecture planning,
which became necessary in order to suitably map out the role
for the `TurnoutSystem` — which can now be defined as ''mediator''
to connect and forward control- and parameter data while specific
render invocation proceeds through the render node network.
After the actual processing functions are defined,
the "next level" of test framework building is to find a way
how these bare bone operations can be used easily from a test
with the goal to ''build and invoke a Render-Node''
* we need some descriptor
* the bare bone operation must be packaged into an ''Invocation-Adapter''
* we need some means to configure variants of the setup
The overall goal is eventually to arrive at something akin to a ''»Dummy Media-processing Library«''
* this will offer some „Functionality“
* it will work on different ''kinds'' or ''flavours'' of data
* it should provide operations that can be packaged into ''Nodes''
However — at the moment I have no clue how to get there...
And thus I'll start out with some rather obvious basic data manipulation functions,
and then try to give them meaningful names and descriptors. This in turn
will allow to build some multi-step processing netwaorks — which actually
is the near-term goal for the ''main effort'' (which is after all, to get
the Render Node code into some sufficient state of completion)...
Bugfix: should use the full bit-range for randomised data in `TestFrame`
Bugfix: prevent division by zero for approximate floatingpoint equality
...and use the new zip()-itertor to simplify the loops
As follow-up from the preceding refactorings,
it is now possible to drastically simplify several type signatures.
Generally speaking, iterator pipelines can now pass-through the result type,
and thus it is no longer necessary to handle this result type explicitly
In the case of `IterStateWrapper`, the result type parameter was retained,
but moved to the second position and defaulted; sometimes it can be relevant
to force a specific type; this is especially useful when defining an
`iterator` and a `const_iterator` based on the same »state-core«
For sake of completeness, since the `IterExplorer` supports building extended
search- and evaluation patterns, a tuple-zipping adapter can be expected
to handle these extended usages transparently.
While the idea is simple, making this actually happen had several ramifications
and required to introduce additional flexibility within the adaptor-framework
to cope better with those cases were some iterator must return a value, not a ref.
In the end, this could be solved with a bit of metaprogramming based on `std::common_type`
...and indeed, this is all quite nasty stuff — in hindsight, my initial intuition
to shy away from this topic was spot-on....
This involves some quite tricky changes in the way types are composed to form an iterator-pipeline.
Some wrappers are added as adaptors or for additional safety-checks, and to provide a builder-API.
Unfortunately, when building a new `IterExplorer` iterator pipeline from an existing pipeline naively,
composing all those types will add several unecessary intermediary wrapper-layers.
Worse even, the handling of `BaseAdapter` prevents the new tuple-zipping iterator
actually to pass-through any `expandChildren()` call.
These issues are a consequence of using templated types, instead of fixed types with an interface;
we can not just determine if some wrapper is present — unless the wrapper itself ''helps by exposing a tag.''
Even while I must admit that the whole packaging and adaptation machinery of `IterExplorer`
looks dangerously complex already, using dedicated type tags for this single purpose
seems like a tenable soulution.
and yes ... this revealed a **long standing bug**
The `Filter::pullFilter()` invocation in the ctor may produce dangling refs,
whenever an underlying source-iterator generates a reference that points
into the iterator itself.
The reason is: due to the »onion shell« design of the iterator pipeline,
we are bound to move a source iterator into the next layer constructor.
With this minor change, the internal result-tuple may now also hold references,
in case a source iterator exposes a reference (which is in fact the standard case).
Under the right circumstances, source-manipulation through the iterator becomes possible.
Moreover, the optimiser should now be able to elide the result-value tuple in many cases.
and access the iterator internals directly instead.
Obviously this is an advanced and possibly dangerous feature, and only possible
when no additional transformer functions are interspersed; moreover this prompted
a review of some long standing type definitions to more precisely reflect the intention.
Note: most deliberately, the Transformer element in IterExplorer must expose a reference type,
and capture the results into an internal ItemWrapper. This is the only way we can support arbitrary functions.
Indeed the solution worked out yesterday could be extracted and turned generic.
Some in-depth testing is necessary though, and possibly some qualifications to allow pass-through of references...
Moreover, last days I started collecting notes regarding problem solving patterns,
which I tend to use frequently, but which might not be obvious and thus can easily
be forgotten. In fact, I had encountered several cases, where I did invent some
roughly similar solution repeatedly, having forgotten about already settled matters.
Hopefully the habit of collecting notes and hints at a central location serves to remedy
Basically I am sick of writing for-loops in those cases
where the actual iteration is based on one or several data sources,
and I just need some damn index counter. Nothing against for-loops
in general — they have their valid uses — sometimes a for-loop is KISS
But in these typical cases, an iterator-based solution would be a
one-liner, when also exploiting the structured bindings of C++17
''I must admit that I want this for a loooooong time —''
...but always got intimidated again when thinking through the fine points.
Basically it „should be dead simple“ — as they say
Well — — it ''is'' simple, after getting the nasty aspects of tuple binding
and reference data types out of the way. Yesterday, while writing those
`TestFrame` test cases (which are again an example where you want to iterate
over two word sequences simultaneously and just compare them), I noticed that
last year I learned about the `std::apply`-to-fold-expression trick, and
that this solution pattern could be adapted to construct a tuple directly,
thereby circumventing most of the problems related to ''perfect forwarding''
So now we have a new util function `mapEach` (defined in `tuple-helper.hpp`)
and I have learned how to make this application completely generic.
As a second step, I implemented a proof-of-concept in `IterZip_test`,
which indeed was not really challenging, because the `IterExplorer`
is so very sophisticated by now and handles most cases with transparent
type-driven adaptors. A lot of work went into `IterExplorer` over the years,
and this pays off now.
The solution works as follows:
* apply the `lib::explore()` constructor function to the varargs
* package the resulting `IterExplorer` instantiations into a tuple
* build a »state core« implementation which just lifts out the three
iterator primitives onto this ''product type'' (i.e. the tuple)
* wrap it in yet another `IterExplorer`
* add a transformer function on top to extract a value-tuple for each ''yield'
As expected, works out-of-the-box, with all conceivable variants and wild
mixes of iterators, const, pointers, references, you name it....
PS: I changed the rendering of unsigned types in diagnostic output
to use the short notation, e.g. `uint` instead of `unsigned int`.
This dramatically improves the legibility of verification strings.
I changed the rendering of unsigned types in diagnostic output
to use the short notation, e.g. `uint` instead of `unsigned int`.
This dramatically improves the legibility of verification strings.
Moreover, I took the opportunigy to look through the existing page
with codeing style guides to explicitly write down some conventions
formed over years of usage.
I did not just »make up« those light heartedly, rather these conventions
are the result of a craftsman's ''attentive observation and self-reflection.''
* based on reproducible data in `TestFrame`
* using Murmur64A hash-chaining to »mark« with a parameter
This emulates the simplest case of 1:1 processing and can also be applied ''in-place''
For simplified tests there is a helper function to attain a reference to some `TestFrame` data, created on-demand and maintained in a repository in heap memory.
This storage has now be switched to `std::deque`
* provided addresses are stable
* less memory waste
__note__: `TestFrame::reseed()` will discard this repository, and draw a new (reproducible) seed.
Since each `TestFrame` now has a metadata header,
we can store an additional data checksum there,
so that it is now possible both to detect if data
is in pristine state, or if it matches a changed state
recorded in the additional checksum.
So we have now three different levels of verification
isSane:: consistent metadata header found
isValid:: metadata header found and checksum there matches data
isPristine:: in addition, the data is exactly as generated from the `(frameNr,family)`
Change data layout to place a metadata record ''behind the'' payload data,
and add a checksum to allow for validating dummy calculations and also
detect data corruption on data modified after initial generation.
By virtue of a marker data word, the presence of a valid metadata record can be confirmed.
Based on the recent work it is now possible to generate reproducible yet randomly distributed data content.
A new `TestFrame::reseed()` operation is introduced, which attaches to the `lib::defaultGen`
Using the linear-congruential engine for the actual data generation.
* Lumiera source code always was copyrighted by individual contributors
* there is no entity "Lumiera.org" which holds any copyrights
* Lumiera source code is provided under the GPL Version 2+
== Explanations ==
Lumiera as a whole is distributed under Copyleft, GNU General Public License Version 2 or above.
For this to become legally effective, the ''File COPYING in the root directory is sufficient.''
The licensing header in each file is not strictly necessary, yet considered good practice;
attaching a licence notice increases the likeliness that this information is retained
in case someone extracts individual code files. However, it is not by the presence of some
text, that legally binding licensing terms become effective; rather the fact matters that a
given piece of code was provably copyrighted and published under a license. Even reformatting
the code, renaming some variables or deleting parts of the code will not alter this legal
situation, but rather creates a derivative work, which is likewise covered by the GPL!
The most relevant information in the file header is the notice regarding the
time of the first individual copyright claim. By virtue of this initial copyright,
the first author is entitled to choose the terms of licensing. All further
modifications are permitted and covered by the License. The specific wording
or format of the copyright header is not legally relevant, as long as the
intention to publish under the GPL remains clear. The extended wording was
based on a recommendation by the FSF. It can be shortened, because the full terms
of the license are provided alongside the distribution, in the file COPYING.
⚠ __This is a problematic decision__
It temporarily **breaks compatibility with 32bit** until this issue is resolved.
== Explanation ==
Lumiera relies on a mix of the Standard library and Lib-Boost for calculation of hash values.
Before C++11, the Standard did not support and hashtable implementation; meanwhile, we
got several hash based containers in the STL and a framework for hashes,
which unfortunately is incomplete and cumbersome to use.
The C++ Committee has spend endless discussions and was not able to settle
on a convincing solution without major drawbacks regarding one aspect or the other.
This situation is problematic, since Lumiera relies heavily on the technique
of building stable systematic identifiers based on chained hash values.
It is thus essential to use a strong, reliable and portable hash function.
But unfortunately...
* the standard-fallback solution is known to be weak.
* Lib-Boost automatically uses stronger implementations for 64bit systems
* this implies that Hash-Values **are non-portable**
As the Lumiera project currently has no developer time to expend on such a
difficult and deep topic of fundamental research, today I decided to go down
the path of least resistance and **effectively abandon any system
that can not compile and use the 64bit `hash_combine` implementation.
This changeset extracts code from Lib-Boost 1.67 and adds a static assertion
to **break compilation** on non-64bit-platforms (whatever this means)
After augmenting our `lib/random.hpp` abstraction framework to add the necessary flexibility,
a common seeding scheme was ''built into the Test-Runner.''
* all tests relying on some kind of randomness should invoke `seedRand()`
* this draws a seed from the `entropyGen` — which is also documented in the log
* individual tests can now be launched with `--seed` to force a dedicated seed
* moreover, tests should build a coherent structure of linked generators,
especially when running concurrently. The existing tests were adapted accordingly
All usages of `rand()` in the code base were investigated and replaced
by suitable calls to our abstraction framework; the code base is thus
isolated from the actual implementation, simplifying further adaptation.
A deeper investigation revealed that we can show the result of glitches
for each relevant situation, simply by scrutinising the produced distribution.
Even the 64-bit-Variant shows a skewed distribuion, in spite of all numbers
being within definition range.
So the conclusion is: we can expect tilted results, but in many cases
this might not be an issue, if the result range is properly wrapped / clipped.
Notably this is the case if we just want to inject a randomised sleep into a multithreaded test setup
Build a self-contained test case to document these findings.
Further investigation shows that the ''data type used for computation'' plays a crucial role.
The (recommended) 64bit mersenne twister uses the full value range of the working data type,
which on a typical 64bit system is also `uint64_t`. In this case, values corrupted by concurrency
go unnoticed. This can be **verified empirically** : the distribution
of shifts from the theoretical mean value is in the expected low range < 2‰
However, when using the 32bit mersenne engine, the working data type is still uint64_t.
In this case a **significant number of glitches** can be shown empricially.
When drawing 1 Million values, in 80% of all runs at least one glitch and up to 5 glitches
can happen, and the mean values are **significantly skewed**
''In theory,'' the random number generators are in no way threadsafe,
neither the old `rand()`, nor the mersenne twister of the C++ standard.
However, since all we want is some arbitrarily diffused numbers,
chances are that this issue can be safely ignored; because a random
number computation broken by concurrency will most likely generate --
well, a garbled number or "randomly" corrupted internal state.
Validating this reasoning by an empiric investigation seems advisable though.
- SchedulerStress_test simply takes too long to complete (~4 min)
and is thus aborted by the testrunner. Add a switch to allow for
a quick smoke test.
- SchedulerCommutator_test aborts due to an unresolved design problem,
which I marked as failure
- add some convenience methods for passing arguments to tests
...which turn out not to be due to the PRNG changes
* the SchedulerCommutator_test was inadvertently broken 2024-04-10
* SchedulerStress_test simply runs for 4min, which is not tolerated by our Testsuite setup
see also:
5b62438eb
We use the memory address to detect reference to ''the same language object.''
While primarily a testing tool, this predicate is also used in the
core application at places, especially to prevent self-assignment
and to handle custom allocations.
It turns out that actually we need two flavours for convenient usage
- `isSameObject` uses strict comparison of address and accepts only references
- `isSameAdr` can also accept pointers and even void*, but will dereference pointers
This leads to some further improvements of helper utilities related to memory addresses...
Problems in `Rational_test` were caused by `#include' reorderings regarding ''rational'' and ''intgral'' numbers.
The actual root cause is the fact that `FSecs` is only a typedef,
which prevents us from providing a string conversion for rational numbers without ambiguity
* most usages are drop-in replacements
* occasionally the other convenience functions can be used
* verify call-paths from core code to identify usages
* ensure reseeding for all tests involving some kind of randomness...
__Note__: some tests were not yet converted,
since their usage of randomness is actually not thread-safe.
This problem existed previously, since also `rand()` is not thread safe,
albeit in most cases it is possible to ignore this problem, as
''garbled internal state'' is also somehow „random“
As it turns out, by far margin we mostly use rand() to generate
test values within a limited interval, using the ''modulo trick''
and thus excluding the upper bound.
Looking into the implementation of the distributions in the
libStdC++ shows that ''constructing'' a distribution on-the-fly
is cheap and boils down to checking and then storing the bounds;
so basically there is no need to keep ''cached distribution objects''
around, because for all practical purposes these behave like free functions
What is required occasionally is a non-zero HashValue, and sometimes
an interval of floating-point number or a normal distribution seem useful.
Providing these as free-standing convenience functions,
implicitly accessing the default PRNG.
* add new option to the commandline option parser
* pass this as std::optional to the test-suite constructor
* use this value optionally to inject a fixed value on re-seeding
* provide diagnostic output to show the actual seed value used
...to the base-class of all tests
* `seedRand()` shall be invoked by every test using randomisation
* it will draw a new seed for the implicit default-PRNG
* it will document this seed value
* but when a seed was given via cmdline, it will inject that instead
* `makeRandGen()` will create a new dedicated generator instance,
attached (by seeding) to the current default-PRNG
It is not clear yet how to pass the actual `SeedNucleus`, which
for obvious reasons must be maintained by the `test::Suite`
Using random or pseudo-random numbers as input for tests
can be a very effective tool to spot unintended behaviour in
corner cases, and also helps writing more principled test verifications.
However, investigating failures in randomised tests can be challenging.
A well-proven solution is to exploit the **determinism** of pseudo-random-numbers
by documenting a randomly generated seed, that can be re-injected for investigation.
Up to now, most tests rely on the old library function `rand()`, while
at some places already the C++ standard framework for random number generation
is used, packaged into a custom wrapper. Adding adequate support for
documented seed values seems to be easy to achieve, after switching
existing usages of `rand()` to a suitable drop-in replacement.
After some consideration, I decided ''against'' wiring random generator instances
explicitly, while allowing to do so on occasion, when necessary. Thus
the planned seeding mechanism will rather re-seed a ''implicit default''
generator, which could then be used to construct explicit generator instances
when required (e.g. for multithreaded tests)
As a starting point, this changeset replaces the `randomise()` API call
by a direct access to the ''reseeding functionality'' exposed by the
C++ framework and all default generators. Since we already provide a
dedicated static instance of the plattform entropy source, re-randomisation
can be achieved by seeding from there.
NOTE: there was extended debate in the net, questioning the viability
of the `std::random_seq` -- these arguments, while valid from a theoretical
point of view, seem rather moot when placed into a practical context,
where even 2^32 different generation-paths(cycles) are more than enough
to provide sufficient diffusion of results (unless the goal is really to
engage into Monte-Carlo simulations for scientific research or large model
simulations).
Notable most of the more catchy reprovals raised by Melissa O'Neill
have been refuted by experts of the field, even while being still propagated
at various places in the net, often combined with promoting PCG-Random.
Originally, this helper was called `IterIndex`, thereby following a
common naming scheme of iteration-related facilities in Lumiera, e.g.
* `IterAdapter`
* `IterExplorer`
* `IterSource`
However, I myself was not able to recall this name, and found myself
now for the second time unable to find this piece of code, even while
still able to recall vaguely that I had written something of this kind.
(and unable to find it by a text search for "index", for obvious reasons)
So, on a second thought, the original name is confusing: we do not create
an index of / for iterators; rather we are iterating an index. So this
is what it should be called...
This is the first step towards a »Test Domain Ongology« #1372,
which is a systematic arrangement of test-dummy functionality assumed
to mirror the actual media processing functionality present in external libs.
Each media-processing library not only provides functions to crunch data,
but also establishes a framework of entities and classification to determine
what »media« is an how it is structured and can be generated, transformed
and qualified. Since a essential goal for Lumiera is to be **library agnostic,**
it is important to avoid naïvely to take some popular library's choices
as universal truth regarding structure and nature of »media« as such.
Rather, the architecture of the Lumiera Render Engine must be kept
sufficiently open to accommodate the working style of various libraries,
even ones not known today.
To validate this architectural openness, we use a set of test functions
unrelated to any existing library to validate access to and usage of
rendering functionality — followed by further steps to adopt existing
popular libraries like **FFmpeg** or **Gstreamer**, without tilting
the basic structure of the Render Engine one way or the other.
showing the Node-symbol and a reduced rendering of
either the predecessor or a collection of source nodes.
For this we need functionality to traverse the node graph depth-first
and collect all leaf nodes (which are the source nodes without predecessor);
such can be implemented with the help of the expandAll() functionality
of `lib::IterExplorer`. In addition we need to collect, sort and deduplicate
all the source-node specs; since this is a common requirement, a new
convenience builder was added to `lib::IterExplorer`
...taking into account the prospecive usage context
where the builder expressions will be invoked from within
a media-library plug-in, using std::string_view to pass
the symbolic information seems like a good fit, because
the given spec will typically be assembled from some
building blocks, and thus in itself not be literal data.
Building a precise Frame Cache is a tough job, and is doomed to fail
when attempting to tie cache invalidation to state changes. The only
viable path is to create a system of systematic tagging of processing
steps, and use this as foundation for chained hash values, linked
in accordance to the actual processing structure.
This is complicated by the secondary concern of maintaining memory efficacy
for the render node model, which can be expected to grow to massive scale.
And even while this invocation can not be fully devised right now,
an attempt can be made to build a foundation that is not outright
wasteful, by detaching the logical information from the specific
weaving pattern used for implementation, and by minimising the
representation in memory and computing the compound information
on-demand....
The immediate next goal is to verify properties of render nodes
generated by the builder framework; two kinds of validations
can be distinguished
* structural aspects of the wiring
* the fact that processing functionality is invoked in proper order
Looking into the structural aspects brings about the necessity
to identify the actual processing function bound into some functor.
Some recapitulation of goals and requirements revealed, that this
can not be a merely technical identity record — because the intention
is to base the ''cache key'' on chained processing node identities,
so that the key is stable as long as the user-visible results will be
equivalent. And while structural data can be aggregated, at the
core this information must be provided by the scheme embedded
into the domain ontology, which is tasked with invoking the
builder in order to implement a ''specific processing-asset''
Review the achievements from the last days and map out the further path
for test-driven build-up of a render-node network and invocation.
Notably ''several layers of prototyping'' are in the works now;
it is important to understand the purpose of each such round of
prototyping and to draw the necessary conclusions after closing out.
The next topic to investigate relates to the ''identity'' of nodes and
ports within nodes; this entails to generate a ''symbolic spec'' that
can be verified and used as base for a systematic hash-ID and cache-key...
Since it would in fact be possible to access and write beyond the configured storage,
simply by using the builder API without considering consistency,
it seems advisable to use explicit runtime checks here, instead of
only assertions, and to throw an exception when violating bounds.
Moreover, unsuccessfully attempted to better arrange the functionality
between PortBuilder and WeavingBuilder; seemingly we have an rather tight
coupling here, and also the expectations regarding the processing function
seem to be too tight (but that's the reason why it's an prototype...)
...which then also allow to fill in the missing parts for the
default 1:1 wiring scheme, which connects each »input slot«
of the processing function with the corresponding ''lead node''
- the chaining constructor is picked reliably when the
slicing is done by a direct static_cast
- the function definition can be passed reliably in all cases
after it has been ''decayed,'' which is done here simply by
taking it by-value. This is adequate, since the function
definition must be copied / inlined for each invocation.
With these fixes, the simplest test case now for the first time
**runs through without failure**
This change allows to disentangle the usages of `lib::SeveralBuilder`,
so that at any time during the build process only a single instance is
actively populated, all in one row — and thus the required storage can
either be pre-allocated, or dynamically extended and shrinked (when
filling elements into the last `SeveralBuilder` currently activated)
By packaging into a λ-closure, the building of the actual `Port`
implementation objects (≙ `Turnout` instances) is delayed until the
very end of the build process, and then unloaded into yet another
`lib::Several` in one strike. Temporarily, those building functor
objects are „hidden“ in the current stack frame, as a new `NodeBuilder`
instance is dropped off with an adapted type parameter (embedding the
λ-type produced by the last nested `PortBuilder` invocation, while
inheriting from previous ones.
However, defining a special constructor to cause this »chaining«
poses some challenge (regarding overload resolution). Moreover,
since the actual processing function shall be embedded directly
(as opposed to wrapping it into a `std::function`), further problems
can arise when this function is given as a ''function reference''
...and as expected, this turns up quite some inconsistencies,
especially regarding usage of the »buffer types«.
Basically, the `PortBuilder` is responsible for the high-level functionality
and thus must ensure the nested `WiringBuilder` is addressed and parameterised
properly to connect all »slots« of the processing function.
- can use a helper function in the WiringBuilder to fill in connections
- but the actual buffer types passed over these connectinos are totally
unchecked at that level, and can not see yet how this danger can be
mitigated one level above, where the PortBuilder is used.
- it is still unclear what a »buffer type« actually means; it could
be the pointer type, but it could also imply a class or struct type
to be emplaced into the buffer, which is a special extension to the
`BufferProvider` protocol, yet seems to be used here rather to transport
specific data types required by the actual media handling library (e.g. FFmpeg)
__Analysis__: what kind of verifications are sensible to employ
to cover building, wiring and invocation of render nodes?
Notably, a test should cover requirements and observable functionality,
while ''avoiding direct hard coupling to implementation internals...''
__Draft__: the most simple node builder invocation conceivable...
Code clean-up: mark all buffers with a dedicated tagging type
The point in question is: if we work the LocalTag into the type-hash,
could it be possible to miss an existing entry in the metadata registry?
This could cause two entries to be locked for a single buffer address,
leading to data corruption.
As far as I can see, in the current usage this would not happen,
but unfortunately this problem can not be ruled out, since the BufferProvider
API and protocol is designed to be open for various usage patterns.
However, the same potentially disastrous pattern could also materialise
when registering two different buffer types, and then locking each
for the same buffer location.
...this is a surprisingly tricky issue, since it undercuts the
generic and recursive implementation of buffer handling;
fortunately I've foreseen such demands may arise down the road
and I've reserved an »Local Key« (now renamed into `LocalTag`),
whose meaning is implementation defined and interpreted by
the specific `BufferProvider`
It became clear that a secondary system of connections must be added,
running top-down from a global model context, and thus contrary to the
regular orientation of the node network, which connects upwards from
predecessor to successor, in accordance with the pull principle.
If we accept this wiring as part of the primary structure, it can be
established immediately while building the nodes, thus adding a preconfigured
''pattern of Buffer Descriptors'' to each node, since there is no further
''moving part'' — beyond the wiring to the `BufferProvider`, which thus
becomes part of a global `ModelContext`
As an immediate consequence, the storage for this configuraion should
also be switched to `lib::Several` and handled similar to the primary
node wiring in the Builder...
* conduct analysis regarding allocator handling in the Builder
* turns out we'll have to keep around two different allocators while building
* ⟹ establish the goal to confine usage of the Node allocator to the lower Levels
* consequently must open up the `lib::SeveralBuilder` to be usable
as an intermediary data structure, while building up the target data
* in the initial design, the `SeveralBuilder` was kept opaque, since
contents can be expected to be re-located frequently and thus exposing
elements and taking references could be dangerous — yet this is also
true for `std::vector` however, so people are assumed to know
when they want to shoot themselves into their own foot
...especially what is necessary to represent at this level and what information
is implicit; notably there will be an implicit default wiring, but we allow
for case-by-case deviations
To escape a possible deadlock in analysis, I resort to developing
some kind of free-wheeling presupposition how the **Builder** could
be implemented — a centrepiece of the Lumiera architecture envisioned
thus far — which ''unfortunately'' can only be planned and developed
in a more solid way ''after'' the current »Vertical Slice« is completed.
Thus I find myself in the uncomfortable situation of having to work towards
a core piece, which can not yet be built, since it relies heavily on
the very structures to be built...
...and this line of analysis brings us deep into the ''Buffer Provider''
concept developed in 2012 — which appears to be very well to the point
and stands the test of time.
Adding some ''variadic arguments'' at the right place surprisingly leads
to an ''extension point'' — which in turn directly taps into the
still quite uncharted territory interfacing to a **Domain Ontology**;
the latter is assumed to define how to deal with entities and relationships
defined by some media handling library like e.g. FFmpeg.
So what we're set to do here is actually ''ontology mapping....''
The immediate next step is to build some render nodes directly
in a test setting, without using any kind of ''node factory.''
Getting ahead with this task requires to identify the constituents
to be represented on the first code layer for the reworked code
(here ''first layer'' means any part that are ''not'' supplied
by generic, templated building blocks).
Notably we need to build a descriptor for the `FeedManifold` —
which in turn implies we have to decide on some fundamental aspects
of handling buffers in the render process.
To allow rework of the `ProcNode` connectivity, a lot of presumably obsoleted
draft code from 2011 has to be detached, to be able to keep it in-tree
for further reference (until the rework and refactoring is settled).
As outlined in #1367, the integration effort requires some rework
of existing code, which will be driven ahead by the `NodeLinkage_test`
* redefine Node Connectivity
* build simple `ProcNode` directly in scope
* create an `TurnoutSystem` instance
* perform a ''dummy Node-Invocation''
As a replacement for the `RefArray` a new generic container
has been implemented and tested, in interplay with `AllocationCluster`
* the front-end container `lib::Several<I>` exposes only a reference
to the ''interface type'' `I`, while hiding any storage details
* data can only be populated through the `lib::SeveralBuilder`
* a lot of flexibility is allowed for the actual element data types
* element storage is maintained in a storage extent, managed through
a custom allocator (defaulting to `std::allocator` ⟹ heap storage)
The `SeveralBuilder` employs the same tactic as `std::vector`,
by over-allocating a reserve buffer, which grows in exponential
increments, to amortise better the costs of re-allocation.
This tactic does not play well with space limited allocators
like `AllocationCluster` however; it is thus necessary to provide
an extension point where the actuall allocator's limitation can be
queried, allowing to use what is available as reserve, but not more.
With these adaptations, a full usage cycle backed by `AllocationCluster`
can be demonstrated, including variations of dynamic allocation adjustment.
...identified as part of bug investigation
* make clear that reserve() prepares for an absolute capacity
* clarify that, to the contrary, ensureStorageCapaciy() means the delta
Moreover, it turns out that the assertion regarding storage limits
triggers frequently while writing the test code; so we can conclude
that the `AllocationCluster` interface lures into allocating without
previous check. Consequently, this check now throws a runtime exception.
As an aside, the size limitation should be accessible on the interface,
similar to `std::vector::max_size()`
- decided to allow creating empty lib::Several;
no need to be overly rigid in this point,
since it is move-assignable anyway...
- populate with enough elements to provoke several reallocations
with copying over the existing elements
- precisely calculate and verify the expected allocation size
- verify the use-count due to dedicated allocator instances
being embedded into both the builder and hidden in the deleter
- move-assign data
- all checksums go to zero at end
The setup for `ArrayBucket` is special, insofar it shell de-allocate itself,
which creates the danger of re-entrant calls, or to the contrary, the danger
to invoke this clean-up function without actually invoking the destructor.
These problems become relevant once the destructor function itself is statefull,
as is the case when embedding a non-trivial, instance bound allocator
to be used for the clean-up work. Using the new `lib::TrackingAllocator`
highlighted this potential problem, since the allocator maintains a use-count.
Thus I decided to move the »destruction mechanics« one level down into
a dedicated and well encapsulated base class; invoking ArrayBucket's destructor
thereby becomes the only way to trigger the clean-up, and even ElementFactory::destroy()
can now safely check if the destructor was already invoked, and otherwise
re-invoke itself through this embedded destructor function. Moreover,
as an additional safety measure, the actual destructor function is now
moved into the local stack frame of the object's destructor call, removing
any possibility for the de-allocation to interfere with the destructor
invokation itself
part of the observed deviation stems form bugs in logging and checksum calculation;
but there seems to be a real problem hidden in the allocator usage of the
new component, since the use-cnt of the handle does not drop to zero
While there might be the possibility to use the magic of the standard library,
it seems prudent rather to handle this insidious problem explicitly,
to make clear what is going on here.
To allow for such explicit alignment handling, I have now changed the
scheme of the storage definition; the actual buffer now starts ''behind''
the `ArrayBucket<I>` object, which thereby becomes a metadata managing header.
__To summarise the problem__: since we are maintaining a dynamically sized buffer,
and since we do not want to expose the actual element type through the
front-end object, we're necessarily bound to perform a raw-memory allocation.
This is denoted in bytes, and thus the allocator can no longer manage
the proper alignment automatically. Rather, we get a storage buffer with
just ''some accidental'' alignment, and we must care to request a sufficient
overhead to be able to shift the actual storage area forward to the next
proper alignment boundary. Obviously this also implies that we must
store this individual padding adjustment somewhere in the metadata,
in order to be able to report the correct size of the block later
on de-allocation.
The solution implemented thus far turns out to be not sufficient
for ''over-aligned-data'', as the raw-allocator can not perform the
''magic work'' because we're exposing only `std::byte` data.
This adaptor works in concert with the generic allocator
building blocks (prospective ''Concepts'') and automatically
registers a either static or dynamic back-link to the factory
for clean-up.
Use this wrapper fore more in-depth test of the new `TrackingAllocator`
and verify proper behaviour through the `EventLog`
- ability to verify a hash-checksum
- ability to watch number of allocations and allotted bytes
- using either a common global pool or a separate dedicated pool
- log all operations into a common `EventLog` instance
- front-end adaptors for use as C++ custom allocator
...these features are now used quite regularly,
and so a dedicated documentation test seems indicated.
Actually my intention is to add a tracking allocator to these test helpers
(and then to use that to verify the custom allocator usage of `lib::Several`)
Phew... this was a tough one — and not sure yet if this even remotely works...
Anyway, the `lib::SeveralBuilder` is already prepared for collaboration with a
custom allocator, since it delegates all memory handling through a base policy,
which in turn relies on std::allocator_traits.
The challenge however is to find a way...
* to make this clear and easy to use
* to expose an extension point for specific tweaks
* and to make all this work without excessive header cross dependencies
This is a low-level interface to allow changing the size of
the currently latest allocation in `AllocationCluster`; a client
aware of this capability can perform a real »in-place re-alloc«,
assuming the very specific usage constraints can be met.
`lib::Several<X>` will use this feature when attached to an
`AllocationCluster`; with this special setup, an previously
unknown number of non-copyable objects can be built without
wasting any storage, as long as the storage reserve in the
current extent of the `AllocationCluster` is sufficient.
...use some pointer arithmetic for this test to verify
some important cases of object placement empirically.
Note: there is possibly a very special problematic case
when ''over aligned objects'' are not placed in accordance
to their alignment requirements. Fixing this problem would
be non-trivial, and thus I have only left a note in #1204
...including the interesting cases where objects are relocated
and the element spread is changed. With the help of the checksum
feature built into the test-dummy objects, the properly balanced
invocation of constructors can be demonstrated
PS: for historical context...
Last week the "Big F**cking Rocket" successfully performed the
test flight 4; both booster and Starship made it back to the
water surface and performed a soft splash-down after decelerating
to speed zero. The Starship was even able to maintain control
in spite of quite some heat damage on the steering flaps.
Yes ... all techies around the world are thrilled...
- spread change now retains the nominal element reserve
- `capacity()` and `capReserve()` now exposed on the builder API
- factor out the handling check safety functions
- rewrite the `resize()` builder function to be more generic
__Test now covers__ example with trivial data type, which can
indeed be resized and allows to grow buffer on-the fly without
requiring any knowledge of the actual type (due to using `memmove`)
building on the preceding analysis, we can now demonstrate that
the container is initially able to grow, but looses this capability
after accepting one element of unknown subclass type...
`lib::Several` is designed to be highly adaptable, allowing for
several quite distinct usage styles. On the downside, this requires
to perform some checks at runtime only, since the ability to handle
some element depends on specific circumstances.
This is a notable difference to `std::vector`, which is simply not capable
of handling ''non-copyable'' types, even if given an up-front memory reservation.
The last test case provided with the previous changeset did not trigger
an exception, but closer investigation revealed that this is correct,
since in this specific situation the container can accept this object type,
thereby just loosing the ability to move-relocate further objects.
A slightly re-arranged test scenario can be used to demonstrate this fine point.
- the test-dummy objects need a `noexcept` move ctor
- **bug** here: need an explicit check to prevent other types
than the known element type from ''sneaking in''
The `SeveralBuilder` is very flexible with respect to added elements,
but it will investigate the provided type information and reject any
further build operation that can not be carried out safely.
...turns out that we must ensure to pass a plain "object" type
to the standard allocator framework (no const, no references).
Here, ''object in C++ terminology'' means a scalar or record type,
but no functor, no references and no void,
