Requirement analysis indicates that a »Node ID« is rather tangential
to the core operation of calculating media; the only infliction point
seems to be the generation of ''systematic cache keys.''
A spec — especially for the `Turnout` however is very relevant for
diagnostics, error reporting and unit testing. So we are in the
difficult situation where rather elaborate functionality is
required only for a secondary concern, and moreover the
node data structure imposes a critical memory leverage.
The immediate next goal is to verify properties of render nodes
generated by the builder framework; two kinds of validations
can be distinguished
* structural aspects of the wiring
* the fact that processing functionality is invoked in proper order
Looking into the structural aspects brings about the necessity
to identify the actual processing function bound into some functor.
Some recapitulation of goals and requirements revealed, that this
can not be a merely technical identity record — because the intention
is to base the ''cache key'' on chained processing node identities,
so that the key is stable as long as the user-visible results will be
equivalent. And while structural data can be aggregated, at the
core this information must be provided by the scheme embedded
into the domain ontology, which is tasked with invoking the
builder in order to implement a ''specific processing-asset''
Review the achievements from the last days and map out the further path
for test-driven build-up of a render-node network and invocation.
Notably ''several layers of prototyping'' are in the works now;
it is important to understand the purpose of each such round of
prototyping and to draw the necessary conclusions after closing out.
The next topic to investigate relates to the ''identity'' of nodes and
ports within nodes; this entails to generate a ''symbolic spec'' that
can be verified and used as base for a systematic hash-ID and cache-key...
Since it would in fact be possible to access and write beyond the configured storage,
simply by using the builder API without considering consistency,
it seems advisable to use explicit runtime checks here, instead of
only assertions, and to throw an exception when violating bounds.
Moreover, unsuccessfully attempted to better arrange the functionality
between PortBuilder and WeavingBuilder; seemingly we have an rather tight
coupling here, and also the expectations regarding the processing function
seem to be too tight (but that's the reason why it's an prototype...)
...which then also allow to fill in the missing parts for the
default 1:1 wiring scheme, which connects each »input slot«
of the processing function with the corresponding ''lead node''
The intention is to offer an automatic 1:1 association
between the »input parameter slots« of the processing function
and the ''lead nodes,'' thereby always using the same default
port, corresponding to the current port number under construction.
Unfortunately, the preceding refactoring removed the information
necessary for a simple implementation, as the port array is now
built up late, in the final build() function...
The next step is to round out the first prototypical implementation,
which requires access to ''lead node ports'' and thereby generally
places focus on the interplay of ''data builders'' within the ongoing
build process. While the prototype still uses the fall-back to simple
heap allocation, notably the intended usage will require to wire-through
the connection to a single `AllocationCluster`. This poses some
challenge, since further ''data builders'' will be added step-wise,
implying that this wiring can not be completed at construction time.
Thus it seems indicated to slightly open-up the internal allocator
policy base template used by `lib::SeveralBuilder` to allow for some
kind of ''cross building'' based on a shared compatible base allocator
type, so that the allocation policy wiring can be passed-on from an
existing `SeveralBuilder`
- the chaining constructor is picked reliably when the
slicing is done by a direct static_cast
- the function definition can be passed reliably in all cases
after it has been ''decayed,'' which is done here simply by
taking it by-value. This is adequate, since the function
definition must be copied / inlined for each invocation.
With these fixes, the simplest test case now for the first time
**runs through without failure**
This change allows to disentangle the usages of `lib::SeveralBuilder`,
so that at any time during the build process only a single instance is
actively populated, all in one row — and thus the required storage can
either be pre-allocated, or dynamically extended and shrinked (when
filling elements into the last `SeveralBuilder` currently activated)
By packaging into a λ-closure, the building of the actual `Port`
implementation objects (≙ `Turnout` instances) is delayed until the
very end of the build process, and then unloaded into yet another
`lib::Several` in one strike. Temporarily, those building functor
objects are „hidden“ in the current stack frame, as a new `NodeBuilder`
instance is dropped off with an adapted type parameter (embedding the
λ-type produced by the last nested `PortBuilder` invocation, while
inheriting from previous ones.
However, defining a special constructor to cause this »chaining«
poses some challenge (regarding overload resolution). Moreover,
since the actual processing function shall be embedded directly
(as opposed to wrapping it into a `std::function`), further problems
can arise when this function is given as a ''function reference''
Conduct in-depth analysis to handle a secondary, implementation-related
(and frankly quite challenging) concern regarding the placement of node
and port connectivity data in memory. The intention is for the low-level
model to use a custom data structure based on `lib::Several`, allowing for
flexible and compact arrangement of the connectivity descriptors within
tiled memory blocks, which can then later be discarded in bulk, whenever
a segment of the render graph is superseded. Yet since the generated
descriptors are heterogeneous and, due to virtual functions, can not be
trivially copied, the corresponding placement invocations on the
data builder API must not be mixed, but rather given in ordered strikes
and preceded by a dimensioning call to pre-reserve a bulk of storage
However, doing so directly would jeopardise the open and flexible nature
of the node builder API, thereby creating a dangerous coupling between
the implementation levels of the node graph and of prospective library
wrapper plug-ins in charge of controlling details of the graph layout.
The solution devised here entails a functional helper data structure
created temporarily within the builder API stack frames; the detailed
and local type information provided from within the library plug-in
can thereby be embedded into opaque builder functors, allowing to
delay the actual data generation up until the final builder step,
at which point the complete number and size requirements of
connectivity data is known and can be used for dimensioning.
This investigation was set off by a warning regarding an
unused argument in `SeveralBuilder`, using `AllocationPolicy::moveElem()`
This warning is correct and easy to fix, but (luckily) it brought my
attention to the fact that a `SeveralBuilder<Port>` can not grow dynamically,
which is somewhat mitigated by the default policy to pre-allocate several
elements, which would work to some degree but waste a lot of memory.
This points to a deeper problem with the implementation pattern used for
all those Builders: they create their product by-value, which must then
be moved into the intended target location.
And doing so is **extremely dangerous**, given that our very goal is to
build a complex data structure internally connected by direct references
and ideally also allocated with a high degree of memory locality.
Unfortunately I do not see any favourable alternative yet;
Ideally all products should be `NonCopyable` — but then, the builder
implementation scheme would become even more complicated and less intuitive
and additionally the client code would need to pre-declare the number of
expected Leads and Ports (not clear if this is even feasible)
...and as expected, this turns up quite some inconsistencies,
especially regarding usage of the »buffer types«.
Basically, the `PortBuilder` is responsible for the high-level functionality
and thus must ensure the nested `WiringBuilder` is addressed and parameterised
properly to connect all »slots« of the processing function.
- can use a helper function in the WiringBuilder to fill in connections
- but the actual buffer types passed over these connectinos are totally
unchecked at that level, and can not see yet how this danger can be
mitigated one level above, where the PortBuilder is used.
- it is still unclear what a »buffer type« actually means; it could
be the pointer type, but it could also imply a class or struct type
to be emplaced into the buffer, which is a special extension to the
`BufferProvider` protocol, yet seems to be used here rather to transport
specific data types required by the actual media handling library (e.g. FFmpeg)
__Analysis__: what kind of verifications are sensible to employ
to cover building, wiring and invocation of render nodes?
Notably, a test should cover requirements and observable functionality,
while ''avoiding direct hard coupling to implementation internals...''
__Draft__: the most simple node builder invocation conceivable...
* decision how to provide a default service for tests
while also allow for configuration of more specific services
* as starting point for the prototype: use the `TrackingHeapBlockProvider`
(simply because this is the only implementation available and tested)
Prototyping and analysis revealed that some aspects of the render node wiring
refers to effectively global services and can thus be taken out of the picture
by relying on classical ''Dependency Injection''
Consequently, `EngineCtx` needs a default implementation, which brings up
a simplistic fall-back version of those services in support for prototyping.
Moreover, dedicated lifecycle functionality must be provided to bring up
and shut down the actual service instances intended for operational use.
...need to pass a binding for the actual processing function
in a way that it acts as a ''prototype'' — since the `Feed`,
i.e. the ''Invocation Adapter'' must be generated for each
invocation anew within the current stack frame
(so to avoid spurious heap allocations)
...seems that the former is well suited to serve as detail builder
used internally by the latter to provide a simplified standard adaptation
for a given processing function.
The integration can be achieved to layer a specialised detail builder class
on top, which can be entered only by specifying the concrete function or lambda
to wrap for the processing; the further builder-API-functions to control
the wiring in detail become thus only accessible after the function type
is known; this allows to place the detail builder as member into the
enclosing port builder and thus to allocate everything within the current
stack frame invoking the builder chain.
...after having determined the several levels of prototyping
currently employed, an important step ahead could be achieved
by analysing the intended and implied usage context of this
builder scheme, while still assuming the simplifications
related to prototyping.
It can be assumed that
* the Level-2 builder object is ''somehow provided''
* the invocation happens from within a media-handling lib-plugin
* alongside with the desired `ProcAsset` spec, an `ExpectationContext`
will be provided, allowing to pass-through additional semantic tags
The implementation in the lib-plugin is then able to draw from specific
knowledge related to the **Domain Ontology** for ''especially for this library''
and provide the necessary wrappers and parameter mapping information.
⟹ the **Level-2**-builder should thus expose an API to
* set up a straight forward mapping, based on a given wrapper functor
to delegate to the actual library invocation
* allow optionally to override some of the input connections
* alternatively allow to use a complete `InvocationAdapter`,
including a `FeedManifold`, as provided directly by the library-plugin
...caused by personal circumstances
...attempt to understand the context I was working on
* Integration is driven by the `NodeLinkage_test`
* the near-term goal is to ''get any node built'' — simplified
* the outline of the `NodeBuilder` and `PortBuilder` is settled
* the task at hand is how to fill in the definition of a `Port`
* which in turn ''requires prototyping'' — to establish a kind of weaving-pattern
* the immediate next thing to do is to ''build an `InvocationAdapter` within the »test-ontology«''
...by relying on DI for some effectively global services, notably
the cache provider, the API for building and wiring render nodes
can be simplified to cover only the actual node connectivity
Doing so directly seems to be a better solution than to inject an OutputBufferProvider;
the latter will still be needed, yet will not be part of the regular weaving pattern,
but used directly at top-level to obtain the output `BuffHandle`, which is then
passed to the `Port::weave()` call
...still not convinced that this is a good design,
since it seems to subvert the general design to treat one special case.
However, I can't see a good way to address this special case directly
There might be one specific output result buffer at top level
for each invocation, which must be delivered into a prepared
output sink. This amounts to one special case, cross-cutting
an otherwise completely generic data flow scheme.
After considering several solutions, it seems most straight-forward
to configure a specific `OutputBufferProvider` to serve as a proxy for
the `OutputSlot` / `DataSink` provided at top-level to the Render-Job.
As an asside, this analyis reveals that the result-slot number does
not belong into the `FeedManifold`, which is dynamic (on the stack);
rather, it's a fixed value configured as part of the `WeavingPattern`
Code clean-up: mark all buffers with a dedicated tagging type
The point in question is: if we work the LocalTag into the type-hash,
could it be possible to miss an existing entry in the metadata registry?
This could cause two entries to be locked for a single buffer address,
leading to data corruption.
As far as I can see, in the current usage this would not happen,
but unfortunately this problem can not be ruled out, since the BufferProvider
API and protocol is designed to be open for various usage patterns.
However, the same potentially disastrous pattern could also materialise
when registering two different buffer types, and then locking each
for the same buffer location.
...this seems to be a tricky aspect; we use hash-chaining to create
derived entries, which may cause the identity of an entry to depend
on the order of specialisation. Looked through the possible code paths,
but these seem to be quite complicated; I see the lurking danger of
creating a second entry (with a different hash), and then in worst case
even locking/unlocking a given buffer twice....
...this is a surprisingly tricky issue, since it undercuts the
generic and recursive implementation of buffer handling;
fortunately I've foreseen such demands may arise down the road
and I've reserved an »Local Key« (now renamed into `LocalTag`),
whose meaning is implementation defined and interpreted by
the specific `BufferProvider`
Requirement analysis shows that the ''actual buffer provider'' to use
constitutes yet another independent degree of freedom, which conceivably
must be handled by the Builder internals rather than by the Domain Ontology.
Thus the simple solution to use a `BuffDescr` to mark the type must be augmented
to also allow configuration of the underlying `BufferProvider`, which generates
the descriptor and can later be invoked with this descriptor to ''lock an actual Buffer.''
In some cases, setup of the buffer types could even be more complicated and require
access to the actual (runtime) invocaton context; such extreme cases however
could be rendered as an extension of the scheme established here,
by storing the (up to now transient) constructor functors persistently.
Which leads to the decision not to care for those extremely complicated
corner cases right now, and thus to construct all buffer descriptors
in the `build()` call
...still fighting to find a suitable API to define
how inputs and outputs are connected and mapped to function parameters.
The solution drafted here uses the reshaped `DataBuilder` (≙`lib::SeveralBuilder`)
to add up connections for each »slot«, disregarding the possibility of permutations.
Similar to `NodeBuilder`, a policy template is used to pass down the setup
for an actual custom allocator.
After applying all the preceding refactorings, it turns out that
the `DataBuilder` defined here ''is essentially `lib::SeveralBuilder`'',
only with a different arrangement of the type parameters, due to the
specific usage context here.
It is thus possible to replace all the interim / helper / rebinding templates
by simple templated typedefs. The only tangible difference is that for
usage in the Builder, a ''selector policy'' is passed as a simple type argument,
which in practice wires the concrete allocator information down into each
sub-builder created during the ongoing construction of a node structure.
redefine the policy for `lib::SeveralBuilder` to be a template-template parameter.
In fact it should have been this way from start, yet defining this kind of
very elaborate code bottom-up lets you sometime miss the wood for the trees
So to restate: `lib::SeveralBuilder` takes a ''policy template,''
which then in turn will be instantiated with the same types `I` (interface)
and `E` (element type) used on `SeveralBuilder` itself. Obviously, there can be
further types involved and thus additional type parameters may be necessary,
notably the ''Allocator'' — yet these are better injected when ''defining''
the policy template itself.
The default binding for this policy template is defined as `allo::HeapOwn`,
which causes the builder to allocate the storage extents through the standard
heap allocator, and for the created `lib::Several` to take full ownership of
embedded objects, invoking their destructors when falling out of scope.
As a direct consequence of the insights regarding Dependency-Injection,
a ''Builder Toolkit'' is required, which can be used to adapt various
kinds of ''Weaving Patterns'' — since obviously it is not possible to
settle down on a single Pattern, and thus several ''families of builders''
will emerge, one for each ''line of construction'' for ''Weaving Patterns''.
To stress this point, what I am coding here is a prototype, aimed at
being used as part of a **Test Domain Ontology** — and other Domain Ontologies
(e.g. für FFmpeg) will certainly require other construction schemes
for their Weaving Patterns. So this is an open field, and can not be
settled once and for all.
This immediately leads to another, rather technical problem:
If we're about to work with ''delegate Builders,'' then also
a way to pass-down the allocator configuration is required.
We had settled on a preliminary solution with the helper `DataBuilder`,
yet this solution looks like it defines how `lib::SeveralBuilder`
should be used in most of the cases. So there is now a conflict
between the existing definition scheme for `lib::SeveralBuilder`,
which was achieved in a bottom-up way, and a slightly different
definition scheme ''as it should be''
Starting to attack this latter detail problem, as a first step,
the definition of `DataBuilder` can be simplified by collapsing
it with the `lib::allo::SetupSeveral`
It became clear that a secondary system of connections must be added,
running top-down from a global model context, and thus contrary to the
regular orientation of the node network, which connects upwards from
predecessor to successor, in accordance with the pull principle.
If we accept this wiring as part of the primary structure, it can be
established immediately while building the nodes, thus adding a preconfigured
''pattern of Buffer Descriptors'' to each node, since there is no further
''moving part'' — beyond the wiring to the `BufferProvider`, which thus
becomes part of a global `ModelContext`
As an immediate consequence, the storage for this configuraion should
also be switched to `lib::Several` and handled similar to the primary
node wiring in the Builder...
It seems we need a `WeavingPattern`-Builder, which obviously
must be rather flexible, since those patterns are to be composed
from several layers, which should be extensible within a given ''Domain Ontology''
So this seems to lead to a builder-DSL which creates »**onion layers**«
of builders, with the ability to extend and specialise the type on each layer.
''As it will be quite challenging to get this into usable shape,
it seems best to approach this step by step through prototyping...''
Not entirely sure how to use the `emit()` call properly,
assuming that it means that data is complete in buffer,
but can still be read after that point
* at least for a simple, prototypical setup
* and actually shifting the onerous into the Level-1 builder \\
''(which is precisely the intention here)''
The deeper problem is that we must not engage into any premature decisions
regarding the structure or layout of the actual processing function invocation.
Thus attempting to create a kind of »firewall« of sorts, by connecting
the building blocks strictly through template parameter and preferably
figuring out any detailed knowledge locally, through ''compile-time introspection...''
...even the initial effort to stub its operation turns into a
challenge, since honestly there is near nothing we can assume safely,
without sliding into uncovered provisions regarding the ''Domain Ontology''
- it is clear that this adaptor will be a ''Concept''
- yet it must in some way access the `FeedManifold` and also control additional storage
- a rather obvious solution is to layer it ''on top'' of the manifold
...which brings about various (preliminary) decisions regarding
Metadata storage in the `Turnout`-object, which acts as a guidance
and specification for the actual invocation for this specific node.
As starting point, I choose the ''KISS'' solution of embedding some
blocks of `UninitialisedStorage` directly into the `Turnout`; obviously
these blocks must be oversized, since we can not effort emitting a
dedicated template instance for each different count of input / output
feeds. Moreover, these data buffers are assumed to be filled with
valid objects by the builder ''(this is a lurking danger)''
...turns out that the intended structure is still too fine grained
and explicit and many operational steps can be collapsed into a single
virtual scope, wherein they can be deemed implementation detail...
...so the solution is to build up the working data as `lib::SeveralBuilder`;
however, a more concise notation can be achieved with a suitably configured
wrapping subclass; together with the cross-builder trick, this allows
to write the allocation configuration in a clearly libelled way,
while the field definition and the builder constructor hides the
complexities of picking up the extension point and passing on the
wiring to the allocator instance.
...using the same pattern here as was successful also for the underlying lib::SeveralBuilder;
even if it may seem logically backwards, it just reads much better and
is more understandable, and has the added benefit of providing a dedicated
definition scope, which can be kept separate from the constructor definition
of the actual builder
{{{
prepareNode()
.withAllocator<XYZ>()
.addLead(predecessor)
.build()
}}}
...turns out to be surprisingly tricky, since the nested
lib::SeveralBuilder instances require parametrisation by a
''policy template,'' which in turn relies on the actual allocator.
And we want to provide the allocator as a constructor parameter,
including the ability to pick up a custom specialisation for
some specific allocator (notably AllocationCluster requires
to hook into this kind of extension point, to be able to
employ its dedicated API for dynamic allocation adjustment)
* conduct analysis regarding allocator handling in the Builder
* turns out we'll have to keep around two different allocators while building
* ⟹ establish the goal to confine usage of the Node allocator to the lower Levels
* consequently must open up the `lib::SeveralBuilder` to be usable
as an intermediary data structure, while building up the target data
* in the initial design, the `SeveralBuilder` was kept opaque, since
contents can be expected to be re-located frequently and thus exposing
elements and taking references could be dangerous — yet this is also
true for `std::vector` however, so people are assumed to know
when they want to shoot themselves into their own foot
...especially what is necessary to represent at this level and what information
is implicit; notably there will be an implicit default wiring, but we allow
for case-by-case deviations
The Builder will have to perform several passes, gradually refining
the model into the low-level Render Node network. Right now, some
guesses regarding the last steps of this process are possible,
thus defining the lowest level of a model builder structure
* Level-3 : mapping data flow paths
* Level-2 : detailed configuration of data buffer passing
* Level-1 : build the actual parameter structures for invocation
In the current »Vertical Slice« we're able to fully define Level-1
and maybe Level-2
