...taking into account the prospecive usage context
where the builder expressions will be invoked from within
a media-library plug-in, using std::string_view to pass
the symbolic information seems like a good fit, because
the given spec will typically be assembled from some
building blocks, and thus in itself not be literal data.
Building a precise Frame Cache is a tough job, and is doomed to fail
when attempting to tie cache invalidation to state changes. The only
viable path is to create a system of systematic tagging of processing
steps, and use this as foundation for chained hash values, linked
in accordance to the actual processing structure.
This is complicated by the secondary concern of maintaining memory efficacy
for the render node model, which can be expected to grow to massive scale.
And even while this invocation can not be fully devised right now,
an attempt can be made to build a foundation that is not outright
wasteful, by detaching the logical information from the specific
weaving pattern used for implementation, and by minimising the
representation in memory and computing the compound information
on-demand....
The immediate next goal is to verify properties of render nodes
generated by the builder framework; two kinds of validations
can be distinguished
* structural aspects of the wiring
* the fact that processing functionality is invoked in proper order
Looking into the structural aspects brings about the necessity
to identify the actual processing function bound into some functor.
Some recapitulation of goals and requirements revealed, that this
can not be a merely technical identity record — because the intention
is to base the ''cache key'' on chained processing node identities,
so that the key is stable as long as the user-visible results will be
equivalent. And while structural data can be aggregated, at the
core this information must be provided by the scheme embedded
into the domain ontology, which is tasked with invoking the
builder in order to implement a ''specific processing-asset''
Review the achievements from the last days and map out the further path
for test-driven build-up of a render-node network and invocation.
Notably ''several layers of prototyping'' are in the works now;
it is important to understand the purpose of each such round of
prototyping and to draw the necessary conclusions after closing out.
The next topic to investigate relates to the ''identity'' of nodes and
ports within nodes; this entails to generate a ''symbolic spec'' that
can be verified and used as base for a systematic hash-ID and cache-key...
Since it would in fact be possible to access and write beyond the configured storage,
simply by using the builder API without considering consistency,
it seems advisable to use explicit runtime checks here, instead of
only assertions, and to throw an exception when violating bounds.
Moreover, unsuccessfully attempted to better arrange the functionality
between PortBuilder and WeavingBuilder; seemingly we have an rather tight
coupling here, and also the expectations regarding the processing function
seem to be too tight (but that's the reason why it's an prototype...)
...which then also allow to fill in the missing parts for the
default 1:1 wiring scheme, which connects each »input slot«
of the processing function with the corresponding ''lead node''
- the chaining constructor is picked reliably when the
slicing is done by a direct static_cast
- the function definition can be passed reliably in all cases
after it has been ''decayed,'' which is done here simply by
taking it by-value. This is adequate, since the function
definition must be copied / inlined for each invocation.
With these fixes, the simplest test case now for the first time
**runs through without failure**
This change allows to disentangle the usages of `lib::SeveralBuilder`,
so that at any time during the build process only a single instance is
actively populated, all in one row — and thus the required storage can
either be pre-allocated, or dynamically extended and shrinked (when
filling elements into the last `SeveralBuilder` currently activated)
By packaging into a λ-closure, the building of the actual `Port`
implementation objects (≙ `Turnout` instances) is delayed until the
very end of the build process, and then unloaded into yet another
`lib::Several` in one strike. Temporarily, those building functor
objects are „hidden“ in the current stack frame, as a new `NodeBuilder`
instance is dropped off with an adapted type parameter (embedding the
λ-type produced by the last nested `PortBuilder` invocation, while
inheriting from previous ones.
However, defining a special constructor to cause this »chaining«
poses some challenge (regarding overload resolution). Moreover,
since the actual processing function shall be embedded directly
(as opposed to wrapping it into a `std::function`), further problems
can arise when this function is given as a ''function reference''
...and as expected, this turns up quite some inconsistencies,
especially regarding usage of the »buffer types«.
Basically, the `PortBuilder` is responsible for the high-level functionality
and thus must ensure the nested `WiringBuilder` is addressed and parameterised
properly to connect all »slots« of the processing function.
- can use a helper function in the WiringBuilder to fill in connections
- but the actual buffer types passed over these connectinos are totally
unchecked at that level, and can not see yet how this danger can be
mitigated one level above, where the PortBuilder is used.
- it is still unclear what a »buffer type« actually means; it could
be the pointer type, but it could also imply a class or struct type
to be emplaced into the buffer, which is a special extension to the
`BufferProvider` protocol, yet seems to be used here rather to transport
specific data types required by the actual media handling library (e.g. FFmpeg)
__Analysis__: what kind of verifications are sensible to employ
to cover building, wiring and invocation of render nodes?
Notably, a test should cover requirements and observable functionality,
while ''avoiding direct hard coupling to implementation internals...''
__Draft__: the most simple node builder invocation conceivable...
Code clean-up: mark all buffers with a dedicated tagging type
The point in question is: if we work the LocalTag into the type-hash,
could it be possible to miss an existing entry in the metadata registry?
This could cause two entries to be locked for a single buffer address,
leading to data corruption.
As far as I can see, in the current usage this would not happen,
but unfortunately this problem can not be ruled out, since the BufferProvider
API and protocol is designed to be open for various usage patterns.
However, the same potentially disastrous pattern could also materialise
when registering two different buffer types, and then locking each
for the same buffer location.
...this is a surprisingly tricky issue, since it undercuts the
generic and recursive implementation of buffer handling;
fortunately I've foreseen such demands may arise down the road
and I've reserved an »Local Key« (now renamed into `LocalTag`),
whose meaning is implementation defined and interpreted by
the specific `BufferProvider`
It became clear that a secondary system of connections must be added,
running top-down from a global model context, and thus contrary to the
regular orientation of the node network, which connects upwards from
predecessor to successor, in accordance with the pull principle.
If we accept this wiring as part of the primary structure, it can be
established immediately while building the nodes, thus adding a preconfigured
''pattern of Buffer Descriptors'' to each node, since there is no further
''moving part'' — beyond the wiring to the `BufferProvider`, which thus
becomes part of a global `ModelContext`
As an immediate consequence, the storage for this configuraion should
also be switched to `lib::Several` and handled similar to the primary
node wiring in the Builder...
* conduct analysis regarding allocator handling in the Builder
* turns out we'll have to keep around two different allocators while building
* ⟹ establish the goal to confine usage of the Node allocator to the lower Levels
* consequently must open up the `lib::SeveralBuilder` to be usable
as an intermediary data structure, while building up the target data
* in the initial design, the `SeveralBuilder` was kept opaque, since
contents can be expected to be re-located frequently and thus exposing
elements and taking references could be dangerous — yet this is also
true for `std::vector` however, so people are assumed to know
when they want to shoot themselves into their own foot
...especially what is necessary to represent at this level and what information
is implicit; notably there will be an implicit default wiring, but we allow
for case-by-case deviations
To escape a possible deadlock in analysis, I resort to developing
some kind of free-wheeling presupposition how the **Builder** could
be implemented — a centrepiece of the Lumiera architecture envisioned
thus far — which ''unfortunately'' can only be planned and developed
in a more solid way ''after'' the current »Vertical Slice« is completed.
Thus I find myself in the uncomfortable situation of having to work towards
a core piece, which can not yet be built, since it relies heavily on
the very structures to be built...
...and this line of analysis brings us deep into the ''Buffer Provider''
concept developed in 2012 — which appears to be very well to the point
and stands the test of time.
Adding some ''variadic arguments'' at the right place surprisingly leads
to an ''extension point'' — which in turn directly taps into the
still quite uncharted territory interfacing to a **Domain Ontology**;
the latter is assumed to define how to deal with entities and relationships
defined by some media handling library like e.g. FFmpeg.
So what we're set to do here is actually ''ontology mapping....''
The immediate next step is to build some render nodes directly
in a test setting, without using any kind of ''node factory.''
Getting ahead with this task requires to identify the constituents
to be represented on the first code layer for the reworked code
(here ''first layer'' means any part that are ''not'' supplied
by generic, templated building blocks).
Notably we need to build a descriptor for the `FeedManifold` —
which in turn implies we have to decide on some fundamental aspects
of handling buffers in the render process.
To allow rework of the `ProcNode` connectivity, a lot of presumably obsoleted
draft code from 2011 has to be detached, to be able to keep it in-tree
for further reference (until the rework and refactoring is settled).
As outlined in #1367, the integration effort requires some rework
of existing code, which will be driven ahead by the `NodeLinkage_test`
* redefine Node Connectivity
* build simple `ProcNode` directly in scope
* create an `TurnoutSystem` instance
* perform a ''dummy Node-Invocation''
As a replacement for the `RefArray` a new generic container
has been implemented and tested, in interplay with `AllocationCluster`
* the front-end container `lib::Several<I>` exposes only a reference
to the ''interface type'' `I`, while hiding any storage details
* data can only be populated through the `lib::SeveralBuilder`
* a lot of flexibility is allowed for the actual element data types
* element storage is maintained in a storage extent, managed through
a custom allocator (defaulting to `std::allocator` ⟹ heap storage)
The `SeveralBuilder` employs the same tactic as `std::vector`,
by over-allocating a reserve buffer, which grows in exponential
increments, to amortise better the costs of re-allocation.
This tactic does not play well with space limited allocators
like `AllocationCluster` however; it is thus necessary to provide
an extension point where the actuall allocator's limitation can be
queried, allowing to use what is available as reserve, but not more.
With these adaptations, a full usage cycle backed by `AllocationCluster`
can be demonstrated, including variations of dynamic allocation adjustment.
...identified as part of bug investigation
* make clear that reserve() prepares for an absolute capacity
* clarify that, to the contrary, ensureStorageCapaciy() means the delta
Moreover, it turns out that the assertion regarding storage limits
triggers frequently while writing the test code; so we can conclude
that the `AllocationCluster` interface lures into allocating without
previous check. Consequently, this check now throws a runtime exception.
As an aside, the size limitation should be accessible on the interface,
similar to `std::vector::max_size()`
- decided to allow creating empty lib::Several;
no need to be overly rigid in this point,
since it is move-assignable anyway...
- populate with enough elements to provoke several reallocations
with copying over the existing elements
- precisely calculate and verify the expected allocation size
- verify the use-count due to dedicated allocator instances
being embedded into both the builder and hidden in the deleter
- move-assign data
- all checksums go to zero at end
The setup for `ArrayBucket` is special, insofar it shell de-allocate itself,
which creates the danger of re-entrant calls, or to the contrary, the danger
to invoke this clean-up function without actually invoking the destructor.
These problems become relevant once the destructor function itself is statefull,
as is the case when embedding a non-trivial, instance bound allocator
to be used for the clean-up work. Using the new `lib::TrackingAllocator`
highlighted this potential problem, since the allocator maintains a use-count.
Thus I decided to move the »destruction mechanics« one level down into
a dedicated and well encapsulated base class; invoking ArrayBucket's destructor
thereby becomes the only way to trigger the clean-up, and even ElementFactory::destroy()
can now safely check if the destructor was already invoked, and otherwise
re-invoke itself through this embedded destructor function. Moreover,
as an additional safety measure, the actual destructor function is now
moved into the local stack frame of the object's destructor call, removing
any possibility for the de-allocation to interfere with the destructor
invokation itself
part of the observed deviation stems form bugs in logging and checksum calculation;
but there seems to be a real problem hidden in the allocator usage of the
new component, since the use-cnt of the handle does not drop to zero
While there might be the possibility to use the magic of the standard library,
it seems prudent rather to handle this insidious problem explicitly,
to make clear what is going on here.
To allow for such explicit alignment handling, I have now changed the
scheme of the storage definition; the actual buffer now starts ''behind''
the `ArrayBucket<I>` object, which thereby becomes a metadata managing header.
__To summarise the problem__: since we are maintaining a dynamically sized buffer,
and since we do not want to expose the actual element type through the
front-end object, we're necessarily bound to perform a raw-memory allocation.
This is denoted in bytes, and thus the allocator can no longer manage
the proper alignment automatically. Rather, we get a storage buffer with
just ''some accidental'' alignment, and we must care to request a sufficient
overhead to be able to shift the actual storage area forward to the next
proper alignment boundary. Obviously this also implies that we must
store this individual padding adjustment somewhere in the metadata,
in order to be able to report the correct size of the block later
on de-allocation.
The solution implemented thus far turns out to be not sufficient
for ''over-aligned-data'', as the raw-allocator can not perform the
''magic work'' because we're exposing only `std::byte` data.
This adaptor works in concert with the generic allocator
building blocks (prospective ''Concepts'') and automatically
registers a either static or dynamic back-link to the factory
for clean-up.
Use this wrapper fore more in-depth test of the new `TrackingAllocator`
and verify proper behaviour through the `EventLog`
- ability to verify a hash-checksum
- ability to watch number of allocations and allotted bytes
- using either a common global pool or a separate dedicated pool
- log all operations into a common `EventLog` instance
- front-end adaptors for use as C++ custom allocator
...these features are now used quite regularly,
and so a dedicated documentation test seems indicated.
Actually my intention is to add a tracking allocator to these test helpers
(and then to use that to verify the custom allocator usage of `lib::Several`)
Phew... this was a tough one — and not sure yet if this even remotely works...
Anyway, the `lib::SeveralBuilder` is already prepared for collaboration with a
custom allocator, since it delegates all memory handling through a base policy,
which in turn relies on std::allocator_traits.
The challenge however is to find a way...
* to make this clear and easy to use
* to expose an extension point for specific tweaks
* and to make all this work without excessive header cross dependencies
This is a low-level interface to allow changing the size of
the currently latest allocation in `AllocationCluster`; a client
aware of this capability can perform a real »in-place re-alloc«,
assuming the very specific usage constraints can be met.
`lib::Several<X>` will use this feature when attached to an
`AllocationCluster`; with this special setup, an previously
unknown number of non-copyable objects can be built without
wasting any storage, as long as the storage reserve in the
current extent of the `AllocationCluster` is sufficient.
...use some pointer arithmetic for this test to verify
some important cases of object placement empirically.
Note: there is possibly a very special problematic case
when ''over aligned objects'' are not placed in accordance
to their alignment requirements. Fixing this problem would
be non-trivial, and thus I have only left a note in #1204
...including the interesting cases where objects are relocated
and the element spread is changed. With the help of the checksum
feature built into the test-dummy objects, the properly balanced
invocation of constructors can be demonstrated
PS: for historical context...
Last week the "Big F**cking Rocket" successfully performed the
test flight 4; both booster and Starship made it back to the
water surface and performed a soft splash-down after decelerating
to speed zero. The Starship was even able to maintain control
in spite of quite some heat damage on the steering flaps.
Yes ... all techies around the world are thrilled...
- spread change now retains the nominal element reserve
- `capacity()` and `capReserve()` now exposed on the builder API
- factor out the handling check safety functions
- rewrite the `resize()` builder function to be more generic
__Test now covers__ example with trivial data type, which can
indeed be resized and allows to grow buffer on-the fly without
requiring any knowledge of the actual type (due to using `memmove`)
building on the preceding analysis, we can now demonstrate that
the container is initially able to grow, but looses this capability
after accepting one element of unknown subclass type...
`lib::Several` is designed to be highly adaptable, allowing for
several quite distinct usage styles. On the downside, this requires
to perform some checks at runtime only, since the ability to handle
some element depends on specific circumstances.
This is a notable difference to `std::vector`, which is simply not capable
of handling ''non-copyable'' types, even if given an up-front memory reservation.
The last test case provided with the previous changeset did not trigger
an exception, but closer investigation revealed that this is correct,
since in this specific situation the container can accept this object type,
thereby just loosing the ability to move-relocate further objects.
A slightly re-arranged test scenario can be used to demonstrate this fine point.
- the test-dummy objects need a `noexcept` move ctor
- **bug** here: need an explicit check to prevent other types
than the known element type from ''sneaking in''
The `SeveralBuilder` is very flexible with respect to added elements,
but it will investigate the provided type information and reject any
further build operation that can not be carried out safely.
...turns out that we must ensure to pass a plain "object" type
to the standard allocator framework (no const, no references).
Here, ''object in C++ terminology'' means a scalar or record type,
but no functor, no references and no void,
Consider what (not) to support.
Notably I decided ''not to support'' moving out of an iterator,
since doing so would contradict the fundamental assumptions of
the »Lumiera Forward Iterator« Concept.
Start verifying some variations of element placement,
still focussing on the simple cases
Parts of the decision logic for element handling was packaged
as separate »strategy« class — but this turned out to be neither
a real abstraction, nor configurable in any way. Thus it is better
to simplify the structure and turn these type predicates into simple
private member functions of the SeveralBuilder itself
...and the nice thing is, the recently built `IterIndex` iteration wrapper
covers this functionality right away, simply because `lib::Several`
is a generic container with subscript operator.
...passes the simplest unit test
* create a Several<int>
* populate from `std::initializer_list`
* random-access to elements
''next step would be to implement iteration''
Some decisions
- use a single template with policy base
- population via separate builder class
- implemented similar to vector (start/end)
- but able to hold larger (subclass) objects
