...which happens when a buffer type is referred for the first time
from within a new worker thread, while it is already stored in the
global registry; the full key-chain is down-synced then.
...using the first usage situation as guideline,
and verify proper operation through unit-test
- the LocalBufferStage gets a set of thread-local BufferMetadata
- furthermore, it accesses the central hub via dependency-injection
- after each type registration ''from a worker thread'' data is published
As an asside: ensure most metadata registration can ''move'' the TypeHandler
This test setup defines one of the problematic situations
where a synchronisation is necessary.
__TODO__: actually add locking and that synchronisation to LocalBufferStage
What I am building step by step right now is the first draft
of a production-grade render buffer management scheme.
For this I need suitable implementations of `BufferStag` (and `BufferStore`)
These will include a `LocalSlice` to hold thread-local »satelitte services«,
which in turn need a connection to some central coordination hub.
The thread-local instances can access the hub via our `lib::Depend` (I have just
verified that this works as intended, with on-demand creation of the
central instance on first access from some thread. This is important,
since very likely we need to install a dedicated and speficically wired
service instance, which must happen prior to first dependency-request
via `lib::Depend`. And since the worker-threads are created by the Engine,
we get ample opportunity to do so before starting those threads.
Included in this changeset is a first rough draft of the central registry
`EngineBufferMetadata` — I have modelled it analogous to the `SimpleBufferStateRegistry`,
which is the test-level implementation of the `BufferStage` API.
However, we do not want to expose direct buffer allocation operations
on the central hub, and thus I have left out the respecive parts of the API.
Furthermore, I have added an `abandon()` operation, without there beeing
a need for such an operation right now. But since this operation is tricky
and potentially dangerous, I prefer implementing it the correct way,
since I have a clear understanding what this entails at the moment.
The recursive synchronisation up the parent-chain can in most cases be skipped,
by relying on the fact that inconsistent data is never stored in the BufferMetadata
table. If thus an entry is locally present and matches the entry from the
other side, then it can be concluded that its parent must also be already in sync.
Since the intention is to run several thread-local instances of `BufferMetadata`
in a concurrent setup, we need a way to publish and exchange chains of
registered type-keys and metadata entries.
The first step is to develop a mechanic for data exchange,
and to define the rules when such synchronisation is (not) allowed.
Since thread_local implies a static setup, we need a way how to
build a new service instance. Usually this would be a default construction,
yet there might be services with special init requirements.
Using an explicit template parameter for this setup configuration
allows to make the primary template parameter just the service itself,
which makes the default usage much clearer.
- Use a ''monostate'' as access point to thread-local service instances
- rely on a configuration-context to setup the actual service
- investigate runtime overhead using a multithreaded microbenchmark
During the last days, I started with analysis to find a way
to operate the `BufferProvider`, as implemented thus far,
within a massively concurrent environment.
Since I intend to create thread-local caches,
the initialisation of the `BufferMetadata` store must be static.
It turns out that the root-ID, which acts like a mandator ID,
is only used at a single code path, when creating or accessing
an abstract type-Key. It is thus easy to extract and relocate
this distinguishing ID into the actual `BufferStage` implementation.
Looking through all my work notes, it seems that all the
problematic aspects are now solved in a satisfactory way.
* it is clear how to dependency-inject a `BufferProvider`
* the internals of the `BufferProvider` front-End have been restructured
* there are now two ''pillars'' of the implementation: `BufferStage` and `BufferStore`
* both can be subclassed independently
* `BufferProviderSetup` exposes a proxying-mechanism (used for `DiagnosticBufferProvider`)
* the `BufferMetadata` have been reviewed and some minor details were rectified
* after a re-Design of `OutputSlot`, `BuffHandle` can now be used as unified front-End
* both `DiagnosticBufferProvider` and `DiagnosticOutputSlot` have been rewritten
* they are now independent, self-contained specialisations, layered on top
Overall, what might seem like a detour, was an important step towards
well-balanced structures and a better understanding of the Render Engine.
The actual mystery, that could ''not'' be clarified,
is why that test did pass previously. Because it was
always broken, insofar it pretended that the output buffer
does hold a valid test frame, while in fact the buffer is
just newly allocated.
The deeper reason behind this mismatch lies in the concept of
''in-place processing'' — which is a very obvious, nullary optimisation,
and thus we want our engine to be ''in-place capable'' in the long run.
Which explains why I did write the dummy-test operations with in-place
processing in mind. Yet the actual version of the render engine leaves
out a lot of future improvements, amongst others the in-place processing.
This implies that — at least at the moment — it is clearly the responsibility
of the processing function to prepare the output buffer, in cases where such
preparation is necessary (and this applies in the case of `TestFrame`,
which needs to be primed with a valid header mark)
This change touches on another unsolved problem in the `TestDomainOntology`:
we need a mechanism to allow the generic builder front-end to set
specific configuration for some kind of processing. Generally speaking,
such things are possible in C++, yet somewhat tricky — and thus I postponed
that aspect and just included a configuration variant, hard-coded for the moment.
After an extended round of analysis and cascading reworks,
the long-standing discrepancy between `OutputSlot` and the
''Buffer Provider Protocol'' (#1395) has finally been resolved.
The foundation for this success was laid by means of a restructured
implementation within `BufferProvider` – which has now been
successfully completed and validated.
__note__: regarding hash-standard.hpp see Ticket #1422
Mostly this hack worked -- but it is very likely not needed any more.
Today I had to re-wire a partial specialisation for hashing pointers,
that was added somewhere in the past into the STL.
Now able to reap the rewards: since all that convoluted and tangled
old ''state tracking'' has been resolved and replaced by dedicated
and self-contained diagnostic subclasses, the old dummy BufferProvider
implementation can be detached, using a similar yet very simple
fake implementation.
For context: the `HeapMemBufferStore` is part of the default configuration
and thus automatically used whenever a unit test needs a `BufferProvider`.
This default implementation is not optimised for speed, and actually
never releases any memory, which allows to verify proper handling
of buffer memory allocation.
''This is a breakthrough...''
Using the realigned implementation structure in `BufferProvider`,
and especially the ability to establish an overarching context by
passing an connection-ID through the `LocalTag` in the `BufferMetadata`,
it was possible to resolve the longstanding mismatch buried deep within
the `OutputSlot` implementation.
Based on the new internal APIs in `BufferProvider`, it was thus possible
to setup an `OututBufferProxy` that acts as a hub to connect the
`DataSink` handles with the actual `OutputSlot::Connection` implementation.
This change allowed to rectify the convoluted control structure in
`OutputSlot` and finally to manage the Lifecycle properly. Furthermore,
one level of unnecessary implementation-inheritance could be removed,
so that it is no longer necessary to subclass `OutputSlot` in order to
configure the link to a specific output backend.
Using this redesigned internal structure, it was now fairly simple to
re-implement a tracking `DiagnosticOutputSlot` and then to verify proper
integration by passing the `OutputSlotProtocol_test` again.
This is a design study to find out what would be required
to build an opaque function wrapper, similar to `std::function`,
yet without using any heap memory ever.
Such a building block would allow for some interesting options
for a well structured design of elaborate data structures.
This changeset provides an implementation draft.
__Two important findings to note__:
* it is impossible to get away with just a single function pointer
* because at the very least we need to invoke the destructor of the
embedded delegate. Thus something akin to a "manager" is required.
* this implies that the component would not be so efficient as I hoped,
since it requires two pointers and requires to emit code for each
type of delegate.
* by making the component **NonCopyable**, I avoided the complexities
of virtual copy support. In case the latter is desired, it would be
much simpler to switch to a classical OO approach with a VTable
and use our `PolymorphicValue` wrapper to handle copy and cloning.
This commit is the aggregated result of extended research
that is only loosely related to current development topics.
One theme was the construction of nested data structures using tuples,
which faces some difficulties due to tuple manipulations being limited
almost entirely to compile-time processing. However, the barrier between
compile-time and runtime can be overcome with the help of a dispatcher
table -- which leads to the secondary problem to handle the diverse
tuple elements with a common, generic functor. Such a functor can be
constructed, if and only if those element types are ''reconcilable''
The STDLIB provides two metafunctions, `std::common_type` and
`std::common_refernce`, which however pose some challenges for the
task at hand, since they include additional processing. In the end,
a custom version of the ''basic type reconciliation'' was implemented
and tested, and then used as foundation to build a dispatcher table
for runtime-indexed access and iteration over tuples.
As an aside, several deeply hidden bugs in our library code were resolved,
and the tuple-helpers, tuple string rendering and associated utilities
were regrouped and modernised.
As result of this research work, the following components are provided:
- meta/common-result
- meta/fun-trampoline
- meta/tuple-index-adaptor
The detailed analysis of corner cases indicated a lot of hidden dangers
when stretching the `IterExplorer` design beyond its limits.
By deliberate choice, this design was kept open so that advanced cohesive
implementation structures can be built. It is even possible ''and tolerated''
to rely on ''references'' to some degree -- assuming however that the
referred memory location remains stable.
As it turned out, these basic assumptions can be broken by a combination
of read-ahead processing (as is necessary for grouping and filtering),
together with adaptor objects embedded as value, especially when
the resulting pipeline object is then moved or copied during evaluation.
While my general attitude towards such extremely clever implementation
constructs is "don't shoot yourself into your foot unless you need to" --
it seems prudent to clean-up some crappy code from the first attempts
towards an iterator pipeline -- notably some of the »Itertools«.
This expands the processing scheme developed thus far, so that
an arbitrary nested iterable can be handled, as long as it can be
adapted by the generic mechanisms provided by IterExplorer.
Notably there is now a type pattern match to distinguish various cases,
even spanning the ''rather esoteric'' possibility that an underlying
iterator produces STL collections by-value.
By examining these special cases, the limitations of the `IteratorExplorer`
design were highlighted, and a number of tricky bugs could be fixed.
Since the new `flatten()` operation was implemented in the `IterExplorer` framework,
some non-obvious corner cases must be considered; notably the source iterator
might yield something that is not by itself a Lumiera Iterator, yet could be
adapted by applying those generic mechanisms already in place to handle such cases;
however there is the special twist in the case at hand that we do not have to adapt
the ''source iterator'', yet rather the ''product'' of the source iterator —
a requirement that, once understood, can be easily adopted with
yet another iterator-decorator layered below.
This is a feature I wasn’t sure for a long time how best to tackle.
It’s worth noting that an initial attempt at `IterExplorer` was based
on the monad pattern, which turned out to be a mistake, since monads
are more of an anti-pattern, than being genuinely useful.
The actual use-case that made me reconsider such a feature it
to build an seamlessly chained sequence from several iterators.
It would be simple to develop a suitable utility as a stand-alone helper,
yet it seemed worth the effort to integrate this into the `IterExplorer`
ecosystem, due to the flexible automatic adaptation to iterable source
pipelines provided therein.
Going this route incurred some cost however, unveiling several defects
related to handling nested iterable types in the generic processing framework.
On occasion of the recent improvements, I noticed that somewhere
in the past I had misunderstood the meaning of the typedefs in iterators:
* `reference` is the central definition; this is what the iterator yields
* `value_type` is secondary; this is the type you use to store results
So at times I defined only a `value_type`, wrongly assuming this to be
the primary definition, and all the other ones to be ephemeral.
Consequently the trait to detect a »Lumiera Forward Iterator« should be sharpened...
...which unveils a long-standing bug in our ''Duck Detector''
(it is not possible to define a pointer to a reference, causing
a check for the `reference` typedef erroneously to fail.
- use uniform naming scheme
- extract type-binding from RangeIter into a trait-template
- use the C++20 iterator concept and trait to detect a STL iterator
After some investigation, I seem to understand the problem better now.
This builds upon the Introduction of our `ValueTraits´ two years ago:
e176e54004
The root cause to these ongoing sporadic problems with type deduction
seems to be that we encounter a **clash of concepts**: What Lumiera calls
an »Iterator« is essentially different to the order of terms used by the STL.
In the STL, an iterator can be seen as an ''abstracted pointer'', and is
thus only meaningful in conjunction with a backing container. It is also
this backing STL container that provides the value-type definitions.
In Lumiera however, an ''Iterator is a promise to retrieve a sequence of evaluations''.
For that reason, the Iterator itself provides value-type definitions,
and the notion of "iterable" can largely be conflated with "Lumiera Iterator".
Now, the `lib::RangeIter` can act as a bridge between both usage patterns,
and thus it is possible to create a `RangeIter` both from a Lumiera Iterator
or from a STL container. (As an aside, this is a similar idea as promoted
by the "Range TS" in C++20 -- yet the Lumiera iterator concept is much more
abstract and allows evaluations like tree expansion and search algorithms,
which would be never possible with the STL based notion of iteration)...
Thus the value type binding in `RangeIter` is the only correct place
where to mitigate this ''clash of concepts:''
* if the `IT` parameter is a Lumiera iterator, we need to retrieve
the type bindings defined therein
* otherwise, we can assume that `IT` is an STL iterator and thus
essentially a pointer, and we must strip that pointer and work from there.
...to emplace some object into the buffer space.
I wouldn't expect that to be necessary for the real usage,
but it comes in handy for tests, where we often want to place
some sentinel value into a buffer.
- visible Signature of TestFrame changed due to being relocated into the Vault
- attempt to make the WorkForce_test more robust;
frequently fails on testsuite run after full build;
so better ''wait'' for the workers to have started working,
instead of using a fixed delay (that must be extremely short
and is hard to predict)
Straight-forward:
- add a map keyed by observed frame numbers
- individual connection needs to know its own seqNr :-(
- also need to store and recall the frameNr in the buffer-Index
Fix bug with DataSink constructor
(value constructor shadows copy constructor => Stackoverflow)
''Complete construction and wiring sequence works as intended''
- can now throw away the old convoluted `TrackingInMemoryBlockSequence`
- instead use a simplistic implementation of the Connection interface
- do not perform any kind of validation, allocate new buffer for each frame
- use the `NaiveBufferSetup` (`BufferProvider`) as implementation backend.
This will be the first ''reality test'' for the new `OutputSlot` internal design:
Is it possible to implement a tracking test implementation in a clean
and straight-forward way, avoiding all the tangling and indirection
that plagued the old prototype implementation from 2012?
After resolving the inner tensions in the old prototype from 2012,
it was now possible to resolve all issues in a single strike
* `DataSink` becomes a functor, yet embeds a shared ref-count
* in the λ-closure a ''specific Buffer Descriptor'' is prepared for each feed
* building upon the prototyping work, a dedicated `OutputBufferProxy`
exposes a `BuffHandle` to the client, with `emit()` and `release()` calls
* the implementation ''delegates directly'' to the `OutputSlot::Connection`
* an this is where all actual output technology is encapsulated
...this is the gordian λ
Note
- embedding a copy of the shared-ptr in the closure
- pre-configure a buffer descriptor (buffer type)
* with the buffer size as retrieved from the connection
* and pass the connection itself marked as LocalTag
Thus, on invocation of the DataSink (functor), it suffices
to invoke the lock() function on the BufferDescriptor,
using the new additional argument introduced with the
preceding commit
Reaping the benefits of the preceding refactorings, most of the complex
wiring within `OutputBufferProxy` can be omitted, and the proxy calls
can be delegated directly to the actual `Connection`
There is a catch however: we need to create an additional pathway
to feed the frame-number information into the `BufferStore` backend.
The solution adopted here is the result of extensive reflection.
Taken literally, or naively, the reworked architecture would imply
to route every call through a delegation-chain that is dispatched
in the state-managing "PImpl" (`AllocState<CON>`).
However, in keeping with the idea of an opaque functor embedded into
`DataSink`, direct and explicitly typed calls can be bound in the
associated closure. Doing so will strengthen the role of the
interface `Connection` — which defines the boundary to the
actual domain of output technology.
This changeset attempts to breach into the implementation design of OutputSlot
(while largely retaining the API and the »Output Slot Protocol«).
While the actual problem that brought me to this point can be traced back to some
rather obvious mistake, namely to conflate the `BufferProvider` used as
backing implementation with a `BufferProvider` used as front-end API,
creating the new connections to use an `OutputBufferProxy` unveils some
further long-standing structural mismatch; notably this leads to the
difficulties to get the lifecylce management of `OutputSlot` and `DataSink`
under control.
So I remove one entire level from the implementation,
in a bold move to realign the design with the prospective usage pattern.
Obviously, this breaks a lot of further code, and I need to rewire
the Allocation implementation altogether, and provide a new functor based
implementation for `DataSink`
After a lot of pondering over various solution schemes,
it seems that `BufferProxyAdaptor` is too rigid for the task at hand.
It could be retained as a technology demonstration, and used as blueprint
for a custom-made implementation. In a nutshell, we need a `BufferStore`
that directly delegates to the respective `Connection` — which also implies
to use the `OutputSlot::AllocState` as an intermediate to hendle that delegation.
...which still looks like a nasty task
* the good part is, we have a working `BufferProxyAdaptor` now
* yet this setup does not seem to fit into the `OutputSlot::AllocState`
As an aside, several test definitions need to be relocated to the Vault,
since the whole compound around `BufferProvider` migrated down
This is a nasty new requirement, discovered as an aside while looking
into ways to resolve the ''crucial mismatch'' between the `OutputSlot`
and `BufferProvider`, using the proxy-implementation of `BufferProvider`
(thereby expanding on the recent draft with the `BufferProxyAdaptor`)
The point is, if the BufferMetadata will be used beyond the internal coordination
within BufferProvider, then it will be used as part of actual buffer storage.
And this implies that the actual size of the allocation may differ (be greater)
than the nominally required size (since we'll very likely employ some kind
of memory tiling, for reasons of performance).
This leads again to an analysis of `BufferMetadata` usage,
with the conclusion to add the actual storage size directly into the API,
but treat it as an ''extra'' and not integrated into the ''buffer type''
Phew ... quite complex alltogether
This might have seemed like a good idea, adding an additional ''consistency check''...
However — we ''do have'' a generic check, insofar `BufferMetadata` is a **state engine**.
As long as we're able to prevent, reliably, that the same buffer address
ends up being classified as a different buffer type, this should be enough
to prevent double-use of any buffer (letting aside that `BufferProvider` is
backed by an implementing memory allocator, which in itself should prevent
any double-use.
Removing this feature resolves some ambiguity in the overall design,
and allows to turn `BufferMetadata::lock(type, address)` into a private
implementation function.
In the old code, there were some external access paths directly entering
into that implementation function, which caused a lot of confusion (at least for me...)
This refacotring prepares for the planned changes in `OutputSlot`
* introduce a new compound parameter (tuple) BufferAlloc
* wire this through all calls involved with storage handling
__Background__:
* `BufferMetadata` deemed important for the implementation of buffer storage
* ''new requirement detected'': must be able to handle an actual storage block
that is ''larger'' than the requested buffer size (⟼ tiling)
* need compact and uniform info, also for the `BufferProxyAdaptor` callbacks
There was an old UML diagram created at the time of the first draft.
It is integrated into the »OutputSlot« page in TiddlyWiki. This description
and the diagram are highly relevant for understanding the design in this case.
Unfortunately, Umbrello was "improved" with a lot of features since I used it
for the last time, but the **quality** of the software has been **degraded**,
it is now buggy and the layout is imprecise and gets messed up easily.
__Warning__: the old diagrams look ''slighly off alignment'',
this is very likely due to some change in font handling.
I have set the default font now to 'DejaVu Sans Condensed' 9pt
The old implementation relied on two layered APIs:
one for the client and one for the implementer of
the Allocation state. This is logically sound, yet
clumsy in terms of design, since the internal API
turns out to be void of actual meaning.
It seems possible to eliminate this confusing structure
by a clear separation of ''implementation realms''
* the ''mechanics'' of connection state management
* the actual implementation of the connection
The first can be provided as generic template subclass,
while the latter should be confined entirely into the
implementation of the `Connection` interface.
Doing so requires to keep each part of the implementation
strictly focused onto its purpose — something I obviously
was not able to see at the time when writing this old
implementation, notably because I was completely guided
and driven by the test/dummy implementation without
considering how an actual implementation of
low-level data output would work.
This API function was the reason why the handling of
`OutputSlot` and `BufferProvider` do not mesh up well,
so that the former could not yet be integrated properly
into the engine.
''Obviously the proper solution still needs to be implemented''
The state of connectedness is represented as class,
which actually is a three step deep hierarchy.
This design seems plausible, since we get
- an interface visible for clients (`Allocation`)
- an internal interface
- the actual implementation, templated to the actual connection
At least we can try to choose names that better indicate
these three related levels.
The old implementation of `DiagnosticOutputSlot` is now disabled
and will be re-written from scratch, and no longer tangled to `BufferProvider`
- `OutputManager` remains in Steam-Layer for now
- the `Timings` and `OutputSlot` are now a self-contained
One of the goals of this extended clean-up and refactoring is to allow
access to the data buffers of a `DataSink` through a `BuffHandle` as front-end.
Which essentially means that the ''Buffer Provider Protocol'' and the
''Output Slot Protocol'' mesh together well.
After the clean-up of the `BufferProvider` implementation hierarchy,
the viability of this idea was demonstrated with the `BufferProxyAdapter` draft;
so it is now roughly clear what must be changed within the existing implementation
of `OutputSlot`. More specifically, the only actual implementation ever completed
was a dummy implementation for test (`DiagnosticOutputSlot`), which hooked
confusingly into the corresponding dummy implementation of `BufferProvider`,
that was written one year earlier (and was also without actual implementation
at that time).
So the second half of the task ahead seems very similar to the first part:
- the base implementation of the ''Output Slot Protocol'' seems roughly acceptable
- but the actual dummy implementation was somewhat confused and should be
re-written from scratch, without ''any attempt to cut corners by re-using other dummy code''
__Note__ as part of this analysis I discovered some gap in the design
regarding timeout handling ⟶ Ticket #1417
building upon all the preparations done during the last days,
the compound of code connected to the `BufferProvide` can now
be migrated down into the vault:
- `BuffHandle`
- `BufferProvider`
- `BufferMetadata`
- `BufferProviderSetup`
- the simple standard implementations
- `DiagnosticBufferProvider`
- all directly related unit tests
Gradually, the header buffhandle.hpp became the minimal include
to use basic `BufferProvider` functionality, since `BuffHandle`
provides proxy functionality to lock and access a data buffer.
The whole compound connected to the `BufferProvider` has evolved
into a cohesive yet central connection hub related to buffer memory;
it should thus be extracted, moved down to »Vault« layer and used
4as a foundation for various high-level tasks. In preparation of
this step, relevant marker typedefs are re-arranged;
notably ''Stream Type'' now includes `buffhandle.hpp`
