The test suite is now **green** again.
The design rework and clean-up of unnecessary test-driven functionality
in the `DiagnosticBufferProvider` and `DiagnosticOutputSlot` required
to break some very old test-setup, using fake instances of the relevant
context facilities. This setup however was used by the newer family
of test support facilities, notable the `MockJob`, `MockSegmentation`
and `MockDispatcher`. These required a set of `DataSink` handles to be
present (even while they are not used currently).
This could be fixed easily now, relying on the simplified internal structure
of the `OutputSlot`: by re-arranging the types, the implied identity of the
`DataSink` handles could be made explicit, allowing for equality comparison.
__Remark__: even while `DataSink` with only a single abstracted function,
using function notation turned out to be rather confusing at usage site.
Naming this function as `lockFrame(frameNo)` clarifies its purpose nicely.
That call is effectively abstracted by means of the ''buffer type'',
relying on the VTable in the `BufferProvider` implementation.
- some tests are notoriously flaky
- increase resource limits for the testsuite to allow stronger stress tests
- provide a "quick" variant of BufferProviderLoad_test with lowered requirements
Furthermore, investigate test scaffolding in `DummyBuilderContext`,
which was broken due to the clean-up in `DiagnosticOutputSlot`.
It seems that only some placeholder instances are required here,
so that we might be successful with just instantiating the `UnimplementedConnection`
__TODO__: need to supply slightly more implementation into the latter,
so that we're able at least to ''create'' such a connection stub.
Within the reworked design of the `OutputSlot` a proxy implementation
of the `BufferProvider` API is used (class `OutputBufferProxy`).
After analysis of the invocation patterns, I concluded that it seems adequate
to attach the special-purpose metadata generated by this integration
to the generic system of thread-local replicated `BufferMetadata` storage.
However, since thread-local instances are effectively static(global),
I was concerned that unit-testing might be impeded by accumulation of
stale metadata in the store linked to the main thread. Thus I created
a special configuration variant to retain the previously used setup
for the sake of unit testing (which applies especially to use of the
`DiagnosticOutputSlot`). Using a `SimpleBufferStateRegistry` for in
that case has the benefit that simply a local instance can be used
in the test and all metadata will be discarded afterwards.
Furthermore the factory syntax for obtaining an `OutputSlot` was improved,
drawing on the recent changes to `lib::ScopedCollection`.
Also I looked through some of the documentation and updated
the UML sequence diagram to reflect `OutputSlot`'s improved internal structure
With this third test case, a fairly continuous processing
can be demonstrated, while also coordinating allocations
in a decentralised way.
The implementation behaves correct, and seemingly with
satisfactory performance -- while there is certainly
some room for improvement: with the processing pattern
in this test, a nominal computation load of 4 seconds
can be processed with 4 cores in ~ 1.3 seconds
Yet only in a simple way, by pressing with 100 threads....
This test runs for about 10 seconds, shows a massive allocation
of more than 1 GiB, but comes down clean, without deadlocks,
using all cores and returning all memory blocks at the end.
This test uncovered some missing safety checks,
and it highlighted (again) that this memory management scheme is
elaborate and potentially fragile: I had to deal with carried-on
state in the local metadata registry associated with the main thread
(in fact I did not ever want that registry to be used, but turns out
that the initialisation check, that runs in the constructor, already
pulls up an instance for the main thread, which carries stale state
after the test case is complete)
This is the meat of this test case...
- generate a ''random walk'' with buffer allocation requests
- attempt to make the load pattern nasty, since the goal of this test
is to verify correctness of the complex interplay between local pools
and central allocator.
''correct pattern was verified single-threaded, with print statements''
- provide mock instances of the relevant global services
- use a base config record for the worker thread(s)
- pregenerate ''Buffer Descriptors'' commonly used by the workers
- further access to the buffer manager should then be possible
by locking one of these descriptors...
The intention is to utilise this helper class, that was developed for the
scheduler stress testing; but this time, it should access the actual allocated
render buffers to generate the load.
This implies to use buffers of a significant size; it turned out that
the existing implementation was imprecise, insofar it always used full multiples
of the configured buffer size, which defeats the purpose of performing
a precisely calibrated number of calculation steps. As long as the buffer size
would remain comparatively small, this drift does not matter much.
Thus I changed the implementation now to ensure that exactly the calibrated
number of steps is performed, using a modulus to map it to the buffer memory.
Furthermore, I plan to use a random-walk pattern in the worker thread
to select from a collection of pre-established »buffer types«, similar
to what happens in the actual engine
To verify integration of this new engine-level facility,
it seems prudent to build a load test (without going into
extended stress- and performance testing).
For that we need
- random buffer demand
- a computational load that actually uses the buffer.
As part of the ''scheduler stress testing'', a self-calibrating
`ComputationalLoad` was developed, which even implements both a
merely computational load, but also a memory accessing computation.
This can be extended to optionally use an external buffer, instead
of allocating a precisely dimensioned memory block directly from heap.
Obviously this extension is ''dangerous'', since there is no way to
verify that the storage passed in can accommodated the storage required
based on the current calibration of the `ComputationalLoad`
As an aside: identified a timing check in `SchedulerService_test`
that showed the tendency to fail when invoked immediately after a build.
The mechanism is tuned now to engage rather quickly
whenever excess capacity builds up in a local allocator pool,
but when all buffers are used most of the time, they will be retained.
Can now demonstrate by integration test that capacity is sent back.
The goal is to develop a heuristic when to trigger clean-up of the local pool.
For developing this heuristic, I'll use a unit-test
- represent "allocations" by dummy buffer pointers
- use a queue and a stack to push those "allocations" around
- the "global pool" is just a queue with such dummy pointers
- next step is to define some λ-primitives
- finally, a λ-DSL to allow to perform "N operations of kind X"
This is a fall-back situation,
in case the allocation ''was not announced beforehand''
A direct locked global access is used then, which is verified by this test
...need an elaborate test setup, with the test worker thread
and the main thread proceeding in lockstep, otherwise the
effect would not be visible. Furthermore, the diagnostic
access for LocalMemPool can be re-used here, thereby
mapping requests onto the pool for the current thread.
This makes the test case pass, validating the theory.
Each `AllocReceiver` now integrates a `lib::LiveMark` instance,
and the callbacks are dispatched using the corresponding
`LiveMark::SafeHandle`, so that it is no longer possible
to dispatch allocation messages to a defunct receiver.
Furthermore, the `LiveMark` is explicitly disengaged prior
to unwinding a local pool, so that no further pending requests
can be handled ans dispatched concurrently. The local pool
sends back all the remaining allocations via async queue
to the `EngineBufferManager`
Not that such a shutdown can be triggered any time,
since an exception or failure might cause a worker thread
to unwind, thereby destroying the thread-local `LocalMemPool`
without further notice.
...even while this situation is unlikely, it needs to be considered:
A worker thread might terminate unexpectedly, due to an error;
all thread-local objects will be destroyed automatically — while the
`EngineBufferManager` might be processing further allocations requested
by that very thread, and dispatch them towards an already destroyed `LocalMemPool`
This turns out to be a nasty problem, since the whole setup was created
to allow the workers to operate independently from each other; any handshake
or registration/deregistration mechanism would require a lock of sorts,
thereby defeating the whole purpose of the setup.
The best solution I could find is to rely on a instance marker hash,
that can be checked with an atomic read prior to each dispatch of a
memory allocation. The destructor of the local pool would then be able
to ''disengage'', by invalidating that hash.
To evaluate this solution, I developed this mechanism
as a free-standing library component, called `LiveMark`
Most relevant parts of the `LocalBufferStore` are already available
and just need to be connected. One point poses a challenge however:
when and how to trigger the heuristic clean-up of the local pool,
so that ''on average neither too much memory is wasted, nor
to much unnecessary allocations'' are performed...
Signed-off-by: Ichthyostega <prg@ichthyostega.de>
This is a collection with unique-ownership,
and the ability to create all content objects in ''RAII style''
from the constructor call, using a ''population functor''.
It is often used to maintain a collection of resource handles,
and especially to start a series of threads (for testing).
With the variadic λ-captures from C++20, it is possible to rewrite
the most frequently used population functors to be easier to use.
Notably we can now bind constructor arguments, that will be passed
to each of the created elements — a prominent example being the
thread function when launching several threads. Furthermore,
I added a constructor variant that consumes a Lumiera Iterator.
Adding this kind of generic variants is possible now with C++20,
since overload resolution takes concepts into account.
The test performs a massively concurrent scenario
that presses with 100 threads and 1 MiB allocation for each.
This highlighted my still limited understanding of the situation
and leads to a much better solution, where, on-demand, some worker
performs all the accrued allocation and management work.
So effectively we do not even need to use the synchronous API
Implementing the `EngineBufferManager` the desired way,
with the ability to post allocation requests asynchronously,
raises the difficult question of who will ultimately perform
the allocation / deallocation and further management work.
The point is that this management effort takes up capacity
originally assigned for the render workers themselves, either
by implicitly pushing aside those threads, or as a dedicated
processing assigned to some worker.
In the current stage of development it seems impossible to decide
upon the proper mechanism to schedule this excess work explicitly.
The solution proposed hereby however circumvents this dilemma,
insofar it allows to implement the synchronous, fallback processing
in terms of the asynchronous processing, just performed exclusively
within the current thread. Unless some processing will be scheduled
explicitly (which can be added as performance optimisation later on),
the handling of a direct allocation request will ''fold down'' into
the current worker eventually, thereby using it's processing capability
without the need of any explicit scheduling.
While all of this aggregate can be considered ''deep implementation'',
the structure of dependencies bears danger to get confusing. Even more so,
since an elaborate pooling memory allocator will be added to and interconnected
with this structure.
Now considering the following route:
- `engine-buffer-allocator.hpp` has elements of a common service / interface
- but `engine-buffer-allocator.cpp `is also the most deeply nested implementation backend
- `local-buffer-store.hpp` is a junction point
- `local-mem-pool.hpp` is considered an insolated implementation detail
- ultimately, this will be included and wired in `render-enviroment.cpp`
This implementation and the accompanying test might seem pointless,
yet the point is that the interface is aligned to the emerging
concepts for memory allocators in Lumiera, while also directly
delegating to the std::allocator for the time being.
Before we're able to observe the actual allocator trend and behaviour,
it is pretty much moot to engage into any kind of optimisation;
the generic heap allocator performs quite well on recent C++ runtimes,
and, furthermore, the most important performance tweak is to use
a thread-local pool of recent allocations, which should allow
our jobs to run without impeding each other.
Seems to work now roughly as expected, so that the next layers
of functionality can be built on top. It seems impossible
to determine the viability of this whole approach without
taking the full buffer memory management into use with
a real-world load scenario.
At the begin of each Render Node `pull()`, the render job is expected
to ''announce'' a maximum expected allocation requirement to the `BufferProvider`,
which information is passed-on to the `LocalMemPool` to determine
to which degree it can be satisfied with allocations from that local pool.
Without actual empiric observation, it is impossible to determine
the relevance of this step, how it should be resolved heuristically
against the known contents of the pool, and how much effort to spend
on that step.
Thus I'll settle for the ''simplest thing that can possibly work'' —
which is to check only for close matches and rather rely on additional
allocations, forwarded asynchronously, assuming that the clean-up
heuristics will deal with excess allocations quickly.
- take into account how much memory will be wasted when
using some allocation to satisfy a (smaller) allocation request
- furthermore prefer allocations used frequently in the past
- when an allocation is used, tag it with a score that also
reflects this degree of waste
- select buffers with low score first for clean-up
- use boost::lockfree to implement the in-queue with new allocations
- use a std::list with an internal administrative record
- automatically ingest the in-queue prior to accessing the pool
After several fruitless attempts, I have decided now
to settle for self-contained ''memory pool'' component,
that can be operated **thread-locally** and employs a
**lock-free** message queue for the allotment of memory.
Seemingly the question of how this relates to `BufferMetadata`
must remain unresolved for the time being, until the entire
buffer memory subsystem is available for empirical investigation.
As staring point, it is clear that the `BufferStore` side of the
implementation will rely on a similar structure with a thread-local
base service — which in this case will have to maintain a small pool
of recently used buffer allocations...
...which happens when a buffer type is referred for the first time
from within a new worker thread, while it is already stored in the
global registry; the full key-chain is down-synced then.
...using the first usage situation as guideline,
and verify proper operation through unit-test
- the LocalBufferStage gets a set of thread-local BufferMetadata
- furthermore, it accesses the central hub via dependency-injection
- after each type registration ''from a worker thread'' data is published
As an asside: ensure most metadata registration can ''move'' the TypeHandler
This test setup defines one of the problematic situations
where a synchronisation is necessary.
__TODO__: actually add locking and that synchronisation to LocalBufferStage
What I am building step by step right now is the first draft
of a production-grade render buffer management scheme.
For this I need suitable implementations of `BufferStag` (and `BufferStore`)
These will include a `LocalSlice` to hold thread-local »satelitte services«,
which in turn need a connection to some central coordination hub.
The thread-local instances can access the hub via our `lib::Depend` (I have just
verified that this works as intended, with on-demand creation of the
central instance on first access from some thread. This is important,
since very likely we need to install a dedicated and speficically wired
service instance, which must happen prior to first dependency-request
via `lib::Depend`. And since the worker-threads are created by the Engine,
we get ample opportunity to do so before starting those threads.
Included in this changeset is a first rough draft of the central registry
`EngineBufferMetadata` — I have modelled it analogous to the `SimpleBufferStateRegistry`,
which is the test-level implementation of the `BufferStage` API.
However, we do not want to expose direct buffer allocation operations
on the central hub, and thus I have left out the respecive parts of the API.
Furthermore, I have added an `abandon()` operation, without there beeing
a need for such an operation right now. But since this operation is tricky
and potentially dangerous, I prefer implementing it the correct way,
since I have a clear understanding what this entails at the moment.
The recursive synchronisation up the parent-chain can in most cases be skipped,
by relying on the fact that inconsistent data is never stored in the BufferMetadata
table. If thus an entry is locally present and matches the entry from the
other side, then it can be concluded that its parent must also be already in sync.
Since the intention is to run several thread-local instances of `BufferMetadata`
in a concurrent setup, we need a way to publish and exchange chains of
registered type-keys and metadata entries.
The first step is to develop a mechanic for data exchange,
and to define the rules when such synchronisation is (not) allowed.
During the last days, I started with analysis to find a way
to operate the `BufferProvider`, as implemented thus far,
within a massively concurrent environment.
Since I intend to create thread-local caches,
the initialisation of the `BufferMetadata` store must be static.
It turns out that the root-ID, which acts like a mandator ID,
is only used at a single code path, when creating or accessing
an abstract type-Key. It is thus easy to extract and relocate
this distinguishing ID into the actual `BufferStage` implementation.
Looking through all my work notes, it seems that all the
problematic aspects are now solved in a satisfactory way.
* it is clear how to dependency-inject a `BufferProvider`
* the internals of the `BufferProvider` front-End have been restructured
* there are now two ''pillars'' of the implementation: `BufferStage` and `BufferStore`
* both can be subclassed independently
* `BufferProviderSetup` exposes a proxying-mechanism (used for `DiagnosticBufferProvider`)
* the `BufferMetadata` have been reviewed and some minor details were rectified
* after a re-Design of `OutputSlot`, `BuffHandle` can now be used as unified front-End
* both `DiagnosticBufferProvider` and `DiagnosticOutputSlot` have been rewritten
* they are now independent, self-contained specialisations, layered on top
Overall, what might seem like a detour, was an important step towards
well-balanced structures and a better understanding of the Render Engine.
''This is a breakthrough...''
Using the realigned implementation structure in `BufferProvider`,
and especially the ability to establish an overarching context by
passing an connection-ID through the `LocalTag` in the `BufferMetadata`,
it was possible to resolve the longstanding mismatch buried deep within
the `OutputSlot` implementation.
Based on the new internal APIs in `BufferProvider`, it was thus possible
to setup an `OututBufferProxy` that acts as a hub to connect the
`DataSink` handles with the actual `OutputSlot::Connection` implementation.
This change allowed to rectify the convoluted control structure in
`OutputSlot` and finally to manage the Lifecycle properly. Furthermore,
one level of unnecessary implementation-inheritance could be removed,
so that it is no longer necessary to subclass `OutputSlot` in order to
configure the link to a specific output backend.
Using this redesigned internal structure, it was now fairly simple to
re-implement a tracking `DiagnosticOutputSlot` and then to verify proper
integration by passing the `OutputSlotProtocol_test` again.
This is a design study to find out what would be required
to build an opaque function wrapper, similar to `std::function`,
yet without using any heap memory ever.
Such a building block would allow for some interesting options
for a well structured design of elaborate data structures.
This changeset provides an implementation draft.
__Two important findings to note__:
* it is impossible to get away with just a single function pointer
* because at the very least we need to invoke the destructor of the
embedded delegate. Thus something akin to a "manager" is required.
* this implies that the component would not be so efficient as I hoped,
since it requires two pointers and requires to emit code for each
type of delegate.
* by making the component **NonCopyable**, I avoided the complexities
of virtual copy support. In case the latter is desired, it would be
much simpler to switch to a classical OO approach with a VTable
and use our `PolymorphicValue` wrapper to handle copy and cloning.
...to emplace some object into the buffer space.
I wouldn't expect that to be necessary for the real usage,
but it comes in handy for tests, where we often want to place
some sentinel value into a buffer.
- visible Signature of TestFrame changed due to being relocated into the Vault
- attempt to make the WorkForce_test more robust;
frequently fails on testsuite run after full build;
so better ''wait'' for the workers to have started working,
instead of using a fixed delay (that must be extremely short
and is hard to predict)
This changeset attempts to breach into the implementation design of OutputSlot
(while largely retaining the API and the »Output Slot Protocol«).
While the actual problem that brought me to this point can be traced back to some
rather obvious mistake, namely to conflate the `BufferProvider` used as
backing implementation with a `BufferProvider` used as front-end API,
creating the new connections to use an `OutputBufferProxy` unveils some
further long-standing structural mismatch; notably this leads to the
difficulties to get the lifecylce management of `OutputSlot` and `DataSink`
under control.
So I remove one entire level from the implementation,
in a bold move to realign the design with the prospective usage pattern.
Obviously, this breaks a lot of further code, and I need to rewire
the Allocation implementation altogether, and provide a new functor based
implementation for `DataSink`
...which still looks like a nasty task
* the good part is, we have a working `BufferProxyAdaptor` now
* yet this setup does not seem to fit into the `OutputSlot::AllocState`
As an aside, several test definitions need to be relocated to the Vault,
since the whole compound around `BufferProvider` migrated down
This is a nasty new requirement, discovered as an aside while looking
into ways to resolve the ''crucial mismatch'' between the `OutputSlot`
and `BufferProvider`, using the proxy-implementation of `BufferProvider`
(thereby expanding on the recent draft with the `BufferProxyAdaptor`)
The point is, if the BufferMetadata will be used beyond the internal coordination
within BufferProvider, then it will be used as part of actual buffer storage.
And this implies that the actual size of the allocation may differ (be greater)
than the nominally required size (since we'll very likely employ some kind
of memory tiling, for reasons of performance).
This leads again to an analysis of `BufferMetadata` usage,
with the conclusion to add the actual storage size directly into the API,
but treat it as an ''extra'' and not integrated into the ''buffer type''
Phew ... quite complex alltogether
This might have seemed like a good idea, adding an additional ''consistency check''...
However — we ''do have'' a generic check, insofar `BufferMetadata` is a **state engine**.
As long as we're able to prevent, reliably, that the same buffer address
ends up being classified as a different buffer type, this should be enough
to prevent double-use of any buffer (letting aside that `BufferProvider` is
backed by an implementing memory allocator, which in itself should prevent
any double-use.
Removing this feature resolves some ambiguity in the overall design,
and allows to turn `BufferMetadata::lock(type, address)` into a private
implementation function.
In the old code, there were some external access paths directly entering
into that implementation function, which caused a lot of confusion (at least for me...)
This refacotring prepares for the planned changes in `OutputSlot`
* introduce a new compound parameter (tuple) BufferAlloc
* wire this through all calls involved with storage handling
__Background__:
* `BufferMetadata` deemed important for the implementation of buffer storage
* ''new requirement detected'': must be able to handle an actual storage block
that is ''larger'' than the requested buffer size (⟼ tiling)
* need compact and uniform info, also for the `BufferProxyAdaptor` callbacks
