...the plan is to wrap the `DataSink` and also to expose possible further control parameters here...
Rather difficult to get started, since mostly related to ''planned features''
This is the result of an requirement analysis,
based on the current (better) understanding of ''render job invocation''
* the `InvocationInstanceID` is @deprecated, because rather a data based hash is required
* instead, two free parameters are introduced, with a meaning defined by the specific `JobFunctor`
* notably the functor for rendering, the `RenderInvocation´ can use these for the exit node and an output context
**Note**: the C-structs for the `job` are no longer considered an interface (and will be remoulded into an internal detail of the dispatcher)
While the setup here in the test seems trivial,
this touches on some challenging questions of design:
Presumably the invocation of a Render Job will need some kind of
''Render Context'' — since the relation of the absolute nominal time
to some output frame number is strictly ''a concern of the render process''.
However, it seems prudent to keep that relation separate from the `RenderInvocation`,
so that access to all invocation parameters has to be supplied by the
Render Job Functor.
This change in the test setup now uses a `DiagnosticOutputSlot`
instead of directly allocating some buffer for output; the data
written into the `DataSink` can be investigated after-the-fact.
turns out that was not (yet) tested and did not even compile,
due to a missing cross-constructor to get from one data allocation
policy to another one backed by the same cluster.
- added test case to `NodeBuilder_test` to demonstrate basic usage
- use `AllocationCluster` in NodeOpera_test and verify that allocations took place
Works as expected, since it was directly adapted from NodeLink_test
The preceding changeset however caused a SEGFAULT,
since the initialisation order in the helper class `RenderPipeline`
was not correct, so that the node builder used a not-yet initialised
`AllocationCluster`....
In NodeLink_test, a processing pipeline was established that
works on `TestFrame` and is quite complex; here I use a slightly
simplified variant (featuring only a single port) to perform
the same kind of computations.
However, node memory allocations are managed by an `AllocationCluster`
The purpose of this test is to invoke a Render Node graph
using the full service environment as present in the Render Engine.
However, for sake of clarity, I do not want to pull up the Engine subsystem,
but rather use a local / mocked setup of all the services involved.
The node invocation is performed from within a worker thread,
to demonstrate integration with the thread-local memory pools.
The first simple test case builds and then pulls a source node.
...after having settled the problems related to buffer management
and the interplay between buffers and output sinks, the ''next goal''
is to drive ahead the integration between job planning and invocation...
The test suite is now **green** again.
The design rework and clean-up of unnecessary test-driven functionality
in the `DiagnosticBufferProvider` and `DiagnosticOutputSlot` required
to break some very old test-setup, using fake instances of the relevant
context facilities. This setup however was used by the newer family
of test support facilities, notable the `MockJob`, `MockSegmentation`
and `MockDispatcher`. These required a set of `DataSink` handles to be
present (even while they are not used currently).
This could be fixed easily now, relying on the simplified internal structure
of the `OutputSlot`: by re-arranging the types, the implied identity of the
`DataSink` handles could be made explicit, allowing for equality comparison.
__Remark__: even while `DataSink` with only a single abstracted function,
using function notation turned out to be rather confusing at usage site.
Naming this function as `lockFrame(frameNo)` clarifies its purpose nicely.
That call is effectively abstracted by means of the ''buffer type'',
relying on the VTable in the `BufferProvider` implementation.
These provide a faked context setup for unit testing.
I created them ''out of sheer desperation'' several years ago,
since it is impossible to verify the building blocks if you
don't know what the building itself will look like.
The recent design clean-up of OutputSlot has unfortunately
removed a bunch of »hexagonal hooks« — and now I seem to be unable
to hook into anything with my tests. Oh my!
- some tests are notoriously flaky
- increase resource limits for the testsuite to allow stronger stress tests
- provide a "quick" variant of BufferProviderLoad_test with lowered requirements
Furthermore, investigate test scaffolding in `DummyBuilderContext`,
which was broken due to the clean-up in `DiagnosticOutputSlot`.
It seems that only some placeholder instances are required here,
so that we might be successful with just instantiating the `UnimplementedConnection`
__TODO__: need to supply slightly more implementation into the latter,
so that we're able at least to ''create'' such a connection stub.
Integrate the work related to buffer management in the render engine.
* in-depth analysis and rework of the `BufferProvider` framework (see #1395)
* build a prototype for production-grade buffer allocation handling (closes: #1415)(see #1396)
* as an aside: Application code and namespace reorganisation
Within the reworked design of the `OutputSlot` a proxy implementation
of the `BufferProvider` API is used (class `OutputBufferProxy`).
After analysis of the invocation patterns, I concluded that it seems adequate
to attach the special-purpose metadata generated by this integration
to the generic system of thread-local replicated `BufferMetadata` storage.
However, since thread-local instances are effectively static(global),
I was concerned that unit-testing might be impeded by accumulation of
stale metadata in the store linked to the main thread. Thus I created
a special configuration variant to retain the previously used setup
for the sake of unit testing (which applies especially to use of the
`DiagnosticOutputSlot`). Using a `SimpleBufferStateRegistry` for in
that case has the benefit that simply a local instance can be used
in the test and all metadata will be discarded afterwards.
Furthermore the factory syntax for obtaining an `OutputSlot` was improved,
drawing on the recent changes to `lib::ScopedCollection`.
Also I looked through some of the documentation and updated
the UML sequence diagram to reflect `OutputSlot`'s improved internal structure
With this third test case, a fairly continuous processing
can be demonstrated, while also coordinating allocations
in a decentralised way.
The implementation behaves correct, and seemingly with
satisfactory performance -- while there is certainly
some room for improvement: with the processing pattern
in this test, a nominal computation load of 4 seconds
can be processed with 4 cores in ~ 1.3 seconds
Yet only in a simple way, by pressing with 100 threads....
This test runs for about 10 seconds, shows a massive allocation
of more than 1 GiB, but comes down clean, without deadlocks,
using all cores and returning all memory blocks at the end.
This test uncovered some missing safety checks,
and it highlighted (again) that this memory management scheme is
elaborate and potentially fragile: I had to deal with carried-on
state in the local metadata registry associated with the main thread
(in fact I did not ever want that registry to be used, but turns out
that the initialisation check, that runs in the constructor, already
pulls up an instance for the main thread, which carries stale state
after the test case is complete)
This is the meat of this test case...
- generate a ''random walk'' with buffer allocation requests
- attempt to make the load pattern nasty, since the goal of this test
is to verify correctness of the complex interplay between local pools
and central allocator.
''correct pattern was verified single-threaded, with print statements''
- provide mock instances of the relevant global services
- use a base config record for the worker thread(s)
- pregenerate ''Buffer Descriptors'' commonly used by the workers
- further access to the buffer manager should then be possible
by locking one of these descriptors...
The intention is to utilise this helper class, that was developed for the
scheduler stress testing; but this time, it should access the actual allocated
render buffers to generate the load.
This implies to use buffers of a significant size; it turned out that
the existing implementation was imprecise, insofar it always used full multiples
of the configured buffer size, which defeats the purpose of performing
a precisely calibrated number of calculation steps. As long as the buffer size
would remain comparatively small, this drift does not matter much.
Thus I changed the implementation now to ensure that exactly the calibrated
number of steps is performed, using a modulus to map it to the buffer memory.
Furthermore, I plan to use a random-walk pattern in the worker thread
to select from a collection of pre-established »buffer types«, similar
to what happens in the actual engine
To verify integration of this new engine-level facility,
it seems prudent to build a load test (without going into
extended stress- and performance testing).
For that we need
- random buffer demand
- a computational load that actually uses the buffer.
As part of the ''scheduler stress testing'', a self-calibrating
`ComputationalLoad` was developed, which even implements both a
merely computational load, but also a memory accessing computation.
This can be extended to optionally use an external buffer, instead
of allocating a precisely dimensioned memory block directly from heap.
Obviously this extension is ''dangerous'', since there is no way to
verify that the storage passed in can accommodated the storage required
based on the current calibration of the `ComputationalLoad`
As an aside: identified a timing check in `SchedulerService_test`
that showed the tendency to fail when invoked immediately after a build.
The mechanism is tuned now to engage rather quickly
whenever excess capacity builds up in a local allocator pool,
but when all buffers are used most of the time, they will be retained.
Can now demonstrate by integration test that capacity is sent back.
The goal is to develop a heuristic when to trigger clean-up of the local pool.
For developing this heuristic, I'll use a unit-test
- represent "allocations" by dummy buffer pointers
- use a queue and a stack to push those "allocations" around
- the "global pool" is just a queue with such dummy pointers
- next step is to define some λ-primitives
- finally, a λ-DSL to allow to perform "N operations of kind X"
This is a fall-back situation,
in case the allocation ''was not announced beforehand''
A direct locked global access is used then, which is verified by this test
...need an elaborate test setup, with the test worker thread
and the main thread proceeding in lockstep, otherwise the
effect would not be visible. Furthermore, the diagnostic
access for LocalMemPool can be re-used here, thereby
mapping requests onto the pool for the current thread.
This makes the test case pass, validating the theory.
Each `AllocReceiver` now integrates a `lib::LiveMark` instance,
and the callbacks are dispatched using the corresponding
`LiveMark::SafeHandle`, so that it is no longer possible
to dispatch allocation messages to a defunct receiver.
Furthermore, the `LiveMark` is explicitly disengaged prior
to unwinding a local pool, so that no further pending requests
can be handled ans dispatched concurrently. The local pool
sends back all the remaining allocations via async queue
to the `EngineBufferManager`
Not that such a shutdown can be triggered any time,
since an exception or failure might cause a worker thread
to unwind, thereby destroying the thread-local `LocalMemPool`
without further notice.
...even while this situation is unlikely, it needs to be considered:
A worker thread might terminate unexpectedly, due to an error;
all thread-local objects will be destroyed automatically — while the
`EngineBufferManager` might be processing further allocations requested
by that very thread, and dispatch them towards an already destroyed `LocalMemPool`
This turns out to be a nasty problem, since the whole setup was created
to allow the workers to operate independently from each other; any handshake
or registration/deregistration mechanism would require a lock of sorts,
thereby defeating the whole purpose of the setup.
The best solution I could find is to rely on a instance marker hash,
that can be checked with an atomic read prior to each dispatch of a
memory allocation. The destructor of the local pool would then be able
to ''disengage'', by invalidating that hash.
To evaluate this solution, I developed this mechanism
as a free-standing library component, called `LiveMark`
Basically the setup should be symmetrical to the `BufferStage` setup,
which implies to install the `LocalMemPool` as a thread-local »slice«.
However, in addition we need a mechanism to trigger an automatic clean-up
of the local pool whenever a worker-thread terminates, with the additional
complication that thread-local variables are effectively static.
Since `LocalMemPool` is written such as to abstract from the setup environment,
we need to inject a shutdown function from the setup within `LocalSlice`,
thereby directly accessing the `EngineBufferManager` via dependency-injection.
Most relevant parts of the `LocalBufferStore` are already available
and just need to be connected. One point poses a challenge however:
when and how to trigger the heuristic clean-up of the local pool,
so that ''on average neither too much memory is wasted, nor
to much unnecessary allocations'' are performed...
Signed-off-by: Ichthyostega <prg@ichthyostega.de>
This is a collection with unique-ownership,
and the ability to create all content objects in ''RAII style''
from the constructor call, using a ''population functor''.
It is often used to maintain a collection of resource handles,
and especially to start a series of threads (for testing).
With the variadic λ-captures from C++20, it is possible to rewrite
the most frequently used population functors to be easier to use.
Notably we can now bind constructor arguments, that will be passed
to each of the created elements — a prominent example being the
thread function when launching several threads. Furthermore,
I added a constructor variant that consumes a Lumiera Iterator.
Adding this kind of generic variants is possible now with C++20,
since overload resolution takes concepts into account.
Well, finally.
Lumiera Iterators were conceived as a concept from the very beginning,
and this idea dates back even before C++11. Furthermore, over time some
''trait templates'' have been developed, to detect if some template argument
might match the known pattern of functions associated with a Lumiera Iterator.
Meanwhile this concept is used pervasively in the whole code base, and also
the traits templates proved to be reliable. It therefore seems that the time is ripe
to gradually introduce a genuine **concept** `lum_iter`
I'll take a defensive approach here, since any mistake would lead to
a massive breakage of compilation. The new concept is defined simply
by delegating to to the tried-and-tested trait template, which is
also renamed `can_IterForEach` ⟼ `can_LumieraIter`
One immediate benefit is that the compiler is now able to select
overloads based on that concept, which allows to drop the tricky
selection of variants based on the SFINAE technique and `enable_if`.
For validation, one very heavily used case of that pattern was switched
in `lib/format-util.hpp', which notably lies in the code-path used by `util::join`
The test performs a massively concurrent scenario
that presses with 100 threads and 1 MiB allocation for each.
This highlighted my still limited understanding of the situation
and leads to a much better solution, where, on-demand, some worker
performs all the accrued allocation and management work.
So effectively we do not even need to use the synchronous API
Implementing the `EngineBufferManager` the desired way,
with the ability to post allocation requests asynchronously,
raises the difficult question of who will ultimately perform
the allocation / deallocation and further management work.
The point is that this management effort takes up capacity
originally assigned for the render workers themselves, either
by implicitly pushing aside those threads, or as a dedicated
processing assigned to some worker.
In the current stage of development it seems impossible to decide
upon the proper mechanism to schedule this excess work explicitly.
The solution proposed hereby however circumvents this dilemma,
insofar it allows to implement the synchronous, fallback processing
in terms of the asynchronous processing, just performed exclusively
within the current thread. Unless some processing will be scheduled
explicitly (which can be added as performance optimisation later on),
the handling of a direct allocation request will ''fold down'' into
the current worker eventually, thereby using it's processing capability
without the need of any explicit scheduling.
- use a base class with the lock-free queue as building block
- both `LocalMemPool` and `EngineBufferManager` are built on top,
allowing them to exchange `Alloc` entries asynchronously
- ''Allocation requests'' include a receiver callback,
so that a newly allocated memory block can be dispatched
While all of this aggregate can be considered ''deep implementation'',
the structure of dependencies bears danger to get confusing. Even more so,
since an elaborate pooling memory allocator will be added to and interconnected
with this structure.
Now considering the following route:
- `engine-buffer-allocator.hpp` has elements of a common service / interface
- but `engine-buffer-allocator.cpp `is also the most deeply nested implementation backend
- `local-buffer-store.hpp` is a junction point
- `local-mem-pool.hpp` is considered an insolated implementation detail
- ultimately, this will be included and wired in `render-enviroment.cpp`
Images were extracted form the PDF provided by Wouter with the command
pdfimages -png -j wouter260316v4.pdf wouter/v4
As usual, I have added the images separately, into the Website-repository,
in order to save storage space in the main Lumiera repo (since storage in
a Git-repository is permanent and can not be shrinked later without
rewriting the history)
Furthermore, I have changed the naming scheme of the images,
adding a prefix with the version number where some image was added.
...added new chapter 1: working with source material
===========================================================
Remark from Ichthyo(committer):
Converted from PDF to raw text with
pdftotext wouter260316v4.pdf WorkflowProposals.txt
Images extracted with:
pdfimages -png -j wouter260316v4.pdf wouter/v4
Again adding the images separately to the Website-repository,
in order to save storage space in the main repository....
This implementation and the accompanying test might seem pointless,
yet the point is that the interface is aligned to the emerging
concepts for memory allocators in Lumiera, while also directly
delegating to the std::allocator for the time being.
Before we're able to observe the actual allocator trend and behaviour,
it is pretty much moot to engage into any kind of optimisation;
the generic heap allocator performs quite well on recent C++ runtimes,
and, furthermore, the most important performance tweak is to use
a thread-local pool of recent allocations, which should allow
our jobs to run without impeding each other.
Seems to work now roughly as expected, so that the next layers
of functionality can be built on top. It seems impossible
to determine the viability of this whole approach without
taking the full buffer memory management into use with
a real-world load scenario.
At the begin of each Render Node `pull()`, the render job is expected
to ''announce'' a maximum expected allocation requirement to the `BufferProvider`,
which information is passed-on to the `LocalMemPool` to determine
to which degree it can be satisfied with allocations from that local pool.
Without actual empiric observation, it is impossible to determine
the relevance of this step, how it should be resolved heuristically
against the known contents of the pool, and how much effort to spend
on that step.
Thus I'll settle for the ''simplest thing that can possibly work'' —
which is to check only for close matches and rather rely on additional
allocations, forwarded asynchronously, assuming that the clean-up
heuristics will deal with excess allocations quickly.
- take into account how much memory will be wasted when
using some allocation to satisfy a (smaller) allocation request
- furthermore prefer allocations used frequently in the past
- when an allocation is used, tag it with a score that also
reflects this degree of waste
- select buffers with low score first for clean-up
- use boost::lockfree to implement the in-queue with new allocations
- use a std::list with an internal administrative record
- automatically ingest the in-queue prior to accessing the pool
After several fruitless attempts, I have decided now
to settle for self-contained ''memory pool'' component,
that can be operated **thread-locally** and employs a
**lock-free** message queue for the allotment of memory.
Seemingly the question of how this relates to `BufferMetadata`
must remain unresolved for the time being, until the entire
buffer memory subsystem is available for empirical investigation.
As staring point, it is clear that the `BufferStore` side of the
implementation will rely on a similar structure with a thread-local
base service — which in this case will have to maintain a small pool
of recently used buffer allocations...
...which happens when a buffer type is referred for the first time
from within a new worker thread, while it is already stored in the
global registry; the full key-chain is down-synced then.
