From ace20660aac4d160e036e98793f00796f52a6fe7 Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Wed, 11 Feb 2026 20:05:08 +0100 Subject: [PATCH] BufferProvider: analyse the existing API - write some documentation - identify some add-on API that must be retracted - consider further how to disentangle ''diagnostics'' and ''implementation'' --- src/steam/engine/buffer-provider.hpp | 87 ++++++++++++++++--- src/steam/engine/buffhandle-attach.hpp | 34 ++++---- src/steam/engine/buffhandle.hpp | 4 +- src/steam/engine/weaving-pattern-builder.hpp | 2 +- .../engine/buffer-provider-protocol-test.cpp | 2 +- .../diagnostic-buffer-provider-test.cpp | 2 +- tests/core/steam/engine/node-base-test.cpp | 2 +- .../steam/play/output-slot-protocol-test.cpp | 2 +- wiki/renderengine.html | 4 +- 9 files changed, 105 insertions(+), 34 deletions(-) diff --git a/src/steam/engine/buffer-provider.hpp b/src/steam/engine/buffer-provider.hpp index 58170c6ec..a6e8c8e1b 100644 --- a/src/steam/engine/buffer-provider.hpp +++ b/src/steam/engine/buffer-provider.hpp @@ -13,20 +13,87 @@ /** @file buffer-provider.hpp ** Abstraction to represent buffer management and lifecycle within the render engine. - ** It turns out that -- throughout the render engine implementation -- we never need - ** direct access to the buffers holding media data. Buffers are just some entity to be \em managed, - ** i.e. "allocated", "locked" and "released"; the actual meaning of these operations is an implementation detail. - ** The code within the render engine just pushes around BufferHandle objects, which act as a front-end, - ** being created by and linked to a BufferProvider implementation. There is no need to manage the lifecycle - ** of buffers automatically, because the use of buffers is embedded into the render calculation cycle, - ** which follows a rather strict protocol anyway. Relying on the capabilities of the scheduler, - ** the sequence of individual jobs in the engine ensures... - ** - that the availability of a buffer was ensured prior to planning a job ("buffer allocation") + ** When looking at the render engine implementation at a high level, buffers are a scarce + ** resource to be managed. Looking into the implementation in detail however reveals — maybe + ** surprisingly — that we never need _direct access_ to the buffers holding media data. This is + ** a consequence of the Engine's design, which treats _actual media processing_ as an external concern, + ** handled by specialised libraries and Plug-ins. Buffers are thus treated as some entity to be _managed_, + ** i.e. "allocated", "locked" and "released"; the actual meaning of these operations is considered a detail + ** of the implementation. + ** + ** The code within the Render Engine is centred around _invocation_ and the transport of data; the latter is + ** marked and represented as BufferHandle, that can be passed form invocation to invocation. For this to work, + ** the _type_ of the buffers must be compatible with the data and the processing functions — and this condition + ** is accommodated already in the preparation of the rendering, which is the purpose of the Builder. In this + ** framework, BufferHandle objects act as a front-end, as being created by and linked to a BufferProvider + ** implementation. There is no need to manage the lifecycle of buffers explicitly because the use of buffers + ** is embedded into the fabric of the render calculation, which follows a rather strict protocol anyway. + ** Relying on the capabilities of the scheduler, the sequence of individual jobs in the engine ensures... + ** - that the availability of sufficient buffer memory was accommodated as part of the planning + ** - that overall buffer capacity will be announced at start of the actual render invocation ** - that a buffer handle was obtained ("locked") prior to any operation requiring a buffer ** - that buffers are marked as free ("released") after doing the actual calculations. ** - ** @see state.hpp + ** # Buffer Provider Protocol + ** + ** BufferProvider is a public interface, that also defines an interaction protocol. + ** This implies that any usage of _buffer memory_ has to proceed through the stages of a process, + ** which is flexible enough however to accommodate the various usage situations. At the bare minimum, + ** a buffer can be _obtained and locked_, to get a fixed amount of memory, accessible through a BuffHandle. + ** After usage, the buffer _must be released._ There is no tracking and no automatisation to ensure this + ** duty is fulfilled — failure to release will block the buffer memory indeterminately. Usually this + ** simple arrangement does not cause any serious problems, as the process of rendering media is + ** bound to operate within very precisely defined boundaries anyway. Furthermore, the C++ + ** language offers various general-purpose capabilities to automate resource clean-up. + ** + ** For the general case, the usage cycle is defined to be more elaborate and flexible: + ** - in a pre-planning step, a [Buffer Type Descriptor](\ref BuffDescr) can be obtained + ** + this descriptor is copyable and embeds a back-link to the BufferProvider + ** + obtaining such a descriptor implies a _type registration_ + ** + in this context a **Buffer Type** describes at least a fixed amount of storage, + ** and optionally also an _access type,_ or even an attached _object creation_ and + ** automatic _destruction_. + ** - from a given BuffDescr, the usage of a number of buffers can optionally be _announced;_ + ** doing so is highly recommended, since it confirms the availability of an actual number + ** of buffers (which can be lower than requested); these buffers are guaranteed to be + ** available, and this availability will even be pre-arranged for the actual thread, + ** as an asynchronous process + ** - for the actual usage, individual buffers can be _locked_ on the BuffDescr, thereby + ** providing a BuffHandle. The number of actually obtained buffers need not match the + ** announced number (yet obviously it should, overall). Buffers exceeding the announced + ** capacity _can_ be obtained, if possible — in that case the operation may block or + ** even fail with an exception. + ** - a BuffHandle, once obtained, represents the client's ownership and liability + ** - optionally, the client may _emit_ on the BuffHandle, to indicate that data + ** in the Buffer is complete and will not be touched anymore. The interpretation + ** of this call is reserved to special services attached behind the BufferProvider + ** interface, e.g. a Cache service or an DataSink. + ** - as the last step, a BuffHandle _must_ be _released_. It must not be used further + ** after that point, since the buffer memory might have been re-assigned to another + ** client, or way even released altogether. + ** + ** # Providing an Implementation + ** + ** The class BufferProvider is abstract, yet contains some orchestration and clue code + ** to provide the high-level operations. Functionality is broken down in terms of + ** two _implementation interfaces:_ + ** - BufferProvider::BufferStage describes the type registration and state transitions + ** - BufferProvider::BufferStore handles coordination of memory access + ** + ** For creating actual implementations, buffer-provider-setup.hpp defines a framework. + ** A simplistic heap-based implementation for demonstration and unit test usage is + ** available through NaiveBufferSetup. This can be extended with tracking functionality + ** to verify actual operations from a test setup, with the help of DiagnosticBufferProvider. + ** The [Render Engine setup](\ref engine-facilities.hpp) includes a suitable BufferProvider + ** implementation; when used from the actual [Render Environment](\ref render-environment.hpp), + ** this will be backed by a production-strenght and concurrency-safe implementation. + ** + ** @see buffer-provider-protocol-test.cpp ** @see output-slot.hpp + ** @see engine-ctx.hpp + ** @see proc-node.hpp + ** @see media-weaving-pattern.hpp + ** @see weaving-pattern-builder.hpp */ #ifndef STEAM_ENGINE_BUFFR_PROVIDER_H diff --git a/src/steam/engine/buffhandle-attach.hpp b/src/steam/engine/buffhandle-attach.hpp index b5c5b7c56..6d92fdefe 100644 --- a/src/steam/engine/buffhandle-attach.hpp +++ b/src/steam/engine/buffhandle-attach.hpp @@ -28,6 +28,10 @@ ** header and compilation footprint low, the implementation of the functions supporting ** this special case was split out of the basic buffhandle.hpp ** + ** @deprecated 2026-01 this extension-API does not fit into the overall concept. + ** It was created under the impression that the Render Engine will work "with void pointers" + ** Meanwhile, the BuffHandle::accessAs() turned into a core-API now; + ** all the other functionality was never used and is considered pointless //////////////////////TICKET 1410 : remove unnecessary API ** @see BuffHandle ** @see BufferProviderProtocol_test usage demonstration */ @@ -71,13 +75,13 @@ namespace engine { * in charge of managing the buffer contents, or when the object to create * would not fit into this buffer. */ - template - inline BU& - BuffHandle::create() - { - takeOwnershipFor(); - _EXCEPTION_SAFE_INVOKE (BU()); - } +// template +// inline BU& +// BuffHandle::create() +// { +// takeOwnershipFor(); +// _EXCEPTION_SAFE_INVOKE (BU()); +// } #undef _EXCEPTION_SAFE_INVOKE @@ -90,14 +94,14 @@ namespace engine { * in charge of managing the buffer contents, or when the object to create * would not fit into this buffer. */ - template - inline void - BuffHandle::takeOwnershipFor() - { - BuffDescr howto_attach_object_automatically - = descriptor_.provider_->getDescriptor(); - takeOwnershipFor (howto_attach_object_automatically); // EX_STRONG - } +// template +// inline void +// BuffHandle::takeOwnershipFor() +// { +// BuffDescr howto_attach_object_automatically +// = descriptor_.provider_->getDescriptor(); +// takeOwnershipFor (howto_attach_object_automatically); // EX_STRONG +// } /** convenience shortcut: access the buffer contents casted to a specific type. diff --git a/src/steam/engine/buffhandle.hpp b/src/steam/engine/buffhandle.hpp index 4b62d5ce0..0a6d18269 100644 --- a/src/steam/engine/buffhandle.hpp +++ b/src/steam/engine/buffhandle.hpp @@ -136,7 +136,7 @@ namespace engine { template - BU& create(); + BU& create(); /////////////////////TICKET #1410 : this API needs to be retracted; it does not fit into the concept! template BU& accessAs(); @@ -159,7 +159,7 @@ namespace engine { } HashVal - entryID() const + entryID() const ///////////////////////TICKET 1410 : why can't we just expose operator HashVal() ?? { return HashVal(descriptor_); } diff --git a/src/steam/engine/weaving-pattern-builder.hpp b/src/steam/engine/weaving-pattern-builder.hpp index b99a09a16..dec25a2c9 100644 --- a/src/steam/engine/weaving-pattern-builder.hpp +++ b/src/steam/engine/weaving-pattern-builder.hpp @@ -136,7 +136,7 @@ #include "steam/engine/proc-id.hpp" #include "steam/engine/engine-ctx.hpp" #include "steam/engine/buffer-provider.hpp" -#include "steam/engine/buffhandle-attach.hpp" /////////////////OOO why do we need to include this? we need the accessAs() template function +#include "steam/engine/buffhandle-attach.hpp" ///////////////////////////////////////////////////////////TICKET 1410 : unnecessary API -- only accessAs() needs to be retained! #include "steam/engine/media-weaving-pattern.hpp" #include "lib/meta/tuple-closure.hpp" #include "lib/meta/tuple-helper.hpp" diff --git a/tests/core/steam/engine/buffer-provider-protocol-test.cpp b/tests/core/steam/engine/buffer-provider-protocol-test.cpp index 396875e8d..dd7145a2a 100644 --- a/tests/core/steam/engine/buffer-provider-protocol-test.cpp +++ b/tests/core/steam/engine/buffer-provider-protocol-test.cpp @@ -24,7 +24,7 @@ #include "steam/engine/testframe.hpp" #include "steam/engine/test-rand-ontology.hpp" #include "steam/engine/diagnostic-buffer-provider.hpp" -#include "steam/engine/buffhandle-attach.hpp" +#include "steam/engine/buffhandle-attach.hpp" ///////////////////////////////////////////////////////////TICKET 1410 : remove unnecessary API #include "steam/engine/bufftable.hpp" using util::isSameObject; diff --git a/tests/core/steam/engine/diagnostic-buffer-provider-test.cpp b/tests/core/steam/engine/diagnostic-buffer-provider-test.cpp index b8538b2bc..cc6780418 100644 --- a/tests/core/steam/engine/diagnostic-buffer-provider-test.cpp +++ b/tests/core/steam/engine/diagnostic-buffer-provider-test.cpp @@ -19,7 +19,7 @@ #include "lib/error.hpp" #include "lib/test/run.hpp" #include "steam/engine/diagnostic-buffer-provider.hpp" -#include "steam/engine/buffhandle-attach.hpp" +#include "steam/engine/buffhandle-attach.hpp" ///////////////////////////////////////////////////////////TICKET 1410 : remove unnecessary API #include "steam/engine/testframe.hpp" #include diff --git a/tests/core/steam/engine/node-base-test.cpp b/tests/core/steam/engine/node-base-test.cpp index b7a343586..e8a8be806 100644 --- a/tests/core/steam/engine/node-base-test.cpp +++ b/tests/core/steam/engine/node-base-test.cpp @@ -26,7 +26,7 @@ #include "steam/engine/turnout-system.hpp" #include "steam/engine/feed-manifold.hpp" #include "steam/engine/node-builder.hpp" -#include "steam/engine/buffhandle-attach.hpp" +#include "steam/engine/buffhandle-attach.hpp" ///////////////////////////////////////////////////////////TICKET 1410 : remove unnecessary API #include "lib/test/test-helper.hpp" #include "lib/util.hpp" diff --git a/tests/core/steam/play/output-slot-protocol-test.cpp b/tests/core/steam/play/output-slot-protocol-test.cpp index 0460e28fb..19eaae3dc 100644 --- a/tests/core/steam/play/output-slot-protocol-test.cpp +++ b/tests/core/steam/play/output-slot-protocol-test.cpp @@ -20,7 +20,7 @@ #include "steam/play/diagnostic-output-slot.hpp" #include "steam/engine/buffhandle.hpp" -#include "steam/engine/buffhandle-attach.hpp" +#include "steam/engine/buffhandle-attach.hpp" ///////////////////////////////////////////////////////////TICKET 1410 : remove unnecessary API #include "steam/engine/testframe.hpp" diff --git a/wiki/renderengine.html b/wiki/renderengine.html index 306616a17..400a34ba4 100644 --- a/wiki/renderengine.html +++ b/wiki/renderengine.html @@ -5142,7 +5142,7 @@ Thus the mapping is a copyable value object, using an associative array. It may First and foremost, mapping can be seen as a //functional abstraction.// As it's used at implementation level, encapsulation of detail types in't the primary concern, so it's a candidate for generic programming: For each of those use cases outlined above, a distinct mapping type is created by instantiating the {{{OutputMapping<DEF>}}} template with a specifically tailored definition context ({{{DEF}}}), which takes on the role of a strategy. Individual instances of this concrete mapping type may be default created and copied freely. This instantiation process includes picking up the concrete result type and building a functor object for resolving on the fly. Thus, in the way typical for generic programming, the more involved special details are moved out of sight, while being still in scope for the purpose of inlining. But there //is// a concern better to be encapsulated and concealed at the usage site, namely accessing the rules system. Thus mapping leads itself to the frequently used implementation pattern where there is a generic frontend as header, calling into opaque functions embedded within a separate compilation unit. -
+
Within the Lumiera player and output subsystem, actually sending data to an external output requires to allocate an ''output slot''
 This is the central metaphor for the organisation of actual (system level) outputs; using this concept allows to separate and abstract the data calculation and the organisation of playback and rendering from the specifics of the actual output sink. Actual output possibilities (video in GUI window, video fullscreen, using a beamer, sound, Jack, rendering to file) can be added and removed dynamically from various components (Vault, Stage), all using the same resolution and mapping mechanisms (&rarr; OutputManagement)
 
@@ -5166,7 +5166,7 @@ The assumption is for the client to have elaborate timing capabilities at his di
 some concrete output connections and drivers embody a specific operation mode (e.g. sample rate or number of channels). The decision and setup of these operational configuration is initiated together with the [[resolution|OutputMapping]] of an OutputDesignation within the OutputManager, finally leading to an output slot (reference), which can be assumed to be suitably configured, before the client allocates this slot for active use. Moreover, an individual output sink (corresponding to a single channel) may just remain unused -- until there is an {{{emit()}}} call and successful data handover, this channel will just feature silence or remain black. (There are some output systems with additional flexibility, e.g. ~Jack-Audio, that allow to generate an arbitrary number of output pins -- Lumiera will support this by allowing to set up additional output slots and attach this information to the current session &rarr; SessionConfigurationAttachment)
 
 !!!Lifecycle and storage
-The concrete OutputSlot implementation is owned and managed by the facility actually providing the output possibility in question. For example, the GUI provides viewer widgets, while some sound output backend provides sound ports. The associated OutputSlot implementation object is required to stay alive as long as it's registered with some OutputManager. It needs to be de-registered explicitly prior to destruction -- and this deregistration may block until all clients using this slot did terminate. Beyond that, an output slot implementation is expected to handle all kinds of failures gracefully -- preferably just emitting a signal (callback functor).
+The concrete OutputSlot implementation is owned and managed by the facility that actually provides the output capability in question. For example, the GUI provides viewer widgets, while some sound output backend provides sound ports. The associated OutputSlot implementation object is required to stay alive as long as it's registered with some OutputManager. It needs to be de-registered explicitly prior to destruction -- and this deregistration may block until all clients using this slot did terminate. Beyond that, an output slot implementation is expected to handle all kinds of failures gracefully -- preferably just emitting a signal (callback functor).
 {{red{TODO 7/11: Deregistration is an unsolved problem....}}}
 
 !!!unified data exchange cycle