From 57cd51e3a29590f4c686b16950fd488d7fcd9dd1 Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Sun, 19 Apr 2026 18:02:38 +0200 Subject: [PATCH] BufferProvider: draft setup for engine-wide coordination hub MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit What I am building step by step right now is the first draft of a production-grade render buffer management scheme. For this I need suitable implementations of `BufferStag` (and `BufferStore`) These will include a `LocalSlice` to hold thread-local »satelitte services«, which in turn need a connection to some central coordination hub. The thread-local instances can access the hub via our `lib::Depend` (I have just verified that this works as intended, with on-demand creation of the central instance on first access from some thread. This is important, since very likely we need to install a dedicated and speficically wired service instance, which must happen prior to first dependency-request via `lib::Depend`. And since the worker-threads are created by the Engine, we get ample opportunity to do so before starting those threads. Included in this changeset is a first rough draft of the central registry `EngineBufferMetadata` — I have modelled it analogous to the `SimpleBufferStateRegistry`, which is the test-level implementation of the `BufferStage` API. However, we do not want to expose direct buffer allocation operations on the central hub, and thus I have left out the respecive parts of the API. Furthermore, I have added an `abandon()` operation, without there beeing a need for such an operation right now. But since this operation is tricky and potentially dangerous, I prefer implementing it the correct way, since I have a clear understanding what this entails at the moment. --- src/vault/mem/buffer-metadata.hpp | 34 ++-- src/vault/mem/engine-buffer-manager.hpp | 68 ++++++++ src/vault/mem/engine-buffer-metadata.hpp | 145 ++++++++++++++++++ .../mem/simple-buffer-state-registry.hpp | 2 +- tests/vault/mem/local-buffer-stage-test.cpp | 4 +- wiki/thinkPad.ichthyo.mm | 129 +++++++++++++++- 6 files changed, 367 insertions(+), 15 deletions(-) create mode 100644 src/vault/mem/engine-buffer-manager.hpp create mode 100644 src/vault/mem/engine-buffer-metadata.hpp diff --git a/src/vault/mem/buffer-metadata.hpp b/src/vault/mem/buffer-metadata.hpp index 3af6d4826..1867889e9 100644 --- a/src/vault/mem/buffer-metadata.hpp +++ b/src/vault/mem/buffer-metadata.hpp @@ -2,7 +2,7 @@ BUFFER-METADATA.hpp - internal metadata for data buffer providers Copyright (C) - 2011, Hermann Vosseler + 2011,2026 Hermann Vosseler   **Lumiera** is free software; you can redistribute it and/or modify it   under the terms of the GNU General Public License as published by the @@ -41,11 +41,29 @@ ** ** @todo 2/2026 After refactoring the BufferProvider and splitting up the implementation ** in two realms (state and storage), the purpose of this code her became much - ** clearer. The problem remains however, that the BufferMetadata directly links - ** to the one central metadata hashtable. We'll have to reconsider this aspect - ** once we have to build a metadata store that works well in a massively - ** concurrent environment, where requests for state transition emanate - ** any time form within all worker threads! + ** clearer. The problem remains however, that the BufferMetadata directly + ** maintains its metadata in an embedded hashtable. This is problematic + ** when operating within a massively concurrent environment. + ** @todo 4/2026 After resolving the problems related to BufferProvidder and OutputSlot, + ** the next task is to build a first draft implementation for a production-grade + ** buffer allocator and manager. I decided to employ thread-local »satellite« + ** tables, and a central EngineBufferMetadata hub. It remains to be seen + ** if this design is sustainable. + ** + ** @warning BufferMetadata entails an intricate setup of chained hash keys, which relies + ** on several implicit assumptions regarding consistency. These are largely unchecked + ** (beyond the unit test) and guaranteed only through the call sequence in proper use. + ** - there are Key entries that represent a _buffer type_ — these must not be mutated. + ** - the hash value for these typeKeys is totally determined by their properties + ** - a child entry can only vary _one_ of these properties, and the change should + ** be applied in the fixed sequence as defined in BufferMetadata::key() + ** - any violation to this scheme might lead to *duplicate entries* -> buffer corruption + ** - active state entries should only added as leaves, and should be removed after use. + ** - the hash of an active entry must reflect its associated memory address. + ** - Warning: this is only ensured by invoking Key::forEntry, but not persisted in the + ** Key itself. A directly computed hash of such a Key differs from the explicitly + ** associated hashID_. If the associated buffer* is changed, the entry can not be + ** found anymore, which might lead to double allocation and memory corruption. ** ** @see buffer-provider.hpp ** @see BufferMetadata_test @@ -591,11 +609,9 @@ namespace mem { using Entry = metadata::Entry; /** establish a metadata registry. - * Such will maintain a family of buffer type entries + * Such will maintain families of chained buffer type entries * and provide a service for storing and retrieving metadata * for concrete buffer entries associated with these types. - * @param implementationID to distinguish families of - * type keys belonging to different registries. */ BufferMetadata() : table_{} diff --git a/src/vault/mem/engine-buffer-manager.hpp b/src/vault/mem/engine-buffer-manager.hpp new file mode 100644 index 000000000..62928edc8 --- /dev/null +++ b/src/vault/mem/engine-buffer-manager.hpp @@ -0,0 +1,68 @@ +/* + ENGINE-BUFFER-MANAGER.hpp - central buffer allocation coordinator + + Copyright (C) + 2026, Hermann Vosseler + +  **Lumiera** is free software; you can redistribute it and/or modify it +  under the terms of the GNU General Public License as published by the +  Free Software Foundation; either version 2 of the License, or (at your +  option) any later version. See the file COPYING for further details. + +*/ + +/** @file engine-buffer-manager.hpp + ** Central hub of buffer management within the Lumiera Render Engine. + ** + ** @todo WIP-WIP-WIP 4/2026 + ** + ** @see buffer-provider.hpp + ** @see LocalBufferStore_test + ** @see BufferProviderProtocol_test + */ + +#ifndef VAULT_MEM_ENGINE_BUFFR_MANAGA_H +#define VAULT_MEM_ENGINE_BUFFR_MANAGA_H + + +#include "lib/error.hpp" +#include "lib/hash-value.h" +#include "include/logging.h" +#include "vault/mem/buffhandle.hpp" +#include "vault/mem/type-handler.hpp" +#include "vault/mem/buffer-local-tag.hpp" +#include "lib/nocopy.hpp" +#include "lib/util.hpp" + +#include + + +namespace vault{ +namespace mem { + + using lib::HashVal; + using util::unConst; + + namespace err = lumiera::error; + + + + + /** */ + + + + + namespace { // internal blah + + } + + + + /* === deaf === */ + + + + +}} // namespace vault::mem +#endif /*VAULT_MEM_ENGINE_BUFFR_MANAGA_H*/ diff --git a/src/vault/mem/engine-buffer-metadata.hpp b/src/vault/mem/engine-buffer-metadata.hpp new file mode 100644 index 000000000..7c1470f41 --- /dev/null +++ b/src/vault/mem/engine-buffer-metadata.hpp @@ -0,0 +1,145 @@ +/* + ENGINE-BUFFER-METADATA.hpp - central buffer metadata registry + + Copyright (C) + 2026, Hermann Vosseler + +  **Lumiera** is free software; you can redistribute it and/or modify it +  under the terms of the GNU General Public License as published by the +  Free Software Foundation; either version 2 of the License, or (at your +  option) any later version. See the file COPYING for further details. + +*/ + +/** @file engine-buffer-metadata.hpp + ** Coordination of buffer type registration at Render Engine level. + ** Within the Render Node Network, access to buffer management is abstracted in the form + ** of [Buffer Descriptors](\ref BuffDescr) and [Buffer Handles](\ref BuffHandle). And since + ** processing of media data is organised into jobs that are performed by worker(threads), + ** the necessity arises to handle parts of the buffer allocation and coordination locally, + ** within thread-local »satellite« services — which implies there must be some central hub + ** to exchange and coordinate metadata globally. Data processing within the Engine requires + ** all kind of different buffer sizes and memory layout, and so this diversity is abstracted + ** into the notion of a **Buffer Type**. The corresponding metadata entry defines, first and + ** foremost, the _size_ of the allocation. And, in addition, it _can_ define a TypeHandler + ** with a constructor- and destructor functor, to allow automated setup of specialised layout + ** within the buffer allocation. Furthermore, a LocalTag can be added that provides additional + ** implementation level instructions to the underlying memory manager. + ** + ** # Setup + ** + ** The EngineBufferMetadata hub is a central service, to be accessed through dependency-injection. + ** A single service instance will be created as part of booting up the Render Engine. At this point, + ** a corresponding setup for the BufferProvider is established, which relies on thread-local parts + ** within the implementation backend services. Notable the BufferProvider::BufferStage implementation, + ** which is provided by class LocalBufferStage, establishes a buffer metadata registry table for each + ** worker thread; under specific and well defined conditions, at certain points in the buffer lifecycle + ** a synchronisation must be made with the central metadata hub, which also involves a global lock to + ** prevent data corruption by concurrent access, since at those synchronisation points, a complete + ** exchange of metadata must be ensured, and completed in a single synchronous call. These data + ** synchronisation activities are initiated by the thread-local BufferMetadata tables, and the + ** implementation will rely on lib::Depend for access. + ** + ** @todo WIP-WIP 4/2026 prototyping to work towards a buffer management scheme. + ** + ** @see LocalBufferStage_test + ** @see BufferMetadata_test + ** @see buffer-provider.hpp + ** @see BufferProviderProtocol_test + */ + +#ifndef VAULT_MEM_ENGINE_BUFFR_METADAZ_H +#define VAULT_MEM_ENGINE_BUFFR_METADAZ_H + + +#include "lib/error.hpp" +//#include "lib/hash-value.h" +#include "include/logging.h" +#include "vault/mem/buffer-metadata.hpp" +#include "lib/nocopy.hpp" +#include "lib/util.hpp" + +#include + +namespace vault{ +namespace mem { +// namespace err = lumiera::error; + + using std::move; + using util::unConst; + + + + + + /** + * Exchange hub for buffer metadata in the Render Engine. + */ + class EngineBufferMetadata + : util::NonCopyable + { + BufferMetadata registry_; + ///////////////////////////TICKET #854 : ensure proper locking happens "somewhere" when mutating metadata + + public: + using Key = metadata::Key; + using Entry = metadata::Entry; + + EngineBufferMetadata() + : registry_{} + { } + + + bool + isKnown (HashVal metaID) const + { + return registry_.isKnown (metaID); + } + + bool + isAllotted (HashVal metaID) const + { + return registry_.isLocked (metaID); + } + + bool + isAccessible (HashVal metaID) const + { + return registry_.isAccessible (metaID); + } + + Key const& + lookup (HashVal metaID) const + { + return isKnown (metaID)? registry_.get (metaID) + : metadata::Key::INVALID; + } + + Key const& + defineBufferType (HashVal familyID, size_t buffSiz, TypeHandler handlerFunctions, LocalTag localTag) + { + return lookup (registry_.key (familyID, buffSiz, move(handlerFunctions), localTag)); + } // possibly create new entry, and return (stable) reference to it + + + /** @warning potentially dangerous; + * must ensure that the entry is not used and registered in any worker thread! + */ + bool + abandon (HashVal metaID, bool invokeDtor =true) + { + Key const& key = lookup (metaID); + Entry& metaEntry = static_cast (unConst(key)); + if (metaEntry.isTypeKey()) + return false; + metaEntry.invalidate (invokeDtor); + registry_.release (metaEntry); + return true; + } + }; + + + + +}} // namespace vault::mem +#endif /*VAULT_MEM_ENGINE_BUFFR_METADAZ_H*/ diff --git a/src/vault/mem/simple-buffer-state-registry.hpp b/src/vault/mem/simple-buffer-state-registry.hpp index 190d58a0b..8819ceee6 100644 --- a/src/vault/mem/simple-buffer-state-registry.hpp +++ b/src/vault/mem/simple-buffer-state-registry.hpp @@ -69,7 +69,7 @@ namespace mem { ID defineBufferType (size_t buffSiz, TypeHandler handlerFunctions, LocalTag localTag) { - return lookup (metadata_.key (familyID_, buffSiz, move (handlerFunctions), localTag)); + return lookup (metadata_.key (familyID_, buffSiz, move(handlerFunctions), localTag)); } // deliberately: maybe create storage, and return reference to it ID diff --git a/tests/vault/mem/local-buffer-stage-test.cpp b/tests/vault/mem/local-buffer-stage-test.cpp index 170335d4b..762cf94e5 100644 --- a/tests/vault/mem/local-buffer-stage-test.cpp +++ b/tests/vault/mem/local-buffer-stage-test.cpp @@ -21,8 +21,9 @@ #include "test/test-frame.hpp" #include "test/test-helper.hpp" #include "vault/mem/buffhandle.hpp" -#include "vault/mem/buffer-metadata.hpp" +#include "vault/mem/engine-buffer-metadata.hpp" #include "vault/mem/local-buffer-stage.hpp" +#include "lib/depend-inject.hpp" #include "lib/symbol.hpp" #include "lib/util.hpp" @@ -177,6 +178,7 @@ namespace test { void verify_publishNewKey() { + lib::DependInject::Local metaHub; } diff --git a/wiki/thinkPad.ichthyo.mm b/wiki/thinkPad.ichthyo.mm index 8f06c621c..181ffbae1 100644 --- a/wiki/thinkPad.ichthyo.mm +++ b/wiki/thinkPad.ichthyo.mm @@ -144503,9 +144503,51 @@ StM_bind(Builder<R1> b1, Extension<R1,R2> extension) - - - + + + + + + + + + + + + + + +

+ also entweder wenn jeman eine lokale Instanz aufruft, um einen neuen Typ zu registrieren, oder wenn ein lokal nicht bekannter type-Key auftritt +

+ + + + + + + + + + + + +

+ soll auf jeden Fall ausschließen, daß dieser Test die zentral-Instanz verschmutzt +

+ + + + + + + + + + + + + @@ -144569,7 +144611,86 @@ StM_bind(Builder<R1> b1, Extension<R1,R2> extension) - + + + + + + + +

+ zwar könnte man einfach eine Instanz von BufferMetadata hinstellen — ich halte das aber nicht für klug, denn BufferMetadata ist Implementierungs-Code (und ich bin nicht glücklich damit, wiewohl ich keine Argumente dagegen finde) +

+ + + + + + + + + + +

+ Keine Funktionalität zum Bedienen konkreter Allokationen +

+ + + + + + + + + + + +
    +
  • + ist NOP wenn es sich um einen typeKey handelt. Diese löschen wir niemals, weil dadruch die Hierarchie zerstört werden könnte +
    • +
  • + kann auf einem stateKey möglicherweise auch die Destruktoren aufrufen (default: true ) +
    • +
  • + aber dann auch den Entry aus der Tabelle tilgen. Sonst könnte er wiederverwendet werden ⟹ würde Konsistenz der Keys brechen +
    • +
+ + + + + + + + + + + +

+ darf also keinerlei Funktionalität aus dem Steam-Layer voraussetzen +

+ + + + + + + + + + +

+ das hier ist keine generische Library, sondern ein Application-Framework. Auch »Vault« ist inhaltlich genau auf Lumiera ausgerichtet, der Unterschied liegt nur im Grad der Integration mit Application-level Services +

+ + + + + + + +