benn/lumiera_documentation
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

BufferProvider: draft setup for engine-wide coordination hub

Browse source 
What I am building step by step right now is the first draft
of a production-grade render buffer management scheme.
For this I need suitable implementations of `BufferStag` (and `BufferStore`)
These will include a `LocalSlice` to hold thread-local »satelitte services«,
which in turn need a connection to some central coordination hub.

The thread-local instances can access the hub via our `lib::Depend` (I have just
verified that this works as intended, with on-demand creation of the
central instance on first access from some thread. This is important,
since very likely we need to install a dedicated and speficically wired
service instance, which must happen prior to first dependency-request
via `lib::Depend`. And since the worker-threads are created by the Engine,
we get ample opportunity to do so before starting those threads.

Included in this changeset is a first rough draft of the central registry
`EngineBufferMetadata` — I have modelled it analogous to the `SimpleBufferStateRegistry`,
which is the test-level implementation of the `BufferStage` API.
However, we do not want to expose direct buffer allocation operations
on the central hub, and thus I have left out the respecive parts of the API.

Furthermore, I have added an `abandon()` operation, without there beeing
a need for such an operation right now. But since this operation is tricky
and potentially dangerous, I prefer implementing it the correct way,
since I have a clear understanding what this entails at the moment.
This commit is contained in:
Fischlurch 2026-04-19 18:02:38 +02:00
parent 1601838037
commit 57cd51e3a2
6 changed files with 367 additions and 15 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

34
src/vault/mem/buffer-metadata.hpp
View file

@ -2,7 +2,7 @@
BUFFER-METADATA.hpp - internal metadata for data buffer providers
Copyright (C)
2011, Hermann Vosseler <Ichthyostega@web.de>
2011,2026 Hermann Vosseler <Ichthyostega@web.de>
  **Lumiera** is free software; you can redistribute it and/or modify it
  under the terms of the GNU General Public License as published by the
@ -41,11 +41,29 @@
**
** @todo 2/2026 After refactoring the BufferProvider and splitting up the implementation
** in two realms (state and storage), the purpose of this code her became much
** clearer. The problem remains however, that the BufferMetadata directly links
** to the one central metadata hashtable. We'll have to reconsider this aspect
** once we have to build a metadata store that works well in a massively
** concurrent environment, where requests for state transition emanate
** any time form within all worker threads!
** clearer. The problem remains however, that the BufferMetadata directly
** maintains its metadata in an embedded hashtable. This is problematic
** when operating within a massively concurrent environment.
** @todo 4/2026 After resolving the problems related to BufferProvidder and OutputSlot,
** the next task is to build a first draft implementation for a production-grade
** buffer allocator and manager. I decided to employ thread-local »satellite«
** tables, and a central EngineBufferMetadata hub. It remains to be seen
** if this design is sustainable.
**
** @warning BufferMetadata entails an intricate setup of chained hash keys, which relies
** on several implicit assumptions regarding consistency. These are largely unchecked
** (beyond the unit test) and guaranteed only through the call sequence in proper use.
** - there are Key entries that represent a _buffer type_ — these must not be mutated.
** - the hash value for these typeKeys is totally determined by their properties
** - a child entry can only vary _one_ of these properties, and the change should
** be applied in the fixed sequence as defined in BufferMetadata::key()
** - any violation to this scheme might lead to *duplicate entries* -> buffer corruption
** - active state entries should only added as leaves, and should be removed after use.
** - the hash of an active entry must reflect its associated memory address.
** - Warning: this is only ensured by invoking Key::forEntry, but not persisted in the
** Key itself. A directly computed hash of such a Key differs from the explicitly
** associated hashID_. If the associated buffer* is changed, the entry can not be
** found anymore, which might lead to double allocation and memory corruption.
**
** @see buffer-provider.hpp
** @see BufferMetadata_test
@ -591,11 +609,9 @@ namespace mem {
using Entry = metadata::Entry;
/** establish a metadata registry.
* Such will maintain a family of buffer type entries
* Such will maintain families of chained buffer type entries
* and provide a service for storing and retrieving metadata
* for concrete buffer entries associated with these types.
* @param implementationID to distinguish families of
* type keys belonging to different registries.
*/
BufferMetadata()
: table_{}

68
src/vault/mem/engine-buffer-manager.hpp Normal file
View file

@ -0,0 +1,68 @@
/*
ENGINE-BUFFER-MANAGER.hpp - central buffer allocation coordinator
Copyright (C)
2026, Hermann Vosseler <Ichthyostega@web.de>
  **Lumiera** is free software; you can redistribute it and/or modify it
  under the terms of the GNU General Public License as published by the
  Free Software Foundation; either version 2 of the License, or (at your
  option) any later version. See the file COPYING for further details.
*/
/** @file engine-buffer-manager.hpp
** Central hub of buffer management within the Lumiera Render Engine.
**
** @todo WIP-WIP-WIP 4/2026
**
** @see buffer-provider.hpp
** @see LocalBufferStore_test
** @see BufferProviderProtocol_test
*/
#ifndef VAULT_MEM_ENGINE_BUFFR_MANAGA_H
#define VAULT_MEM_ENGINE_BUFFR_MANAGA_H
#include "lib/error.hpp"
#include "lib/hash-value.h"
#include "include/logging.h"
#include "vault/mem/buffhandle.hpp"
#include "vault/mem/type-handler.hpp"
#include "vault/mem/buffer-local-tag.hpp"
#include "lib/nocopy.hpp"
#include "lib/util.hpp"
#include <unordered_map>
namespace vault{
namespace mem {
using lib::HashVal;
using util::unConst;
namespace err = lumiera::error;
/** */
namespace { // internal blah
}
/* === deaf === */
}} // namespace vault::mem
#endif /*VAULT_MEM_ENGINE_BUFFR_MANAGA_H*/

145
src/vault/mem/engine-buffer-metadata.hpp Normal file
View file

@ -0,0 +1,145 @@
/*
ENGINE-BUFFER-METADATA.hpp - central buffer metadata registry
Copyright (C)
2026, Hermann Vosseler <Ichthyostega@web.de>
  **Lumiera** is free software; you can redistribute it and/or modify it
  under the terms of the GNU General Public License as published by the
  Free Software Foundation; either version 2 of the License, or (at your
  option) any later version. See the file COPYING for further details.
*/
/** @file engine-buffer-metadata.hpp
** Coordination of buffer type registration at Render Engine level.
** Within the Render Node Network, access to buffer management is abstracted in the form
** of [Buffer Descriptors](\ref BuffDescr) and [Buffer Handles](\ref BuffHandle). And since
** processing of media data is organised into jobs that are performed by worker(threads),
** the necessity arises to handle parts of the buffer allocation and coordination locally,
** within thread-local »satellite« services — which implies there must be some central hub
** to exchange and coordinate metadata globally. Data processing within the Engine requires
** all kind of different buffer sizes and memory layout, and so this diversity is abstracted
** into the notion of a **Buffer Type**. The corresponding metadata entry defines, first and
** foremost, the _size_ of the allocation. And, in addition, it _can_ define a TypeHandler
** with a constructor- and destructor functor, to allow automated setup of specialised layout
** within the buffer allocation. Furthermore, a LocalTag can be added that provides additional
** implementation level instructions to the underlying memory manager.
**
** # Setup
**
** The EngineBufferMetadata hub is a central service, to be accessed through dependency-injection.
** A single service instance will be created as part of booting up the Render Engine. At this point,
** a corresponding setup for the BufferProvider is established, which relies on thread-local parts
** within the implementation backend services. Notable the BufferProvider::BufferStage implementation,
** which is provided by class LocalBufferStage, establishes a buffer metadata registry table for each
** worker thread; under specific and well defined conditions, at certain points in the buffer lifecycle
** a synchronisation must be made with the central metadata hub, which also involves a global lock to
** prevent data corruption by concurrent access, since at those synchronisation points, a complete
** exchange of metadata must be ensured, and completed in a single synchronous call. These data
** synchronisation activities are initiated by the thread-local BufferMetadata tables, and the
** implementation will rely on lib::Depend<EngineBufferMetadata> for access.
**
** @todo WIP-WIP 4/2026 prototyping to work towards a buffer management scheme.
**
** @see LocalBufferStage_test
** @see BufferMetadata_test
** @see buffer-provider.hpp
** @see BufferProviderProtocol_test
*/
#ifndef VAULT_MEM_ENGINE_BUFFR_METADAZ_H
#define VAULT_MEM_ENGINE_BUFFR_METADAZ_H
#include "lib/error.hpp"
//#include "lib/hash-value.h"
#include "include/logging.h"
#include "vault/mem/buffer-metadata.hpp"
#include "lib/nocopy.hpp"
#include "lib/util.hpp"
#include <utility>
namespace vault{
namespace mem {
// namespace err = lumiera::error;
using std::move;
using util::unConst;
/**
* Exchange hub for buffer metadata in the Render Engine.
*/
class EngineBufferMetadata
: util::NonCopyable
{
BufferMetadata registry_;
///////////////////////////TICKET #854 : ensure proper locking happens "somewhere" when mutating metadata
public:
using Key = metadata::Key;
using Entry = metadata::Entry;
EngineBufferMetadata()
: registry_{}
{ }
bool
isKnown (HashVal metaID) const
{
return registry_.isKnown (metaID);
}
bool
isAllotted (HashVal metaID) const
{
return registry_.isLocked (metaID);
}
bool
isAccessible (HashVal metaID) const
{
return registry_.isAccessible (metaID);
}
Key const&
lookup (HashVal metaID) const
{
return isKnown (metaID)? registry_.get (metaID)
: metadata::Key::INVALID;
}
Key const&
defineBufferType (HashVal familyID, size_t buffSiz, TypeHandler handlerFunctions, LocalTag localTag)
{
return lookup (registry_.key (familyID, buffSiz, move(handlerFunctions), localTag));
} // possibly create new entry, and return (stable) reference to it
/** @warning potentially dangerous;
* must ensure that the entry is not used and registered in any worker thread!
*/
bool
abandon (HashVal metaID, bool invokeDtor =true)
{
Key const& key = lookup (metaID);
Entry& metaEntry = static_cast<Entry&> (unConst(key));
if (metaEntry.isTypeKey())
return false;
metaEntry.invalidate (invokeDtor);
registry_.release (metaEntry);
return true;
}
};
}} // namespace vault::mem
#endif /*VAULT_MEM_ENGINE_BUFFR_METADAZ_H*/

2
src/vault/mem/simple-buffer-state-registry.hpp
View file

@ -69,7 +69,7 @@ namespace mem {
ID
defineBufferType (size_t buffSiz, TypeHandler handlerFunctions, LocalTag localTag)
{
return lookup (metadata_.key (familyID_, buffSiz, move (handlerFunctions), localTag));
return lookup (metadata_.key (familyID_, buffSiz, move(handlerFunctions), localTag));
} // deliberately: maybe create storage, and return reference to it
ID

4
tests/vault/mem/local-buffer-stage-test.cpp
View file

@ -21,8 +21,9 @@
#include "test/test-frame.hpp"
#include "test/test-helper.hpp"
#include "vault/mem/buffhandle.hpp"
#include "vault/mem/buffer-metadata.hpp"
#include "vault/mem/engine-buffer-metadata.hpp"
#include "vault/mem/local-buffer-stage.hpp"
#include "lib/depend-inject.hpp"
#include "lib/symbol.hpp"
#include "lib/util.hpp"
@ -177,6 +178,7 @@ namespace test {
void
verify_publishNewKey()
{
lib::DependInject<EngineBufferMetadata>::Local<EngineBufferMetadata> metaHub;
}

129
wiki/thinkPad.ichthyo.mm
View file

@ -144503,9 +144503,51 @@ StM_bind(Builder&lt;R1&gt; b1, Extension&lt;R1,R2&gt; extension)
<node CREATED="1776566302722" ID="ID_563548473" MODIFIED="1776566323627" TEXT="ein release / dtor-call kann nicht synchronisiert werden"/>
</node>
</node>
<node CREATED="1776566422332" ID="ID_773286904" MODIFIED="1776566436334" TEXT="Verbindung zur zentral-Registry">
<node CREATED="1776566441432" ID="ID_83703051" MODIFIED="1776566458537" TEXT="neue Typ-Registrierung im Thread &#x27f9; zur Zentrale publiziert"/>
<node CREATED="1776566459396" ID="ID_20651536" MODIFIED="1776566492396" TEXT="Verwendung im Thread &#x27f8; Typ aus Zentrale synchronisiert"/>
<node BACKGROUND_COLOR="#eef0c5" COLOR="#990000" CREATED="1776566422332" ID="ID_773286904" MODIFIED="1776604505367" TEXT="Verbindung zur zentral-Registry">
<icon BUILTIN="pencil"/>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1776566441432" ID="ID_83703051" MODIFIED="1776604516229" TEXT="neue Typ-Registrierung im Thread &#x27f9; zur Zentrale publiziert">
<node CREATED="1776604591195" ID="ID_1133340482" MODIFIED="1776604688880" TEXT="brauche daf&#xfc;r die zentralen EngineMetadata">
<node COLOR="#5b280f" CREATED="1776604776169" ID="ID_1295263836" MODIFIED="1776604807258" TEXT="Aua &#x2014; jetzt gibt&apos;s doch eine statische Initialisierung">
<icon BUILTIN="button_cancel"/>
</node>
<node CREATED="1776604808417" ID="ID_1839658680" MODIFIED="1776604847186" TEXT="besser &#xfc;ber lib::Depend l&#xf6;sen &#x27f9; initialisiert sich lazy"/>
<node CREATED="1776604894666" ID="ID_1374623359" MODIFIED="1776605011172" TEXT="wird nur relevant an den zweien hier besprochenen Synchronisations-Szenarien">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
also entweder wenn jeman eine lokale Instanz aufruft, um einen neuen Typ zu registrieren, oder wenn ein lokal nicht bekannter type-Key auftritt
</p>
</body>
</html>
</richcontent>
</node>
</node>
<node CREATED="1776604670575" ID="ID_1448023528" MODIFIED="1776604672346" TEXT="Test">
<node COLOR="#338800" CREATED="1776604673415" ID="ID_1473562963" MODIFIED="1776612748341" TEXT="die zentralen EngineMetadata mocken">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
soll auf jeden Fall ausschlie&#223;en, da&#223; dieser Test die zentral-Instanz verschmutzt
</p>
</body>
</html>
</richcontent>
<icon BUILTIN="button_ok"/>
<node CREATED="1776612751815" ID="ID_423973357" MODIFIED="1776612774748" TEXT="verifiziert(Debugger): die globale Instanz ist vorher undefiniert"/>
<node CREATED="1776612775704" ID="ID_1966664848" MODIFIED="1776612787131" TEXT="und wird nachher wieder in undefinierten Zustand zur&#xfc;ckversetzt"/>
<node CREATED="1776612790102" ID="ID_798359000" MODIFIED="1776612794696" TEXT="geiler code">
<icon BUILTIN="ksmiletris"/>
</node>
</node>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1776566459396" ID="ID_20651536" MODIFIED="1776604516229" TEXT="Verwendung im Thread &#x27f8; Typ aus Zentrale synchronisiert"/>
</node>
</node>
<node CREATED="1776204575221" ID="ID_869340284" MODIFIED="1776204579271" TEXT="Store-Slice"/>
@ -144569,7 +144611,86 @@ StM_bind(Builder&lt;R1&gt; b1, Extension&lt;R1,R2&gt; extension)
</node>
<node CREATED="1776204702568" ID="ID_1197574712" MODIFIED="1776204706747" TEXT="WorkerBufferStore"/>
<node CREATED="1776204787207" ID="ID_1425789996" MODIFIED="1776204792254" TEXT="vorerst nur anlegen">
<node CREATED="1776204765269" ID="ID_1404790840" MODIFIED="1776204785977" TEXT="EngineBufferMetadata"/>
<node CREATED="1776204765269" ID="ID_1404790840" MODIFIED="1776204785977" TEXT="EngineBufferMetadata">
<node CREATED="1776605957061" ID="ID_405169256" MODIFIED="1776606058887" TEXT="eigenst&#xe4;ndiger Service">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
zwar k&#246;nnte man einfach eine Instanz von BufferMetadata hinstellen &#8212; ich halte das aber nicht f&#252;r klug, denn BufferMetadata ist Implementierungs-Code (und ich bin nicht gl&#252;cklich damit, wiewohl ich keine Argumente dagegen finde)
</p>
</body>
</html>
</richcontent>
</node>
<node CREATED="1776606067153" ID="ID_1057221878" MODIFIED="1776611491005" TEXT="bekommt ein reduziertes Interface">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
Keine Funktionalit&#228;t zum Bedienen konkreter Allokationen
</p>
</body>
</html>
</richcontent>
<node CREATED="1776611243004" ID="ID_1857604671" MODIFIED="1776611263510" TEXT="nehme die SimpleBufferStateRegistry als Vorbild"/>
<node CREATED="1776611264755" ID="ID_403795481" MODIFIED="1776611278322" TEXT="lasse alle Allokations-bezogenen Aufrufe weg"/>
<node CREATED="1776611279319" ID="ID_436854947" MODIFIED="1776611476802" TEXT="biete eine generalisierte discard()-Operation">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<ul>
<li>
ist NOP wenn es sich um einen typeKey handelt. Diese <b>l&#246;schen wir niemals</b>, weil dadruch die <font color="#bf0707">Hierarchie zerst&#246;rt</font>&#160;werden k&#246;nnte
</li>
<li>
kann auf einem stateKey m&#246;glicherweise auch die Destruktoren aufrufen (default: <font face="Monospaced" color="#644545">true </font>)
</li>
<li>
aber dann auch den Entry aus der Tabelle tilgen. Sonst k&#246;nnte er wiederverwendet werden &#10233; w&#252;rde Konsistenz der Keys <b>brechen</b>
</li>
</ul>
</body>
</html></richcontent>
<icon BUILTIN="messagebox_warning"/>
</node>
</node>
<node CREATED="1776605772600" ID="ID_1300641128" MODIFIED="1776605856910" TEXT="mu&#xdf; aber auf Vault-Level definiert sein">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
darf also keinerlei Funktionalit&#228;t aus dem Steam-Layer voraussetzen
</p>
</body>
</html>
</richcontent>
</node>
<node CREATED="1776605858192" ID="ID_563549971" MODIFIED="1776605951752" TEXT="bleibe aber bei dem Namen &#x2014; weil er klar ist">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
das hier ist keine generische Library, sondern ein Application-Framework. Auch &#187;Vault&#171; ist inhaltlich genau auf Lumiera ausgerichtet, der Unterschied liegt nur im Grad der Integration mit Application-level Services
</p>
</body>
</html>
</richcontent>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1776611511064" ID="ID_419155046" MODIFIED="1776611525083" TEXT="Mutex-Locking einf&#xfc;hren">
<icon BUILTIN="flag-yellow"/>
</node>
</node>
<node CREATED="1776204795573" ID="ID_1001789565" MODIFIED="1776204802934" TEXT="EngineBufferManager"/>
</node>
<node CREATED="1776442826894" ID="ID_509761355" MODIFIED="1776442834087" TEXT="OutputSlot mit integrieren">