From 0144049f9d8f577c8ccd1c0de1955169a7017cc4 Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Wed, 23 Oct 2024 16:27:09 +0200 Subject: [PATCH] Invocation: reconsider data access and allocator usage from Builder The next step is to round out the first prototypical implementation, which requires access to ''lead node ports'' and thereby generally places focus on the interplay of ''data builders'' within the ongoing build process. While the prototype still uses the fall-back to simple heap allocation, notably the intended usage will require to wire-through the connection to a single `AllocationCluster`. This poses some challenge, since further ''data builders'' will be added step-wise, implying that this wiring can not be completed at construction time. Thus it seems indicated to slightly open-up the internal allocator policy base template used by `lib::SeveralBuilder` to allow for some kind of ''cross building'' based on a shared compatible base allocator type, so that the allocation policy wiring can be passed-on from an existing `SeveralBuilder` --- src/lib/several-builder.hpp | 14 +- src/steam/engine/node-builder.hpp | 30 ++-- src/steam/engine/proc-node.hpp | 8 +- src/steam/engine/weaving-pattern-builder.hpp | 55 ++++---- wiki/thinkPad.ichthyo.mm | 137 +++++++++++++------ 5 files changed, 158 insertions(+), 86 deletions(-) diff --git a/src/lib/several-builder.hpp b/src/lib/several-builder.hpp index e9add188c..f454c2e94 100644 --- a/src/lib/several-builder.hpp +++ b/src/lib/several-builder.hpp @@ -210,6 +210,15 @@ namespace lib { : Allo{std::move (allo)} { } + /** allow cross-initialisation when using same kind of base allocator */ + template + ElementFactory (ElementFactory& relatedFac) + : ElementFactory{relatedFac.baseAllocator()} + { } + + template class XALO> + friend class ElementFactory; + Bucket* create (size_t cnt, size_t spread, size_t alignment =alignof(I)) @@ -403,12 +412,15 @@ namespace lib { SeveralBuilder() = default; /** start Several build using a custom allocator */ - template>> + template>> SeveralBuilder (ARGS&& ...alloInit) : Several{} , Policy{forward (alloInit)...} { } + /** expose policy to configure other ServeralBuilder */ + Policy& policyConnect() { return *this; } + /* ===== Builder API ===== */ diff --git a/src/steam/engine/node-builder.hpp b/src/steam/engine/node-builder.hpp index 93b03ce98..b1fb5d70a 100644 --- a/src/steam/engine/node-builder.hpp +++ b/src/steam/engine/node-builder.hpp @@ -33,6 +33,7 @@ ** some goal oriented building blocks, that can be combined and directed with greater ** clarity by the control structure to govern the build process. ** + ** ** # Levels of connectivity building ** ** The actual node connectivity is established by a process of gradual refinement, @@ -40,7 +41,7 @@ ** builder and descriptor records to collect information, which is then emitted by a ** _terminal invocation_ to produce the result; the higher levels thereby rely on the ** lower levels to fill in and elaborate the details. - ** - Level-1 is the preparation of an actual frame processing operation; the Level-1-builder + ** - *Level-1* is the preparation of an actual frame processing operation; the Level-1-builder ** is in fact the implementation class sitting behind a Render Node's _Port._ It is called ** a _Turnout_ and contains a preconfigured »blue print« for the data structure layout ** used for the invocation; its purpose is to generate the actual data structure on the @@ -48,12 +49,12 @@ ** library functions. Since the actual data processing is achieved by a _pull processing,_ ** originating at the top level exit nodes and propagating down towards the data sources, ** all the data feeds at all levels gradually link together, forming a _TurnoutSystem._ - ** - Level-2 generates the actual network of Render Nodes, which in turn will have the - ** Turnout instances for Level-1 embedded into its internal ports. Conceptually, a + ** - *Level-2* generates the actual network of Render Nodes, which in turn will have the + ** Turnout instances for Level-1 embedded into their internal ports. Conceptually, a ** _Port_ is where data production can be requested, and the processing will then ** retrieve its prerequisite data from the ports of the _Leads,_ which are the ** prerequisite nodes situated one level below or one step closer to the source. - ** - Level-3 establishes the processing steps and data retrieval links between them; + ** - *Level-3* establishes the processing steps and data retrieval links between them; ** at this level, thus the outline of possible processing pathways is established. ** After spelling out the desired connectivity at a high level, the so called »Level-3 build ** walk« is triggered by invoking the [terminal builder operation](\ref ProcBuilder::build() @@ -66,17 +67,17 @@ ** Since the low-level-Model is a massive data structure comprising thousands of nodes, each with ** specialised parametrisation for some media handling library, and a lot of cross-linking pointers, ** it is important to care for efficient usage of memory with good locality. Furthermore, the higher - ** levels of the build process will generate additional temporary data structures, which is gradually - ** refined until the actual render node network can be emitted. Each builder level can thus be - ** outfitted with a custom allocator — typically an instance of lib::AllocationCluster. Notably - ** the higher levels can be attached to a separate AllocationCluster instance, which will be - ** discarded when the build process is complete, while Level-2 (and below) uses the allocator - ** for the actual target data structure, which will be retained and until a complete segment - ** of the timeline is superseded and has been re-built. + ** levels of the build process will generate additional temporary data structures, refined gradually + ** until the actual render node network can be emitted. Each builder level can thus be outfitted + ** with a custom allocator — typically an instance of lib::AllocationCluster. Notably the higher + ** levels can be attached to a separate AllocationCluster instance, which will be discarded once + ** the build process is complete, while Level-2 (and below) uses the allocator for the actual + ** target data structure, which has to be retained while the render graph is used; more + ** specifically until a complete segment of the timeline is superseded and has been re-built. ** @remark syntactically, the custom allocator specification is given after opening a top-level ** builder, by means of the builder function `.withAllocator (args...)` ** - ** @todo WIP-WIP-WIP 7/2024 Node-Invocation is reworked from ground up -- some parts can not be + ** @todo WIP-WIP-WIP 10/2024 Node-Invocation is reworked from ground up -- some parts can not be ** spelled out completely yet, since we have to build this tightly interlocked system of ** code moving bottom up, and then filling in further details later working top-down. ** @@ -129,7 +130,8 @@ namespace engine { template using DataBuilder = lib::SeveralBuilder; - + + template class NodeBuilder; @@ -349,7 +351,7 @@ namespace engine { template PortBuilder(_Par&& base, FUN&& fun) : _Par{move(base)} - , weavingBuilder_{forward (fun)} + , weavingBuilder_{forward (fun), _Par::leads_.policyConnect()} { } friend class PortBuilderRoot; diff --git a/src/steam/engine/proc-node.hpp b/src/steam/engine/proc-node.hpp index b5725cad9..8e8ad3f7e 100644 --- a/src/steam/engine/proc-node.hpp +++ b/src/steam/engine/proc-node.hpp @@ -130,7 +130,7 @@ namespace engine { /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1367 : Rebuild the Node Invocation /* ==== strategy API for configuring the node operation ==== */ - friend class ProcNode; /////////////////////////////////TODO 1/12 : wouldn't it be better to extract that API into a distinct strategy? + friend class ProcNode; /////////////////////////////////OOO who needs friendship? /** the wiring-dependent part of the node operation. * Includes the creation of a one-way state object on the stack @@ -177,6 +177,12 @@ namespace engine { public: /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1367 : Rebuild the Node Invocation + Port& + getPort (uint portIdx) + { + REQUIRE (portIdx <= wiring_.ports.size()); + return wiring_.ports[portIdx]; + } /** Engine Core operation: render and pull output from this node. * On return, currentProcess will hold onto output buffer(s) diff --git a/src/steam/engine/weaving-pattern-builder.hpp b/src/steam/engine/weaving-pattern-builder.hpp index afb9e5db0..e10f1bf15 100644 --- a/src/steam/engine/weaving-pattern-builder.hpp +++ b/src/steam/engine/weaving-pattern-builder.hpp @@ -39,12 +39,13 @@ ** inherits from an *Invocation Adapter* given as template parameter. So this constitutes ** an *extension point* where other, more elaborate invocation schemes could be integrated. ** + ** ** # Interplay of NodeBuider, PortBuilder and WeavingBuilder ** - ** The steam::engine::WeavingBuilder defined here serves as the low-level builder and adapter to - ** prepare the wiring and invocation. The builder-API allows to setup the wiring of input and - ** output-»slots« and control some detail aspects like caching. However, without defining any - ** connections explicitly, a simple 1:1 wiring scheme is employed + ** The steam::engine::WeavingBuilder defined here serves as the low-level builder and adapter + ** to prepare the wiring and invocation. The builder-API allows to setup the wiring of input + ** and output-»slots« and control some detail aspects like caching. However, without defining + ** any connections explicitly, a simple 1:1 wiring scheme is employed ** - each _input slot_ of the function gets an input buffer, which is filled by _pulling_ ** (i.e. invoking) a predecessor node (a so called »lead«). ** - for each _output slot_ a buffer is allocated for the processing function to drop off @@ -52,8 +53,8 @@ ** - only one of these output buffers is used as actual result, while the other buffers ** are just discarded (but may possibly be fed to the frame cache). ** - ** Each [Processing Node](\ref ProcNode) represents one specific processing functionality on - ** a logical level; yet such a node may be able to generate several „flavours“ of this processing, + ** Each [Processing Node](\ref ProcNode) represents one specific processing functionality on a + ** logical level; yet such a node may be able to generate several „flavours“ of this processing, ** which are represented as *ports* on this node. Actually, each such port stands for one specific ** setup of a function invocation, with appropriate _wiring_ of input and output connections. ** For example, an audio filtering function may be exposed on port-#1 for stereo sound, while @@ -62,13 +63,13 @@ ** The WeavingBuilder is used to generate a single \ref Turnout object, which corresponds to ** the invocation of a single port and thus one flavour of processing. ** - ** On the architectural level above, the \ref NodeBuilder exposes the ability to set up a + ** At one architectural level above, the \ref NodeBuilder exposes the ability to set up a ** ProcNode, complete with several ports and connected to possibly several predecessor nodes. - ** Using several NodeBuilder invocations, the _processing node graph_ can be built up starting - ** from the source (predecessors) and moving up to the _exit nodes,_ which produce the desired - ** calculation results. The NodeBuilder offers a function to define the predecessor nodes - ** (also designated as _lead nodes_), and it offers an entrance point to descend into a - ** PortBuilder, allowing to add the port definitions for this node step by step. + ** Using a sequence of NodeBuilder invocations, the _processing node graph_ can be built gradually, + ** starting from the source (predecessors) and moving up to the _exit nodes,_ which produce the + ** desired calculation results. The NodeBuilder offers a function to define the predecessor nodes + ** (also designated as _lead nodes_), and it offers an [entrance point](\ref NodeBuilder::preparePort) + ** to descend into a \ref PortBuilder, allowing to add the port definitions for this node step by step. ** ** On the implementation level, the PortBuilder inherits from the NodeBuilder and embeds a ** WeavingBuilder instance. Moreover, the actual parametrisations of the NodeBuilder template @@ -80,7 +81,7 @@ ** the low-level memory allocation and object creation functionality. The purpose of this ** admittedly quite elaborate scheme is to generate a compact data structure, with high ** cache locality and without wasting too much memory. Since the exact number of elements - ** and the size of those elements can be concluded only after the builder-API usage has + ** and the size of those elements can be deduced only after the builder-API usage has ** been completed, the aforementioned functional datastructure is used to collect the ** parametrisation information for all ports, while delaying the actual object creation. ** With this technique, it is possible to generate all descriptors or entries of one @@ -355,9 +356,9 @@ namespace engine { using TypeMarker = std::function; using ProviderRef = std::reference_wrapper; - DataBuilder leadPort; - std::vector buffTypes; - std::vector providers; + DataBuilder leadPorts; + std::vector buffTypes; + std::vector providers; uint resultSlot{0}; @@ -365,16 +366,18 @@ namespace engine { FUN fun_; - WeavingBuilder(FUN&& init) - : fun_{move(init)} + template + WeavingBuilder(FUN&& init, INIT&& ...alloInit) + : leadPorts{forward (alloInit)...} + , fun_{move(init)} { } WeavingBuilder attachToLeadPort(ProcNode& lead, uint portNr) { - PortRef portRef; /////////////////////////////////////OOO TODO need Accessor on ProcNode!!!!! - leadPort.append (portRef); - ENSURE (leadPort.size() <= N); + PortRef portRef{lead.getPort (portNr)}; + leadPorts.append (portRef); + ENSURE (leadPorts.size() <= N); return move(*this); } @@ -411,23 +414,23 @@ namespace engine { build() { // discard excess storage prior to allocating the output types sequence - leadPort.shrinkFit(); + leadPorts.shrinkFit(); maybeFillDefaultProviders (buffTypes.size()); REQUIRE (providers.size() == buffTypes.size()); - auto outTypes = DataBuilder{} + auto outTypes = DataBuilder{leadPorts.policyConnect()} .reserve (buffTypes.size()); uint i=0; for (auto& typeConstructor : buffTypes) outTypes.append ( typeConstructor (providers[i++])); - ENSURE (leadPort.size() <= N); - ENSURE (outTypes.size() <= N); + ENSURE (leadPorts.size() <= N); + ENSURE (outTypes.size() <= N); using PortDataBuilder = DataBuilder; // provide a free-standing functor to build a suitable Port impl (≙Turnout) - return [leads = move(leadPort.build()) + return [leads = move(leadPorts.build()) ,types = move(outTypes.build()) ,procFun = move(fun_) ,resultIdx = resultSlot diff --git a/wiki/thinkPad.ichthyo.mm b/wiki/thinkPad.ichthyo.mm index 6dbff3030..2ad191fa3 100644 --- a/wiki/thinkPad.ichthyo.mm +++ b/wiki/thinkPad.ichthyo.mm @@ -8852,9 +8852,7 @@ - - - +

...und wenn die Scheitert, ist das ein compile-Fehler @@ -9065,9 +9063,7 @@ - - - +

Aufrufpunkt: invokeTransformation() @@ -9563,9 +9559,7 @@ - - - +

implementieren ebenfalls expandChildren() @@ -10979,9 +10973,7 @@ - - - +

rLet(77943 < 18446744073709551615) → R @@ -13033,9 +13025,7 @@ - - - +

...denn Reolver ist ein UICoord::Builder und als Solcher non-copyable. @@ -49085,9 +49075,7 @@ - - - +

Abstraktion @@ -49912,9 +49900,7 @@ - - - +

mark: Nachricht downstream @@ -50173,9 +50159,7 @@ - - - +

...ist mir nicht völlig klar, warum das bei einigen Includes auftritt, @@ -50487,9 +50471,7 @@ - - - +

empfängt alle state mark notificatons @@ -50569,9 +50551,7 @@ - - - +

einer könnte für @@ -88172,9 +88152,13 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
- + + + + + @@ -88398,19 +88382,75 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
 + + + + + + + + + + + + + + + + + + + + + + + +

+ die Policies und Adapter haben alle einen pass-through-ctor, welcher letztlich den ganz innen (protected) liegenden C++ -  Allokator initialisiert. Alles, was über diese Standard-C++-Mechanismen hinausgeht, muß über die Policy geregelt und eingebunden werden.  Relevantes Beispiel ist der AllocationCluster, für den in der Policy ein Zugangspunkt für dynamisches Justieren der aktuell letzten Allokation herausgeführt ist. +

+ + + + + + + + + + + + +

+ denn das Standard-C++-Front-end für Allocation-Cluster ist ein std::allocator<byte>, und deshalb sind alle Varianten untereinander kompatibel (und das einzige verbleibende Problem wäre, daß der Allokator eine protected-Basisklasse ist) +

+ + + + + + + + + + + + + - + + + - - - - + + + + - - + + @@ -89341,6 +89381,15 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
+ + + + + + + + + @@ -90103,7 +90152,8 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
 - + + @@ -90371,8 +90421,7 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
Denn std::decay macht genau das, was auch passiert, wenn man ein Argument  by-value nimmt. Und das ist hier auch aus anderen Gründen genau das, was wir brauchen: Egal ob Lambda oder Funktor order Funktions-Referenz, es wird erst mal im Builder materialisiert, und von dort weiter geschoben in die λ-closure, welche im Builder für den InvocationAdapter abgelegt ist; im Prototyp-Beispiel ist das die Klasse DirectFunctionInvocation. Das »Protokoll« für den Turnout und das SimpleWeavingPattern erwartet, daß in diesem Builder eine Fuktion oder ein Funktor buildFeed() gegeben ist. Das bedeutet, für jede Invocation wird eine Kopie der processing-Function gezogen und direkt in den code des InvocationAdapters geinlined. Typischerweise ist die processing-Function selber wiederum als Lamda definiert, und damit erfolgt ein sehr direkter und effizienter Aufruf der eigentlichen Library-Funktion

- - + @@ -90411,8 +90460,7 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
diese setzen den jeweiligen Turnout direkt per emplace() in die Ziel-Storage

- - + @@ -90571,7 +90619,7 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
 - + @@ -90580,6 +90628,7 @@ Date:   Thu Apr 20 18:53:17 2023 +0200

 +