/* NODE-BUILDER.hpp - Setup of render nodes connectivity Copyright (C) 2009, Hermann Vosseler 2024, Hermann Vosseler   **Lumiera** is free software; you can redistribute it and/or modify it   under the terms of the GNU General Public License as published by the   Free Software Foundation; either version 2 of the License, or (at your   option) any later version. See the file COPYING for further details. */ /** @file node-builder.hpp ** Specialised shorthand notation for building the Render Node network. ** During the Builder run, the Render Node network will be constructed by gradually ** refining the connectivity structure derived from interpreting the »high-level Model« ** from the current Session. At some point, it is essentially clear what data streams ** must be produced and what media processing functionality from external libraries ** will be utilised to achieve the goal. This is when the fluent builder notation ** defined in this header comes into play, allowing to package the fine grained and ** in part quite confusing details of parameter wiring and invocation preparation into ** some goal oriented building blocks, that can be combined and directed with greater ** clarity by the control structure to govern the build process. ** ** ** # Levels of connectivity building ** ** The actual node connectivity is established by a process of gradual refinement, ** operating over several levels of abstraction. Each of these levels uses its associated ** builder and descriptor records to collect information, which is then emitted by a ** _terminal invocation_ to produce the result; the higher levels thereby rely on the ** lower levels to fill in and elaborate the details. ** - *Level-1* is the preparation of an actual frame processing operation; the Level-1-builder ** is in fact the implementation class sitting behind a Render Node's _Port._ It is called ** a _Turnout_ and contains a preconfigured »blue print« for the data structure layout ** used for the invocation; its purpose is to generate the actual data structure on the ** stack, holding all the necessary buffers and parameters ready for invoking the external ** library functions. Since the actual data processing is achieved by a _pull processing,_ ** originating at the top level exit nodes and propagating down towards the data sources, ** all the data feeds at all levels gradually link together, forming a _TurnoutSystem._ ** - *Level-2* generates the actual network of Render Nodes, which in turn will have the ** Turnout instances for Level-1 embedded into their internal ports. Conceptually, a ** _Port_ is where data production can be requested, and the processing will then ** retrieve its prerequisite data from the ports of the _Leads,_ which are the ** prerequisite nodes situated one level below or one step closer to the source. ** - *Level-3* establishes the processing steps and data retrieval links between them; ** at this level, thus the outline of possible processing pathways is established. ** After spelling out the desired connectivity at a high level, the so called »Level-3 build ** walk« is triggered by invoking the [terminal builder operation](\ref ProcBuilder::build() ** on the [processing builder](\ref ProcBuilder) corresponding to the topmost node. This ** build walk will traverse the connectivity graph depth-first, and then start invoking the ** Level-2 builder operations bottom-up to generate and wire up the corresponding Render Nodes. ** ** ## Using custom allocators ** ** Since the low-level-Model is a massive data structure comprising thousands of nodes, each with ** specialised parametrisation for some media handling library, and a lot of cross-linking pointers, ** it is important to care for efficient usage of memory with good locality. Furthermore, the higher ** levels of the build process will generate additional temporary data structures, refined gradually ** until the actual render node network can be emitted. Each builder level can thus be outfitted ** with a custom allocator — typically an instance of lib::AllocationCluster. Notably the higher ** levels can be attached to a separate AllocationCluster instance, which will be discarded once ** the build process is complete, while Level-2 (and below) uses the allocator for the actual ** target data structure, which has to be retained while the render graph is used; more ** specifically until a complete segment of the timeline is superseded and has been re-built. ** @remark syntactically, the custom allocator specification is given after opening a top-level ** builder, by means of the builder function `.withAllocator (args...)` ** ** @todo WIP-WIP-WIP 10/2024 Node-Invocation is reworked from ground up -- some parts can not be ** spelled out completely yet, since we have to build this tightly interlocked system of ** code moving bottom up, and then filling in further details later working top-down. ** ** @see steam::engine::NodeFactory ** @see nodewiring.hpp ** @see node-basic-test.cpp ** */ #ifndef ENGINE_NODE_BUILDER_H #define ENGINE_NODE_BUILDER_H #include "lib/error.hpp" #include "lib/nocopy.hpp" #include "steam/engine/weaving-pattern-builder.hpp" #include "steam/engine/media-weaving-pattern.hpp" #include "steam/engine/param-weaving-pattern.hpp" #include "steam/engine/proc-node.hpp" #include "steam/engine/turnout.hpp" #include "lib/several-builder.hpp" #include "lib/format-string.hpp" #include "lib/index-iter.hpp" #include #include namespace steam { namespace engine { namespace err = lumiera::error; using util::_Fmt; using std::forward; using std::move; namespace { // default policy configuration to use heap allocator struct UseHeapAlloc { template using Policy = lib::allo::HeapOwn; }; // }//(End) policy /** * A builder to collect working data. * Implemented through a suitable configuration of lib::SeveralBuilder, * with a policy configuration parameter to define the allocator to use. */ template using DataBuilder = lib::SeveralBuilder; template class NodeBuilder; template class PortBuilderRoot; template class NodeBuilder : util::MoveOnly { using PortData = DataBuilder; using LeadRefs = DataBuilder; protected: StrView symbol_; LeadRefs leads_; DAT patternData_; public: template NodeBuilder (StrView nodeSymbol, INIT&& ...alloInit) : symbol_{nodeSymbol} , leads_{forward (alloInit)...} { } template NodeBuilder (NodeBuilder&& pred, SizMark, BUILD&& entryBuilder) : symbol_{pred.symbol_} , leads_{move (pred.leads_)} , patternData_{move (pred.patternData_), forward (entryBuilder)} { } template friend class NodeBuilder; NodeBuilder addLead (ProcNode const& lead) { leads_.append (lead); return move(*this); } /** recursively enter detailed setup of a single processing port */ PortBuilderRoot preparePort(); /** * cross-builder function to specify usage of a dedicated *node allocator* * @tparam ALO (optional) spec for the allocator to use * @tparam INIT (optional) initialisation arguments for the allocator * @remarks this is a front-end to the extension point for allocator specification * exposed through lib::SeveralBuilder::withAllocator(). The actual meaning * of the given parameters and the choice of the actual allocator happens * through resolution of partial template specialisations of the extension * point lib::allo::SetupSeveral. Some notable examples * - withAllocator() attaches to a _monostate_ allocator type. * - `withAllocator (ALO allo)` uses a C++ standard allocator * instance `allo`, dedicated to produce objects of type `X` * - `withAllocator (AllocationCluster&)` attaches to a specific * AllocationCluster; this is the most relevant usage pattern */ template class ALO =std::void_t, typename...INIT> auto withAllocator (INIT&& ...alloInit) { using AllocatorPolicy = lib::allo::SetupSeveral; return NodeBuilder{symbol_, forward(alloInit)...}; } /************************************************************//** * Terminal: complete the ProcNode Connectivity defined thus far. */ Connectivity build() { PortData ports; patternData_.collectEntries(ports); return Connectivity{ports.build() ,leads_.build() }; } }; /** Deduction Guide: help the compiler with deducing follow-up NodeBuilder parameters */ template NodeBuilder (NodeBuilder&&, SizMark, BUILD&&) -> NodeBuilder>; template class PortBuilderRoot : protected NodeBuilder { public: NodeBuilder completePort() { static_assert(not sizeof(POL), "can not build a port without specifying a processing function"); } /** setup standard wiring to adapt the given processing function. * @return a PortBuilder specialised to wrap the given \a FUN */ template auto invoke (StrView portSpec, FUN fun); /** setup a »ParamAgentNode« to compute additional parameters * and then delegate into an existing node invocation. */ template auto computeParam(SPEC&&); private: PortBuilderRoot(NodeBuilder&& anchor) : NodeBuilder{move(anchor)} { } friend PortBuilderRoot NodeBuilder::preparePort(); }; /** * @remark while _logically_ this builder-function _descends_ into the * definition of a port, for the implementation we _wrap_ the existing * NodeBuilder and layer a PortBuilder subclass „on top“ — thereby shadowing * the enclosed original builder temporarily; the terminal builder operation * PortBuilder::completePort() will unwrap and return the original NodeBuilder. */ template inline PortBuilderRoot NodeBuilder::preparePort () { return PortBuilderRoot{move(*this)}; } template class PortBuilder : public PortBuilderRoot { using _Par = PortBuilderRoot; WAB weavingBuilder_; uint defaultPort_; public: template PortBuilder createBuffers (ARGS&& ...args) { UNIMPLEMENTED ("define builder for all buffers to use"); return move(*this); } /** define the output slot to use as result * @remark default is to use the first one */ PortBuilder asResultSlot (uint r) { weavingBuilder_.selectResultSlot(r); return move(*this); } /** connect the next input slot to existing lead-node given by index */ PortBuilder connectLead (uint idx) { return connectLeadPort (idx, this->defaultPort_); } /** connect the next input slot to either existing or new lead-node" */ PortBuilder conectLead (ProcNode& leadNode) { return connectLeadPort (leadNode, this->defaultPort_); } /** connect next input to lead-node, using a specific port-number */ PortBuilder connectLeadPort (uint idx, uint port) { if (idx >= _Par::leads_.size()) throw err::Logic{_Fmt{"Builder refers to lead-node #%d, yet only %d are currently defined."} % idx % _Par::leads_.size() ,LERR_(INDEX_BOUNDS) }; weavingBuilder_.attachToLeadPort (_Par::leads_[idx], port); return move(*this); } /** connect next input to existing or new lead-node, with given port-number */ PortBuilder connectLeadPort (ProcNode& leadNode, uint port) { uint knownEntry{0}; for (auto& lead : lib::IndexIter{_Par::leads_}) if (util::isSameObject (leadNode, lead)) break; else ++knownEntry; if (knownEntry == _Par::leads_.size()) _Par::addLead (leadNode); ENSURE (knownEntry < _Par::leads_.size()); weavingBuilder_.attachToLeadPort (knownEntry, port); return move(*this); } /** use given port-index as default for all following connections */ PortBuilder useLeadPort (uint defaultPort) { this->defaultPort_ = defaultPort; return move(*this); } /** * Embed the explicitly given parameter-functor into the FeedPrototype, * so that it will be called on each Node invocation to generate parameters * to be passed into the actual processing function. The TurnoutSystem acts * as source for the base coordinates, typically the _absolute nominal Time._ * @return adapted PortBuilder marked with the `FeedPrototype` holding \a PFX */ template auto attachParamFun (PFX paramFunctor) { using AdaptedWeavingBuilder = typename WAB::template Adapted; using AdaptedPortBuilder = PortBuilder; // return AdaptedPortBuilder{move(*this) ,weavingBuilder_.adaptParam (move (paramFunctor)) }; } template auto attachAutomation (AUTO&& aFun) { return attachParamFun ([automation = forward(aFun)] (TurnoutSystem& turnoutSys) { Time nomTime = Time::ZERO; ////////////////////OOO need to retrieve that from turnoutSys return automation(nomTime); }); } template auto setParam (PAR paramVal) { return attachParamFun ([=](TurnoutSystem&) -> PAR { return paramVal; }); } /*************************************************************//** * Terminal: complete the Port wiring and return to the node level. * @remark this prepares a suitable Turnout instance for a port; * but due to constraints with memory allocation, actual build * is delayed and packaged as functor into a PatternData instance. */ auto completePort() { weavingBuilder_.connectRemainingInputs (_Par::leads_, this->defaultPort_); return NodeBuilder{static_cast&&> (*this) // slice away PortBulder subclass data ,weavingBuilder_.sizMark ,weavingBuilder_.build()}; } // chain to builder with extended patternData private: template PortBuilder(_Par&& base, FUN&& fun, StrView portSpec) : _Par{move(base)} , weavingBuilder_{forward (fun), _Par::symbol_, portSpec, _Par::leads_.policyConnect()} , defaultPort_{_Par::patternData_.size()} { } // ^^^ by default use next free port friend class PortBuilderRoot; /** cross-builder to adapt embedded WeavingBuilder type */ template PortBuilder (PortBuilder&& prevBuilder, WAB&& adaptedWeavingBuilder) : _Par{move(prevBuilder)} , weavingBuilder_{move (adaptedWeavingBuilder)} , defaultPort_{prevBuilder.defaultPort_} { } template friend class PortBuilder; }; /** * @param qualifier a semantic distinction of the implementation function * @param fun invocation of the actual _data processing operation._ * @remarks * - a _»weaving pattern«_ is applied for the actual implementation, which amounts * to a specific style how to route data input and output and how to actually integrate * with the underlying media handling library, which exposes the processing functionality. * - the standard case of this connectivity is to associate input and output connections * directly with the »parameter slots« of the processing function; a function suitable * for this pattern takes two arguments (input, output) — each of which is a std::array * of buffer pointers, corresponding to the »parameter slots« * - what is bound as \a FUN here thus typically is either an adapter function provided by * the media-library plug-in, or it is a lambda directly invoking implementation functions * of the underlying library, using a buffer type (size) suitable for this library and for * the actual media frame data to be processed. * - the `fun` is deliberately _taken by-value_ and then moved into a »prototype copy« within * the generated `Turnout`, from which an actual copy is drawn anew for each node invocation. * - notably this implies that the implementation code of a lambda will be _inlined_ into the * actual invocation call, while possibly _creating a copy_ of value-captured closure data; * this arrangement aims at exposing the actual invocation for the optimiser. */ template template auto PortBuilderRoot::invoke (StrView portSpec, FUN fun) { using Prototype = typename FeedManifold::Prototype; using WeavingBuilder_FUN = WeavingBuilder; return PortBuilder{move(*this), move(fun), portSpec}; } /** * Nested sub-Builder analogous to \ref PortBuilder, but for building a _»Param Agent Node«._ * This will compute additional parameters and make them temporarily accessible through the * TurnoutSystem of the invocation, but only while delegating recursively to another * computation node, which can then draw upon these additional parameter values. * @tparam SPEC a ParamBuildSpec, which is a sub-builder to define the parameter-functors * evaluated on each invocation to retrieve the actual parameter values */ template class ParamAgentBuilder : public PortBuilderRoot { using _Par = PortBuilderRoot; using BlockBuilder = typename SPEC::BlockBuilder; using PostProcessor = function; BlockBuilder blockBuilder_; PostProcessor postProcessor_; Port* delegatePort_; uint defaultPortNr_; public: /*********************************************************************//** * Terminal: complete the Param-Agent wiring and return to the node level. * @remark this prepares a suitable Turnout instance for a port; it will * actually built later, together with other ports of this Node. */ auto completePort() { if (not delegatePort_) throw err::Logic{"Building a ParamAgentNode requires a delegate node " "to perform within the scope with extended parameters" ,LERR_(BOTTOM_VALUE)}; string portSpec = "Par+"+delegatePort_->procID.genProcSpec(); using WeavingPattern = ParamWeavingPattern; using TurnoutWeaving = Turnout; using PortDataBuilder = DataBuilder; return NodeBuilder ( static_cast&&> (*this) // slice away PortBulder subclass data , SizMark{} ,// prepare a builder-λ to construct the actual Turnout-object [procID = ProcID::describe(_Par::symbol_,portSpec) ,builder = move(blockBuilder_) ,postProc = move(postProcessor_) ,delegate = delegatePort_ ] (PortDataBuilder& portData) mutable -> void { portData.template emplace (procID ,move(builder) ,move(postProc) ,*delegate ); }); } // chain back up to Node-Builder with extended patternData private: template ParamAgentBuilder(_Par&& base, BlockBuilder&& builder) : _Par{move(base)} , blockBuilder_{move(builder)} , delegatePort_{nullptr} , defaultPortNr_{_Par::patternData_.size()} { } // ^^^ by default use next free port friend class PortBuilderRoot; }; template template auto PortBuilderRoot::computeParam(SPEC&& spec) { using ParamBuildSpec = std::decay_t; return ParamAgentBuilder{spec.makeBlockBuilder()}; } /** * Entrance point for building actual Render Node Connectivity (Level-2) * @note when using a custom allocator, the first follow-up builder function * to apply should be `withAllocator(args...)`, prior to adding * any further specifications and data elements. */ inline auto prepareNode (StrView nodeSymbol) { return NodeBuilder{nodeSymbol}; } class ProcBuilder : util::MoveOnly { public: void //////////////////////////////////////////////////////////OOO return type requiredSources () { UNIMPLEMENTED ("enumerate all source feeds required"); // return move(*this); } void //////////////////////////////////////////////////////////OOO return type retrieve (void* streamType) { UNIMPLEMENTED ("recursively define a predecessor feed"); // return move(*this); } /****************************************************//** * Terminal: trigger the Level-3 build walk to produce a ProcNode network. */ void //////////////////////////////////////////////////////////OOO return type build() { UNIMPLEMENTED("Level-3 build-walk"); } }; class LinkBuilder : util::MoveOnly { public: void //////////////////////////////////////////////////////////OOO return type from (void* procAsset) { UNIMPLEMENTED ("recursively enter definition of processor node to produce this feed link"); // return move(*this); } }; /** * Entrance point for defining data flows and processing steps. */ inline auto retrieve(void* streamType) { UNIMPLEMENTED("start a connectivity definition at Level-3"); return LinkBuilder{}; ///////////////////////////////////////////////////////////////////OOO this is placeholder code; should at least open a ticket } }} // namespace steam::engine #endif /*ENGINE_NODE_BUILDER_H*/