Invocation: complete demonstration of Node tree with Param Agent (closes #1386)

This is a high-level integration test to sum up this development effort
 * an advanced refactoring was carried out to introduce a
   flexible and fully-typed binding for the ''processing-functor''
 * this entailed a complete rework of the `FeedManifold` to integrate
   inline storage for a ''parameter tuple'' and input / output ''buffer tuples''
 * optional ''parameter functors'' were included into the design at a deep level,
   closely related to the binding of the processing-functor
 * the chosen design is thus a compromise between ''everything nodes''
   and a ''dedicated parameter-handling'' at invocation level

As a proof-of-concept, an scheme to handle extended parameters was devised,
using a special »Param Agent Node« and extension storage blocks in stack memory.
While not immediately necessary, this design exercise proves the overall design
is flexible enough to accommodate future extended needs.

src/steam/engine/node-builder.hpp

@@ -105,6 +105,7 @@
  ** @todo WIP-WIP 1/2025 Node-Invocation is reworked from ground up -- some parts can not be
  **       spelled out completely yet, since we have to build this tightly interlocked system of
  **       code moving bottom up, and then filling in further details later working top-down.
+ ** @todo as of 1/2025, the Layer-3 builder seems to be beyond the current integration scope.  //////////////TICKET #1389 : develop a processing builder
  ** 
  ** @see steam::engine::NodeFactory
  ** @see nodewiring.hpp
@@ -455,6 +456,17 @@ namespace engine {
                                     });
         }
       
+      template<typename GET>
+      auto
+      retrieveParam (GET&& getter)
+        {
+          return attachParamFun ([accessor=forward<GET>(getter)]
+                                 (TurnoutSystem& turnoutSys)
+                                    {
+                                      return accessor.getParamVal (turnoutSys);
+                                    });
+        }
+      
       
       
       /*************************************************************//**
@@ -733,16 +745,17 @@ namespace engine {
     {
     public:
       
-      void //////////////////////////////////////////////////////////OOO return type
+      void //////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1389 : return type
       requiredSources ()
         {
           UNIMPLEMENTED ("enumerate all source feeds required");
 //        return move(*this);
         }
       
-      void //////////////////////////////////////////////////////////OOO return type
+      void //////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1389 : return type
       retrieve (void* streamType)
         {
+          (void)streamType;
           UNIMPLEMENTED ("recursively define a predecessor feed");
 //        return move(*this);
         }
@@ -750,7 +763,7 @@ namespace engine {
       /****************************************************//**
        * Terminal: trigger the Level-3 build walk to produce a ProcNode network.
        */
-      void //////////////////////////////////////////////////////////OOO return type
+      void //////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1389 : return type
       build()
         {
           UNIMPLEMENTED("Level-3 build-walk");
@@ -763,9 +776,10 @@ namespace engine {
     {
     public:
       
-      void //////////////////////////////////////////////////////////OOO return type
+      void //////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1389 : return type
       from (void* procAsset)
         {
+          (void)procAsset;
           UNIMPLEMENTED ("recursively enter definition of processor node to produce this feed link");
 //        return move(*this);
         }
@@ -779,8 +793,9 @@ namespace engine {
   inline auto
   retrieve(void* streamType)
   {
+    (void)streamType;
     UNIMPLEMENTED("start a connectivity definition at Level-3");
-    return LinkBuilder{};  ///////////////////////////////////////////////////////////////////OOO this is placeholder code; should at least open a ticket
+    return LinkBuilder{};  //////////////////////////////////////////////////////////////////////////////////TICKET #1389 : this is placeholder code....
   }
   
   

tests/46node.tests

@@ -6,7 +6,7 @@ PLANNED "Proc Node basics" NodeBase_test <<END
 END
 
 
-PLANNED "Proc Node creation" NodeBuilder_test <<END
+TEST "Proc Node creation" NodeBuilder_test <<END
 END
 
 

tests/core/steam/engine/node-builder-test.cpp

@@ -24,7 +24,6 @@
 #include "lib/time/timequant.hpp"
 #include "lib/time/timecode.hpp"
 #include "lib/symbol.hpp"
-//#include "lib/util.hpp"
 
 #include <array>
 
@@ -180,23 +179,23 @@ namespace test  {
       build_connectedNodes()
         {
           using SrcBuffs = array<uint*, 2>;
-          auto detailFun = [](uint param, uint* out)  { *out = 1 + param; };
+          auto sourceFun = [](uint param, uint* out)  { *out = 1 + param; };
           auto joinerFun = [](SrcBuffs src, uint* out){ *out = *src[0] + *src[1]; };
           
-          int peek{0};
+          int peek{-1};
           auto randParam = [&](TurnoutSystem&){ return peek = rani(100); };
           
           
           ProcNode n1{prepareNode("Src1")
                           .preparePort()
-                            .invoke ("fix-val()", detailFun)
+                            .invoke ("fix-val()", sourceFun)
                             .setParam (LIFE_AND_UNIVERSE_4EVER)
                             .completePort()
                           .build()};
           
           ProcNode n2{prepareNode("Src2")
                           .preparePort()
-                            .invoke ("ran-val()", detailFun)
+                            .invoke ("ran-val()", sourceFun)
                             .attachParamFun (randParam)
                             .completePort()
                           .build()};
@@ -211,16 +210,69 @@ namespace test  {
           
           uint res = invokeRenderNode(n3);
           CHECK (res == peek+1 + LIFE_AND_UNIVERSE_4EVER+1 );
+          CHECK (peek != -1);
         }
       
       
-      /** @test TODO 
+      /** @test demonstrate the setup of a »Param Agent Node«
-       * @todo WIP 12/24 🔁 define ⟶ implement
+       *      - perform effectively the same computation as the preceding test
+       *      - but use two new custom parameters in the Param Agent Node
+       *      - pick them up from the nested source nodes by accessor-functors
+       * @todo 12/24 ✔ define ⟶ ✔ implement
        */
       void
       build_ParamNode()
         {
-          UNIMPLEMENTED ("build ParamNode + follow-up-Node");
+          // Note: using exactly the same functors as in the preceding test
+          using SrcBuffs = array<uint*, 2>;
+          auto sourceFun = [](uint param, uint* out)  { *out = 1 + param; };
+          auto joinerFun = [](SrcBuffs src, uint* out){ *out = *src[0] + *src[1]; };
+          
+          int peek{-1};
+          auto randParam = [&](TurnoutSystem&){ return peek = rani(100); };
+          
+          // Step-1 : build a ParamSpec
+          auto spec = buildParamSpec()
+                          .addValSlot (LIFE_AND_UNIVERSE_4EVER)
+                          .addSlot (randParam)
+                          ;
+          auto get0 = spec.makeAccessor<0>();
+          auto get1 = spec.makeAccessor<1>();
+          
+          // Step-2 : build delegate Node tree
+          ProcNode n1{prepareNode("Src1")
+                          .preparePort()
+                            .invoke ("fix-val()", sourceFun)
+                            .retrieveParam (get0)
+                            .completePort()
+                          .build()};
+          
+          ProcNode n2{prepareNode("Src2")
+                          .preparePort()
+                            .invoke ("ran-val()", sourceFun)
+                            .retrieveParam (get1)
+                            .completePort()
+                          .build()};
+          
+          ProcNode n3{prepareNode("Join")
+                          .preparePort()
+                            .invoke ("add()", joinerFun)
+                            .connectLead(n1)
+                            .connectLead(n2)
+                            .completePort()
+                          .build()};
+          
+          // Step-3 : build Param Agent as entry point
+          ProcNode n4{prepareNode("Param")
+                          .preparePort()
+                            .computeParam(spec)
+                            .delegateLead(n3)
+                            .completePort()
+                          .build()};
+          
+          uint res = invokeRenderNode(n4);
+          CHECK (res == peek+1 + LIFE_AND_UNIVERSE_4EVER+1 );
+          CHECK (peek != -1);
         }
     };
   

wiki/renderengine.html

@@ -4767,6 +4767,13 @@ Moreover, the design of coordinate matching and resolving incurs a structure sim
 <div title="NodeCreatorTool" modifier="Ichthyostega" created="200712100626" modified="201006250229" tags="def">
 <pre>~NodeCreatorTool is a [[visiting tool|VisitorUse]] used as second step in the [[Builder]]. Starting out from a [[Fixture]], the builder first [[divides the Timeline into segments|SegmentationTool]] and then processes each segment with the ~NodeCreatorTool to build a render nodes network (Render Engine) for this part of the timeline. While visiting individual Objects and Placements, the ~NodeCreatorTool creates and wires the necessary [[nodes|ProcNode]]</pre>
 </div>
+<div title="NodeFeedManifold" creator="Ichthyostega" modifier="Ichthyostega" created="202501051951" modified="202501051952" tags="Rendering spec draft" changecount="2">
+<pre>//Junction element between Render Node invocation and the binding to an external [[Media-Library Plug-in|MediaLibPlugin]].//
+The Lumiera Render Engine relies on a »working substrate« of [[Render Nodes|ProcNode]], interconnected in accordance with the structure of foreseeable computations. Yet the actual media processing functionality is provided by external libraries — while the engine is arranged in a way to remain //agnostic// regarding any details of actual computation. Those external libraries are attached into the system by means of a Plug-in, which is tasked with the translation of external capabilities into a representation as [[Processing Assets|ProcAsset]]. These can be picked up and used in the Session, and will eventually be visited by the [[Builder]] as part of the effort to establish the aforementioned »[[network of Render Nodes|LowLevelModel]]«.
+
+At one point however, external functionality must actually be connected to internal structures: this is the purpose of the {{{FeedManifold}}}.
+</pre>
+</div>
 <div title="NodeFrameNumbering" modifier="Ichthyostega" created="200810140254" modified="200810160129" tags="spec draft">
 <pre>!Problem of Frame identification
 
@@ -6331,8 +6338,8 @@ In 2018, the middle Layer was renamed into &amp;rarr; Steam-Layer
 
 </pre>
 </div>
-<div title="ProcNode" modifier="Ichthyostega" created="200706220409" modified="202501040420" tags="def spec" changecount="24">
+<div title="ProcNode" modifier="Ichthyostega" created="200706220409" modified="202501052001" tags="def spec" changecount="26">
-<pre>//A data processing node within the Render Engine.// Its key feature is the ability to pull a [[Frame]] of calculated data -- which may involve the invocation of //predecessor nodes// and the evaluation of [[Parameter Providers|ParamProvider]]. Together, those nodes form a »node-network«, a ''D''irected ''A''cyclic ''G''raph (DAG) known as the LowLevelModel, which is //explicit// to the degree that it can be //performed// to generate or process media. As such it is an internal structure and //compiled// by evaluating and interpreting the HighLevelModel within the [[Session]], which is the symbolic or logical representation of »the film edit« or »media product« established and created by the user.
+<pre>//A data processing node within the Render Engine.// Its key feature is the ability to pull a [[Frame]] of calculated data -- which may involve the invocation of //predecessor nodes// and the [[evaluation of Parameters|RenderParamHandling]]. Together, those nodes form a »node-network«, a ''D''irected ''A''cyclic ''G''raph (DAG) known as the LowLevelModel, which is //explicit// to the degree that it can be //performed// to generate or process media. As such it is an internal structure and //compiled// by evaluating and interpreting the HighLevelModel within the [[Session]], which is the symbolic or logical representation of »the film edit« or »media product« established and created by the user.
 
 So each node and all interconnections are //oriented.// Calculation starts from the output side and propagates to acquire the necessary inputs, thereby invoking the predecessor nodes (also known as »leads«). This recursive process leads to performing a sequence of calculations -- just to the degree necessary to deliver the desired result. And since the Render Node Network is generated by a compilation step, which is conducted repeatedly and on-demand by the part of the Lumiera application known as the [[Builder]], all node connectivity can be assumed to be adequate and fitting, each predecessor can be assumed to deliver precisely the necessary data into buffers with the correct format to be directly consumed for the current calculation step. The actual processing algorithm working on these buffers is provided by an ''external Media-processing-library'' -- the LowLevelModel can thus be seen as a pre-determined organisation and orchestration structure to enact externally provided processing functionality.
 &amp;rarr; [[Description of the render process|RenderProcess]]
@@ -7102,9 +7109,12 @@ An Activity is //performed// by invoking its {{{activate(now, ctx)}}} function -
 In a similar vein, also ''dependency notifications'' need to happen decoupled from the activity chain from which they originate; thus the Post-mechanism is also used for dispatching notifications. Yet notifications are to be treated specially, since they are directed towards a receiver, which in the standard case is a {{{GATE}}}-Activity and will respond by //decrementing its internal latch.// Consequently, notifications will be sent through the ''λ-post'' -- which operationally re-schedules a continuation as a follow-up job. Receiving such a notification may cause the Gate to become opened; in this case the trigger leads to //activation of the chain// hooked behind the Gate, which at some point typically enters into another calculation job. Otherwise, if the latch (in the Gate) is already zero (or the deadline has passed), nothing happens. Thus the implementation of state transition logic ensures the chain behind a Gate can only be //activated once.//
 </pre>
 </div>
-<div title="RenderParamHandling" creator="Ichthyostega" modifier="Ichthyostega" created="202501050139" modified="202501050143" changecount="6">
+<div title="RenderParamHandling" creator="Ichthyostega" modifier="Ichthyostega" created="202501050139" modified="202501052000" changecount="8">
 <pre>//Invocation- and ~Configuration-Parameters are essential for rendering.//
 Typically, a processing operation can be configured in various ways, by passing additional setup- and invocation parameters. This entails both technical aspects (like picking some specific data format), organisational concerns (like addressing a specific frame-number) and elements of artistic control, like choosing the settings of a media processing effect. Parameters will thus be collected from various sources, which leads to an additional binding step, where all these sources are retrieved and the actual parameter value or value tuple is produced. This specific //parameter binding// is represented as a [[Parameter-functor|NodeParamFunctor]]. Whenever the processing-function accepts a parameter argument, optionally a such parameter-functor can be installed; this functor is supplied with the TurnoutSystem of the actual invocation, which acts as front-end to access contextual parameters.
+
+The signature of the [[processing-functor|NodeProcFunctor]] is pivotal for configuring the storage layout and invocation structure used for Node activation. Accepting parameters is optional, but when parameters are used, they must be taken as a first argument, either as a single value or as a //C++ tuple of values.// The storage for the parameter values is incorporated into the [[Feed-Manifold|NodeFeedManifold]], which also implies that the [[parameter-functor|NodeParamFunctor]] is evaluated at this point, when the actual {{{FeedManifold}}} is dropped into the local stack frame.
+
 !Synthesised additional parameters
 As an extension for (rare and elaborate) special cases, a special evaluation scheme is provided, which relies on a »''Param Agent Node''« as entry point, to invoke additional functors and compute additional parameter values, which can then be used in a nested //delegate Node tree.//  The solution relies on placing those additional data values into a tuple, which is then stored directly in the render invocation stack frame, prior to descending into the recursive Node evaluations. Parameter-functors within the scope of this evaluation tree can then access these additional parameters through the TurnoutSystem of the overall invocation.