WIP investigating the relation of Jobs, JobTicket and Closure in detail

src/lib/scoped-collection.hpp

@@ -30,7 +30,7 @@
  ** and never adjusted.
  ** 
  ** \par usage patterns
- ** The common ground for all usage of this container is to hold and some
+ ** The common ground for all usage of this container is to hold some elements
  ** with exclusive ownership; when the enclosing container goes out of scope,
  ** all the dtors of the embedded objects will be invoked. Frequently this
  ** side effect is the reason for using the container: we want to own some

src/proc/engine/job-ticket.cpp

@@ -90,6 +90,12 @@ namespace engine {
   
   
   /**  */
+  Job
+  JobTicket::createJobFor (FrameCoord coordinates)
+  {
+    UNIMPLEMENTED ("job planning and generation");
+  }
+  
   
   
 }} // namespace proc::engine

src/proc/engine/job-ticket.hpp

@@ -101,11 +101,8 @@ namespace engine {
         }
       
       
-      Job
-      createJobFor (FrameCoord coordinates)
-        {
-          UNIMPLEMENTED ("job planning and generation");
-        }
+      Job createJobFor (FrameCoord coordinates);
+      
       
       bool
       isValid()  const

tests/components/proc/engine/dispatcher-interface-test.cpp

@@ -156,7 +156,7 @@ namespace test  {
           JobTicket& executionPlan = dispatcher.accessJobTicket (coordinates);
           CHECK (executionPlan.isValid());
           
-          Job frameJob = executionPlan.createJobFor (coordinates);
+          Job frameJob = executionPlan.createJobFor (coordinates); //////////////////////////////TODO this is wrong: we never create a single job!
           CHECK (frameJob.getNominalTime() == coordinates.absoluteNominalTime);
           CHECK (0 < frameJob.getInvocationInstanceID());
 #if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////UNIMPLEMENTED :: TICKET #880

wiki/renderengine.html

@@ -2797,13 +2797,19 @@ From experiences with other middle scale projects, I prefer having the test code
 [img[Example: Interfaces/Namespaces of the ~Session-Subsystems|uml/fig130053.png]]
 </pre>
 </div>
-<div title="JobTicket" modifier="Ichthyostega" modified="201202161712" created="201202120018" tags="spec Rendering draft" changecount="7">
-<pre>The actual media data is rendered by [[individually scheduled render jobs|RenderJob]]. When preparing such jobs, in order to [[dispatch|FrameDispatcher]] the individual frames calculations required to make a given [[stream of calculations|CalcStream]] happen, a node planning phase is performed. The goal is to find out
+<div title="JobTicket" modifier="Ichthyostega" modified="201204290241" created="201202120018" tags="spec Rendering draft" changecount="12">
+<pre>The actual media data is rendered by [[individually scheduled render jobs|RenderJob]]. When preparing such jobs, in order to [[dispatch|FrameDispatcher]] the individual frames calculations required to make a given [[stream of calculations|CalcStream]] happen, a ''node planning phase'' is performed. The goal is to find out
 * what channel(s) to pull
 * what prerequisites to prepare
 * what parameters to provide
 The result of this planning phase is the {{{JobTicket}}}, a complete ''execution plan''.
-This planning is uniform for each segment and treated for all channels together, resulting in a nested tree structure of sub job tickets, allocated and stored alongside with the processing nodes and wiring descriptors to form the segment's data and descriptor network. Job tickets are //higher order functions:// entering a concrete frame number and channel into a given job ticket will produce an actual job descriptor, which in itself is again a function, to be invoked through the scheduler when it's time to trigger the actual calculations.</pre>
+This planning is uniform for each segment and treated for all channels together, resulting in a nested tree structure of sub job tickets, allocated and stored alongside with the processing nodes and wiring descriptors to form the segment's data and descriptor network. Job tickets are //higher order functions:// entering a concrete frame number and channel into a given job ticket will produce an actual job descriptor, which in itself is again a function, to be invoked through the scheduler when it's time to trigger the actual calculations.
+
+!Structure of the render Jobs created
+To be more precise: in the general case, invoking the ~JobTicket with a given frame number will produce //multiple jobs//  -- typically each frame rendering will require at least one further source media frame; and because Lumiera render jobs will //never block waiting on IO,// this source media access will be packaged as a separate [[resource retrieving job|ResourceJob]], to be treated specifically by the scheduler.
+
+To support the generation of multiple dependent jobs, a ~JobTicket might refer to further ~JobTickets corresponding to the prerequisites. These prerequisite ~JobTickets are the result of a classical recursive descent call into the top level ProcNode, to perform the planning step. There is always an 1:1 relation between actual jobs generated and the corresponding tickets. More precisely, for each possible job to generate there is a suitable ''job closure'', representing exactly the context information necessary to get that job started. To stress that point: a ProcNode might be configured such as to perform a series of recursive invocations of other prerequisite nodes right away (especially when those recursive invocations correspond just to further CPU bound calculations) -- yet still there is only //one//&amp;nbsp; ~JobTicket, because there will be only one Job to invoke the whole sequence. On the other hand, when there is a prerequisite requiring an operation to be scheduled separately, a corresponding separate {{{JobClosure}}} will be referred.
+</pre>
 </div>
 <div title="LayerSeparationInterface" modifier="Ichthyostega" modified="200904302314" created="200902080635" tags="def" changecount="2">
 <pre>A major //business interface// &amp;mdash; used by the layers for interfacing to each other; also to be invoked externally by scripts.
@@ -5305,6 +5311,10 @@ __see also__
 
 Rendering can be seen as a passive service available to the Backend, which remains in charge what to render and when. Render processes may be running in parallel without any limitations. All of the storage and data management falls into the realm of the Backend. The render nodes themselves are ''completely stateless'' &amp;mdash; if some state is necessary for carrying out the calculations, the backend will provide a //state frame// in addition to the data frames.</pre>
 </div>
+<div title="ResourceJob" modifier="Ichthyostega" created="201204290102" tags="def Rendering" changecount="1">
+<pre>a special kind of [[render job|RenderJob]], used to retrieve input data relying on external IO.
+Since a primary goal for the design of Lumiera's render engine is to use the available resources effectively, we try to avoid the situation where a working thread gets blocked waiting  for external IO to deliver data. Thus we create special marker jobs to keep track of any prerequisites and start the actual render calculations only if all these prerequisites are already fulfilled and all the required input data is available in RAM.</pre>
+</div>
 <div title="Rules" modifier="Ichthyostega" modified="200910171620" created="200910171618" tags="overview" changecount="5">
 <pre>A distinct property of the Lumiera application is to rely on a rules based approach rather then on hard wired logic. When it comes to deciding and branching, a [[Query]] is issued, resulting either immediately in a {{{bool}}} result, or creating a //binding// for the variables used within the query. Commonly, there is more than one solution for a given query, allowing the result set to be enumerated.