benn/lumiera_
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

initial draft for the job representation

Browse source 
this draft is based on
 - Cehteh's draft for the scheduler
 - my plannings about segmentation and JobTicket

it defines "Job" as a closure which can be invoked
from plain-C, using the information in the
job descriptor datastructure
This commit is contained in:
Fischlurch 2012-02-17 01:54:51 +01:00
parent e9dbb3bdb1
commit 875342fa40
3 changed files with 170 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

90
src/proc/engine/job.cpp Normal file
View file

@ -0,0 +1,90 @@
/*
Job - render job closure
Copyright (C) Lumiera.org
2012, Hermann Vosseler <Ichthyostega@web.de>
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of
the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
* *****************************************************/
#include "proc/engine/job.hpp"
#include "proc/engine/job-ticket.hpp"
namespace proc {
namespace engine {
namespace { // Details...
} // (END) Details...
// using mobject::Placement;
// using mobject::session::Effect;
/** @todo WIP-WIP 2/12
*/
void
Job::triggerJob (lumiera_jobParameter param) const
{
UNIMPLEMENTED ("how to access the JobTicket and build the RenderInvocation");
}
void
Job::signalFailure (lumiera_jobParameter) const
{
UNIMPLEMENTED ("how to organise job failure and abortion");
}
}} // namespace proc::engine
namespace {
using proc::engine::Job;
inline Job&
forwardInvocation (LumieraJobClosure jobFunctor)
{
Job* job = static_cast<Job*> (jobFunctor);
REQUIRE (job);
REQUIRE (job->isValid());
return *job;
}
}
extern "C" { /* ==== implementation C interface for job invocation ======= */
void
lumiera_job_invoke (LumieraJobClosure jobFunctor, lumiera_jobParameter param)
{
forwardInvocation(jobFunctor).triggerJob (param);
}
void
lumiera_job_failure (LumieraJobClosure jobFunctor, lumiera_jobParameter param)
{
forwardInvocation(jobFunctor).signalFailure (param);
}
}

58
src/proc/engine/job.hpp
View file

@ -29,6 +29,7 @@
typedef void* LList; ////////////////////////////////////TODO
typedef uint64_t InvocationInstanceID; /////////////////TODO
enum JobState
@ -42,6 +43,36 @@ enum JobState
};
/**
* closure representing the execution context of a job.
* The information reachable through this closure is specific
* for this kind of job, but static and typically shared among
* all jobs for a given feed and segment of the timeline
*/
struct lumiera_jobClosure { /* placeholder */ };
typedef struct lumiera_jobClosure* LumieraJobClosure;
/**
* invocation parameter for the individual
* frame calculation job. Embedded into the job descriptor
* and passed to #lumiera_job_invoke when triggering
*/
struct lumiera_jobParameter_struct
{
InvocationInstanceID invoKey;
gavl_time_t nominalTime;
//////////////////////////////////////////////////////////////TODO: place a parameter value here, or make the instanceID globally unique?
};
typedef struct lumiera_jobParameter_struct lumiera_jobParameter;
typedef lumiera_jobParameter* LumieraJobParameter;
/**
* descriptor record used by the scheduler to organise job invocation.
* The invocation parameter and job closure necessary to invoke this
* job as a function is embedded into this descriptor.
*/
struct lumiera_jobDescriptor_struct
{
gavl_time_t when;
@ -53,10 +84,10 @@ struct lumiera_jobDescriptor_struct
JobState jobstate;
void (*jobfn)();
void (*failfn)();
LumieraJobClosure jobClosure;
lumiera_jobParameter parameter;
};
typedef struct lumiera_JobDescriptor_struct lumiera_jobDescriptor;
typedef struct lumiera_jobDescriptor_struct lumiera_jobDescriptor;
typedef lumiera_jobDescriptor* LumieraJobDescriptor;
@ -70,7 +101,6 @@ typedef lumiera_jobDescriptor* LumieraJobDescriptor;
//#include "lib/time/timevalue.hpp"
//#include "lib/time/timequant.hpp"
#include <tr1/functional>
namespace proc {
@ -85,7 +115,7 @@ namespace engine {
/**
* Frame rendering task, represented as closure.
* This functor encodes all information necessary actually to
* This functor encodes all information necessary to actually
* trigger and invoke the rendering operation. It will be embedded
* into a job descriptor and then enqueued with the scheduler for
* invocation just in time.
@ -93,9 +123,11 @@ namespace engine {
* @todo 1/12 WIP-WIP-WIP defining the invocation sequence and render jobs
*/
class Job
: public lumiera_jobClosure
{
public:
//////////////////////////////TODO: value semantics or turn this into an interface?
Job()
{
@ -105,6 +137,10 @@ namespace engine {
// using standard copy operations
void triggerJob (lumiera_jobParameter) const;
void signalFailure (lumiera_jobParameter) const;
bool
isValid() const
{
@ -123,7 +159,17 @@ extern "C" {
#endif /* =========================== CL Interface ===================== */
////////////////////////////////////TODO define a C binding for use by the scheduler
/** trigger execution of a specific job,
* assuming availability of all prerequisites */
void lumiera_job_invoke (LumieraJobClosure, lumiera_jobParameter);
/** signal inability to invoke this job
* @todo decide what and how to communicate details of the failure
* @remarks the purpose of this function is to allow for reliable checkpoints
* within the network of dependent jobs invocations, even after
* missing deadlines or aborting a sequence of jobs */
void lumiera_job_failure (LumieraJobClosure, lumiera_jobParameter);
#ifdef __cplusplus

32
wiki/renderengine.html
View file

@ -2785,13 +2785,13 @@ From experiences with other middle scale projects, I prefer having the test code
[img[Example: Interfaces/Namespaces of the ~Session-Subsystems|uml/fig130053.png]]
</pre>
</div>
<div title="JobTicket" modifier="Ichthyostega" created="201202120018" tags="spec Rendering draft" changecount="1">
<pre>The actual media data is rendered by [[individually scheduled render jobs|RenderJob]]. When preparing such jobs, in order to [[dispatch|FrameDispatch]] the individual frames to be calculated to make a given [[stream of calculations|CalcStream]] happen, a node planning phase is performed to find out
<div title="JobTicket" modifier="Ichthyostega" modified="201202161712" created="201202120018" tags="spec Rendering draft" changecount="7">
<pre>The actual media data is rendered by [[individually scheduled render jobs|RenderJob]]. When preparing such jobs, in order to [[dispatch|FrameDispatcher]] the individual frames calculations required to make a given [[stream of calculations|CalcStream]] happen, a node planning phase is performed. The goal is to find out
* what channel(s) to pull
* what prerequisites to prepare
* what parameters to provide
The result of this planning phase is the JobTicket, a complete execution plan.
This planning is uniform for each segment and treated for all channels together, resulting in a nested tree structure of sub job tickets, allocated and stored alongside with the processing nodes and wiring descriptors forming the segment's data and descriptor network. Job tickets are //higher order functions:// entering a concrete frame number and channel into a given job ticket will produce an actual job descriptor, which in itself is again a function, to be invoked through the scheduler when it's time to trigger the actual calculations.</pre>
The result of this planning phase is the {{{JobTicket}}}, a complete ''execution plan''.
This planning is uniform for each segment and treated for all channels together, resulting in a nested tree structure of sub job tickets, allocated and stored alongside with the processing nodes and wiring descriptors to form the segment's data and descriptor network. Job tickets are //higher order functions:// entering a concrete frame number and channel into a given job ticket will produce an actual job descriptor, which in itself is again a function, to be invoked through the scheduler when it's time to trigger the actual calculations.</pre>
</div>
<div title="LayerSeparationInterface" modifier="Ichthyostega" modified="200904302314" created="200902080635" tags="def" changecount="2">
<pre>A major //business interface// &amp;mdash; used by the layers for interfacing to each other; also to be invoked externally by scripts.
@ -5198,6 +5198,30 @@ We //do employ//&amp;nbsp; some virtual calls for the buffer management in order
@@clear(right):display(block):@@
</pre>
</div>
<div title="RenderJob" modifier="Ichthyostega" modified="201202162316" created="201202162156" tags="spec Rendering" changecount="6">
<pre>An unit of operation, to be [[scheduled|Scheduler]] for calculating media frame data just in time.
Within each CalcStream, render jobs are produced by the associated FrameDispatcher, based on the corresponding JobTicket used as blue print (execution plan).
!Anatomy of a render job
Basically, each render job is a //closure// -- hiding all the prepared, extended execution context and allowing the scheduler to trigger the job as a simple function.
When activated, by virtue of this closure, the actual ''node invocation'' is constructed, which is a private and safe execution environment for the actual calculations. This (mostly stack based) environment embodies a StateProxy, acting as communication hub for accessing anything possibly stateful within the larger scope of the currently ongoing render process. The node invocation sequence is what actually implements the ''pulling of data'': on exit, all cacluated data is expected to be available in the output buffers. Typically (but not necessarily) each node embodies a ''calculation function'', holding the actual data processing algorithm.
!{{red{open questions 2/12}}}
* what are the job's actual parameters?
* how is prerequisite data passed? &amp;rarr; maybe by an //invocation key?//
!Input and closure
each job gets only the bare minimum information required to trigger the execution: the really variable part of the node's invocation. The job uses this pieces of information to re-activate a pre-calculated closure, representing a wider scope of environment information. Yet the key point is for this wider scope information to be //quasi static.// It is shared by a whole [[segment|Segmentation]] of the timeline in question, and it will be used and re-used, possibly concurrently. From the render job's point of view, the engine framework just ensures the availability and accessibility of all this wider scope information.
Prerequisite data for the media calculations can be considered just part of that static environment, as far as the node is concerned. Actually, this prerequisite data is dropped off by other nodes, and the engine framework and the builder ensure the availability of this data just in time.
!observations
* the job's scope represents a strictly local view
* the job doesn't need to know about its output
* the job doesn't need to know anything about the frame grid or frame number
* all it needs to know is the ''effective nominal time'' and an ''invocation instance ID''
</pre>
</div>
<div title="RenderMechanics" modifier="Ichthyostega" modified="201109172221" created="200806030230" tags="Rendering operational impl img" changecount="32">
<pre>While the render process, with respect to the dependencies, the builder and the processing function is sufficiently characterized by referring to the ''pull principle'' and by defining a [[protocol|NodeOperationProtocol]] each node has to adhere to &amp;mdash; for actually get it coded we have to care for some important details, especially //how to manage the buffers.// It may well be that the length of the code path necessary to invoke the individual processing functions is finally not so important, compared with the time spent at the inner pixel loop within these functions. But my guess is (as of 5/08), that the overall number of data moving and copying operations //will be//&amp;nbsp; of importance.
{{red{WIP as of 9/11 -- need to mention the planning phase more explicitly}}}