Job-Planning: start rework of the planning data aggregation

The initial implementation effort for Player and Job-Planning
has been reviewed and largely reworked, and some parts are now
obsoleted by the reworked alternative and can be disabled.

The basic idea will be retained though: JobPlanning is a
data aggregator and performs the final step of creating a Job
This commit is contained in:
Fischlurch 2023-06-15 03:51:07 +02:00
parent f84517547b
commit a551314e80
9 changed files with 214 additions and 46 deletions

View file

@ -40,6 +40,7 @@ namespace engine {
FrameSequencer::~FrameSequencer() { }
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
/////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1301 : likely to become obsolete
/** @todo WIP */
FrameCoord
@ -56,6 +57,7 @@ namespace engine {
return dispatcher_->locateRelative (frame, frameOffset);
}
/////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1301 : likely to become obsolete
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...

View file

@ -91,6 +91,7 @@ namespace engine {
class Dispatcher
: public FrameLocator
{
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
/////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1301 : obsolete
struct JobBuilder
{
@ -109,6 +110,7 @@ namespace engine {
}
};
/////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1301 : obsolete
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
struct PipeFrameTick;
template<class IT>
@ -161,6 +163,7 @@ namespace engine {
protected:
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
/** core dispatcher operation: based on the coordinates of a reference point,
* establish binding frame number, nominal time and real (wall clock) deadline.
* @return new FrameCoord record (copy), with the nominal time, frame number
@ -187,6 +190,7 @@ namespace engine {
////////////TODO: remaining issues
//////////// - the TimeAnchor needs to be created directly from the JobParameter. No mutable state!
//////////// - but this leads to a lot of duplicated Timings records, unless we rewrite the TimeAnchor to be noncopyable and use a Timings const&
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
virtual JobTicket& accessJobTicket (size_t, TimeValue nominalTime) =0;
};

View file

@ -21,40 +21,20 @@
*/
/** @file job-planning.hpp
** The "mechanics" of discovering and planning frame calculation jobs.
** This is a rather abstract chunk of code, to deal especially with the technicalities
** of \em organising the discovery of prerequisites and of joining all the discovered operations
** into a sequence of planning steps. The net result is to present a <i>sequence of job planing</i>
** to the user, while actually encapsulating a depth-first tree exploration, which proceeds on demand.
**
** # participating elements
** All of these job planning operations are implemented on top of the JobTicket. This is where to look
** for "actual" implementation code. Here, within this header, the following entities cooperate to
** create a simple sequence out of this implementation level tasks:
** - JobPlanningSequence is the entry point for client code: it allows to generate a sequence of jobs
** - JobPlanning is a view on top of all the collected planning information for a single job
** - PlanningState is an iterator, successively exposing a sequence of JobPlanning views
** - steam::engine::expandPrerequisites(JobPlanning const&) is the operation to explore further prerequisite Jobs recursively
** - PlanningStepGenerator yields the underlying "master beat": a sequence of frame locations to be planned
**
** # how the PlanningState (sequence) is advanced
** PlanningState is an iterator to expose a sequence of JobPlanning elements. On the implementation level,
** there is always just a single JobPlanning element, which represents the \em current element; this element
** lives as "state core" within the PlanningState object. Advancing to the next JobPlanning element (i.e. to
** consider the next job or prerequisite job to be planned for scheduling) is performed through the iteration
** control API exposed by JobPlanning (the functions `checkPoint()`, `yield()` and `iterNext()`. Actually,
** these functions are invoked through the depth-first tree exploration performed by JobPlaningSequence.
** The implementation of these invocations can be found within the IterExplorer strategy
** lib::iter_explorer::RecursiveSelfIntegration. The net result is
** - the current element is always accessed through `yield()`
** - advancing to the next element happens \em either
**
** - by invoking `iterNext()` (when processing a sequence of sibling job prerequisites)
** - by invoking `integrate()` (when starting to explore the next level of children)
** Aggregation of planning data to generate actual frame calculation jobs.
** These render jobs are generated periodically by an ongoing process while rendering is underway.
** For this purpose, each CalcStream of the play/render process operates a RenderDrive with a
** _job-planning pipeline_, rooted at the »master beat« as defined by the frame grid from the
** Timings spec of the current render process. This pipeline will assemble the specifications
** for the render jobs and thereby possibly discover prerequisites, which must be calculated first.
** From a usage point of view, the _job-planning pipeline_ is an _iterator:_ for each independent
** calculation step a new JobPlanning record appears at the output side of the pipeline, holding
** all collected data, sufficient to generate the actual job definition, which can then be
** handed over to the Scheduler.
**
** @warning as of 4/2023 a complete rework of the Dispatcher is underway ///////////////////////////////////////////TICKET #1275
**
** @see DispatcherInterface_test simplified usage examples
** @see JobPlanning_test
** @see JobTicket
** @see Dispatcher
** @see EngineService
@ -70,8 +50,8 @@
#include "steam/engine/job-ticket.hpp"
#include "steam/engine/frame-coord.hpp"
#include "lib/time/timevalue.hpp"
#include "lib/iter-explorer.hpp"
#include "lib/iter-adapter.hpp"
//#include "lib/iter-explorer.hpp"
//#include "lib/iter-adapter.hpp"
#include "lib/util.hpp"
@ -100,13 +80,11 @@ namespace engine {
* on a recursive exploration of the corresponding JobTicket, which acts as
* a general blueprint for creating jobs within this segment of the timeline.
*
* @remarks on the implementation level, JobPlanning is used as "state core"
* for a PlanningState iterator, to visit and plan subsequently all
* the individual operations necessary to render a timeline chunk.
* @todo WIP-WIP 6/2023 reworking the job-planning pipeline for »PlaybackVerticalSlice«
*/
class JobPlanning
{
JobTicket::ExplorationState plannedOperations_;
// JobTicket::ExplorationState plannedOperations_;
FrameCoord point_to_calculate_;
public:
@ -117,6 +95,7 @@ namespace engine {
JobPlanning()
{ }
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
/** further job planning can be initiated by continuing off a given previous planning state.
* This is how the forks are created, expanding into a multitude of prerequisites for
* the job in question.
@ -125,6 +104,7 @@ namespace engine {
: plannedOperations_(startingPoint)
, point_to_calculate_(requestedFrame)
{ }
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
// using the standard copy operations
@ -134,15 +114,18 @@ namespace engine {
*/
operator Job()
{
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
if (isnil (plannedOperations_))
throw error::Logic("Attempt to plan a frame-Job based on a missing, "
"unspecified, exhausted or superseded job description"
,error::LUMIERA_ERROR_BOTTOM_VALUE);
return plannedOperations_->createJobFor (point_to_calculate_);
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
}
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
/////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1301 : likely to become obsolete
/** build a new JobPlanning object,
* set to explore the prerequisites
@ -159,8 +142,6 @@ namespace engine {
}
/////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1301 : likely to become obsolete
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
/** integrate another chain of prerequisites into the current evaluation line.
* Further evaluation will start to visit prerequisites from the new starting point,
* and return to the current evaluation chain later on exhaustion of the side chain.
@ -199,13 +180,13 @@ namespace engine {
plannedOperations_.pullNext();
plannedOperations_.markTreeLocation();
}
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
};
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
/**
* iterator, exposing a sequence of JobPlanning elements
*/
@ -286,6 +267,7 @@ namespace engine {
calculationStep.discoverPrerequisites());
return newSubEvaluation;
}
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
@ -323,7 +305,6 @@ namespace engine {
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
/**
* Generate a sequence of starting points for Job planning,
* based on the underlying frame grid. This sequence will be
@ -457,6 +438,7 @@ namespace engine {
>>= expandPrerequisites) // "flat map" (monad operation)
{ }
};
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...

View file

@ -133,12 +133,14 @@ using lib::LUID;
static JobTicket NOP;
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
/////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 : likely to become obsolete
class ExplorationState;
friend class ExplorationState;
ExplorationState startExploration() const; ////////////////////////////TICKET #1276 : likely to become obsolete
ExplorationState discoverPrerequisites (uint channelNr =0) const; ////////////////////////////TICKET #1276 : likely to become obsolete
/////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 : likely to become obsolete
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
Job createJobFor (FrameCoord coordinates) const;
@ -179,6 +181,7 @@ using lib::LUID;
};
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
/////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 : likely to become obsolete
class JobTicket::ExplorationState
{
@ -274,6 +277,7 @@ using lib::LUID;
}
};
/////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 : likely to become obsolete
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
@ -309,6 +313,7 @@ using lib::LUID;
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
/////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 : likely to become obsolete
/// @deprecated : could be expendable ... likely incurred solely by the use of Monads as design pattern
inline JobTicket::ExplorationState
@ -332,6 +337,7 @@ using lib::LUID;
: ExplorationState (util::unConst(provision_).prerequisites);
}
/////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 : likely to become obsolete
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #1276 :: to be refactored...
}} // namespace steam::engine

View file

@ -54,6 +54,11 @@ return: 0
END
PLANNED "Render job planning calculation" JobPlanning_test <<END
return: 0
END
TEST "Mock support for render job planning" MockSupport_test <<END
return: 0
END

View file

@ -0,0 +1,140 @@
/*
JobPlanning(Test) - data evaluation for frame job creation
Copyright (C) Lumiera.org
2023, Hermann Vosseler <Ichthyostega@web.de>
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of
the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
* *****************************************************/
/** @file job-planning-test.cpp
** unit test \ref JobPlanning_test
**
** @warning as of 4/2023 a complete rework of the Dispatcher is underway ///////////////////////////////////////////TICKET #1275
*/
#include "lib/test/run.hpp"
#include "lib/error.hpp"
//#include "steam/engine/procnode.hpp"
//#include "steam/play/dummy-play-connection.hpp"
//#include "steam/mobject/model-port.hpp"
//#include "steam/engine/dispatcher.hpp"
#include "steam/play/timings.hpp"
#include "lib/time/timevalue.hpp"
//#include "lib/time/timequant.hpp"
//#include "lib/format-cout.hpp"
//#include "lib/depend.hpp"
//#include "lib/itertools.hpp"
//#include "lib/util-coll.hpp"
//#include "lib/util.hpp"
//#include <functional>
//#include <vector>
using test::Test;
//using util::isnil;
//using util::last;
//using std::vector;
//using std::function;
//using std::rand;
namespace steam {
namespace engine{
namespace test {
// using lib::time::FrameRate;
// using lib::time::Duration;
// using lib::time::Offset;
// using lib::time::TimeVar;
// using lib::time::Time;
// using mobject::ModelPort;
// using play::Timings;
namespace { // used internally
// ModelPort
// getTestPort()
// {
// return mockDispatcher().provideMockModelPort();
// }
} // (End) internal defs
#if false /////////////////////////////////////////////////////////////////////////////////////////////////////////////UNIMPLEMENTED :: TICKET #1301
#endif /////////////////////////////////////////////////////////////////////////////////////////////////////////////UNIMPLEMENTED :: TICKET #1301
/***************************************************************//**
* @test document and verify the data aggregation and the calculations
* necessary to prepare render jobs for scheduling.
* @todo WIP-WIP 6/2023
*/
class JobPlanning_test : public Test
{
virtual void
run (Arg)
{
simpleUsage();
calculateDeadline();
setupDependentJob();
}
/** @test demonstrate a simple usage scenario
*/
void
simpleUsage()
{
}
/** @test verify the timing calculations to establish
* the scheduling deadline of a simple render job
*/
void
calculateDeadline()
{
}
/** @test verify the setup of a prerequisite job in relation
* to the master job depending on this prerequisite
*/
void
setupDependentJob()
{
}
};
/** Register this test class... */
LAUNCHER (JobPlanning_test, "unit engine");
}}} // namespace steam::engine::test

View file

@ -135,12 +135,14 @@ namespace test {
CHECK (nopJob.parameter.nominalTime == coord.absoluteNominalTime);
InvocationInstanceID empty; ///////////////////////////////////////////////////////////////////////TICKET #1287 : temporary workaround until we get rid of the C base structs
CHECK (lumiera_invokey_eq (&nopJob.parameter.invoKey, &empty));
CHECK (MockJob::isNopJob(nopJob)); // this diagnostic helper checks the same conditions as done here explicitly
MockJobTicket mockTicket;
CHECK (mockTicket.discoverPrerequisites().empty());
CHECK (not mockTicket.empty());
Job mockJob = mockTicket.createJobFor (coord);
CHECK ( mockTicket.verify_associated (mockJob)); // proof by invocation hash : is indeed backed by this JobTicket
CHECK (not mockTicket.verify_associated (nopJob)); // ...while some random other job is not related
CHECK (not MockJob::isNopJob(mockJob));
}

View file

@ -20,8 +20,8 @@
* *****************************************************/
/** @file node-graph-attachment-test.cpp
** unit test \ref NodeGraphAttachment_test
/** @file fixture-segment-test.cpp
** unit test \ref FixtureSegment_test
*/

View file

@ -75545,6 +75545,9 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<icon BUILTIN="flag-yellow"/>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1686793340860" HGAP="72" ID="ID_121519729" MODIFIED="1686793358090" TEXT="JobPlanning aus der Pipeline erzeugen" VSHIFT="-12">
<icon BUILTIN="flag-yellow"/>
</node>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1681742264338" ID="ID_627755845" MODIFIED="1681742273104" TEXT="in RenderDrive &#xfc;berf&#xfc;hren">
@ -75596,7 +75599,7 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<node CREATED="1685052033041" MODIFIED="1685052033041" TEXT="TimeVar absoluteRealDeadline;"/>
<node CREATED="1685052033041" MODIFIED="1685052033041" TEXT="ModelPort modelPort;"/>
</node>
<node CREATED="1685050895873" ID="ID_1305413793" MODIFIED="1685050899356" TEXT="JobTicket"/>
<node CREATED="1685050895873" ID="ID_1305413793" MODIFIED="1686789885547" TEXT="JobTicket &#xd83e;&#xdc32; JobFunctor"/>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1685050903817" ID="ID_947140645" MODIFIED="1685058047977" TEXT="konkrete Output-Sink">
<linktarget COLOR="#a95a75" DESTINATION="ID_947140645" ENDARROW="Default" ENDINCLINATION="188;21;" ID="Arrow_ID_686370160" SOURCE="ID_1575034395" STARTARROW="None" STARTINCLINATION="747;-40;"/>
<icon BUILTIN="flag-yellow"/>
@ -75659,7 +75662,6 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<icon BUILTIN="flag-yellow"/>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1685052158905" ID="ID_298268641" MODIFIED="1685802456361" TEXT="ist eine State-Facade f&#xfc;r den aktuellen Planungs-Stand">
<icon BUILTIN="yes"/>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1685802434530" ID="ID_84580124" MODIFIED="1685802448802" TEXT="f&#xfc;gt die DataSink(Handle) hinzu">
<icon BUILTIN="flag-yellow"/>
</node>
@ -75667,6 +75669,19 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<icon BUILTIN="flag-yellow"/>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1686793407407" ID="ID_194609603" MODIFIED="1686793411216" TEXT="Erzeugung">
<icon BUILTIN="flag-yellow"/>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1686793412168" ID="ID_566542363" MODIFIED="1686793437280" TEXT="Deadline festsetzen">
<icon BUILTIN="flag-yellow"/>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1686793444999" ID="ID_890638508" MODIFIED="1686793453060" TEXT="einfacher Job">
<icon BUILTIN="flag-yellow"/>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1686793449011" ID="ID_1790599874" MODIFIED="1686793453060" TEXT="abh&#xe4;ngiger Job">
<icon BUILTIN="flag-yellow"/>
</node>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1685802570984" ID="ID_1417369009" MODIFIED="1685802579931" TEXT="Daten-Anordnung f&#xfc;r Jobs vorbereiten">
<icon BUILTIN="flag-yellow"/>
<node CREATED="1685802581055" ID="ID_1626475025" MODIFIED="1685802602595" TEXT="Umbau der Job-Struktur noch nicht gleich">
@ -75721,6 +75736,18 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
</node>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1686793470108" HGAP="-9" ID="ID_1024827381" MODIFIED="1686793480495" TEXT="JobPlanning_test" VSHIFT="7">
<icon BUILTIN="flag-yellow"/>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1686793496442" ID="ID_1934790434" MODIFIED="1686793516470" TEXT="simpleUsage">
<icon BUILTIN="flag-yellow"/>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1686793496443" ID="ID_1476053415" MODIFIED="1686793514422" TEXT="calculateDeadline">
<icon BUILTIN="flag-yellow"/>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1686793496443" ID="ID_1152978912" MODIFIED="1686793511894" TEXT="setupDependentJob">
<icon BUILTIN="flag-yellow"/>
</node>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1685062609797" ID="ID_51155918" MODIFIED="1685062617634" TEXT="(re)-Trigger-Mechanismus">
<icon BUILTIN="flag-yellow"/>