ichthyo/LUMIERA.clone
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Job-Planning: rework pipeline to enable dependency planning

Browse source 
This finishes the last series of refactorings; the basic concept
remains the same, but in the initial version we arranged the expander
function in the pipeline to maintain a Tuple (parent, child) for the
JobTickets. Unfortunately this turned out to be insufficient, since
JobTicket is effectively const and responsible for a complete Sement,
so there is no room to memorise a Deadline for the parent dependency.

This leads to the better idea to link the JobPlanning aggregators
themselves by parent-child references, which is possible since the
whole dependency chain actually sits in the stack embedded into the
Expander (in the pipeline)
This commit is contained in:
Fischlurch 2023-06-19 03:56:11 +02:00
parent 2b92dab377
commit dc1bbfc918
4 changed files with 148 additions and 60 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

26
src/steam/engine/dispatcher.hpp
View file

@ -280,10 +280,6 @@ namespace engine {
} // expected next to invoke pullFrom(port,sink)
/** Package a Ticket together with a direct dependency,
* to allow setup of schedule times in downstream processing */
using TicketDepend = std::pair<JobTicket*, JobTicket*>;
/**
* Builder: connect to the JobTicket defining the actual processing
@ -295,11 +291,11 @@ namespace engine {
size_t portIDX = SRC::dispatcher->resolveModelPort(port);
return buildPipeline (
SRC::transform(
[portIDX](PipeFrameTick& core) -> TicketDepend
[portIDX](PipeFrameTick& core)
{
return {nullptr
,& core.dispatcher->getJobTicketFor(portIDX, core.currPoint)
};
return JobPlanning{core.dispatcher->getJobTicketFor(portIDX, core.currPoint)
,core.currPoint
,core.frameNr};
}));
}
@ -314,15 +310,9 @@ namespace engine {
{
return buildPipeline (
SRC::expandAll(
[](TicketDepend& currentLevel)
[](JobPlanning& currentLevel)
{
JobTicket* parent = currentLevel.second;
return lib::transformIterator (parent->getPrerequisites()
,[&parent](JobTicket& prereqTicket)
{ // parent shifted up to first pos
return TicketDepend{parent, &prereqTicket};
}
);
return currentLevel.buildDependencyPlanning();
}));
}
@ -335,9 +325,9 @@ namespace engine {
{
return terminatePipeline (
SRC::transform(
[sink](TicketDepend& currentLevel)
[sink](JobPlanning& currentLevel) -> JobPlanning&
{
return currentLevel.second; ///////////////////////////////OOO construct a JobPlanning here
return currentLevel; ///////////////////////////////OOO the purpose of this function is no longer clear
}));
}

78
src/steam/engine/job-planning.hpp
View file

@ -30,7 +30,31 @@
** From a usage point of view, the _job-planning pipeline_ is an _iterator:_ for each independent
** calculation step a new JobPlanning record appears at the output side of the pipeline, holding
** all collected data, sufficient to generate the actual job definition, which can then be
** handed over to the Scheduler.
** handed over to the Scheduler.
**
** # Implementation of the Job-Planning pipeline
**
** JobPlanning acts as _working data aggregator_ within the Job-Planning pipeline; for this reason
** all data fields are references, and the optimiser is expected to elide them, since after template
** instantiation, JobPlanning becomes part of the overall assembled pipeline object, stacked on top
** of the Dispatcher::PipeFrameTick, which holds and increments the current frame number. The
** underlying play::Timings will provide a _frame grid_ to translate these frame numbers into
** the _nominal time values_ used throughout the rest of the render calculations.
**
** There is one tricky detail to note regarding the handling of calculation prerequisites. The
** typical example would be the loading and decoding of media data, which is an IO-bound task and
** must be complete before the main frame job can be started. Since the Job-Planning pipeline is
** generic, this kind of detail dependency is modelled as _prerequisite JobTicket,_ leading to
** an possibly extended depth-first tree expansion, starting from the »master frame ticket« at
** the root. This _tree exploration_ is implemented by the TreeExplorer::Expander building block,
** which obviously has to maintain a stack of expanded child dependencies. This leads to the
** observation, that at any point of this dependency processing, for the complete path from the
** child prerequisite up to the root tick there is a sequence of JobPlanning instances placed
** into this stack in the Explorer object (each level in this stack is actually an iterator
** and handles one level of child prerequisites). The deadline calculation directly exploits
** this known arrangement, insofar each JobPlanning has a pointer to its parent (sitting in
** the stack level above). See the [IterExplorer unit test](\ref lib::IterTreeExplorer_test::verify_expandOperation)
** to understand this recursive on-demand processing in greater detail.
**
** @warning as of 4/2023 a complete rework of the Dispatcher is underway ///////////////////////////////////////////TICKET #1275
**
@ -52,8 +76,7 @@
#include "steam/play/output-slot.hpp"
#include "steam/play/timings.hpp"
#include "lib/time/timevalue.hpp"
//#include "lib/iter-explorer.hpp"
//#include "lib/iter-adapter.hpp"
#include "lib/itertools.hpp"
#include "lib/nocopy.hpp"
#include "lib/util.hpp"
@ -68,7 +91,6 @@ namespace engine {
using play::Timings;
using lib::time::Time;
using lib::time::Duration;
using lib::time::TimeValue; ////////TODO for FrameLocator, could be obsolete
using util::unConst;
using util::isnil;
@ -95,16 +117,19 @@ namespace engine {
JobTicket& jobTicket_;
Time const& nominalTime_;
FrameCnt const& frameNr_;
public:
JobPlanning(JobTicket& ticket, Time const& nominalTime, FrameCnt const& frameNr)
JobPlanning (JobTicket& ticket, Time const& nominalTime, FrameCnt const& frameNr)
: jobTicket_{ticket}
, nominalTime_{nominalTime}
, frameNr_{frameNr}
{ }
// move construction is possible
// using the standard copy operations
JobTicket& ticket() { return jobTicket_; }
bool isTopLevel() const { return not dependentPlan_; }
/**
@ -156,7 +181,46 @@ namespace engine {
}
/**
* Build a sequence of dependent JobPlanning scopes for all prerequisites
* of this current JobPlanning, and internally linked back to `*this`
* @return an iterator which explores the prerequisites of the JobTicket.
* @remark typical example would be to load data from file, or to require
* the results from some other extended media calculation.
* @see Dispatcher::PipelineBuilder::expandPrerequisites()
*/
auto
buildDependencyPlanning()
{
return lib::transformIterator (jobTicket_.getPrerequisites()
,[this](JobTicket& prereqTicket)
{
return JobPlanning{*this, prereqTicket};
});
}
private:
/** link to a dependent JobPlanning, for planning of prerequisites */
JobPlanning* dependentPlan_{nullptr};
/**
* @internal construct a chained prerequisite JobPlanning,
* attached to the dependent »parent« JobPlanning, using the same
* frame data, but chaining up the deadlines, so that a Job created
* from this JobPlanning needs to be completed before the »parent«
* Job (which uses the generated data) can start
* @see #buildDependencyPlanning()
* @see JobPlanning_test::setupDependentJob()
*/
JobPlanning (JobPlanning& parent, JobTicket& prerequisite)
: jobTicket_{prerequisite}
, nominalTime_{parent.nominalTime_}
, frameNr_{parent.frameNr_}
, dependentPlan_{&parent}
{ }
Duration
totalLatency (Timings const& timings)
{

10
tests/core/steam/engine/job-planning-pipeline-test.cpp
View file

@ -201,8 +201,8 @@ namespace test {
.pullFrom (port);
CHECK (not isnil (pipeline));
CHECK (nullptr == pipeline->first); // is a top-level ticket
JobTicket& ticket = *pipeline->second;
CHECK (pipeline->isTopLevel()); // is a top-level ticket
JobTicket& ticket = pipeline->ticket();
Job job = ticket.createJobFor(Time::ZERO); // actual time point is irrelevant here
CHECK (dispatcher.verify(job, port, sink));
@ -239,14 +239,14 @@ namespace test {
// the first element is identical to previous test
CHECK (not isnil (pipeline));
CHECK (nullptr == pipeline->first);
Job job = pipeline->second->createJobFor (Time::ZERO);
CHECK (pipeline->isTopLevel());
Job job = pipeline->ticket().createJobFor (Time::ZERO);
CHECK (11 == job.parameter.invoKey.part.a);
auto visualise = [](auto& pipeline) -> string
{
Time frame{pipeline.currPoint}; // can access the embedded PipeFrameTick core to get "currPoint" (nominal time)
Job job = pipeline->second->createJobFor(frame); // looking always at the second element, which is the current JobTicket
Job job = pipeline->ticket().createJobFor(frame); // looking always at the second element, which is the current JobTicket
TimeValue nominalTime{job.parameter.nominalTime}; // job parameter holds the microseconds (gavl_time_t)
int32_t mark = job.parameter.invoKey.part.a; // the MockDispatcher places the given "mark" here
return _Fmt{"J(%d|%s)"} % mark % nominalTime;

94
wiki/thinkPad.ichthyo.mm
View file

@ -75823,7 +75823,7 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1686964642887" ID="ID_1372706833" MODIFIED="1686964706541" TEXT="verwende diese anstelle der erwarteten real-Grid-Time">
<icon BUILTIN="idea"/>
</node>
<node BACKGROUND_COLOR="#fdfdcf" COLOR="#ff0000" CREATED="1686965001287" ID="ID_1948616906" MODIFIED="1686965057296">
<node BACKGROUND_COLOR="#e0ceaa" COLOR="#690f14" CREATED="1686965001287" ID="ID_1948616906" MODIFIED="1687137549677">
<richcontent TYPE="NODE"><html>
<head>
@ -75841,7 +75841,10 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<node CREATED="1686965163417" ID="ID_1358303460" MODIFIED="1686965216709" TEXT="eine Deadline ist Ergebnis einer Integration">
<icon BUILTIN="messagebox_warning"/>
<node CREATED="1686965322397" ID="ID_966792976" MODIFIED="1686965331679" TEXT="das JobTIcket kann die gar nicht errechnen"/>
<node CREATED="1686965332609" ID="ID_1561329377" MODIFIED="1686965353628" TEXT="Rechnung w&#xe4;re rekursiv, quadratisch"/>
<node BACKGROUND_COLOR="#e0ceaa" COLOR="#690f14" CREATED="1686965332609" ID="ID_1561329377" MODIFIED="1687137693493" TEXT="Rechnung w&#xe4;re rekursiv, quadratisch">
<linktarget COLOR="#e42d77" DESTINATION="ID_1561329377" ENDARROW="Default" ENDINCLINATION="361;27;" ID="Arrow_ID_242786471" SOURCE="ID_1507798641" STARTARROW="None" STARTINCLINATION="481;-20;"/>
<icon BUILTIN="clanbomber"/>
</node>
<node CREATED="1686965354284" ID="ID_172681606" MODIFIED="1687039516255" TEXT="oder erfordert &#xbb;schleppenden&#xab; Zustand">
<arrowlink COLOR="#8e5572" DESTINATION="ID_1929351072" ENDARROW="Default" ENDINCLINATION="28;-47;" ID="Arrow_ID_753643955" STARTARROW="None" STARTINCLINATION="-70;5;"/>
</node>
@ -75865,7 +75868,7 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<node CREATED="1687039664958" ID="ID_1390938163" MODIFIED="1687039709039" TEXT="oder mu&#xdf; in die dort erzeugte Datenstruktur nachtr&#xe4;glich gespeichert werden"/>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1687039717162" ID="ID_59086385" MODIFIED="1687039770869" TEXT="L&#xf6;sungsansatz: Expander arbeitet bereits auf JobPlanning als Datenstruktur">
<node COLOR="#435e98" CREATED="1687039717162" ID="ID_59086385" MODIFIED="1687137591057" TEXT="L&#xf6;sungsansatz: Expander arbeitet bereits auf JobPlanning als Datenstruktur">
<icon BUILTIN="idea"/>
<node CREATED="1687039778410" ID="ID_696364803" MODIFIED="1687039807242" TEXT="damit w&#xe4;re das (etwas sonderbare) Tupel aus JobPlanning-Pointern abzul&#xf6;sen"/>
<node CREATED="1687040058436" ID="ID_1402323400" MODIFIED="1687040071534" TEXT="JobPlanning h&#xe4;tte dann einen &#xbb;chaining-ctor&#xab;"/>
@ -75920,22 +75923,9 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<icon BUILTIN="yes"/>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1687043634555" ID="ID_71179253" MODIFIED="1687044340413" TEXT="Umschichten der Daten">
<icon BUILTIN="flag-yellow"/>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1687044201408" ID="ID_1507798641" MODIFIED="1687047110612" TEXT="die Deadline sollte man als Wert speichern (caching)">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
...denn sonst w&#252;rde es genau zu rekursiven kaskadierenden (exponentiell aufwendigen) Aufrufen der Deadline-Berechnungs-Logik kommen; um das Caching zu steuern, kann ich einen Marker-Wert Time::NEVER verwenden
</p>
</body>
</html></richcontent>
<icon BUILTIN="messagebox_warning"/>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1687043695271" ID="ID_497206730" MODIFIED="1687044319733" TEXT="JobTicket-parent-Pointer in JobPlanning-parent verwandeln">
<node COLOR="#338800" CREATED="1687043634555" ID="ID_71179253" MODIFIED="1687137392702" TEXT="Umschichten der Daten">
<icon BUILTIN="button_ok"/>
<node COLOR="#338800" CREATED="1687043695271" ID="ID_497206730" MODIFIED="1687137643989" TEXT="JobTicket-parent-Pointer in JobPlanning-parent verwandeln">
<richcontent TYPE="NOTE"><html>
<head>
@ -75946,6 +75936,7 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
</p>
</body>
</html></richcontent>
<icon BUILTIN="button_ok"/>
</node>
<node COLOR="#5b280f" CREATED="1687043645122" ID="ID_1471106329" MODIFIED="1687131535956" TEXT="FrameCoord m&#xfc;ssen bereits in den TickGenerator">
<richcontent TYPE="NOTE"><html>
@ -75963,7 +75954,7 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<icon BUILTIN="smily_bad"/>
</node>
<node CREATED="1687047139677" ID="ID_1123653637" MODIFIED="1687047160987" TEXT="es bl&#xe4;ht den TickGenerator auf und macht ihn asymetrisch"/>
<node CREATED="1687047169092" ID="ID_1819824079" MODIFIED="1687047186765">
<node COLOR="#4a4398" CREATED="1687047169092" ID="ID_1819824079" MODIFIED="1687137416207">
<richcontent TYPE="NODE"><html>
<head>
@ -75980,7 +75971,7 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<icon BUILTIN="idea"/>
</node>
<node CREATED="1687047521489" ID="ID_1343365951" MODIFIED="1687047537890" TEXT="meistens sind sie nur halb g&#xfc;ltig &#x2014; oder redundant"/>
<node COLOR="#435e98" CREATED="1687047541216" ID="ID_3859431" MODIFIED="1687131788560" TEXT="wichtige bestehende Verwendungen">
<node COLOR="#435e98" CREATED="1687047541216" FOLDED="true" ID="ID_3859431" MODIFIED="1687131788560" TEXT="wichtige bestehende Verwendungen">
<icon BUILTIN="info"/>
<node COLOR="#338800" CREATED="1687047579697" ID="ID_737360799" MODIFIED="1687052483835" TEXT="JobTicket::createJobFor (FrameCoord)">
<icon BUILTIN="button_ok"/>
@ -76083,13 +76074,13 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<icon BUILTIN="yes"/>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1687053133875" ID="ID_613644358" MODIFIED="1687053158627" TEXT="...und stattdessen wird direkt auf die Felder im TickGenerator verwiesen (const&amp;)">
<icon BUILTIN="flag-yellow"/>
<node COLOR="#338800" CREATED="1687053133875" ID="ID_613644358" MODIFIED="1687137361826" TEXT="...und stattdessen wird direkt auf die Felder im TickGenerator verwiesen (const&amp;)">
<icon BUILTIN="button_ok"/>
</node>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1687131811965" ID="ID_1192855537" MODIFIED="1687131818555" TEXT="Pipeline-Definition umbauen">
<icon BUILTIN="flag-yellow"/>
<node COLOR="#338800" CREATED="1687131811965" ID="ID_1192855537" MODIFIED="1687137517231" TEXT="Pipeline-Definition umbauen">
<icon BUILTIN="button_ok"/>
<node COLOR="#435e98" CREATED="1687131819547" ID="ID_1957351479" MODIFIED="1687133763828" TEXT="Problem: ItemWrapper weist Werte zu">
<icon BUILTIN="messagebox_warning"/>
<node CREATED="1687131845108" ID="ID_1247271270" MODIFIED="1687131867304" TEXT="...und kann deshalb keine non-copyable-Typen handhaben">
@ -76156,16 +76147,59 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<icon BUILTIN="button_ok"/>
</node>
</node>
<node COLOR="#338800" CREATED="1687137207845" ID="ID_41795599" MODIFIED="1687137250587" TEXT="JobPlanning nun bereits auf der pull-From-Ebene erzeugen">
<icon BUILTIN="button_ok"/>
</node>
<node COLOR="#338800" CREATED="1687137268479" ID="ID_1427542554" MODIFIED="1687137280067" TEXT="chained-Planning-Konstruktor einf&#xfc;hren">
<icon BUILTIN="button_ok"/>
</node>
<node COLOR="#435e98" CREATED="1687137281027" ID="ID_387219967" MODIFIED="1687137332964" TEXT="TransformIterator in JobPlanning erstellen">
<icon BUILTIN="idea"/>
</node>
<node COLOR="#1768b8" CREATED="1687137306215" ID="ID_1183352863" MODIFIED="1687137343253" TEXT="Ohh ...alles vieeel besser so">
<icon BUILTIN="ksmiletris"/>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1687044355596" ID="ID_1593217439" MODIFIED="1687044369362" TEXT="JobPlanningPipeline_test anpassen">
<icon BUILTIN="flag-yellow"/>
<node BACKGROUND_COLOR="#fdfdcf" COLOR="#ff0000" CREATED="1687044370372" ID="ID_1608005431" MODIFIED="1687044408084" TEXT="der hat bisher das JobTIcket-Tupel verwendet">
</node>
<node COLOR="#338800" CREATED="1687044355596" ID="ID_1593217439" MODIFIED="1687137510008" TEXT="JobPlanningPipeline_test anpassen">
<icon BUILTIN="button_ok"/>
<node COLOR="#435e98" CREATED="1687044370372" ID="ID_1608005431" MODIFIED="1687137469706" TEXT="der hat bisher das JobTicket-Tupel verwendet">
<icon BUILTIN="info"/>
</node>
<node BACKGROUND_COLOR="#f0d5c5" COLOR="#990033" CREATED="1687044384151" ID="ID_228309908" MODIFIED="1687044397980" TEXT="ggfs speziellen Accessor schaffen?">
<node COLOR="#338800" CREATED="1687044384151" ID="ID_228309908" MODIFIED="1687137487850" TEXT="speziellen Accessoren">
<font NAME="SansSerif" SIZE="12"/>
<icon BUILTIN="help"/>
<icon BUILTIN="button_ok"/>
<node COLOR="#338800" CREATED="1687137489231" ID="ID_641532111" MODIFIED="1687137497724" TEXT="ticket()">
<icon BUILTIN="button_ok"/>
</node>
<node COLOR="#338800" CREATED="1687137491791" ID="ID_1927053233" MODIFIED="1687137497725" TEXT="isTopLevel()">
<icon BUILTIN="button_ok"/>
</node>
</node>
<node COLOR="#338800" CREATED="1687137501852" ID="ID_32393171" MODIFIED="1687137508724" TEXT="Hey... l&#xe4;uft">
<icon BUILTIN="button_ok"/>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1687137618239" ID="ID_1445585564" MODIFIED="1687137626846" TEXT="Verkettung der Deadlines implementieren">
<icon BUILTIN="flag-yellow"/>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1687044201408" ID="ID_1507798641" MODIFIED="1687137690544" TEXT="die Deadline sollte man als Wert speichern (caching)">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
...denn sonst w&#252;rde es genau zu rekursiven kaskadierenden (exponentiell aufwendigen) Aufrufen der Deadline-Berechnungs-Logik kommen; um das Caching zu steuern, kann ich einen Marker-Wert Time::NEVER verwenden
</p>
</body>
</html></richcontent>
<arrowlink COLOR="#e42d77" DESTINATION="ID_1561329377" ENDARROW="Default" ENDINCLINATION="361;27;" ID="Arrow_ID_242786471" STARTARROW="None" STARTINCLINATION="481;-20;"/>
<icon BUILTIN="messagebox_warning"/>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1687137710306" ID="ID_1967222999" MODIFIED="1687137719434" TEXT="Refactoring: Berechnung symmetrisch machen">
<icon BUILTIN="flag-yellow"/>
<node CREATED="1687137733390" ID="ID_1496251627" MODIFIED="1687137744225" TEXT="top-Level: Deadline stammt von den Timings"/>
<node CREATED="1687137744907" ID="ID_348107888" MODIFIED="1687137759424" TEXT="Prerequisite: Deadline stammt vom dependent Plan"/>
</node>
</node>
</node>