Scheduler: complete handling of the grooming-token

- Ensure the grooming-token (lock) is reliably dropped
- also explicitly drop it prior to trageted sleeps
- properly signal when not able to acquire the token before dispatch

- amend tests broken by changes since yesterday
This commit is contained in:
Fischlurch 2023-10-28 03:34:06 +02:00
parent 552d8dec0e
commit 6166ab63f2
6 changed files with 301 additions and 116 deletions

View file

@ -200,6 +200,7 @@ namespace gear {
* - activity::PASS continue processing in regular operation
* - activity::WAIT nothing to do now, check back later
* - activity::HALT serious problem, cease processing
* - activity::SKIP to contend (spin) on GroomingToken
* @note Attempts to acquire the GroomingToken for immediate
* processing, but not for just enqueuing planned tasks.
* Never drops the GroomingToken explicitly (unless when
@ -212,7 +213,7 @@ namespace gear {
,EXE& executionCtx
,SchedulerInvocation& layer1)
{
if (!chain) return activity::WAIT;
if (!chain) return activity::SKIP;
Time now = executionCtx.getSchedTime();
if (decideDispatchNow (when, now))

View file

@ -26,12 +26,66 @@
** The implementation of scheduling services is provided by an integration
** of two layers of functionality:
** - Layer-1 allows to enqueue and prioritise render activity records
** - Layer-2 connects and coordinates activities to conduct complex calculations
** - Layer-2 connects and coordinates activities to conduct complex calculations
** Additionally, a [custom allocation scheme](\ref BlockFlow) is involved,
** a [notification service](\ref EngineObserver) and the execution environment
** for the low-level [»Activity Language](\ref ActivityLang). Some operational
** control and and load management is delegated to the \ref LoadController.
** The *purpose* of the »Scheduler Service« in the lumiera Render Engine
** is to coordinate the execution of [»Render Jobs«](\ref vault::gear::Job),
** which can be controlled by a timing scheme, but also triggered in response
** to some prerequisite event, most notably the completion of IO work.
**
** @see SchedulerUsage_test Component integration test
** @see scheduler.cpp implementation details
** # Thread coordination
** The typical situation found when rendering media is the demand to distribute
** rather scarce computation resources to various self-contained tasks sequenced
** in temporary and dependency order. In addition, some internal management work
** must be conducted to order these tasks, generate further tasks and coordinate
** the dependencies. Overall, any such internal work is by orders of magnitude
** less expensive than the actual media calculations, which reach up into the
** range of 1-10 milliseconds, possibly even way more (seconds for expensive
** computations). For this reason, the Scheduler in the Lumiera Render Engine
** uses a pool of workers, each representing one unit of computation resource
** (a »core«), and these workers will _pull work actively,_ rather then
** distributing, queuing and dispatching tasks to a passive set of workers.
** And notably the »management work« is performed also by the workers themselves,
** to the degree it is necessary to retrieve the next piece of computation.
** So there is no dedicated »queue manager« scheduling is driven by the workers.
**
** Assuming that this internal work is comparatively cheap to perform, a choice
** was made to handle any internal state changes of the Scheduler exclusively
** in single-threaded mode. This is achieved by an atomic lock, maintained in
** [Layer-2 of the Scheduler implementation](\ref SchedulerCommutator::groomingToken_).
** Any thread looking for more work will pull a pre-configured functor, which
** is implemented by the [work-function](\ref Scheduler::getWork()). The thread
** will attempt to acquire the lock, designated as »grooming-token« -- but only
** if this is necessary to perform internal changes. Since workers are calling
** in randomly, in many cases there might be no task to perform at the moment,
** and the worker can be instructed to go to a sleep cycle and call back later.
** On the other hand, when load is high, workers are instructed to call back
** immediately again to find the next piece of work. Based on assessment of
** the current [»head time«](\ref SchedulerInvocation::headTime), a quick
** decision will be made if the thread's capacity is useful right now,
** or if this capacity will be re-focussed into another zone of the
** scheduler's time axis, based on the distance to the next task.
**
** If however a thread is put to work, it will start dequeuing an entry from
** the head of the [priority queue](\ref SchedulerInvocation::pullHead),
** and start interpreting this entry as a _chain of render activities_ with
** the help of the [»Activity Language«](\ref ActivityLang::dispatchChain).
** In the typical scenario, after some preparatory checks and notifications,
** the thread [transitions into work mode](\ref Scheduler::ExecutionCtx::work),
** which entails to [drop the grooming-token](\ref SchedulerCommutator::dropGroomingToken).
** Since the scheduler queue only stores references to render activities, which are
** allocated in a [special arrangement](\ref BlockFlow) exploiting the known deadline
** time of each task, further processing can commence concurrently.
**
** @see SchedulerService_test Component integration test
** @see SchedulerStress_test
** @see SchedulerUsage_test
** @see SchedulerInvocation Layer-1
** @see SchedulerCommutator Layer-2
** @see activity.hpp description of »Render Activities«
**
** @todo WIP-WIP 10/2023 »Playback Vertical Slice«
**
@ -321,31 +375,51 @@ namespace gear {
* - activity::PROC causes the worker to poll again immediately
* - activity::SLEEP induces a sleep state
* - activity::HALT terminates the worker
* @note Under some circumstances, this function depends on acquiring the »grooming-token«,
* which is an atomic lock to ensure only one thread at a time can alter scheduler internals.
* In the regular processing sequence, this token is dropped after dequeuing and processing
* some Activities, yet prior to invoking the actual »Render Job«. Explicitly dropping the
* token at the end of this function is a safeguard against deadlocking the system.
* If some other thread happens to hold the token, SchedulerCommutator::findWork
* will bail out, leading to active spinning wait for the current thread.
*/
inline activity::Proc
Scheduler::getWork()
{
ExecutionCtx& ctx = ExecutionCtx::from(*this);
return WorkerInstruction{}
.performStep([&]{
Time now = ctx.getSchedTime();
Time head = layer1_.headTime();
return scatteredDelay(now,
loadControl_.markIncomingCapacity (head,now));
})
.performStep([&]{
Time now = ctx.getSchedTime();
Activity* act = layer2_.findWork (layer1_,now);
return ctx.post (now, act, ctx);
})
.performStep([&]{
Time now = ctx.getSchedTime();
Time head = layer1_.headTime();
return scatteredDelay(now,
loadControl_.markOutgoingCapacity (head,now));
})
;
auto self = std::this_thread::get_id();
auto& ctx = ExecutionCtx::from (*this);
try {
auto res = WorkerInstruction{}
.performStep([&]{
Time now = ctx.getSchedTime();
Time head = layer1_.headTime();
return scatteredDelay(now,
loadControl_.markIncomingCapacity (head,now));
})
.performStep([&]{
Time now = ctx.getSchedTime();
Activity* act = layer2_.findWork (layer1_,now);
return ctx.post (now, act, ctx);
})
.performStep([&]{
Time now = ctx.getSchedTime();
Time head = layer1_.headTime();
return scatteredDelay(now,
loadControl_.markOutgoingCapacity (head,now));
});
// ensure lock clean-up
if (res != activity::PASS
and layer2_.holdsGroomingToken(self))
layer2_.dropGroomingToken();
return res;
}
catch(...)
{
if (layer2_.holdsGroomingToken (self))
layer2_.dropGroomingToken();
throw;
}
}
@ -366,7 +440,11 @@ namespace gear {
Scheduler::scatteredDelay (Time now, LoadController::Capacity capacity)
{
auto doTargetedSleep = [&]
{
{ // ensure not to block the Scheduler after management work
auto self = std::this_thread::get_id();
if (layer2_.holdsGroomingToken (self))
layer2_.dropGroomingToken();
// relocate this thread(capacity) to a time where its more useful
Offset targetedDelay = loadControl_.scatteredDelayTime (now, capacity);
std::this_thread::sleep_for (std::chrono::microseconds (_raw(targetedDelay)));
};

View file

@ -369,7 +369,7 @@ namespace test {
CHECK (not sched.holdsGroomingToken (myself));
// no effect when no Activity given
CHECK (activity::WAIT == sched.postDispatch (nullptr, now, detector.executionCtx, queue));
CHECK (activity::SKIP == sched.postDispatch (nullptr, now, detector.executionCtx, queue));
CHECK (not sched.holdsGroomingToken (myself));
// Activity immediately dispatched when on time and GroomingToken can be acquired

View file

@ -96,7 +96,7 @@ namespace test {
/** @test verify visible behaviour of the [work-pulling function](\ref Scheduler::getWork)
* - use a rigged Activity probe to capture the schedule time on invocation
* - additionally perform a timing measurement for invoking the work-function
* - empty invocations cost ~5µs (-O3) rsp. ~25µs (debug)
* - invoking the Activity probe itself costs 50...150µs, Scheduler internals < 50µs
* - this implies we can show timing-delay effects in the millisecond range
* - demonstrated behaviour
* + an Activity already due will be dispatched immediately by post()
@ -140,6 +140,7 @@ namespace test {
{ // this test class is declared friend to get a backdoor to Scheduler internals...
auto& schedCtx = Scheduler::ExecutionCtx::from(scheduler);
scheduler.layer2_.acquireGoomingToken();
schedCtx.post (start, &probe, schedCtx);
};

View file

@ -7290,7 +7290,7 @@ The primary scaling effects exploited to achieve this level of performance are t
The way other parts of the system are built, requires us to obtain a guaranteed knowledge of some job's termination. It is possible to obtain that knowledge with some limited delay, but it nees to be absoultely reliable (violations leading to segfault). The requirements stated above assume this can be achieved through //jobs with guaranteed execution.// Alternatively we could consider installing specific callbacks -- in this case the scheduler itself has to guarantee the invocation of these callbacks, even if the corresponding job fails or is never invoked. It doesn't seem there is any other option.
</pre>
</div>
<div title="SchedulerWorker" creator="Ichthyostega" modifier="Ichthyostega" created="202309041605" modified="202310241448" tags="Rendering operational spec draft" changecount="13">
<div title="SchedulerWorker" creator="Ichthyostega" modifier="Ichthyostega" created="202309041605" modified="202310280149" tags="Rendering operational spec draft" changecount="14">
<pre>The Scheduler //maintains a ''Work Force'' (a pool of workers) to perform the next [[render activities|RenderActivity]] continuously.//
Each worker runs in a dedicated thread; the Activities are arranged in a way to avoid blocking those worker threads
* IO operations are performed asynchronously {{red{planned as of 9/23}}}
@ -7298,7 +7298,7 @@ Each worker runs in a dedicated thread; the Activities are arranged in a way to
!Workload and invocation scheme
Using a pool of workers to perform small isolated steps of work atomically and in parallel is an well established pattern in high performance computing. However, the workload for rendering media is known to have some distinctive traits, calling for a slightly different approach compared with an operation system scheduler or a load balancer. Notably, the demand for resources is high, often using „whatever it takes“ -- driving the system into load saturation. The individual chunks of work, which can be computed independently, are comparatively large, and must often be computed in a constrained order. For real-time performance, it is desirable to compute data as late as possible, to avoid blocking memory with computed results. And for the final quality render, for the same reason it is advisable to proceed in data dependency order to keep as much data as possible in memory and avoid writing temporary files.
This leads to a situation where it is more adequate to //distribute the scarce computation resources// to the tasks //sequenced in temporary and dependence order//. The computation tasks must be prepared and ordered -- but beyond that, there is not much that can be »managed« with a computation task. For this reason, the Scheduler in the Lumiera Render Engine uses a pool of workers, each providing one unit of computation resource (a »core«), and these workers will ''pull work'' actively, rather then distributing, queuing and dispatching tasks to a passive set of workers.
This leads to a situation where it is more adequate to //distribute the scarce computation resources// to the tasks //sequenced in temporary and dependency order//. The computation tasks must be prepared and ordered -- but beyond that, there is not much that can be »managed« with a computation task. For this reason, the Scheduler in the Lumiera Render Engine uses a pool of workers, each providing one unit of computation resource (a »core«), and these workers will ''pull work'' actively, rather then distributing, queuing and dispatching tasks to a passive set of workers.
Moreover, the actual computation tasks, which can be parallelised, are at least by an order of magnitude more expensive than any administrative work for sorting tasks, checking dependencies and maintaining process state. This leads to a scheme where a worker first performs some »management work«, until encountering the next actual computation job, at which point the worker leaves the //management mode// and transitions into //concurrent work mode//. All workers are expected to be in work mode almost entirely most of the time, and thus we can expect not much contention between workers performing »management work« -- allowing to confine this management work to //single threaded operation,// thereby drastically reducing the complexity of management data structures and memory allocation.
!!!Regulating workers

View file

@ -80636,7 +80636,7 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<node CREATED="1690741926157" ID="ID_1658082854" MODIFIED="1690741927768" TEXT="F&#xe4;lle">
<node COLOR="#435e98" CREATED="1690741929054" ID="ID_1938875580" MODIFIED="1697664293261" TEXT="PASS">
<richcontent TYPE="NOTE"><html>
<head/
<head/>
<body>
<p>
pass on the activation down the chain
@ -82050,8 +82050,10 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<icon BUILTIN="full-4"/>
</node>
</node>
<node CREATED="1698004064515" ID="ID_1757760062" MODIFIED="1698004067549" TEXT="R&#xfc;ckgabewert">
<node CREATED="1698004069585" ID="ID_1914419321" MODIFIED="1698004104844" TEXT="Reihenfolge der Quellen">
<node COLOR="#338800" CREATED="1698004064515" FOLDED="true" ID="ID_1757760062" MODIFIED="1698459253549" TEXT="R&#xfc;ckgabewert">
<icon BUILTIN="button_ok"/>
<node COLOR="#338800" CREATED="1698004069585" ID="ID_1914419321" MODIFIED="1698459232588" TEXT="Reihenfolge der Quellen">
<icon BUILTIN="yes"/>
<node CREATED="1698004105859" ID="ID_755324904" MODIFIED="1698004117284" TEXT="incomingCapacity"/>
<node CREATED="1698004126047" ID="ID_1317095519" MODIFIED="1698004128011" TEXT="postDispatch">
<node CREATED="1698004165273" ID="ID_1299419336" MODIFIED="1698004170455" TEXT="ist entweder PASS"/>
@ -82084,20 +82086,20 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
</node>
</node>
</node>
<node BACKGROUND_COLOR="#eef0c5" COLOR="#990000" CREATED="1698159488648" ID="ID_1616253878" MODIFIED="1698201681491" TEXT="Entscheidungs-Logik realisieren">
<icon BUILTIN="pencil"/>
<node COLOR="#338800" CREATED="1698159488648" ID="ID_1616253878" MODIFIED="1698459202682" TEXT="Entscheidungs-Logik realisieren">
<icon BUILTIN="button_ok"/>
<node CREATED="1698159497515" ID="ID_1173108252" MODIFIED="1698159508856" TEXT="geschieht weitgehend durch Zusammenf&#xfc;gen">
<font ITALIC="true" NAME="SansSerif" SIZE="12"/>
<icon BUILTIN="idea"/>
</node>
<node BACKGROUND_COLOR="#eef0c5" COLOR="#990000" CREATED="1698159517071" ID="ID_1595260615" MODIFIED="1698198257685" TEXT="Umsetzen der Kapazit&#xe4;ts-Entscheidung f&#xfc;r Worker">
<icon BUILTIN="pencil"/>
<node COLOR="#338800" CREATED="1698159517071" FOLDED="true" ID="ID_1595260615" MODIFIED="1698459193257" TEXT="Umsetzen der Kapazit&#xe4;ts-Entscheidung f&#xfc;r Worker">
<icon BUILTIN="button_ok"/>
<node COLOR="#338800" CREATED="1698198240438" ID="ID_1413581521" MODIFIED="1698198284881" TEXT="auf LoadController::scatteredDelayTime() aufbauen">
<icon BUILTIN="button_ok"/>
</node>
<node BACKGROUND_COLOR="#eef0c5" COLOR="#990000" CREATED="1698198271306" ID="ID_1386652739" MODIFIED="1698198290496" TEXT="brauche GroomingToken f&#xfc;r tendNext()">
<icon BUILTIN="pencil"/>
<node CREATED="1698198292902" ID="ID_519079747" MODIFIED="1698198356507" TEXT="das wird jetzt aber verwirrend">
<node COLOR="#338800" CREATED="1698198271306" ID="ID_1386652739" MODIFIED="1698459119951" TEXT="brauche GroomingToken f&#xfc;r tendNext()">
<icon BUILTIN="button_ok"/>
<node CREATED="1698198292902" ID="ID_519079747" MODIFIED="1698454256066" TEXT="das wird jetzt aber verwirrend">
<richcontent TYPE="NOTE"><html>
<head/>
<body>
@ -82110,6 +82112,9 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<node BACKGROUND_COLOR="#e0ceaa" COLOR="#690f14" CREATED="1698198357383" ID="ID_783687241" MODIFIED="1698198391403" TEXT="und zudem mu&#xdf; ich jetzt in den n&#xe4;chsten Switch-Case durchfallen">
<icon BUILTIN="smily_bad"/>
</node>
<node COLOR="#338800" CREATED="1698459126868" ID="ID_1886805829" MODIFIED="1698459145593" TEXT="besser gel&#xf6;st: in Lambdas extrahiert">
<icon BUILTIN="button_ok"/>
</node>
</node>
<node COLOR="#435e98" CREATED="1698198418607" ID="ID_1659177227" MODIFIED="1698200551931" TEXT="kl&#xe4;ren: woher kommt die &quot;now&quot;-Time">
<icon BUILTIN="messagebox_warning"/>
@ -82175,7 +82180,7 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
</node>
</node>
</node>
<node CREATED="1698198536389" ID="ID_864714571" MODIFIED="1698198582467" TEXT="mu&#xdf; nach Delay vorzeitig ausbrechen">
<node COLOR="#435e98" CREATED="1698198536389" ID="ID_864714571" MODIFIED="1698459171503" TEXT="mu&#xdf; nach Delay vorzeitig ausbrechen">
<icon BUILTIN="messagebox_warning"/>
<node CREATED="1698198602506" ID="ID_1248366700" MODIFIED="1698198624915" TEXT="geht hier mit Trick">
<icon BUILTIN="idea"/>
@ -82195,7 +82200,7 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
</node>
</node>
</node>
<node BACKGROUND_COLOR="#f8f1cb" COLOR="#a50125" CREATED="1698200567151" ID="ID_1980030846" MODIFIED="1698200583814" TEXT="das ist problematischer Code">
<node BACKGROUND_COLOR="#f8f1cb" COLOR="#a50125" CREATED="1698200567151" FOLDED="true" ID="ID_1980030846" MODIFIED="1698200583814" TEXT="das ist problematischer Code">
<icon BUILTIN="messagebox_warning"/>
<node CREATED="1698200588924" ID="ID_1924528218" MODIFIED="1698200607982" TEXT="der Ablauf ist inzwischen ziemlich komplex geworden"/>
<node CREATED="1698200609162" ID="ID_1998499714" MODIFIED="1698200656688" TEXT="viel kontextuelle Annahmen">
@ -82222,13 +82227,89 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
</node>
</node>
</node>
<node BACKGROUND_COLOR="#fdfdcf" COLOR="#ff0000" CREATED="1698286167843" ID="ID_3145780" MODIFIED="1698286178482" TEXT="GroomingToken richtig behandeln">
<icon BUILTIN="flag-pink"/>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1698286228019" ID="ID_1492651640" MODIFIED="1698286242507" TEXT="acquire sollte schon korrekt sein (pr&#xfc;fen!)">
<icon BUILTIN="flag-yellow"/>
<node COLOR="#338800" CREATED="1698286167843" ID="ID_3145780" MODIFIED="1698458668582" TEXT="GroomingToken richtig behandeln">
<icon BUILTIN="button_ok"/>
<node COLOR="#338800" CREATED="1698286228019" ID="ID_1492651640" MODIFIED="1698453144418" TEXT="acquire ist korrekt (nochmal &#xfc;berpr&#xfc;ft)">
<icon BUILTIN="button_ok"/>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1698286180725" ID="ID_1291141353" MODIFIED="1698286214373" TEXT="sicherstellen da&#xdf; es am Ende gedropped wurde">
<icon BUILTIN="flag-yellow"/>
<node COLOR="#338800" CREATED="1698286180725" ID="ID_1291141353" MODIFIED="1698453149250" TEXT="sicherstellen da&#xdf; es am Ende gedropped wurde">
<icon BUILTIN="button_ok"/>
<node COLOR="#338800" CREATED="1698453150435" ID="ID_63679617" MODIFIED="1698453154138" TEXT="try-catch">
<icon BUILTIN="button_ok"/>
</node>
<node COLOR="#435e98" CREATED="1698453154962" ID="ID_1484610310" MODIFIED="1698454206723" TEXT="Abk&#xfc;rzung: weiter halten wenn PASS">
<richcontent TYPE="NOTE"><html>
<head/>
<body>
<p>
...das ist ein kleiner Kniff und f&#252;hrt dazu, da&#223; ein Thread ohne weiteres in einem Schwung alle Management-Aufgaben erledigt, bis er durch eine regul&#228;re Transaktion in den Work-Modus geht, oder eben nichts mehr unmittelbar zu tun ist. F&#252;r die logische Konsistenz ist diese Ausnahme nicht notwendig, aber sie verhindert in einigen F&#228;llen eine unn&#246;tige read-write-barrier und arbeitet weiter aus dem Cache der jeweiligen Core. Cache-Effekte k&#246;nnen locker mal 50-100&#181;s kosten
</p>
</body>
</html></richcontent>
<icon BUILTIN="idea"/>
</node>
<node BACKGROUND_COLOR="#e0ceaa" COLOR="#690f14" CREATED="1698454208365" ID="ID_410477206" MODIFIED="1698455361253" TEXT="Falle! mu&#xdf; explizit droppen vor targeted-sleep">
<icon BUILTIN="broken-line"/>
<node CREATED="1698454352963" ID="ID_1446465614" LINK="#ID_1681258664" MODIFIED="1698454471274" TEXT="Hinweis auf Design-smell">
<icon BUILTIN="messagebox_warning"/>
</node>
<node CREATED="1698454732760" ID="ID_1159336693" MODIFIED="1698454753377" TEXT="direkte Konsequenz der Flexibilit&#xe4;t der Activity-Language"/>
<node CREATED="1698454756045" ID="ID_296403681" MODIFIED="1698454779174" TEXT="niemand verlangt, da&#xdf; das Token nach jedem Dispatch gedroppt wurde"/>
<node CREATED="1698454779833" ID="ID_742947700" MODIFIED="1698454794491" TEXT="tats&#xe4;chlich wei&#xdf; sogar die ActivityLang &#xfc;berhaupt nichts vom Grooming-Token"/>
<node CREATED="1698454804383" ID="ID_747980717" MODIFIED="1698454833154" TEXT="und es sind F&#xe4;lle von internem Processing denkbar, wo es gehalten wird"/>
<node CREATED="1698454838234" ID="ID_1414177644" MODIFIED="1698454849444" TEXT="aber ob das nun richtig ist, h&#xe4;ngt vom Queue-Inhalt ab"/>
<node COLOR="#435e98" CREATED="1698454850565" ID="ID_178396508" MODIFIED="1698455364612" TEXT="&#x27f9; wir m&#xfc;ssen explizit vor jedem Sleep aufpassen"/>
<node BACKGROUND_COLOR="#e0ceaa" COLOR="#690f14" CREATED="1698454870694" ID="ID_1550541513" MODIFIED="1698454924572">
<richcontent TYPE="NODE"><html>
<head/>
<body>
<p>
sonst k&#246;nnte der Scheduler zwischendurch
</p>
<p>
&#160;f&#252;r Zeitspannen bis zu 20ms geblockt sein
</p>
</body>
</html></richcontent>
<icon BUILTIN="broken-line"/>
<node CREATED="1698454926522" ID="ID_417517465" MODIFIED="1698455334934" TEXT="das w&#xe4;re ziemlich schwer festzustellen">
<richcontent TYPE="NOTE"><html>
<head/>
<body>
<p>
...denn typischerweise passiert ja ein <i>targeted sleep </i>genau dann, wenn grade eben nichts zu tun ist; aber ein Teil dieser Verz&#246;gerun k&#246;nnte eben doch &#252;ber eine Zeitspanne reichen, in der Activities geplant sind (oder zwischenzeitlich eingespielt wurden)
</p>
</body>
</html></richcontent>
</node>
<node CREATED="1698454949611" ID="ID_609944433" MODIFIED="1698454955558" TEXT="und wahrscheinlich auch sehr selben"/>
<node CREATED="1698454956218" ID="ID_249261761" MODIFIED="1698454959373" TEXT="umso gef&#xe4;hrlicher"/>
</node>
</node>
</node>
<node COLOR="#338800" CREATED="1698449165927" ID="ID_1758101770" MODIFIED="1698453116175" TEXT="wenn nicht zu erlangen &#x27f6; SKIP">
<richcontent TYPE="NOTE"><html>
<head/>
<body>
<p>
passiert tats&#228;chlich in SchedulerCommutator::postDispatch()
</p>
</body>
</html></richcontent>
<icon BUILTIN="button_ok"/>
<node CREATED="1698449196502" ID="ID_901891757" MODIFIED="1698449202355" TEXT="nicht WAIT"/>
<node CREATED="1698449202906" ID="ID_1519160617" MODIFIED="1698449215348" TEXT="aber auch nicht weitermachen"/>
<node BACKGROUND_COLOR="#f8f1cb" COLOR="#a50125" CREATED="1698449216368" ID="ID_1023610173" MODIFIED="1698450803071" TEXT="&#x27f9; also hot polling / spinning">
<arrowlink COLOR="#7f5d5b" DESTINATION="ID_1388229244" ENDARROW="Default" ENDINCLINATION="1570;-56;" ID="Arrow_ID_1744066292" STARTARROW="None" STARTINCLINATION="1233;109;"/>
<icon BUILTIN="messagebox_warning"/>
</node>
</node>
<node COLOR="#338800" CREATED="1698458707764" ID="ID_17588364" MODIFIED="1698458728498" TEXT="etwas Dokumentation &#x27f6; scheduler.hpp">
<icon BUILTIN="button_ok"/>
</node>
<node BACKGROUND_COLOR="#d2beaf" COLOR="#5c4d6e" CREATED="1698458524913" ID="ID_45287969" MODIFIED="1698458664836" TEXT="Offener Punkt: Grooming-Token nach IO-callbacks">
<arrowlink COLOR="#a48aab" DESTINATION="ID_406351135" ENDARROW="Default" ENDINCLINATION="-1523;-56;" ID="Arrow_ID_1878900565" STARTARROW="None" STARTINCLINATION="702;40;"/>
<icon BUILTIN="bell"/>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1698372042164" ID="ID_1272410719" MODIFIED="1698372168803" TEXT="Clock-Implementierung">
@ -87135,6 +87216,10 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<icon BUILTIN="hourglass"/>
<icon BUILTIN="messagebox_warning"/>
</node>
<node BACKGROUND_COLOR="#f0d5c5" COLOR="#990033" CREATED="1698458564719" ID="ID_406351135" MODIFIED="1698458664836" TEXT="pr&#xfc;fen: wird das Grooming-Token ben&#xf6;tigt?">
<linktarget COLOR="#a48aab" DESTINATION="ID_406351135" ENDARROW="Default" ENDINCLINATION="-1523;-56;" ID="Arrow_ID_1878900565" SOURCE="ID_45287969" STARTARROW="None" STARTINCLINATION="702;40;"/>
<icon BUILTIN="help"/>
</node>
</node>
</node>
<node BACKGROUND_COLOR="#d2beaf" COLOR="#5c4d6e" CREATED="1684980732965" ID="ID_1914429002" MODIFIED="1684980742528" TEXT="Thema: Timing-Updates">
@ -90515,7 +90600,7 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
</node>
<node BACKGROUND_COLOR="#e0ceaa" COLOR="#690f14" CREATED="1689905560194" ID="ID_795058897" MODIFIED="1689905988528">
<richcontent TYPE="NODE"><html>
<head/
<head/>
<body>
<p>
auch mit weiterer Variation der Parameter kommt man kaum unter <b><font color="#440e6a">30</font></b><font color="#440e6a">ns</font>
@ -91251,6 +91336,17 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<node CREATED="1697981980202" ID="ID_162807330" MODIFIED="1697982002392" TEXT="waiting-due time &#x2205; der Activities im Scheduler"/>
<node CREATED="1697982054792" ID="ID_712555328" MODIFIED="1697982065291" TEXT="wieviele Activities werden von Schl&#xe4;fern &#xfc;bernommen"/>
<node CREATED="1697982086540" ID="ID_1123632550" MODIFIED="1697982105925" TEXT="waiting-due time speziell wenn von Schl&#xe4;fer &#xfc;bernommen"/>
<node CREATED="1698450006319" ID="ID_1389677902" MODIFIED="1698450366289" TEXT="wieviel Zeit verbringt das System im Grooming-Mode?">
<richcontent TYPE="NOTE"><html>
<head/>
<body>
<p>
...das k&#246;nnte tats&#228;chlich schwierig herauszufinden sein, weil wir <b>definitiv nicht</b>&#160;jede Handhabung des Grooming-Tokens irgendwo als Event loggen wollen; zudem besteht die Schwierigkeit, da&#223; wir das Grooming-Token nach Aufrufen der work-Funktion oft wieder droppen
</p>
</body>
</html></richcontent>
<linktarget COLOR="#9198a2" DESTINATION="ID_1389677902" ENDARROW="Default" ENDINCLINATION="-297;15;" ID="Arrow_ID_16649799" SOURCE="ID_118653070" STARTARROW="None" STARTINCLINATION="196;9;"/>
</node>
</node>
<node BACKGROUND_COLOR="#f8f1cb" COLOR="#a50125" CREATED="1698200300379" ID="ID_919358146" MODIFIED="1698200324271" TEXT="zus&#xe4;tzliche Detail-Fragen kl&#xe4;ren">
<icon BUILTIN="bell"/>
@ -91291,6 +91387,51 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<node CREATED="1698201429620" ID="ID_1635005942" MODIFIED="1698201443678" TEXT="und dem Vermeiden vorgreifender Festlegungen"/>
</node>
</node>
<node BACKGROUND_COLOR="#f0d5c5" COLOR="#990033" CREATED="1698449820247" ID="ID_759735068" MODIFIED="1698449830478" TEXT="Fragen f&#xfc;r sp&#xe4;ter...">
<icon BUILTIN="help"/>
<node BACKGROUND_COLOR="#d2beaf" COLOR="#5c4d6e" CREATED="1698449617586" ID="ID_1388229244" MODIFIED="1698450803071" TEXT="hot-polling on Grooming-Token?">
<linktarget COLOR="#7f5d5b" DESTINATION="ID_1388229244" ENDARROW="Default" ENDINCLINATION="1570;-56;" ID="Arrow_ID_1744066292" SOURCE="ID_1023610173" STARTARROW="None" STARTINCLINATION="1233;109;"/>
<icon BUILTIN="hourglass"/>
<node CREATED="1698449890835" HGAP="30" ID="ID_913093042" MODIFIED="1698450576260" TEXT="das scheint eine ehr akademische Frage" VSHIFT="28">
<richcontent TYPE="NOTE"><html>
<head/>
<body>
<p>
...denn der Contender hat in dem Moment ohnehin nix anderes zu tun und ist auch von der Kapazit&#228;t her definitiv f&#252;r das Rendering belegt. Zudem bel&#228;stigen wir durch aktives Polling niemanden sonst (abgesehen von dem produzierten CO&#8322;); und der Fall sollte selten sein und idealerweise auch nur kurz dauern
</p>
</body>
</html></richcontent>
</node>
<node CREATED="1698449940984" HGAP="50" ID="ID_118653070" MODIFIED="1698450762218" TEXT="und auch nur durch ein Absch&#xe4;tzungs-Argument zu beantworten" VSHIFT="-7">
<richcontent TYPE="NOTE"><html>
<head/>
<body>
<p>
Eine Wirkung k&#246;nnen wir n&#228;mlich nicht messen &#10233; es l&#228;uft darauf hinaus, ob dieses Vorgehen verh&#228;ltnism&#228;&#223;ig ist, und das ist identisch mit der Frage, ob Contentions wirklich so extrem selten sind, wie vom Konzept her vermutet. Der wichtigste Ansatzpunkt w&#228;re daher, nach zeitlichen Koinzidenzen zu suchen, was allerdings ohne weitere Instrumentierung auch nicht m&#246;glich sein d&#252;rfte
</p>
</body>
</html></richcontent>
<arrowlink COLOR="#9198a2" DESTINATION="ID_1389677902" ENDARROW="Default" ENDINCLINATION="-297;15;" ID="Arrow_ID_16649799" STARTARROW="None" STARTINCLINATION="196;9;"/>
</node>
</node>
<node BACKGROUND_COLOR="#d2beaf" COLOR="#5c4d6e" CREATED="1698454549032" ID="ID_1970723448" MODIFIED="1698454589468" TEXT="Activity-Lang: processing setzt auf einem Proc-State auf">
<icon BUILTIN="hourglass"/>
<node BACKGROUND_COLOR="#f0d5c5" COLOR="#990033" CREATED="1698454408819" ID="ID_1681258664" MODIFIED="1698454466874" TEXT="targeted-sleeps passieren im Scheduler. Sinnvoll?">
<icon BUILTIN="help"/>
<node CREATED="1698454595586" ID="ID_1526136231" MODIFIED="1698454700723" TEXT="das ist direkte Konsequenz von activity::PROC">
<richcontent TYPE="NOTE"><html>
<head/>
<body>
<p>
...da wir nur einen Status-Code haben, aber nicht sagen k&#246;nnen <i>was und wie genau weiter verfahren werden soll</i>. Dadurch ist zwar die Kopplung oberfl&#228;chlich lose und explizit, tats&#228;chlich aber f&#252;hrt es zu sehr komplexen Kollaborationen
</p>
</body>
</html></richcontent>
</node>
<node CREATED="1698454491296" ID="ID_1287561281" LINK="#ID_410477206" MODIFIED="1698454517684" TEXT="macht Code complex, z.B. Grooming-Token-Behandlung"/>
</node>
</node>
</node>
</node>
</node>
</node>
@ -91845,16 +91986,13 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
</node>
<node BACKGROUND_COLOR="#e8e78a" COLOR="#a34037" CREATED="1698431962633" ID="ID_1475863913" MODIFIED="1698433650097" TEXT="Heureka! mit dem probabilstischen Ansatz verbinden">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<head/>
<body>
<p>
Gru&#223; vom Vollmond in Pullach (noch nicht ganz voll, morgen ist Mondfinsternis), der geht das Tal entlang mit
</p>
</body>
</html>
</richcontent>
</html></richcontent>
<arrowlink COLOR="#f5fd9d" DESTINATION="ID_323819142" ENDARROW="Default" ENDINCLINATION="44;-68;" ID="Arrow_ID_116183984" STARTARROW="None" STARTINCLINATION="-157;7;"/>
<linktarget COLOR="#722322" DESTINATION="ID_1475863913" ENDARROW="Default" ENDINCLINATION="888;-37;" ID="Arrow_ID_1254135767" SOURCE="ID_1985773186" STARTARROW="None" STARTINCLINATION="741;41;"/>
<font BOLD="true" NAME="SansSerif" SIZE="12"/>
@ -91876,9 +92014,7 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
</node>
<node CREATED="1698433489965" ID="ID_323819142" MODIFIED="1698433626101">
<richcontent TYPE="NODE"><html>
<head>
</head>
<head/>
<body>
<p>
<font size="4"><u>Priorit&#228;t</u></font>&#160;bestimmt die <b>Wahrscheinlichkeit</b>,
@ -91887,15 +92023,12 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
Kapazit&#228;t zuf&#228;llig zu erlangen
</p>
</body>
</html>
</richcontent>
</html></richcontent>
<linktarget COLOR="#f5fd9d" DESTINATION="ID_323819142" ENDARROW="Default" ENDINCLINATION="44;-68;" ID="Arrow_ID_116183984" SOURCE="ID_1475863913" STARTARROW="None" STARTINCLINATION="-157;7;"/>
<icon BUILTIN="forward"/>
<node CREATED="1698433687865" HGAP="17" ID="ID_1856291258" MODIFIED="1698436739705" TEXT="ein Render-Proze&#xdf; setzt Kapazit&#xe4;t voraus" VSHIFT="16">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<head/>
<body>
<p>
Grunds&#228;tzlich darf ein Render-Proze&#223; nur bei vorhandener Kapazit&#228;t zugelassen werden. Aber die Anwendung dieses Prinzips hat Abstufungen, die sich aus der Art des Prozesses bestimmen.
@ -91920,103 +92053,79 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
</ul>
</ul>
</body>
</html>
</richcontent>
</html></richcontent>
</node>
<node CREATED="1698434259630" ID="ID_1030217927" MODIFIED="1698434360871" TEXT="ein geSchedulter Task hat einen Wahrscheinlichkeitsquerschnitt">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<head/>
<body>
<p>
die aktuelle Situation bestimmt dar&#252;ber, wie Kapazit&#228;t zugef&#252;hrt wird; aber der Querschnitt bestimmt, wie gute Chancen ein konkreter Task hat, davon etwas abzubekommen
</p>
</body>
</html>
</richcontent>
</html></richcontent>
<node CREATED="1698434362407" ID="ID_1045531877" MODIFIED="1698434532770" TEXT="solange etwas davor steht, ist er verdeckt">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<head/>
<body>
<p>
Ein <i>frei stehender</i>&#160;Task bekommt durch den tend-Next-Mechanismus auch schon sehr viel fr&#252;her die n&#228;chste frei werdende Kapazit&#228;t zugeordnet; solange aber in der einfachen zeitlichen Ordnung noch etwas vor ihm steht (selbst wenn &#252;berf&#228;llig), dann zieht dieses die Priorit&#228;t auf sich
</p>
</body>
</html>
</richcontent>
</html></richcontent>
</node>
<node CREATED="1698434413940" ID="ID_1412965140" MODIFIED="1698434678138">
<richcontent TYPE="NODE"><html>
<head>
</head>
<head/>
<body>
<p>
sobald er aber frei steht, bestimmt seine <b>L&#228;nge</b>&#160;die Priorit&#228;t
</p>
</body>
</html>
</richcontent>
</html></richcontent>
<richcontent TYPE="NOTE"><html>
<head>
</head>
<head/>
<body>
<p>
Und zwar durch das Ende (die Deadline), nach deren &#220;berstreichen der Task effektiv unwirksam ist und im Vorr&#252;bergehen entnommen und verworfen wird. Wenn nun verschiedene Tasks jeweils in der L&#228;nge beschr&#228;nkt sind, dann f&#228;llt ihnen diejenige Kapazit&#228;t zu, die zuf&#228;llig in ihrem Wirkradius <i>&#8222;landet&#8220;</i>
</p>
</body>
</html>
</richcontent>
</html></richcontent>
</node>
<node CREATED="1698434700812" ID="ID_1790239617" MODIFIED="1698434978201" TEXT="weitere Faktoren modulieren dieses Schema">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<head/>
<body>
<p>
Da sind zun&#228;chst die Gates von Relevanz. Ein noch geschlossenes Gate kann einen Task nach hinten schieben und damit andere Tasks aufdecken. Ein getriggertes und endg&#252;ltig geschlossenes Gate nimmt den Task aus der Konkurrenz komplett heraus. Und au&#223;erdem werden Tasks auch noch &#252;ber eine Proze&#223;/Kontext-ID gekennzeichnet, wodurch eine Revision und Aktualisierung eines gesamten Planungsvorgangs m&#246;glich wird.
</p>
</body>
</html>
</richcontent>
</html></richcontent>
</node>
</node>
<node CREATED="1698434985257" ID="ID_1456084825" MODIFIED="1698435345311" TEXT="man kann die gleiche Chain mehrfach schedulen &#x2014; sofern sie ein Gate-Enablement hat">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<head/>
<body>
<p>
...das bedeutet, wenn wir einmal durch das Gate gegangen sind, ist es endg&#252;ltig geschlossen. Deshalb k&#246;nnen gewisserma&#223;en mehrere &#187;Instanzen&#171; eingeplant werden, denn das Wegr&#228;umen von M&#252;ll ist in einer Priority-Queue relativ effizient, O(log&#8345;), und l&#228;uft bei uns im einstelligen &#181;s-Bereich pro Einzelschritt
</p>
</body>
</html>
</richcontent>
</html></richcontent>
</node>
<node BACKGROUND_COLOR="#e0ceaa" COLOR="#411d16" CREATED="1698435413188" HGAP="18" ID="ID_988639193" MODIFIED="1698436797466" STYLE="bubble" TEXT="&#x27f9; Wir definieren ein Schema, nach dem Tasks in Zeitfenster gesetzt werden" VSHIFT="4">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<head/>
<body>
<p>
Ein Task der nur Restkapazit&#228;t bekommen soll, darf niemals am Anfang des Fensters stehen, und auch m&#246;glichst nicht ganz am Ende. Je k&#252;rzer und je mehr in der Mitte, desto geringer seine Chancen. Ein Task der in jedem Fall Kapazit&#228;t bekommen soll, mu&#223; nur hinreichend weit nach hinten reichen (aber unter der Einschr&#228;nkung, unseren Epochen-basierten BlockFlow nicht zu &#252;berlasten). Oder er mu&#223; hinreichend oft wiederholt werden. Es wird also ein generelles Segment-Schema etabliert, und in diesem gibt es vorgefertigte &#187;Slots&#171;. Gem&#228;&#223; &#252;bergeordnete Kapazit&#228;tsplanung werden diese Slots in Anspruch genommen. Wenn wir beispielsweise eine Restkapazit&#228;t 10-fach &#252;berprovisionieren, dann bedeutet das, jeden einzelnen Task (ggfs. mit Zeitabstand) 10+x-mal einzuplanen, wobei x einen empirisch bestimmten safety-margin darstellt, um die tats&#228;chlichen Fluktuationen der Kapazit&#228;t abzufedern...
</p>
</body>
</html>
</richcontent>
</html></richcontent>
</node>
<node CREATED="1698436618074" HGAP="23" ID="ID_145123835" MODIFIED="1698436752191" VSHIFT="5">
<richcontent TYPE="NODE"><html>
<head>
</head>
<head/>
<body>
<p style="text-align: center">
dadurch &#252;bersetzen wir multidimensionale Zusammenh&#228;nge
@ -92028,8 +92137,7 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
in ein low-Level-Ausf&#252;hrungsschema
</p>
</body>
</html>
</richcontent>
</html></richcontent>
<icon BUILTIN="yes"/>
</node>
</node>
@ -96655,19 +96763,16 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
09.05.19 17:10
</p>
<p style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px">
<font face="Bitstream Vera Sans Mono" size="9pt">Library: further narrowing down the tuple-forwarding problem</font>
</p>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono" size="9pt"> </font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono" size="9pt"> ...yet still not successful.</font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono" size="9pt"> </font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono" size="9pt"> The mechanism used for std::apply(tuple&amp;) works fine when applied directly to the target function,</font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono" size="9pt"> but fails to select the proper overload when passed to a std::forward-call for</font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono" size="9pt"> &quot;perfect forwarding&quot;. I tried again to re-build the situation of std::forward</font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono" size="9pt"> with an explicitly coded function, but failed in the end to supply a type parameter</font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono" size="9pt"> to std::forward suitably for all possible cases</font></pre>
<p style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px">
<font face="Bitstream Vera Sans Mono">Library: further narrowing down the tuple-forwarding problem</font>
</p>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono"> </font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono"> ...yet still not successful.</font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono"> </font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono"> The mechanism used for std::apply(tuple&amp;) works fine when applied directly to the target function,</font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono"> but fails to select the proper overload when passed to a std::forward-call for</font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono"> &quot;perfect forwarding&quot;. I tried again to re-build the situation of std::forward</font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono"> with an explicitly coded function, but failed in the end to supply a type parameter</font></pre>
<pre style="margin-top: 0px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; text-indent: 0px"><font face="Bitstream Vera Sans Mono"> to std::forward suitably for all possible cases</font></pre>
</body>
</html></richcontent>
<linktarget COLOR="#a98ca6" DESTINATION="ID_1691540337" ENDARROW="Default" ENDINCLINATION="2957;477;" ID="Arrow_ID_1272525320" SOURCE="ID_1773619805" STARTARROW="None" STARTINCLINATION="-1064;82;"/>