From cf7c2d1327972ad00b58407866832905d63dd7c0 Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Wed, 6 Sep 2023 19:18:37 +0200 Subject: [PATCH] Workforce: analysis and design - investigate consistency guarantees through acquire-release ==> turns out we do not need a fence, but it is tantamount to have a guard variable and actually load and check the value to ensure we indeed get a happens-before - elaborate design of the WorkForce + no shared control variables necessary + no ability to forcibly shut-down the WorkForce + rather, all control will be exerted through the return value of the Work-Functor --- src/vault/gear/activity.hpp | 5 +- src/vault/gear/work-force.hpp | 47 +++++ wiki/renderengine.html | 20 ++- wiki/thinkPad.ichthyo.mm | 320 +++++++++++++++++++++++++++++++++- 4 files changed, 382 insertions(+), 10 deletions(-) diff --git a/src/vault/gear/activity.hpp b/src/vault/gear/activity.hpp index 1a09b4d65..11984c997 100644 --- a/src/vault/gear/activity.hpp +++ b/src/vault/gear/activity.hpp @@ -114,11 +114,12 @@ namespace gear { /** * Result instruction from Activity activation. - * These codes are used by the ActivityLang to - * coordinate further processing of activity chains. + * These codes are used by the ActivityLang and WorkForce + * to coordinate further processing of activities. */ enum Proc {PASS ///< pass on the activation down the chain ,SKIP ///< skip rest of the Activity chain for good + ,WAIT ///< nothing to do; wait and re-check for work later ,KILL ///< obliterate the complete Activity-Term and all its dependencies ,HALT ///< abandon this play / render process }; diff --git a/src/vault/gear/work-force.hpp b/src/vault/gear/work-force.hpp index a594e559a..1f08657c9 100644 --- a/src/vault/gear/work-force.hpp +++ b/src/vault/gear/work-force.hpp @@ -59,6 +59,53 @@ namespace gear { using std::atomic; + namespace work { + + struct Config + { + + }; + + template + class Runner + : std::thread + , CONF + { + public: + Runner() + : thread{} + { } + + private: + void + pullWork() + { + try { + while (true) + { + activity::Proc res = CONF::doWork(); + if (res == activity::WAIT) + res = idleWait(); + if (res != activity::PASS) + break; + } + } + ERROR_LOG_AND_IGNORE (threadpool, "defunct worker thread") + ////////////////////////////////////////////////////////////////////////////OOO very important to have a reliable exit-hook here!!! + } + + activity::Proc + idleWait() + { + using namespace std::chrono_literals; /////////////WIP + std::this_thread::sleep_for (100ms); + ////////////////////////////////WIP extended inactivity detector here + return activity::PASS; + } + }; + } + + /** * Pool of worker threads for rendering. * diff --git a/wiki/renderengine.html b/wiki/renderengine.html index 738a6a7ef..ce7df8905 100644 --- a/wiki/renderengine.html +++ b/wiki/renderengine.html @@ -7245,7 +7245,7 @@ The primary scaling effects exploited to achieve this level of performance are t The way other parts of the system are built, requires us to obtain a guaranteed knowledge of some job's termination. It is possible to obtain that knowledge with some limited delay, but it nees to be absoultely reliable (violations leading to segfault). The requirements stated above assume this can be achieved through //jobs with guaranteed execution.// Alternatively we could consider installing specific callbacks -- in this case the scheduler itself has to guarantee the invocation of these callbacks, even if the corresponding job fails or is never invoked. It doesn't seem there is any other option. -
+
The Scheduler //maintains a ''Work Force'' (a pool of workers) to perform the next [[render activities|RenderActivity]] continuously.//
 Each worker runs in a dedicated thread; the Activities are arranged in a way to avoid blocking those worker threads
 * IO operations are performed asynchronously {{red{planned as of 9/23}}}
@@ -7256,6 +7256,24 @@ Using a pool of workers to perform small isolated steps of work atomically and i
 This leads to a situation where it is more adequate to //distribute the scarce computation resources// to the tasks //sequenced in temporary and dependence order//. The computation tasks must be prepared and ordered -- but beyond that, there is not much that can be »managed« with a computation task. For this reason, the Scheduler in the Lumiera Render Engine uses a pool of workers, each providing one unit of computation resource (a »core«), and these workers will ''pull work'' actively, rather then distributing, queuing and dispatching tasks to a passive set of workers.
 
 Moreover, the actual computation tasks, which can be parallelised, are at least by an order of magnitude more expensive than any administrative work for sorting tasks, checking dependencies and maintaining process state. This leads to a scheme where a worker first performs some »management work«, until encountering the next actual computation job, at which point the worker leaves the //management mode// and transitions into //concurrent work mode//. All workers are expected to be in work mode almost entirely most of the time, and thus we can expect not much contention between workers performing »management work« -- allowing to confine this management work to //single threaded operation,// thereby drastically reducing the complexity of management data structures and memory allocation.
+!!!Regulating workers
+The behaviour of individual workers is guided solely by the return-value flag from the work-functor. Consequently, no shared flags and no direct synchronisation whatsoever is required //within the {{{WorkForce}}} implementation.// -- notwithstanding the fact that the implementation //within the work-functor// obviously needs some concurrency coordination to produce these return values, since this the whole point is to invoke this functor concurrently. The following aspects of worker behaviour can be directed:
+* returning {{{activity::PASS}}} instructs the worker to re-invoke the work-functor in the same thread immediately
+* returning {{{activity::WAIT}}} requests an //idle-wait cycle//
+* any other value, notably {{{activity::HALT}}} causes the worker to terminate
+* likewise, an exception from anywhere within the worker shall terminate the worker and activate a »disaster mode«
+Essentially this implies that //the workers themselves// (not some steering master) perform the management code leading to the aforementioned state directing return codes.
+
+The ''idle-wait cycle'' involves some degree of capacity control. First off, a worker receiving the {{{activity::WAIT}}} should perform a sleep (by the OS scheduler) for some small period (configurable as Render Engine parameter). However, if a worker encounters an extended ongoing sequence of idle-wait cycles, beyond a (likewise configurable) threshold, the //worker shall terminate.// This way, the {{{WorkForce}}} gradually scales down when not used any more. On the other hand, //scaling up// is performed stepwise or in one jump, and triggered by an external control call, typically issued by the {{{Scheduler}}} in response to a signal from the {{{Player}}} indicating a newly added workload.
+
+!!!Synchronisation and data consistency
+As pointed out above, the code controlling the worker behaviour must care for data consistency when using shared state. However, in the actual usage scenario for scheduling media computations the amount of »management work« is by orders of magnitude smaller than any actual media computations. For that reason, //all management work// involving shared data structures can be performed //exclusively single-threaded.// This is achieved by acquiring the {{{GroomingToken}}} within the Scheduler control layer (Layer-2). After mutating the queues to determine the next task, this token will be dropped again, prior to engaging into the actual media computations.
+
+Following the rules of [[C++ atomic Release-Acquire ordering|https://en.cppreference.com/w/cpp/atomic/memory_order#Release-Acquire_ordering]], for ensuring concurrent data consistency it is sufficient with this usage pattern to rely on a single atomic variable (used as implementation of the {{{GroomingToken}}}). All the other internal management data (notably the priority queue and the memory manager) can be implemented with ordinary memory accessible by all threads. The important part is actually to establish a »synchronises-with« relation
+# managment data is processed in one thread holding the {{{GroomingToken}}}
+# an atomic store is performed with {{{memory_order_release}}} (thereby dropping the token)
+# now another thread can perform an atomic load with {{{memory_order_acquire}}} -- but it is essential to check and branch on the loaded value
+# and only when the loaded value indicates that the load is //synchronised-with// the preceding store, then the other management data can be processed
diff --git a/wiki/thinkPad.ichthyo.mm b/wiki/thinkPad.ichthyo.mm index dee2fea23..82cfd8e9b 100644 --- a/wiki/thinkPad.ichthyo.mm +++ b/wiki/thinkPad.ichthyo.mm @@ -79625,6 +79625,220 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
+ + + + + + +

+ das bedeutet, daß die Work-Function bereits auf Typ-Ebene definiert sein muß; eine Möglichkeit ist, sie über eine Config-Template-Basis einzumischen +

+ + + + + + + + + +

+ Entsprechend dem Design-Prinzip für die Scheduler-Steuerung möchte ich einen aktivenen Management-Thread und eine Zuweisung von Aktivitäten vermeiden +

+ + + + + + + + +

+ Wir vermeiden Kontrollverlust durch einen Fehler im zentralen Steuer-Thread, Verklemmungen oder Split-brain +

+ + + + + + + + + + + + + +

+ das bedeutet auch: sein dtor muß blocken (oder laufenden Threads einfach abwerfen) +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

+ ...man könnte auch sagen, das ist eine Frage der Logik: die WorkForce ist allein, was den Scheduler aktiv macht. Deshalb sollte der dieser Status nicht woanders sein.... +

+ + + + + + + + +

+ ...es wäre denkbar, daß es sich nicht um einen KILL-Switch handelt, sondern um einen Heartbeat, der z.B. aus den tick()-Aufrufen erneuert werden muß, so daß im Fall einer Verklemmung sich der Scheduler selbst terminiert +

+ + + + + + + + +
    +
  • + der Exception-Handler in einem Worker-Thread muß diese Flag setzen können; dieser Handler kann jedoch jederzeit greifen (nicht nur im Managment-Mode) +
    • +
  • + die Implementierung der Work-Fun im Scheduler-Commutator muß darauf zugreifen, um dann den enstprechenden Rückgabewert zu liefern, der die Worker terminiert +
    • +
+ + + + + + + + + + + + + + + + + + + + + + + + + +
    +
  • + entweder ein terminierende Thread löscht sich selbst aus der Storage (gefährlich) +
    • +
  • + oder es hängen bereits terminierte Threads in der Storage, und jede Kapazitäts-Ermittlung muß O(n) Überprüfungen machen +
    • +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -79672,7 +79886,8 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
- + + @@ -79682,6 +79897,68 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
 + + + + + + + + + + + + + + + + + + + + + +

+ die Verzweigung ist essentiell +

+ + + + + + + + + + +

+ das bedeutet, der Check muß darauf hinauslaufen; es genügt nicht, bloß irgend einen Wert vom Guard gelesen zu haben, sondern man muß sicherstellen, daß der gelesene Wert tatsächlich aus derjenigen Operation stammen muß, welche auf der anderen Seite (beim Producer) als letztes nach der Manipulation der Speicherinhalte stattfand... +

+

+ +

+
    +
  • + andere Operationen-A sequenced-before release +
    • +
  • + acquire sequenced-before andere Operationen-B +
    • +
  • + ⟹ Operationen A  happens-before  Operationen-B +
    • +
+ + + + + + + + + + + @@ -87970,7 +88247,7 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
 - + @@ -88142,7 +88419,7 @@ class Something - + @@ -88628,7 +88905,7 @@ class Something - + @@ -88637,7 +88914,7 @@ class Something - + @@ -88683,7 +88960,36 @@ class Something - + + + + + + +

+ zusätzlich gib es eine daran gebundene Payload aus nicht-atomaren Werten +

+ + + + + +
    +
  • + diese nicht-atomaren Werte sind ganz normale Speicherinhalte, die potentiell von verschiedenen Threads aus zugreifbar wären +
    • +
  • + sie müssen aber an den Guard gebunden sein, und zwar durch einen conditional-Check auf den vom Guard gelesenen Wert +
    • +
+

+ Dieser Zusammenhang fußt auf den exakten Definitionen der release  und acquire-semantics und einer damit verbundenen synchronizes-with  Zusicherung. Das hat nichts mit data-dependency zu tun! Es ist egal, wie auf die "anderen" Werte vor/nach der Synchronisierung zugegriffen wird (pointer, argument, globale Variable). Sondern entscheidend ist, daß dazwischen liegend auf eine bestimmte Eigenschaft/Wert im Guard geprüft wird; diese Eigenschaft muß sicherstellen, daß das synchronizes-with stattgefunden hat. Zugriffe ohne diese Prüfung sind undefiniert (d.h. sie können oder können nicht einen Mix von veränderten Werten sehen). +

+ + + + + @@ -88772,7 +89078,7 @@ class Something

- Allerdinsg nicht auf einer Plattform, die ohnehin sequentiell-konsistent ist. Wie "zum Beispiel" x86/64 + Allerdings nicht auf einer Plattform, die ohnehin sequentiell-konsistent ist. Wie "zum Beispiel" x86/64