Scheduler-test: guard memory allocations by grooming-token

Turns out that we need to implemented fine grained and explicit handling logic
to ensure that Activity planning only ever happens protected by the Grooming-Token.
This is in accordance to the original design, which dictates that all management tasks
must be done in »management mode«, which can only be entered by a single thread at a time.
The underlying assumption is that the effort for management work is dwarfed in comparison
to any media calculation work.

However, in
5c6354882d
...I discovered an insidious border condition, an in an attempt to fix it,
I broke that fundamental assumpton. The problem arises from the fact that we
do want to expose a *public API* of the Scheduler. Even while this is only used
to ''seed'' a calculation stream, because any further planning- and management work
will be performed by the workers themselves (this is a design decision, we do not
employ a "scheduler thread")
Anyway, since the Scheduler API ''is'' public, ''someone from the outside'' could
invoke those functions, and — unaware of any Scheduler internals — will
automatically acquire the Grooming-Token, yet never release it,
leading to deadlock.

So we need a dedicated solution, which is hereby implemented as a
scoped guard: in the standard case, the caller is a management-job and
thus already holds the token (and nothing must be done). But in the
rare case of an »outsider«, this guard now ''transparently'' acquires
the token (possibly with a blocking wait) and ''drops it when leaving scope''
This commit is contained in:
Fischlurch 2023-12-19 23:38:57 +01:00
parent 523586570f
commit b497980522
4 changed files with 367 additions and 8 deletions

View file

@ -93,10 +93,13 @@ namespace gear {
using std::memory_order::memory_order_relaxed;
using std::memory_order::memory_order_acquire;
using std::memory_order::memory_order_release;
using std::chrono_literals::operator ""us;
using std::chrono::microseconds;
namespace { // Configuration / Scheduling limit
Offset FUTURE_PLANNING_LIMIT{FSecs{20}}; ///< limit timespan of deadline into the future (~360 MiB max)
microseconds GROOMING_WAIT_CYCLE{70us}; ///< wait-sleep in case a thread must forcibly acquire the Grooming-Token
/** convenient short-notation, also used by SchedulerService */
auto inline thisThread() { return std::this_thread::get_id(); }
@ -165,6 +168,12 @@ namespace gear {
}
class ScopedGroomingGuard;
/** a scope guard to force acquisition of the GroomingToken */
ScopedGroomingGuard requireGroomingTokenHere();
void
sanityCheck (ActivationEvent const& event, Time now)
{
@ -279,5 +288,59 @@ namespace gear {
};
class SchedulerCommutator::ScopedGroomingGuard
: util::MoveOnly
{
SchedulerCommutator& commutator_;
bool handledActively_;
bool
ensureHoldsToken()
{
if (commutator_.holdsGroomingToken(thisThread()))
return false;
while (not commutator_.acquireGoomingToken())
std::this_thread::sleep_for (GROOMING_WAIT_CYCLE);
return true;
}
public:
/** @warning can block indefinitely if someone hogs the token */
ScopedGroomingGuard (SchedulerCommutator& layer2)
: commutator_(layer2)
, handledActively_{ensureHoldsToken()}
{ }
~ScopedGroomingGuard()
{
if (handledActively_)
commutator_.dropGroomingToken();
}
};
/**
* @warning this provides very specific functionality
* required by the »Scheduler Service« to handle both
* _external_ and _internal_ calls properly.
* - whenever a thread already holds the GroomingToken,
* no further action is performed (so the cost of this
* feature is one additional atomic read on the token)
* - however, a thread coming _from the outside_ and not
* belonging to the Scheduler ecosystem is typically not
* aware of the GroomingToken altogether. The token is
* acquired, possibly incurring a *blocking wait*, and
* it is dropped transparently when leaving the scope.
*/
inline SchedulerCommutator::ScopedGroomingGuard
SchedulerCommutator::requireGroomingTokenHere()
{
return ScopedGroomingGuard(*this);
}
}} // namespace vault::gear
#endif /*SRC_VAULT_GEAR_SCHEDULER_COMMUTATOR_H_*/

View file

@ -336,6 +336,7 @@ namespace gear {
,ManifestationID manID = ManifestationID()
,FrameRate expectedAdditionalLoad = FrameRate(25))
{
auto guard = layer2_.requireGroomingTokenHere(); // allow mutation
layer1_.activate (manID);
activityLang_.announceLoad (expectedAdditionalLoad);
continueMetaJob (RealClock::now(), planningJob, manID);
@ -352,6 +353,7 @@ namespace gear {
{
bool isCompulsory = true;
Time deadline = nextStart + DUTY_CYCLE_TOLERANCE;
auto guard = layer2_.requireGroomingTokenHere(); // protect allocation
// place the meta-Job into the timeline...
postChain ({activityLang_.buildMetaJob(planningJob, nextStart, deadline)
.post()
@ -359,7 +361,6 @@ namespace gear {
, deadline
, manID
, isCompulsory});
ensureDroppedGroomingToken();
}
@ -574,7 +575,8 @@ namespace gear {
*/
inline ScheduleSpec
ScheduleSpec::post()
{
{ // protect allocation
auto guard = theScheduler_->layer2_.requireGroomingTokenHere();
term_ = move(
theScheduler_->activityLang_
.buildCalculationJob (job_, start_,death_));
@ -583,7 +585,6 @@ namespace gear {
, death_
, manID_
, isCompulsory_});
theScheduler_->ensureDroppedGroomingToken();
return move(*this);
}
@ -732,6 +733,8 @@ namespace gear {
inline void
Scheduler::handleDutyCycle (Time now, bool forceContinuation)
{
auto guard = layer2_.requireGroomingTokenHere();
// consolidate queue content
layer1_.feedPrioritisation();
// clean-up of outdated tasks here

View file

@ -95,6 +95,7 @@ namespace test {
{
demonstrateSimpleUsage();
verify_GroomingToken();
verify_GroomingGuard();
torture_GroomingToken();
verify_DispatchDecision();
verify_findWork();
@ -162,6 +163,38 @@ namespace test {
/** @test verify extended logic to protect a scope
* - if the thread already holds the grooming token, nothing happens
* - otherwise, it is acquired (blocking) and dropped on exit
*/
void
verify_GroomingGuard()
{
SchedulerCommutator sched;
// Case-1: if a thread already holds the token....
CHECK (sched.acquireGoomingToken());
CHECK (sched.holdsGroomingToken (thisThread()));
{
auto guard = sched.requireGroomingTokenHere();
CHECK (sched.holdsGroomingToken (thisThread()));
}// leave scope -> nothing happens in this case
CHECK (sched.holdsGroomingToken (thisThread()));
// Case-2: when not holding the token...
sched.dropGroomingToken();
{
// acquire automatically (this may block)
auto guard = sched.requireGroomingTokenHere();
CHECK (sched.holdsGroomingToken (thisThread()));
}// leave scope -> dropped automatically
CHECK (not sched.holdsGroomingToken (thisThread()));
___ensureGroomingTokenReleased(sched);
}
/** @test ensure the GroomingToken mechanism indeed creates mutual
* exclusion to protected against concurrent corruption.
* @remark uses lib::test::threadBenchmark() to put the test-subject

View file

@ -104754,8 +104754,8 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
</node>
</node>
<node BACKGROUND_COLOR="#fafe99" COLOR="#fa002a" CREATED="1702949773611" ID="ID_1506062032" MODIFIED="1702954743993" TEXT="Incident: Assertion-Failure aus dem Allokator">
<linktarget COLOR="#e02174" DESTINATION="ID_1506062032" ENDARROW="Default" ENDINCLINATION="-578;34;" ID="Arrow_ID_622434036" SOURCE="ID_1069075638" STARTARROW="None" STARTINCLINATION="186;12;"/>
<linktarget COLOR="#e92d59" DESTINATION="ID_1506062032" ENDARROW="Default" ENDINCLINATION="-129;68;" ID="Arrow_ID_379365083" SOURCE="ID_1140898245" STARTARROW="None" STARTINCLINATION="-245;34;"/>
<linktarget COLOR="#e02174" DESTINATION="ID_1506062032" ENDARROW="Default" ENDINCLINATION="-578;34;" ID="Arrow_ID_622434036" SOURCE="ID_1069075638" STARTARROW="None" STARTINCLINATION="186;12;"/>
<icon BUILTIN="broken-line"/>
<node CREATED="1702949832876" ID="ID_131344952" MODIFIED="1702950526092" TEXT="versuche grade den Scheduler zu qu&#xe4;len...">
<icon BUILTIN="smiley-oh"/>
@ -105214,10 +105214,128 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<node COLOR="#338800" CREATED="1702945603433" ID="ID_427316058" MODIFIED="1703018549851" TEXT="Klammer in &#x3bb;-post in Hilfsmethode">
<icon BUILTIN="button_ok"/>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1703018528370" ID="ID_1425132739" MODIFIED="1703018548063" TEXT="Umgang mit Grooming-Token f&#xfc;r Planungs-Jobs">
<icon BUILTIN="flag-yellow"/>
<node CREATED="1703018555022" ID="ID_705670834" MODIFIED="1703018575527" TEXT="eigentlich galt das Prinzip vom Desigin: Meta-Jobs haben stets das Grooming-Token"/>
<node CREATED="1703018588843" ID="ID_459468662" MODIFIED="1703018615111" TEXT="aber irgendwie hat sich ein ensureDropGroomingToken() in den postChain() &#x201e;eingeschlichen&#x201c;"/>
<node COLOR="#338800" CREATED="1703018528370" ID="ID_1425132739" MODIFIED="1703027135735" TEXT="Umgang mit Grooming-Token f&#xfc;r Planungs-Jobs">
<icon BUILTIN="button_ok"/>
<node BACKGROUND_COLOR="#ccb59b" COLOR="#6e2a38" CREATED="1703018555022" ID="ID_705670834" MODIFIED="1703027412517" TEXT="eigentlich galt das Prinzip vom Desigin: Meta-Jobs haben stets das Grooming-Token">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
dieses Prinzip <i>k&#246;nnte man aufweichen </i>&#8212; ich halte das aber f&#252;r eine schlechte Idee, denn der Scheduler ist auch so schon schwer genug zu verstehen, auch die Activity-Language beginnt schon zu &#187;br&#246;seln&#171; ... gef&#228;hrliche Tendenz. Meine Einsch&#228;tzung ist, wir planen ehr so 10 Jobs pro Metajob-Lauf, d.h. das w&#228;ren dann Gr&#246;&#223;enordnung 500-1000&#181;s Zeit, in der jedwede andere Management-T&#228;tigkeit geblockt ist (renderjobs laufen trotzdem weiter). Sofern diese Annahme <i>nicht zutrifft, </i>m&#246;chte ich daf&#252;r konkrete Beweise sehen
</p>
</body>
</html>
</richcontent>
<font ITALIC="true" NAME="SansSerif" SIZE="14"/>
<icon BUILTIN="yes"/>
</node>
<node BACKGROUND_COLOR="#e0ceaa" COLOR="#690f14" CREATED="1703018588843" ID="ID_459468662" MODIFIED="1703027427886" TEXT="irgendwie hat sich ein ensureDropGroomingToken() in ScheduleSpec::post() &#x201e;eingeschlichen&#x201c;">
<icon BUILTIN="messagebox_warning"/>
<node CREATED="1703019915394" ID="ID_856788042" MODIFIED="1703019921709" TEXT="wie kam das zustande?"/>
<node CREATED="1703019991131" ID="ID_1967190922" MODIFIED="1703020100227" TEXT="5c63548 2023-11-08">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
<font color="#991c1c">commit 5c6354882de1be63724221022b374bd559a499b0</font>
</p>
<p>
Author: Ichthyostega &lt;prg@ichthyostega.de&gt;
</p>
<p>
Date:&#160;&#160;&#160;Wed Nov 8 20:58:32 2023 +0100
</p>
<p>
</p>
<p>
&#160;&#160;&#160;&#160;Scheduler: solve problem with transport from entrance-queue
</p>
<p>
&#160;&#160;&#160;
</p>
<p>
&#160;&#160;&#160;&#160;The test case &quot;scheduleRenderJob()&quot; -- while deliberately operated
</p>
<p>
&#160;&#160;&#160;&#160;quite artificially with a disabled WorkForce (so the test can check
</p>
<p>
&#160;&#160;&#160;&#160;the contents in the queue and then progress manually -- led to discovery
</p>
<p>
&#160;&#160;&#160;&#160;of an open gap in the logic: in the (rare) case that a new task is
</p>
<p>
&#160;&#160;&#160;&#160;added ''from the outside'' without acquiring the Grooming-Token, then
</p>
<p>
&#160;&#160;&#160;&#160;the new task could sit in the entrace queue, in worst case for 50ms,
</p>
<p>
&#160;&#160;&#160;&#160;until the next Scheduler-&#187;Tick&#171; routinely sweeps this queue. Under
</p>
<p>
&#160;&#160;&#160;&#160;normal conditions however, each dispatch of another activity will
</p>
<p>
&#160;&#160;&#160;&#160;also sweep the entrance queue, yet if there happens to be no other
</p>
<p>
&#160;&#160;&#160;&#160;task right now, a new task could be stuck.
</p>
<p>
&#160;&#160;&#160;
</p>
<p>
<font color="#8f02d4">&#160;&#160;&#160;&#160;Thinking through this problem also helped to amend some aspects </font>
</p>
<p>
<font color="#8f02d4">&#160;&#160;&#160;&#160;of Grooming-Token handling and clarified the role of the API-functions.</font>
</p>
</body>
</html>
</richcontent>
</node>
<node CREATED="1703020110765" ID="ID_1266580870" MODIFIED="1703020121816" TEXT="das war nur ein Nebenschauplatz damals"/>
<node CREATED="1703020122494" ID="ID_699106064" MODIFIED="1703020135942" TEXT="hab die Funktion ensureDropGroomingToken() extrahiert"/>
<node CREATED="1703020137245" ID="ID_219811235" MODIFIED="1703020163494">
<richcontent TYPE="NODE"><html>
<head>
</head>
<body>
<p>
aber sie dann auch <b>zus&#228;tzlich</b>&#160;in das <b><font face="Monospaced">post()</font></b>&#160; eingesetzt
</p>
</body>
</html>
</richcontent>
</node>
<node CREATED="1703020169182" ID="ID_893010007" MODIFIED="1703027441068" TEXT="Erinnerung kommt zur&#xfc;ck....">
<icon BUILTIN="broken-line"/>
<node CREATED="1703020180620" ID="ID_943892166" MODIFIED="1703020193750" TEXT="sollte &#xbb;jemand&#xab; von der Stra&#xdf;e kommen...."/>
<node CREATED="1703020194696" ID="ID_1523406474" MODIFIED="1703020203205" TEXT="...der keinen Zugriff auf das Grooming-Token hat"/>
<node CREATED="1703020204009" ID="ID_1512963681" MODIFIED="1703027463303">
<richcontent TYPE="NODE"><html>
<head>
</head>
<body>
<p>
...dann erlangt er es, und <b><font color="#c100b2">beh&#228;lt es</font></b>
</p>
</body>
</html>
</richcontent>
</node>
<node CREATED="1703020222562" ID="ID_1384686267" MODIFIED="1703020228890" TEXT="...ohne davon etwas zu merken"/>
</node>
</node>
<node BACKGROUND_COLOR="#e0ceaa" COLOR="#690f14" CREATED="1703018615989" ID="ID_1140898245" MODIFIED="1703018734442">
<richcontent TYPE="NODE"><html>
<head/>
@ -105238,6 +105356,148 @@ Date:&#160;&#160;&#160;Thu Apr 20 18:53:17 2023 +0200<br/>
<arrowlink COLOR="#e92d59" DESTINATION="ID_1506062032" ENDARROW="Default" ENDINCLINATION="-129;68;" ID="Arrow_ID_379365083" STARTARROW="None" STARTINCLINATION="-245;34;"/>
<icon BUILTIN="clanbomber"/>
</node>
<node COLOR="#435e98" CREATED="1703021202948" ID="ID_1003848049" MODIFIED="1703027129567" TEXT="Schlu&#xdf;folgerung: dedizierte Behandlungs-Logik hier notwendig">
<icon BUILTIN="yes"/>
<node CREATED="1703021229630" ID="ID_1209181073" MODIFIED="1703021664833" TEXT="F&#xe4;lle">
<icon BUILTIN="info"/>
<node CREATED="1703021235667" ID="ID_904143668" MODIFIED="1703021258344" TEXT="wenn Grooming-Token gehalten &#x27f9; nichts tun (weiter halten)"/>
<node CREATED="1703021259900" ID="ID_634569872" MODIFIED="1703021272553" TEXT="wenn kein Grooming-Token &#x27f9;">
<node CREATED="1703021274594" ID="ID_360050342" MODIFIED="1703021293243" TEXT="spinnen bis man es erlangt hat"/>
<node CREATED="1703021295444" ID="ID_1232790266" MODIFIED="1703021301482" TEXT="beim Verlassen wieder droppen"/>
</node>
</node>
<node COLOR="#435e98" CREATED="1703021338367" ID="ID_111414885" MODIFIED="1703026288571" TEXT="Begr&#xfc;ndung">
<icon BUILTIN="yes"/>
<node CREATED="1703021345502" ID="ID_1982923602" MODIFIED="1703021353707" TEXT="es gibt keine andere M&#xf6;glichkeit">
<node CREATED="1703021419574" ID="ID_671144781" MODIFIED="1703021434072" TEXT="sofern der Zugang von Au&#xdf;en &#xfc;berhaupt m&#xf6;glich sein soll"/>
<node CREATED="1703021637097" ID="ID_1920714295" MODIFIED="1703021653326" TEXT="da&#xdf; meta-Jobs das Grooming-Token schon haben, bleibt unbeommen">
<icon BUILTIN="idea"/>
</node>
</node>
<node CREATED="1703021354295" ID="ID_1760061740" MODIFIED="1703021458990" TEXT="dies ist genau die richtige Stelle">
<node CREATED="1703021460369" ID="ID_490691724" MODIFIED="1703021461556" TEXT="Eingang von Au&#xdf;en"/>
<node CREATED="1703021462240" ID="ID_918170213" MODIFIED="1703021510184">
<richcontent TYPE="NODE"><html>
<head>
</head>
<body>
<p>
und: die Allokation findet <b>hier</b>&#160;statt
</p>
</body>
</html>
</richcontent>
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
&#160;&#160;&#160;&#160;term_ = move(
</p>
<p>
&#160;&#160;&#160;&#160;&#160;&#160;theScheduler_-&gt;activityLang_
</p>
<p>
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;.buildCalculationJob (job_, start_,death_));
</p>
</body>
</html>
</richcontent>
</node>
<node CREATED="1703021561211" ID="ID_271715467" MODIFIED="1703021577825" TEXT="&#x27f9; mu&#xdf; auch in continueMetaJob()"/>
</node>
<node CREATED="1703021371684" ID="ID_1674426596" MODIFIED="1703021394862" TEXT="ben&#xf6;tigt wird eine Behandlungs-Klammer (wenn vorher &#x27f9; dann auch nachher)"/>
<node CREATED="1703021734895" ID="ID_259808319" MODIFIED="1703021821939" TEXT="das mit dem Spinning ist s..schade, aber stellt die Ausnahme dar">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
...eben wenn jemand <i>von au&#223;en kommt...</i>&#160;das ist i.d.R. nur beim seedCalcStream() der Fall
</p>
</body>
</html>
</richcontent>
</node>
</node>
<node COLOR="#338800" CREATED="1703021667261" ID="ID_155074899" MODIFIED="1703026286837" TEXT="Implementierung">
<icon BUILTIN="button_ok"/>
<node COLOR="#338800" CREATED="1703021671644" ID="ID_1072770880" MODIFIED="1703026285675" TEXT="das l&#xe4;uft auf einen Guard hinaus">
<icon BUILTIN="button_ok"/>
</node>
<node COLOR="#338800" CREATED="1703021686181" ID="ID_1736317368" MODIFIED="1703026285022" TEXT="der braucht allerdings Verdrahtung auf Layer-2">
<icon BUILTIN="button_ok"/>
</node>
<node COLOR="#338800" CREATED="1703022329777" ID="ID_1935173830" MODIFIED="1703026283263" TEXT="Warten sollte einen kurzen sleep nutzen">
<icon BUILTIN="button_ok"/>
<node CREATED="1703022402580" ID="ID_750310552" MODIFIED="1703022428611" TEXT="aller Wahrscheinlichkeit wird man ohnehin 100&#xb5;s warten m&#xfc;ssen"/>
<node CREATED="1703022558669" ID="ID_475078987" MODIFIED="1703022598315" TEXT="mein Bauchgef&#xfc;hl sagt: 70&#xb5;s w&#xe4;ren angemessen">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
zum einen ist das nicht genau das feste Raster, und au&#223;erdem braucht der OS-Scheduler sowiso immer etwas l&#228;nger
</p>
</body>
</html>
</richcontent>
</node>
<node CREATED="1703022459903" ID="ID_808951013" MODIFIED="1703022483394" TEXT="Annahme: wer &#x201e;von Au&#xdf;en&#x201c; kommt, hat die Zeit">
<icon BUILTIN="messagebox_warning"/>
</node>
</node>
<node COLOR="#338800" CREATED="1703022609913" ID="ID_1015760107" MODIFIED="1703026278188" TEXT="SchedulerCommutator_test::verify_GroomingGuard">
<icon BUILTIN="button_ok"/>
<node COLOR="#435e98" CREATED="1703022836016" ID="ID_705777663" MODIFIED="1703026275222" TEXT="nur die Basis-Logik dokumentieren">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
sage bewu&#223;t &#187;dokumentieren&#171;, denn diese Logik hier ist derart einfach zu implementieren, da&#223; eigentlich kein Test notwendig w&#228;re
</p>
</body>
</html>
</richcontent>
<icon BUILTIN="yes"/>
</node>
<node COLOR="#435e98" CREATED="1703022843647" ID="ID_269682463" MODIFIED="1703026275222" TEXT="verzichte auf das Abdecken der Warteschleife">
<richcontent TYPE="NOTE"><html>
<head>
</head>
<body>
<p>
das ist n&#228;mlich unverh&#228;ltnism&#228;&#223;ig schwer im Test zu verifizieren, und steht in keinem Verh&#228;ltnis zur Aufgabe
</p>
</body>
</html>
</richcontent>
<icon BUILTIN="yes"/>
</node>
</node>
</node>
</node>
<node COLOR="#338800" CREATED="1703026296184" ID="ID_509070738" MODIFIED="1703027142362" TEXT="einsetzen in">
<icon BUILTIN="button_ok"/>
<node COLOR="#435e98" CREATED="1703026314470" ID="ID_1963009764" MODIFIED="1703027146698" TEXT="ScheduleSpec::post()"/>
<node COLOR="#435e98" CREATED="1703026847718" ID="ID_1408557714" MODIFIED="1703027146697" TEXT="Scheduler::seedCalcStream()">
<node CREATED="1703026900315" HGAP="38" ID="ID_233293317" MODIFIED="1703026945473" TEXT="hier auch wegen ActivityLang::announceLoad()" VSHIFT="2"/>
</node>
<node COLOR="#435e98" CREATED="1703026317184" ID="ID_1335544086" MODIFIED="1703027146697" TEXT="Scheduler::continueMetaJob()"/>
<node COLOR="#435e98" CREATED="1703026856056" ID="ID_606063530" MODIFIED="1703027146697" TEXT="Scheduler::handleDutyCycle()">
<node CREATED="1703026900315" HGAP="37" ID="ID_350652626" MODIFIED="1703026967055" TEXT="hier rein defensiv" VSHIFT="2"/>
</node>
<node COLOR="#338800" CREATED="1703027150432" ID="ID_115696934" MODIFIED="1703027163115" TEXT="das sind alle Verwendungen der ActivityLang">
<icon BUILTIN="button_ok"/>
</node>
</node>
</node>
</node>
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1702944889122" ID="ID_816186443" MODIFIED="1702944903543" TEXT="Berechnung der systematisch-erwartbaren Ausf&#xfc;hrungszeit">