- the starting point is the idea to build a dedicated ''turnout system''
- `StateAdapter`, `BuffTable` ⟶ `FeedManifold` and _Invocation_ will be fused
- actually, the `TurnoutSystem` will be ''pulled'' and orchestrate the invocation
- the structure is assumed to be recursive
The essence of the Node-Invocation, as developed 2009 / 2011 remains intact,
yet it will be organised along a clearer structure
The behaviour seems consistent and the schedule breaks at the expected point.
At first sight, concurrency seems slightly to low; detailed investigation
however shows that this is due to the structure of the load graph,
and in fact the run time comes close to optimal values.
the `BreakingPoint` tool conducts a binary search to find the ''stress factor''
where a given schedule breaks. There are some known deviations related to the
measurement setup, which unfortunately impact the interpretation of the
''stress factor'' scale. Earlier, an attempt was made, to watch those factors
empirically and work a ''form factor'' into the ''effective stress factor''
used to guide this measurement method.
Closer investigation with extended and elastic load patters now revealed
a strong tendency of the Scheduler to scale down the work resources when not
fully loaded. This may be mistaken by the above mentioned adjustments as a sign
of a structural limiation of the possible concurrency.
Thus, as a mitigation, those adjustments are now only performed at the
beginning of the measurement series, and also only when the stress factor
is high (implying that the scheduler is actually overloaded and thus has
no incentive for scaling down).
These observations indicate that the »Breaking Point« search must be taken
with a grain of salt: Especially when the test load does ''not'' contain
a high degree of inter dependencies, it will be ''stretched elastically''
rather than outright broken. And under such circumstances, this measurement
actually gauges the Scheduler's ability to comply to an established
load and computation goal.
- use parameters known to produce a clean linear model
- assert on properties of this linear model
Add extended documentation into the !TiddlyWiki,
with a textual account of the various findings,
also including some of the images and diagrams,
rendered as SVG
Investigate the behaviour over a wider range of job loads,
job count and worker pool sizes. Seemingly the processing
can not fully utilise the available worker pool capacity.
By inspection of trace-dumps, one impeding mechanism could
be identified: the »stickiness« of the contention mitigation.
Whenever a worker encounters repeated contention, it steps up
and adds more and more wait cycles to remove pressure from the
schedule coordination. As such this is fine and prevents further
degradation of performance by repeated atomic synchronisation.
However, this throttling was kept up needlessly after further
successful work-pulls. Since job times of several milliseconds
can be expected on average in media processing, such a long
retention would spread a performance degradation over a duration
of several frames. Thus, the scheme for step-down was changed
to decrease the throttling by a power series rather than just
documenting the level.
- repeated invocations of the same test setup for statistics
- the usual nasty 64-node graph with massive fork out
- limit concurrency to 4 cores
- tabulate data to look for clues regarding a trigger criteria
Hypothesis: The Scheduler slips off schedule when all of the
following three criteria are met:
- more than 55% glitches with Δ > 2ms
- σ > 2ms
- ∅Δ > 4ms
...the idea is to use the sum of node weights per level
to create a schedule, which more closely reflects the distribution
of actual computation time. Hopefully such a schedule can then be
squeezed or stretched by a time factor to find out a ''breaking point'',
at which the Scheduler is no longer able to keep up.
The last round of refactorings yielded significant improvements
- parallelisation now works as expected
- processing progresses closer to the schedule
- run time was reduced
The processing load for this test is tuned in a way to overload the
scheduler massively at the end -- the result must be correct non the less.
There was one notable glitch with an assertion failure from the memory manager.
Hopefully I can reproduce this by pressing and overloading the Scheduler more...
* added benchmark over synchronous execution as point of reference
* verified running times and execution pattern
* Scheduler **behaves as expected** for this example
Some test-runs performed excitingly smooth,
but in one case the processing was was drastically delayed,
due to heavy contention. The relevance of this incident is not clear yet,
since this test run uses a rather atypical load with very short actual work jobs.
Anyway, the dump-logs are documented with this commit.
As follow-up to the rework of thread-handling, likewise also
the implementation base for locking was switched over from direct
usage of POSIX primitives to the portable wrappers available in
the C++ standard library. All usages have been reviewed and
modernised to prefer λ-functions where possible.
With this series of changes, the old threadpool implementation
and a lot of further low-level support facilities are not used
any more and can be dismantled. Due to the integration efforts
spurred by the »Playback Vertical Slice«, several questions of
architecture could be decided over the last months. The design
of the Scheduler and Engine turned out different than previously
anticipated; notably the Scheduler now covers a wider array of
functionality, including some asynchronous messaging. This has
ramifications for the organisation of work tasks and threads,
and leads to a more deterministic memory management. Resource
management will be done on a higher level, partially superseding
some of the concepts from the early phase of the Lumiera project.
The EventLog seems to provide all the building blocks, but we need
some higher level special matchers (and maybe we also want to hide
some of the basic EventLog matchers). A soulution might be to wrap
the EventMatcher and delegate all follow-up builder calls.
This seems adequate, since the EventLog-Matcher is basically used as black box,
building up more elaborate matchers from the provided basic matchers...
Spent some time again to understand how EventLog matching works.
My feelings towards this piece of code are always the same: it is
somewhat too "tricky", but I am not aware of any other technique
to get this degree of elaborate chained matching on structured records,
short of building a dedicated matching engine from scratch.
The other alternative would be to use a flat textual log (instead of
the structured log records from EventLog), but then we'd have to
generate quite intricate regular expressions from the builder,
and I'm really doubtful it would be easier and clearer....
- introduce a new entity: RenderDrive
- it supersedes the CalcPlanCalculation, but is managed by CalcStream
- moreover, the RenderDrive will house a IterTreeExplorer-Pipeline
- define the concerns and relationships more clearly (see Drawing)
- prerequisite to disentangle the Job-planning "mechanics"
- decision: the Monad-style iteration framework will be abandoned
- the job-planning will be recast in terms of the iter-tree-explorer
- job-planning and frame dispatch will be disentangled
- the Scheduler will deliberately offer a high-level interface
- on this high-level, Scheduler will support dependency management
- the low-level implementation of the Scheduler will be based on Activity verbs
The drawing code extracts style information from some "virtual"
widgets, which serve as logical placeholder for the actual nested
structure of tracks.
For sake of demonstration, I used rather obvious colours and
also all kinds of margin and padding; a screenshot was added
with annotations to indicate where some specific style settings
are utilised from the drawing code
- pick up all relevant values from CSS
- also control the width of the StaveBracket
- observe the given overall height
Moreover, complete documentation drawing in Inkscape
and add a page to the TiddlyWiki, describing the principles
underlying this design and construction.
.timeline__head : The complete header container on the left side
.timeline__navi : navigation control at top
.timeline__pbay : container holding the patchbay on the left side
.fork__head : each individual TrackHeadWidget (possibly nested)
.fork__control : container for the control components for each track scope
.fork__bracket : the StaveBracket drawing to indicate the nesting structure
Identify the elements of the construction geometry in the "Sketch"
object in the FreeCAD document and paste the corresponding coordinate
values into the SVG drawing prepared for documentation.
The arc segment parameters seemingly are given in radians;
and while FreeCAD uses the common mathematical right-handed orientation,
the orientation in SVG is applied clockwise rather.
...since the construction is determined now (and was worked out in FreeCAD),
the SVG will serve to document the construction; thus the drawing
primitives are rearranged to use the unscaled reference coordinates
to be extracted from the FreeCAD document; all scaling and placement
in the SVG document will be applied through common groups.
My idea was to use the brackets from musical notation as inspiration;
if you know some principles of typography, it is rather straight-forward
to come up with a pleasing design of such a bracket, using a
cascade of golden ratio relationships.
BUT ... all of this is geometry, and translating that into a symbolic
or numerical calculation is excessively complicated. Thus I looked
for ways to use some geometry or CAD software to build such a construction.
The geometry software I tired was woefully inadequate for this task.
Using the Constraint system in FreeCAD, building the construction went
smooth and straight forward, but then I was unable to export that drawing
in a way indicating the construction clearly.
So in the end, I'll have to hand-pick the resulting numerical coordinates
from the FreeCAD XML document and integrate them directly into Cairo
drawing code...
After sleeping some nights over it, rework the wording to make
my reasoning more clear and remove any possibly insulting undertone.
I have seen what I describe here, happening over and over again --
and several times I myself was the one cooking up "simplifications",
which caused lots of pain further down the road.
During the last years, I became more and more doubtful and regretted
that decision. In hindsight, the fundamental conflict was present
already in the original discussion.
My own experience showed me again and again: skipping the hard work
of specification for sake of some kind of fluid prototyping rarely
leads to anything solid. If "time to market" counts, this can be
a viable strategy though...
this is just an "interpretation" of the current architecture diagram,
created for inclusion into the developer report, indicating those components
to be augmented and integrated to get a simple render/playback to work
We are using buttons now, but the standard theme introduces a lot of padding arount button's contents.
Thus we need to consider ways to address the compound of widgets forming an ElementBox; moreover,
this is the classical situation where the BEM notation helps to clarify the intention....
The problem leading to custom styling here is the padding within buttons;
the default stylesheet seemingly adds a min-width and min-height setting,
and some padding within the Button; based on systematic CSS class names,
it is possible to remove these settings specifically for buttons
within the IDLabel in general (no need to treat only the case of an EventBoxLabel
-- IDLabel could become a custom widget on its own
As we continue with building the backbone of the UI,
and abundance of detail information regaring Layout and styling
will be encountered -- it is tantamount to have a place to
write those findings down....
Further extended GTK code survey to clarify the role of the minimum_size,
it is indeed ignored by most standard containers, but it is actually
used by Gtk::Layout as starting point for the query sequence. Thus
it does not make sense to treat minimum and natural size differently;
both queries should be responded by returning our size constraint.
Unless we define additional borders and margins in the CSS, we can be sure
that GTK will base the size allocation on the exact values returned
from the get_required_* functions.
These functions will be invoked only from within the Event Loop
and after the ctor is finished, but before the first "draw".
They will be re-invoked on each "size change" event and on each
focus change (since a focus change may change the style and thus
the actual extension).
On my visit to Benny in the Black forest,
we decided to concentrate on a "Playback Vertical Slice"
and to announce that in the development report, using an
architecture diagram...
Complete the investigation and turn the solution into a generic
mix-in-template, which can be used in flexible ways to support
this qualifier notation.
Moreover, recapitulate requirements for the ElementBoxWidget
- move construct into the buffer
- directly invoke the payload constructor through PlantingHandle
- reconsider type signature and size constraint
- extend the unit test
- document a corner case of c++ "perfect forwarding",
which caused me some grief here
In 2017, I did a first design draft, followed by a design critique,
which partially obsoleted some ideas regarding command binding.
Mostly, the reason to abandon parts of that initial design was
due to the fact, that to many actual construction details of the
UI framework were not worked out at that time.
Thus I rather focussed on (re)-building a backbone for the timeline display,
in order to support that kind of flexibility aspired within the session model.
Now, when re-visiting the topic of an UI gesture (using simple dragging
of a clip in the timeline as an example for a first draft), I picked up
some of those planned structures, but tend to bind them together in
a slightly different way -- more akin to a state machine and less
in the way of an LR-parser.
This chagneset updates the relevant part within the TiddlyWiki
and the corresponding UML drawing to better reflect my actual thinking.
...in an attempt to clarify why numerous cross links are not generated.
In the end, this attempt was not very successful, yet I could find some breadcrumbs...
- file comments generally seem to have a problem with auto link generation;
only fully qualified names seem to work reliably
- cross links to entities within a namespace do not work,
if the corresponding namespace is not documented in Doxygen
- documentation for entities within anonymous namespaces
must be explicitly enabled. Of course this makes only sense
for detailed documentation (but we do generate detailed
documentation here, including implementation notes)
- and the notorious problem: each file needs a valid @file comment
- the hierarchy of Markdown headings must be consistent within each
documentation section. This entails also to individual documented
entities. Basically, there must be a level-one heading (prefix "#"),
otherwise all headings will just disappear...
- sometimes the doc/devel/doxygen-warnings.txt gives further clues
No new information added, rather removed lots of technical details,
which do not belong into design documentation. And try to present
the existing information more comprehensively
..at that time, that page was hastily written do document an
somewhat controversial implementation draft, which later on evolved
into the Application-main object Lumiera relied on since then.
Recently, a dedicated page dealing with subsystems and lifecycle
has been added to the Architecture section; so this page here
should be rewritten to focus on the "Application realm" as such.
The population message is just made up, in order to create more interesting structures
in the UI and so to further the development of the timeline display.
For the actual structure I choose to mirror my example drawing in draw/UI-TimelineLayout-1.png
which is also used in the TiddlyWiki, on the #GuiTimelineView tiddler
https://lumiera.org/wiki/renderengine.html#GuiTimelineView
Mostly, std::regexp can be used as a drop-in replacement.
Note: unfortunately ECMA regexps do not support lookbehind assertions.
This lookbehind is necesary here because we want to allow parsing values
from strings with additional content, which means we need explicitly to
exclude mismatches due to invalid syntax.
We can work around that issue like "either line start, or *not* one of these characters.
Alternatively we could consider to make the match more rigid,
i.e we would require the string to conain *only* the timecode spec to be parsed.
