DOC: fill in explanations for all subsystems (closes #1145)
This commit is contained in:
parent
4294960940
commit
b002a5e0b3
2 changed files with 112 additions and 50 deletions
|
|
@ -5,37 +5,36 @@ Layers -- Subsystems -- Lifecycle
|
|||
:Date: 2018
|
||||
:Toc:
|
||||
|
||||
WARNING: under construction -- [red]#some parts to be filled in#
|
||||
|
||||
|
||||
Terminology
|
||||
-----------
|
||||
|
||||
Layer::
|
||||
A conceptual realm within the application to group related concerns and define an ordering.
|
||||
Layers are above/below each other and may depend _solely_ on lower layers. The Application
|
||||
may be operated in a partial layer configuration with only some lower layers. Each layer
|
||||
deals with distinct topics and has its own style. In Lumiera, we distinguish three layers
|
||||
A conceptual realm within the application in order to group related topics and to build
|
||||
high-level structures in terms of low-level structures. Layers are located above/below
|
||||
each other and may depend _solely_ on lower layers. The Application may be operated in
|
||||
a partial layer configuration with only some lower layers present. Each layer deals
|
||||
with distinct topics and has its own style. In Lumiera, we distinguish three layers
|
||||
+
|
||||
* Stage Layer -> Interaction
|
||||
* Steam Layer -> Processing
|
||||
* Vault Layer -> Data manipulation
|
||||
|
||||
Subsystem::
|
||||
A runtime entity which acts as anchor point and framework to maintain a well defined lifecycle.
|
||||
While layers are conceptual realms, subsystems can actually be started and stopped and their
|
||||
A runtime entity which serves as anchor point and framework to maintain a well defined lifecycle.
|
||||
While layers are conceptual realms, subsystems can actually be started and stopped, and their
|
||||
dependencies are represented as data structure. A subsystem typically starts one or several
|
||||
primary components, which might spawn a dedicated thread and instantiate further components
|
||||
and services.
|
||||
|
||||
Service::
|
||||
A component within some subsystem is termed _Service_
|
||||
A component within some subsystem is called a _Service_
|
||||
+
|
||||
--
|
||||
* when it exposes an interfaces with an associated contract
|
||||
* provided that it exposes an interface with an associated contract
|
||||
(informal rules about usage pattern and expectations)
|
||||
* when it accepts invocations from arbitrary other components
|
||||
without specific wiring or hard coded knowledge
|
||||
* and given that it accepts invocations from arbitrary other components
|
||||
without mutual interdependency or hard coded knowledge about that other part.
|
||||
--
|
||||
+
|
||||
The service lifecycle is tied to the lifecycle of the related subsystem; whenever the subsystem is ``up and running'',
|
||||
|
|
@ -44,14 +43,15 @@ of elaborate _service discovery_ -- rather, services are accessed *by name*, whe
|
|||
of the service interface.
|
||||
|
||||
Dependency::
|
||||
A relation at implementation level and thus a local property of an individual component. A dependency
|
||||
is something we need in order to complete the task at hand, yet a dependency lies beyond that task and
|
||||
relates to concerns outside the scope and theme of this actual task. Which means, a dependency is not
|
||||
introduced by the task or part of the task, rather the task is the reason why some entity dealing with
|
||||
it needs to _pull_ the dependency, in order to be able to handle the task. So essentially, dependencies
|
||||
are accessed on-demand. Dependencies might be other components or services, and typically the user
|
||||
(consumer) of a dependency just relies on the corresponding interface and remains agnostic with respect
|
||||
to the dependency's actual implementation or lifecycle details.
|
||||
A usage relation at implementation level and thus a local prerequisite of an individual component. A
|
||||
dependency is something we need in order to complete the task at hand, yet a dependency lies beyond that
|
||||
task and is satisfied by means outside the scope and theme of this actual task. Consequently, a dependency
|
||||
is not introduced or provided by the local task or part of the task, rather the task at hand is the reason
|
||||
why some other entity dealing with it needs to _request_ or _pull_ that dependency in to accomplish the
|
||||
task at hand. So essentially, dependencies are accessed on-demand. Dependencies might be satisfied by
|
||||
other components or services, and typically the user (consumer) of a dependency just relies on the
|
||||
corresponding interface and remains agnostic with respect to the dependency's actual implementation,
|
||||
data or lifecycle details.
|
||||
|
||||
Subsystems
|
||||
----------
|
||||
|
|
@ -67,7 +67,7 @@ it depends and relies on the scheduling service of the engine. In the end, it re
|
|||
architecture to keep those dependency chains ordered in a way to form a one-way route: when we start
|
||||
the engine, it must not instantiate a component which _requires the player_ in order to be operative.
|
||||
Yet we can not start the player without having started the engine beforehand; if we do, its services
|
||||
will throw exceptions due to missing dependencies on first use.
|
||||
will throw exceptions on first use, due to missing dependencies.
|
||||
|
||||
However, subsystems as such are not dynamically configured. This was a clear cut design decision (and the
|
||||
result of a heated debate within the Lumiera team at that time). We do _not expect_ to load just some plug-in
|
||||
|
|
@ -78,49 +78,111 @@ parts of the application, each with its own theme, style, relations and continge
|
|||
|
||||
Engine
|
||||
~~~~~~
|
||||
_tbw_
|
||||
The Engine performs small pieces of work known as _render jobs,_ oriented towards a deadline,
|
||||
without much knowledge about the purpose of those jobs, or their further interconnections.
|
||||
And thus the purpose of the *Engine Subsystem* is to provide a thread pool and activate
|
||||
the scheduling mechanism. Consequently, this subsystem belongs into the »Vault Layer«
|
||||
|
||||
_[yellow-background]#this part of the system is barely drafted as of 2020#_
|
||||
|
||||
Player
|
||||
~~~~~~
|
||||
_tbw_
|
||||
The *PlayOut Subsystem* is located above the Engine and belongs into the »Steam Layer« -- and contrary
|
||||
to the Engie (which handles individual jobs), the player creates and organises _calculation streams._
|
||||
|
||||
_[yellow-background]#as of 2020, the actual components to form the player need to be worked out#_ +
|
||||
_^however, a fair amount of the services for dispatching streams into jobs has been prototyped^_
|
||||
|
||||
Session
|
||||
~~~~~~~
|
||||
_tbw_
|
||||
The user performs editing activities within the »Session« -- which is a data structure with associated
|
||||
methods for manipulation. There is a `Session` object and a `SessionManager` to load and save session
|
||||
data and conduct the _session lifecycle._ However, all of this needs to be distinguished from the
|
||||
*Session Subsystem* -- which in essence is a dispatcher thread to receive, enqueue and finally
|
||||
trigger the _session commands,_ as sent from the GUI or the script runner. These activities are
|
||||
conducted and controlled by the `SteamDispatcher`, which also cares for triggering the _Builder,_
|
||||
whenever new commands have been dispatched. Moreover, when instantiating the `DispatcherLoop`,
|
||||
also the `SessionCommand` façade is opened, which is the primary »Steam Layer« interface.footnote:[
|
||||
Note the relation between Session-the-datastructure and Session-the-subsystem is rather indirect:
|
||||
the _dispatching_ of commands is blocked, unless there is also a session-datastructure loaded
|
||||
and fully configured. However, a running dispatcher loop is not a prerequisite for opening
|
||||
a session -- just without a running dispatcher, commands will queue up and nothing else will happen.]
|
||||
|
||||
_[green]#as of 2020, this subsystem is operative and commands can be dispatched#_ +
|
||||
_^...while the session data structure as such is mostly still a skeleton...^_
|
||||
|
||||
User Interface
|
||||
~~~~~~~~~~~~~~
|
||||
_tbw_
|
||||
The Lumiera GUI is loaded as self-contained plug-in, which is the task of the *GUI Subsystem*.
|
||||
As can be expected, this is a rather convoluted process, while the actual name of the UI plug-in module
|
||||
to load is configured in the 'setup.ini', which has been evaluated earlier, in the application init phase.
|
||||
However, as it stands, Lumiera is built with a GTK-3 interface, and within the corresponding plug-in module
|
||||
`gtk_gui.lum`, the class `GtkLumiera` serves as top-level guard to carry on all further activities,
|
||||
when triggered from within the subsystem lifecycle to run in a dedicated GUI thread. It will establish
|
||||
the _UI backbone_ by activating the _UI-Bus_ and building the _UI Manager_ controlling the UI global context.
|
||||
After these systems are established and connected, the GTK windows can be created and finally control is handed
|
||||
over to the GTK (GIO) event loop. Whenever this loop terminates, be it regularly, or by exception, application
|
||||
shutdown is initiated.
|
||||
|
||||
The GUI Subsystem is special, insofar it not only attaches to the session interface, but also opens a
|
||||
_Layer separation interface_ oriented downwards, to be used by the lower layers. This interface -- known
|
||||
as GUI Notification façade -- serves to populate the UI with actual content, to mark and animate the
|
||||
tangible elements visible to and manipulated by the user in turn. It is outfitted with a cross-thread
|
||||
dispatcher mechanism, to forward any invocation as message onto the UI-Bus. This setup allows the
|
||||
lower layers to address the tangible parts in the UI based on their ID, which previously was given
|
||||
alongside with the content when populating the structures.
|
||||
|
||||
_[green]#as of 2020, this backbone is established and connected in both directions#_ +
|
||||
_^...while the large part of the actual widgets still remains to be built...^_
|
||||
|
||||
Script Runner
|
||||
~~~~~~~~~~~~~
|
||||
_tbw_
|
||||
One of the most fundamental design decisions for Lumiera is that everything can be done without GUI.
|
||||
Conceptually, this would allow to instantiate a script execution environment with appropriate bindings,
|
||||
either to conduct operations on an existing session, or to build and render a session from scratch.
|
||||
Alternatively, also a CLI-style shell-like interface is conceivable.
|
||||
|
||||
_[maroon orange-background]#this is a concept without any detailed planning as of 2020#_
|
||||
|
||||
Net Node
|
||||
~~~~~~~~
|
||||
_tbw_
|
||||
In variation to the script runner concept, it is conceivable to send instructions to a Lumiera
|
||||
instance over the net. Expanding on that idea, it would be possible to define a protocol to
|
||||
distribute the session definitions to slave nodes and then to launch distributed render tasks.
|
||||
Since Lumiera is built as a self-contained bundle, it is well suited to run within a containerised
|
||||
environment. However, in the light of current trends towards container orchestration frameworks
|
||||
like Kubernetes, we should refrain from building to much process management functionality into
|
||||
the application itself.
|
||||
|
||||
_[aqua teal-background]#this is a mere idea, and certainly not a priority as of 2020#_
|
||||
|
||||
|
||||
....
|
||||
|
||||
....
|
||||
|
||||
|
||||
Lifecycle
|
||||
---------
|
||||
Dependencies and abstraction through interfaces are ways to deal with complexity getting out of hand.
|
||||
When done well, we can avoid adding _accidental complexity_ -- yet essential complexity as such can not
|
||||
be removed, but with the help of abstractions it can be raised to another level.footnote:[Irony tags here.
|
||||
When done well, we can avoid adding _accidental complexity_ -- but essential complexity as such can not
|
||||
be removed, yet with the help of abstractions it can be raised to another level.footnote:[Irony tags here.
|
||||
There is a lot of hostility towards abstractions, because it is quite natural to conflate the abstraction
|
||||
with the essential complexity it has to deal with. It seems compelling to kill the abstraction, in the
|
||||
hope to kill the complexities as well -- an attitude rather effective, in practice...].
|
||||
When components express their external needs in the form of dependency on an interface, the immediate tangling
|
||||
at the code level is resolved, however, someone needs to implement that interface, and this other entity needs
|
||||
to be _available_. It is now an architecture challenge to get those dependency chains ordered. A way to
|
||||
circumvent this problem is to rely on a _lifecycle_ with several _phases._
|
||||
This is the idea behind the subsystems and the subsystem runner.
|
||||
with the essential complexity it has to deal with. It seems compelling to kill the abstraction, in the hope
|
||||
to kill the complexities as well -- a tremendously effective attitude, as it turns out, especially in practice...]
|
||||
When components express their external needs by depending on an interface, the immediate tangling at the code level
|
||||
is resolved. However, someone needs to implement that interface, and this other entity needs to be _available_.
|
||||
The problem has been shifted, since it is now an architecture level challenge to get those dependency chains
|
||||
satisfied. A clever way to circumvent this problem rather then to deal with it explicitly, is to rely on a
|
||||
_lifecycle_ with several _phases._ This is the idea behind the subsystems and the subsystem runner.
|
||||
|
||||
. First we define an ordering between the subsystems. The most basic subsystem (the Engine) is started first.
|
||||
. Within a subsystem, components may be mutually dependent. However, we establish a new rule, dictating that
|
||||
during the _startup phase_ only local operations within a single component are allowed. The component need
|
||||
to be written in a way that it does not need the help of anything ``remote'' in order to get its inner
|
||||
workings up and ready. The component may rely on its members and on other services it created, _owns and
|
||||
manages._ And sometimes we do need to rely on a more low-level service in another subsystem or in the
|
||||
during the _startup phase_ only local operations within a single component are allowed. Each component must
|
||||
to be written in such a way, not to rely on the help of anything ``remote'' in order to get its inner workings
|
||||
up and ready. The component may rely on its members and on other services it _created itself,_ or which it
|
||||
_owns and manages._
|
||||
. However, sometimes we _do need to rely_ on a more low-level service in another subsystem or in the
|
||||
application core.footnote:[A typical example would be the reliance on threading, locking or application
|
||||
configuration.] -- which then creates a hard dependency on _architecture level_
|
||||
. Moreover, we ensure that all operational activity is generated by actual work tasks, and that such tasks
|
||||
|
|
@ -136,8 +198,8 @@ The problem with emergencies
|
|||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
This concept has a weak spot however: A catastrophic failure might cause any subsystem to break down immediately.
|
||||
The handler within the subsystem's primary component will hopefully detect the corresponding exception and signal
|
||||
emergency to the subsystem runner. However, the working services of that subsystem are already gone at that point.
|
||||
And even while other subsystems get the (emergency) shutdown trigger, some working parts may be already failing
|
||||
emergency to the subsystem runner. Yet the working services of that subsystem are already gone at that point.
|
||||
And even before other subsystems might get the (emergency) shutdown trigger, some working parts may be failing
|
||||
catastrophically due to their dependencies being dragged away suddenly.
|
||||
|
||||
Lumiera is not written for exceptional resilience or high availability. Our attitude towards such failures can
|
||||
|
|
@ -153,20 +215,20 @@ However, anything beyond the scope of `main()` is not meant to be used for regul
|
|||
initialisation, dependency management and decommissioning -- when actually necessary -- should be part of the
|
||||
application code proper.footnote:[this is established ``best practice'' for good reasons. The interplay of
|
||||
static lifespan, various translation units and even dynamically loaded libraries together with shared access
|
||||
becomes intricate and insidious quite easily. And since in theory any static function could use some static
|
||||
tends to becomes intricate and insidious easily. And since, in theory, any static function could use some static
|
||||
variable residing in another translation unit, it is always possible to construct a situation where objects
|
||||
are accessed after being destroyed. Typically such objects do not even look especially ``dead'', since the
|
||||
static storage remains in place and still holds possibly sane values. Static (global) variables, like raw
|
||||
pointers, allow to subvert the deterministic automatic memory management, which otherwise is one of the
|
||||
greatest strengths of C++. Whenever we find ourselves developing extended collaborative logic based on
|
||||
several statics, we should consider to transform this logic into regular objects, which are easier to
|
||||
test and better to reason about. If it really can not be avoided to use such units of logic from a
|
||||
static context, it should at least be packaged as a single object, plus we should ensure this logic
|
||||
can only be accessed through a regular (non static) object as front-end. Packaged this way, the
|
||||
most common and dangerous pitfalls with statics can be avoided.] And since Lumiera indeed allows
|
||||
test and to reason about. If it really can not be avoided to use such units of logic from a static
|
||||
context, it should at least be packaged as a single object, plus we should ensure this logic can
|
||||
only be accessed through a regular (non static) object as front-end. Packaged this way, the most
|
||||
common and dangerous pitfalls with statics can be avoided.] And since Lumiera indeed allows
|
||||
for link:{ldoc}/technical/library/Dependencies.html[lazily initialised dependencies], we
|
||||
establish the policy that *destructors must not rely on dependencies*. In fact, they should
|
||||
not do any tangible work at all, beyond releasing other resources.
|
||||
establish the policy that *destructors must not rely on dependencies*. In fact, they
|
||||
should not do any tangible work at all, beyond releasing other resources.
|
||||
|
||||
anchor:lifecycle[]
|
||||
|
||||
|
|
|
|||
|
|
@ -64,7 +64,7 @@ also able to _build_ that code (even partially) from within the IDE, since the i
|
|||
more cross linking information. However, this is not a strict requirement -- even while `F3` often fails, the
|
||||
``Open Type'' dialog is able to spot the definition in many cases non the less, and when this fails, you can
|
||||
still use ``brute-force'' file search. What turns out to be much more an impediment in practice is the fact
|
||||
that you'll have to jump through that C++ binding layer, and you need to pick up some basic knowledge how
|
||||
that you'll have to jump through that C\++ binding layer, and you need to pick up some basic knowledge how
|
||||
this layer works to wrap the underlying plain-C GTK entities; don't confuse the C++ _wrapper objects_
|
||||
with the _gobject_ (a concept from GLib) used by GTK.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue