diff --git a/doc/design/architecture/Subsystems.txt b/doc/design/architecture/Subsystems.txt index 3184ed146..67406f860 100644 --- a/doc/design/architecture/Subsystems.txt +++ b/doc/design/architecture/Subsystems.txt @@ -5,37 +5,36 @@ Layers -- Subsystems -- Lifecycle :Date: 2018 :Toc: -WARNING: under construction -- [red]#some parts to be filled in# - Terminology ----------- Layer:: - A conceptual realm within the application to group related concerns and define an ordering. - Layers are above/below each other and may depend _solely_ on lower layers. The Application - may be operated in a partial layer configuration with only some lower layers. Each layer - deals with distinct topics and has its own style. In Lumiera, we distinguish three layers + A conceptual realm within the application in order to group related topics and to build + high-level structures in terms of low-level structures. Layers are located above/below + each other and may depend _solely_ on lower layers. The Application may be operated in + a partial layer configuration with only some lower layers present. Each layer deals + with distinct topics and has its own style. In Lumiera, we distinguish three layers + * Stage Layer -> Interaction * Steam Layer -> Processing * Vault Layer -> Data manipulation Subsystem:: - A runtime entity which acts as anchor point and framework to maintain a well defined lifecycle. - While layers are conceptual realms, subsystems can actually be started and stopped and their + A runtime entity which serves as anchor point and framework to maintain a well defined lifecycle. + While layers are conceptual realms, subsystems can actually be started and stopped, and their dependencies are represented as data structure. A subsystem typically starts one or several primary components, which might spawn a dedicated thread and instantiate further components and services. Service:: - A component within some subsystem is termed _Service_ + A component within some subsystem is called a _Service_ + -- - * when it exposes an interfaces with an associated contract + * provided that it exposes an interface with an associated contract (informal rules about usage pattern and expectations) - * when it accepts invocations from arbitrary other components - without specific wiring or hard coded knowledge + * and given that it accepts invocations from arbitrary other components + without mutual interdependency or hard coded knowledge about that other part. -- + The service lifecycle is tied to the lifecycle of the related subsystem; whenever the subsystem is ``up and running'', @@ -44,14 +43,15 @@ of elaborate _service discovery_ -- rather, services are accessed *by name*, whe of the service interface. Dependency:: - A relation at implementation level and thus a local property of an individual component. A dependency - is something we need in order to complete the task at hand, yet a dependency lies beyond that task and - relates to concerns outside the scope and theme of this actual task. Which means, a dependency is not - introduced by the task or part of the task, rather the task is the reason why some entity dealing with - it needs to _pull_ the dependency, in order to be able to handle the task. So essentially, dependencies - are accessed on-demand. Dependencies might be other components or services, and typically the user - (consumer) of a dependency just relies on the corresponding interface and remains agnostic with respect - to the dependency's actual implementation or lifecycle details. + A usage relation at implementation level and thus a local prerequisite of an individual component. A + dependency is something we need in order to complete the task at hand, yet a dependency lies beyond that + task and is satisfied by means outside the scope and theme of this actual task. Consequently, a dependency + is not introduced or provided by the local task or part of the task, rather the task at hand is the reason + why some other entity dealing with it needs to _request_ or _pull_ that dependency in to accomplish the + task at hand. So essentially, dependencies are accessed on-demand. Dependencies might be satisfied by + other components or services, and typically the user (consumer) of a dependency just relies on the + corresponding interface and remains agnostic with respect to the dependency's actual implementation, + data or lifecycle details. Subsystems ---------- @@ -67,7 +67,7 @@ it depends and relies on the scheduling service of the engine. In the end, it re architecture to keep those dependency chains ordered in a way to form a one-way route: when we start the engine, it must not instantiate a component which _requires the player_ in order to be operative. Yet we can not start the player without having started the engine beforehand; if we do, its services -will throw exceptions due to missing dependencies on first use. +will throw exceptions on first use, due to missing dependencies. However, subsystems as such are not dynamically configured. This was a clear cut design decision (and the result of a heated debate within the Lumiera team at that time). We do _not expect_ to load just some plug-in @@ -78,49 +78,111 @@ parts of the application, each with its own theme, style, relations and continge Engine ~~~~~~ -_tbw_ +The Engine performs small pieces of work known as _render jobs,_ oriented towards a deadline, +without much knowledge about the purpose of those jobs, or their further interconnections. +And thus the purpose of the *Engine Subsystem* is to provide a thread pool and activate +the scheduling mechanism. Consequently, this subsystem belongs into the »Vault Layer« + +_[yellow-background]#this part of the system is barely drafted as of 2020#_ Player ~~~~~~ -_tbw_ +The *PlayOut Subsystem* is located above the Engine and belongs into the »Steam Layer« -- and contrary +to the Engie (which handles individual jobs), the player creates and organises _calculation streams._ + +_[yellow-background]#as of 2020, the actual components to form the player need to be worked out#_ + +_^however, a fair amount of the services for dispatching streams into jobs has been prototyped^_ Session ~~~~~~~ -_tbw_ +The user performs editing activities within the »Session« -- which is a data structure with associated +methods for manipulation. There is a `Session` object and a `SessionManager` to load and save session +data and conduct the _session lifecycle._ However, all of this needs to be distinguished from the +*Session Subsystem* -- which in essence is a dispatcher thread to receive, enqueue and finally +trigger the _session commands,_ as sent from the GUI or the script runner. These activities are +conducted and controlled by the `SteamDispatcher`, which also cares for triggering the _Builder,_ +whenever new commands have been dispatched. Moreover, when instantiating the `DispatcherLoop`, +also the `SessionCommand` façade is opened, which is the primary »Steam Layer« interface.footnote:[ +Note the relation between Session-the-datastructure and Session-the-subsystem is rather indirect: +the _dispatching_ of commands is blocked, unless there is also a session-datastructure loaded +and fully configured. However, a running dispatcher loop is not a prerequisite for opening +a session -- just without a running dispatcher, commands will queue up and nothing else will happen.] + +_[green]#as of 2020, this subsystem is operative and commands can be dispatched#_ + +_^...while the session data structure as such is mostly still a skeleton...^_ User Interface ~~~~~~~~~~~~~~ -_tbw_ +The Lumiera GUI is loaded as self-contained plug-in, which is the task of the *GUI Subsystem*. +As can be expected, this is a rather convoluted process, while the actual name of the UI plug-in module +to load is configured in the 'setup.ini', which has been evaluated earlier, in the application init phase. +However, as it stands, Lumiera is built with a GTK-3 interface, and within the corresponding plug-in module +`gtk_gui.lum`, the class `GtkLumiera` serves as top-level guard to carry on all further activities, +when triggered from within the subsystem lifecycle to run in a dedicated GUI thread. It will establish +the _UI backbone_ by activating the _UI-Bus_ and building the _UI Manager_ controlling the UI global context. +After these systems are established and connected, the GTK windows can be created and finally control is handed +over to the GTK (GIO) event loop. Whenever this loop terminates, be it regularly, or by exception, application +shutdown is initiated. + +The GUI Subsystem is special, insofar it not only attaches to the session interface, but also opens a +_Layer separation interface_ oriented downwards, to be used by the lower layers. This interface -- known +as GUI Notification façade -- serves to populate the UI with actual content, to mark and animate the +tangible elements visible to and manipulated by the user in turn. It is outfitted with a cross-thread +dispatcher mechanism, to forward any invocation as message onto the UI-Bus. This setup allows the +lower layers to address the tangible parts in the UI based on their ID, which previously was given +alongside with the content when populating the structures. + +_[green]#as of 2020, this backbone is established and connected in both directions#_ + +_^...while the large part of the actual widgets still remains to be built...^_ Script Runner ~~~~~~~~~~~~~ -_tbw_ +One of the most fundamental design decisions for Lumiera is that everything can be done without GUI. +Conceptually, this would allow to instantiate a script execution environment with appropriate bindings, +either to conduct operations on an existing session, or to build and render a session from scratch. +Alternatively, also a CLI-style shell-like interface is conceivable. + +_[maroon orange-background]#this is a concept without any detailed planning as of 2020#_ Net Node ~~~~~~~~ -_tbw_ +In variation to the script runner concept, it is conceivable to send instructions to a Lumiera +instance over the net. Expanding on that idea, it would be possible to define a protocol to +distribute the session definitions to slave nodes and then to launch distributed render tasks. +Since Lumiera is built as a self-contained bundle, it is well suited to run within a containerised +environment. However, in the light of current trends towards container orchestration frameworks +like Kubernetes, we should refrain from building to much process management functionality into +the application itself. + +_[aqua teal-background]#this is a mere idea, and certainly not a priority as of 2020#_ + + +.... + +.... Lifecycle --------- Dependencies and abstraction through interfaces are ways to deal with complexity getting out of hand. -When done well, we can avoid adding _accidental complexity_ -- yet essential complexity as such can not -be removed, but with the help of abstractions it can be raised to another level.footnote:[Irony tags here. +When done well, we can avoid adding _accidental complexity_ -- but essential complexity as such can not +be removed, yet with the help of abstractions it can be raised to another level.footnote:[Irony tags here. There is a lot of hostility towards abstractions, because it is quite natural to conflate the abstraction -with the essential complexity it has to deal with. It seems compelling to kill the abstraction, in the -hope to kill the complexities as well -- an attitude rather effective, in practice...]. -When components express their external needs in the form of dependency on an interface, the immediate tangling -at the code level is resolved, however, someone needs to implement that interface, and this other entity needs -to be _available_. It is now an architecture challenge to get those dependency chains ordered. A way to -circumvent this problem is to rely on a _lifecycle_ with several _phases._ -This is the idea behind the subsystems and the subsystem runner. +with the essential complexity it has to deal with. It seems compelling to kill the abstraction, in the hope +to kill the complexities as well -- a tremendously effective attitude, as it turns out, especially in practice...] +When components express their external needs by depending on an interface, the immediate tangling at the code level +is resolved. However, someone needs to implement that interface, and this other entity needs to be _available_. +The problem has been shifted, since it is now an architecture level challenge to get those dependency chains +satisfied. A clever way to circumvent this problem rather then to deal with it explicitly, is to rely on a +_lifecycle_ with several _phases._ This is the idea behind the subsystems and the subsystem runner. . First we define an ordering between the subsystems. The most basic subsystem (the Engine) is started first. . Within a subsystem, components may be mutually dependent. However, we establish a new rule, dictating that - during the _startup phase_ only local operations within a single component are allowed. The component need - to be written in a way that it does not need the help of anything ``remote'' in order to get its inner - workings up and ready. The component may rely on its members and on other services it created, _owns and - manages._ And sometimes we do need to rely on a more low-level service in another subsystem or in the + during the _startup phase_ only local operations within a single component are allowed. Each component must + to be written in such a way, not to rely on the help of anything ``remote'' in order to get its inner workings + up and ready. The component may rely on its members and on other services it _created itself,_ or which it + _owns and manages._ +. However, sometimes we _do need to rely_ on a more low-level service in another subsystem or in the application core.footnote:[A typical example would be the reliance on threading, locking or application configuration.] -- which then creates a hard dependency on _architecture level_ . Moreover, we ensure that all operational activity is generated by actual work tasks, and that such tasks @@ -136,8 +198,8 @@ The problem with emergencies ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This concept has a weak spot however: A catastrophic failure might cause any subsystem to break down immediately. The handler within the subsystem's primary component will hopefully detect the corresponding exception and signal -emergency to the subsystem runner. However, the working services of that subsystem are already gone at that point. -And even while other subsystems get the (emergency) shutdown trigger, some working parts may be already failing +emergency to the subsystem runner. Yet the working services of that subsystem are already gone at that point. +And even before other subsystems might get the (emergency) shutdown trigger, some working parts may be failing catastrophically due to their dependencies being dragged away suddenly. Lumiera is not written for exceptional resilience or high availability. Our attitude towards such failures can @@ -153,20 +215,20 @@ However, anything beyond the scope of `main()` is not meant to be used for regul initialisation, dependency management and decommissioning -- when actually necessary -- should be part of the application code proper.footnote:[this is established ``best practice'' for good reasons. The interplay of static lifespan, various translation units and even dynamically loaded libraries together with shared access -becomes intricate and insidious quite easily. And since in theory any static function could use some static +tends to becomes intricate and insidious easily. And since, in theory, any static function could use some static variable residing in another translation unit, it is always possible to construct a situation where objects are accessed after being destroyed. Typically such objects do not even look especially ``dead'', since the static storage remains in place and still holds possibly sane values. Static (global) variables, like raw pointers, allow to subvert the deterministic automatic memory management, which otherwise is one of the greatest strengths of C++. Whenever we find ourselves developing extended collaborative logic based on several statics, we should consider to transform this logic into regular objects, which are easier to -test and better to reason about. If it really can not be avoided to use such units of logic from a -static context, it should at least be packaged as a single object, plus we should ensure this logic -can only be accessed through a regular (non static) object as front-end. Packaged this way, the -most common and dangerous pitfalls with statics can be avoided.] And since Lumiera indeed allows +test and to reason about. If it really can not be avoided to use such units of logic from a static +context, it should at least be packaged as a single object, plus we should ensure this logic can +only be accessed through a regular (non static) object as front-end. Packaged this way, the most +common and dangerous pitfalls with statics can be avoided.] And since Lumiera indeed allows for link:{ldoc}/technical/library/Dependencies.html[lazily initialised dependencies], we -establish the policy that *destructors must not rely on dependencies*. In fact, they should -not do any tangible work at all, beyond releasing other resources. +establish the policy that *destructors must not rely on dependencies*. In fact, they +should not do any tangible work at all, beyond releasing other resources. anchor:lifecycle[] diff --git a/doc/technical/howto/IdeSetup.txt b/doc/technical/howto/IdeSetup.txt index 901e6e920..05741b2a4 100644 --- a/doc/technical/howto/IdeSetup.txt +++ b/doc/technical/howto/IdeSetup.txt @@ -64,7 +64,7 @@ also able to _build_ that code (even partially) from within the IDE, since the i more cross linking information. However, this is not a strict requirement -- even while `F3` often fails, the ``Open Type'' dialog is able to spot the definition in many cases non the less, and when this fails, you can still use ``brute-force'' file search. What turns out to be much more an impediment in practice is the fact -that you'll have to jump through that C++ binding layer, and you need to pick up some basic knowledge how +that you'll have to jump through that C\++ binding layer, and you need to pick up some basic knowledge how this layer works to wrap the underlying plain-C GTK entities; don't confuse the C++ _wrapper objects_ with the _gobject_ (a concept from GLib) used by GTK.