From 2f13b9ee4a08ab3579a2ca8e92270da5aa299763 Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Fri, 20 Nov 2015 02:36:34 +0100 Subject: [PATCH 01/12] fix compilation problem with gcc-5 ..while we should note at this point that the whole techique of hijacking std::hash is superfluous now, since the standard libray does no longer define a static assertion which defeats SFINAE --- src/lib/hash-standard.hpp | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/hash-standard.hpp b/src/lib/hash-standard.hpp index af022da7d..bb307aed1 100644 --- a/src/lib/hash-standard.hpp +++ b/src/lib/hash-standard.hpp @@ -72,6 +72,7 @@ #include +#include #include From 1b2a0c151355dcdd605b5a9c3cd54190c97271e5 Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Fri, 20 Nov 2015 04:31:29 +0100 Subject: [PATCH 02/12] fix missing header include in test This omission was spotted when compiling with GCC-5 on Ubuntu/wily --- tests/library/test/test-helper-demangling-test.cpp | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/library/test/test-helper-demangling-test.cpp b/tests/library/test/test-helper-demangling-test.cpp index c158d1cee..6bee8c2a3 100644 --- a/tests/library/test/test-helper-demangling-test.cpp +++ b/tests/library/test/test-helper-demangling-test.cpp @@ -25,6 +25,7 @@ #include "lib/test/test-helper.hpp" #include +#include using std::string; using std::cout; From 0ce192975db954479c3f23114903ed593dcf02e2 Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Sun, 23 Feb 2020 03:06:31 +0100 Subject: [PATCH 03/12] DOC: rename the old SubsystemLifecycle page ..at that time, that page was hastily written do document an somewhat controversial implementation draft, which later on evolved into the Application-main object Lumiera relied on since then. Recently, a dedicated page dealing with subsystems and lifecycle has been added to the Architecture section; so this page here should be rewritten to focus on the "Application realm" as such. --- ...ubsystemLifecycle.txt => ApplicationStart.txt} | 15 ++++----------- doc/design/application/index.txt | 4 +++- 2 files changed, 7 insertions(+), 12 deletions(-) rename doc/design/application/{SubsystemLifecycle.txt => ApplicationStart.txt} (94%) diff --git a/doc/design/application/SubsystemLifecycle.txt b/doc/design/application/ApplicationStart.txt similarity index 94% rename from doc/design/application/SubsystemLifecycle.txt rename to doc/design/application/ApplicationStart.txt index bff850b03..6b48e57fa 100644 --- a/doc/design/application/SubsystemLifecycle.txt +++ b/doc/design/application/ApplicationStart.txt @@ -1,17 +1,10 @@ -Startup and Shutdown of Subsystems -================================== -:Date: Dec 2008 +The Application : Start-up and Subsystems +========================================= +:Date: 2009 :Author: Ichthyo -//MENU: label Subsystem start +//MENU: label Start-up -.copied from 'pipapo.org' -NOTE: This page was moved from _Cehteh_'s MoinMoin-wiki, - which at that time was the first home of Lumiera development + - The design and techniques outlined here are in use without major - changes as of 2015 (and can be expected to remain this way). The - documentation needs rewording in a more neutral and descriptive - way and probably an introductory paragraph might be helpful [yellow-background]#TODO# ...There is now sort-of an ``application realm'', which doesn't belong strictly to any one of the Layers. It serves the purpose of pulling up and tearing down the diff --git a/doc/design/application/index.txt b/doc/design/application/index.txt index 71b7cebe4..11b8b02d5 100644 --- a/doc/design/application/index.txt +++ b/doc/design/application/index.txt @@ -1,8 +1,10 @@ Design Documents: Application Framework ======================================= +// Menu : prepend child ApplicationStart + * link:Config.html[Configuration] * link:plugin_loader.html[Plugin Loader] - * link:SubsystemLifecycle.html[Application Subsystems and Lifecycle] + * link:ApplicationStart.html[Application Subsystems and Lifecycle] From 42949609404ee1d171a162b2d6bc57e6fbe11d71 Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Sun, 23 Feb 2020 05:28:43 +0100 Subject: [PATCH 04/12] DOC: complete rewrite of the Application-main documentation No new information added, rather removed lots of technical details, which do not belong into design documentation. And try to present the existing information more comprehensively --- doc/design/application/ApplicationStart.txt | 257 ++++++++++---------- doc/design/architecture/Subsystems.txt | 2 + 2 files changed, 125 insertions(+), 134 deletions(-) diff --git a/doc/design/application/ApplicationStart.txt b/doc/design/application/ApplicationStart.txt index 6b48e57fa..41206a4c5 100644 --- a/doc/design/application/ApplicationStart.txt +++ b/doc/design/application/ApplicationStart.txt @@ -5,155 +5,144 @@ The Application : Start-up and Subsystems //MENU: label Start-up +.the purpose of »Application-main« +Lumiera is envisioned as a heavyweight yet coherent »**Application**« -- not so much +as platform, framework or operating system. And, in accordance with this focus, we +place no emphasis on possible configuration changes and state transitions within +the running application _as a whole._ And, while in fact it is possible to close +and load a new Session, most of the time we assume the application just to be +``up and running'' -- all required services to be available and interfaces to +be accessible. Without doubt, these are simplifications, but serve us well +to cut down complexity -- yet still there needs to be one dedicated realm +to deal with these specific concerns of configuration, dependency and lifecyle. -...There is now sort-of an ``application realm'', which doesn't belong strictly to -any one of the Layers. It serves the purpose of pulling up and tearing down the -application in a controlled fashion. And additionally, it provides the Interface -and Config core services. Within the application, we find a small number of -_Subsystems_, which are more or less independent. These subsystems are -conceptual entities and don't correspond 1:1 to a layer, an interface, a class -or a plugin. These subsystems are _of no relevance_ outside the mentioned -"application realm". When the application is in normal operational mode, we have -the usual components, interfaces, services, aggregated into the three Layers. -__Currently I've identified the following subsystems:__ +The Application Realm +--------------------- +So we treat all these concerns within a small and self contained structure, clearly +set apart from all the other layers, services and subsystems. This dedicated +_Application Realm_ is organised around the ``Application main object''.footnote:[ +This is the singleton `lumiera::AppState`, which is triggered by the `main` function +of the Lumiera Application. The sourcecode is kept in a separate folder 'src/common' +and linked into the shared library 'liblumieracommon.so'] +It serves the purpose of pulling up and tearing down the application in a controlled +fashion. And additionally, it provides the Interface and Config core services. +The act of building or tearing down this core realm and main object is what creates +the *Lifecycle* of the application. This is a succession of ``lifecycle phases'' -- +and almost all activities happen within the _operational phase,_ when everything +is ``up and running'' or ``just available''. -- Engine -- Builder -- Session -- Lumiera GUI -- Script runner -- Renderfarm node server +Subsystems +~~~~~~~~~~ +However, initially the application needs to be brought up, and at the end, all +parts need to be unwound cleanly. To organise this process, we identify a limited +number of *Subsystems* within the Application, which are more or less independent. +Each link:{ldoc}/design/architecture/Subsystems.html[Subsystem] is self contained +and groups some other parts and services, which typically work together and may +be mutually dependent. These subsystems represent a grouping, applied for the purpose +of starting and stopping the application in a regular way; they rather do not +correspond 1:1 to a layer, an interface, a class or a plugin. As a matter of fact, +they are _rather irrelevant_ outside the mentioned »Application realm«. A subsystem +may depend on other subsystems, which comprises a clear startup- and shutdown-ordering. +However, once the application is in normal operational mode, the subsystems turn +into a passive, guarding and managing role; the activities relevant for the +application's purpose rather rely on components, interfaces, services, all +aggregated into the three Layers »Stage«, »Steam« and »Vault«. -To deal with those subsystems, I've created an Interface to define the -operations and liabilities of a subsystem. Additionally, I assume that for each -subsystem there is a _Façade_, which the subsystem is free to implement as it +__We expect the following subsystems to be built eventually:__ + +Engine, Session, PlayOut, GUI, Script runner, Renderfarm node. + +Organisation of Subsystems +^^^^^^^^^^^^^^^^^^^^^^^^^^ +Not all subsystems need to be started for any use of the application. A script-driven +use, or a renderfarm node does not need a GUI. So there is an overall global operation +mode of the application, controlled through the launching options, and determined during +the startup phase. It is the responsibility of the _Application main object_ to +pull up required functionality, which in turn might result in pulling up +further subsystems as dependencies. + +Subsystems are defined by implementing the interface `lumiera::Subsys`, which acts +as façade to conduct the lifecycle, find out about dependencies and shut down +the subsystem in the end. So this interface, together with the _Subsystem Runner,_ +define a lifecycle protocol; each subsystem is free to implement this as it sees fit. Typically, this façade will load plugins, register and provide further -"business interfaces", and especially set up the _Layer separation interfaces_ +_business interfaces,_ and especially set up the _Layer separation interfaces_ which canalise any communication going on between the layers. -TIP: The code from my `startup` branch has meanwhile been merged to master. Look -http://git.lumiera.org/gitweb?p=LUMIERA;a=tree;h=a0a0e456a5b149df81b25a08358cd488631639fb;hb=a0a0e456a5b149df81b25a08358cd488631639fb[here] -for the code referred in the following discussion. - -- +common/subsys.hpp+ contains the subsystem interface mentioned above. -- +lumiera/main.cpp+ uses the subsystem instances provided by the facades, and - additionally the services of +lumiera::AppState+ (+common/appstate.hpp+) -- AppState represents the state as relevant for the "application realm", i.e. - it performs global initialisation and shutdown. See especially +AppState::init()+ -- see +vault/enginefacade.hpp|cpp+ as an (unimplemented) façade skeleton. +vault::EngineFacade::getDescriptor()+ - yields the subsystem interface -- the GuiFacade is somewhat special, because we want to load the GUI from a - shared library. This façade is basically completed and working, but it currently - just loads a dummy plugin. The implementation of the GuiFacade needs to be in - core (because it pulls the plugin); that's why I've put it into - +common/guifacade.cpp+, while the interface is in +gui/guifacade.hpp+ as usual. -- as an example for a _Layer separation interface_, I've implemented the - GuiNotificationFacade, which will be used by the lower layers to push - informations into the GUI (and finally to request the GUI to shut down). Layer - separation interfaces are considered part of the public Lumiera API, thus the - headers go into +src/include/**+ - * include/guinotification.h (C/C++ combined header) defines an C++ interface (abstract class) and a CLI interface. - * embedded into the interface is a factory, i.e. by stage::GuiNotification::facade() you get an instance... - * which actually is a proxy and routes any call through the instance of the - accompanying CLI interface which is defined within the interface/plugin system - * this in turn forwards to the implementation class in - gui/guinotificationfacade.cpp, which is assumed to live within the GUI - (shared lib) - +The *GUI Façade* is special, while in compliance with this protocol. The actual +UI is loaded from a plug-in at runtime,footnote:[This corresponds to the vision +to allow for different Lumiera UI's -- maybe to support different working styles +or target audiences. If such is actually feasible remains to be clarified as of +2020; even while decoupled on a technical level, the session still needs to make +a lot of assumptions regarding the UI's capabilities and needs.] +and so the implementation of this façade needs to reside in the application core +realm; it will start a `GuiRunner` to load and activate the GUI plug-in, which +then in turn has to open the public _GUI Notification_ façade. The latter is +one of the _Layer separation interfaces_ and comprises the actual way for the +lower layers to activate and interact with the user interface. Parallelism ------------ -Actually this system builds on the assumption, that starting each subsystem -doesn't block the overall start/main/stop thread. I.e. any subsystem is supposed -to spawn its control/event threads if necessary. Not every subsystem needs to -spawn threads though (for example, the session doesn't). In the current -implementation _no spawning of threads happens_. Likewise, I've commented out -the synchronisation primitives. + -[yellow-background]#TODO 2015# _meanwhile we do spawn threads and perform synchronisation_ +~~~~~~~~~~~ +Actually this scheme builds on the assumption that starting each subsystem will +not block the overall start/main/shutdown thread. Any subsystem is supposed +to spawn its own control/event threads if necessary. The Lumiera application +works fundamentally asynchronous. The user interface operates single threaded, +in accordance to long established best practices of UI programming. However, +any user interaction is translated into commands, sent down into the session +and handled there one by one. The result of processing such commands will be +pushed back up into the UI later and detached from the immediate interaction. +Likewise, the re-rendering caused by changes in the session is carried out +within the engine independently, relying on worker jobs and a thread pool. Initialisation and Lifecycle ---------------------------- -Basically, everything workes as discussed last spring in the -link:{ldoc}/devel/rfc/GlobalInitialization.html[GlobalInitialisation] -design entry. I've considered to switch to simple functions +pre_init(), init(), -...+ as proposed by Cehteh, but I'm not happy with this idea, because it creates -a coupling between anything which needs to be done in a certain initialisation -function. Actually, I prefer the usual approach of lifecycle events (or signals) -used in many application frameworks, i.e. the publisher-subscriber model. This -allows to keep the registration immediately within the implementation of a -facility, and it allows to add an arbitrary number of additional lifecycle -events, like *ON_SESSION_CLOSE*, *ON_BUILD*, *ON_EMERGENCY_EXIT*. +After some discussion,footnote:[See the +link:{ldoc}/devel/rfc/GlobalInitialization.html[GlobalInitialisation] RfC +from spring 2008. In the beginning, we all agreed to ``keep matters simple'' +and build an `init()` function within one central piece of code everyone knows +and hooks into. However, while the outline of the application emerged, there +was a tension between the concern about _over-engineering_ versus the concern +about _tangled and unmanageable complexity._ At some point, an alternative +implementation based on lifecycle callbacks was elaborated, which then turned +into the solution described here. Lumiera then ceased to be the typical UI +application started by GTK, and the existing GTK code was retrofitted to +launch from within a plug-in.] +the design leaned toward loosely coupled parts and a formal lifecycle; which +saves us from investigating and coding up the various interdependencies +explicitly. Rather, the parts of the application have to comply to +link:{ldoc}/design/architecture/Subsystems.html#lifecycle[Lifecycle Phases], +and each part has to care for its own state transitions, invoked through +_lifecycle callbacks._ We can distinguish two distinct models how to deal +with lifecycle, and both are equally acceptable: -Basically we have now the following steps and events happening -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +- Assuming that operations happen in response to some client's request, + this activation should go through a _service interface._ Interfaces + can be opened and closed in Lumiera, and this is accomplished by + hooking them up below some subsystem. +- However, some parts carry out continuous activities, and in that case + a _lifecycle hook_ should be registered, to limit activities to the + appropriate lifecycle phase. - * *ON_BASIC_INIT* runs "somewhere" in the static initialisation phase before - main(). To schedule something there, you need to place a statc C++ variable. - This should be reserved for very basic initialisation tasks which need to be - done even prior to any global initialisation, e.g. the NOBUG_INIT is done this - way, the config system, maybe setting up a dispatcher table based on (compile - time) type information (for the rules engine). - * +AppState::init()+ cares for bringing up the plugin loader and opens the config-interface. +Application Start +~~~~~~~~~~~~~~~~~ + * some fundamental language-level facilities will be prepared during + _static initialisation._ At some point, execution enters `main(argc,arvv)`. + * `AppState::init()` brings up the plugin loader and opens the config-interface. * ...followed by triggering *ON_GLOBAL_INIT* - * +main()+ pulls up the subsystems according to the command line options. - * within each subsystem, façade interfaces shall be opened with the interface - system. Any initialisation of components should be tied to these. - * as a rule and if possible, _any service should be written such as to come up on demand._ + * the main thread then pulls up the subsystems (`AppState::maybeStart(subsystem)`), + according to the command line options. + * within each subsystem, façade interfaces will be opened through the + interface/plug-in system. + * At this point, the GUI plug-in is loaded and launched, the windows created, + the UI event loop starts and the application becomes live. * shutdown or failure of _any given subsystem_ initiates the shutdown sequence - by requesting all other running subsystems to terminate - * there is an *ON_GLOBAL_SHUTDOWN* event, which can be used for normal cleanup. - In case of an emergency exit, this hook may be skipped - * alternatively, the *ON_EMERGENCY_EXIT* event is triggered. In case of nested - exceptions, this may happen twice. - * the AppState destructor tries to bring down the core systems (config-interface and pluginloader). - - -Demo Run --------- - * building with scons (default target) now yields a bin/guistart.so, besides the bin/lumiera exe the scons built created since day one. - * we need to set the LUMIERA_PLUGIN_PATH from the commandline to point to this bin directory, e.g. - * +++LUMIERA_PLUGIN_PATH=/home/hiv/devel/lumi/.libs NOBUG_LOG='test:TRACE,lumiera:TRACE,config:TRACE' bin/lumiera 2>&1 | egrep -v '(TODO)|(FIXME)'+++ ------- -00000392: configfacade.cpp:74: INFO: thread_1: Config: Config system ready. -00000394: main.cpp:55: NOTICE: thread_1: main: *** Lumiera NLE for Linux *** -00000395: appstate.cpp:114: TRACE: thread_1: init: initialising application core... -00000405: interface.c:54: TRACE: thread_1: lumiera_interface_open: lumieraorg_interface -00000406: interface.c:55: WARNING: thread_1: lumiera_interface_open: opening experimental interface: lumieraorg_interface_0_lumieraorg_interface -00000412: interface.c:168: TRACE: thread_1: lumiera_interface_open_interfacenode: lumieraorg_interface 0 () -00000437: config.c:199: NOTICE: thread_1: lumiera_config_get: envvar override for config LUMIERA_PLUGIN_PATH = /home/hiv/devel/lumi/ -00000444: config.c:199: NOTICE: thread_1: lumiera_config_get: envvar override for config LUMIERA_PLUGIN_PATH = /home/hiv/devel/lumi/ -00000467: appstate.cpp:132: TRACE: thread_1: init: Lumiera core started successfully. -00000469: appstate.cpp:140: TRACE: thread_1: maybeStart: maybe startup Builder...? -00000472: appstate.cpp:140: TRACE: thread_1: maybeStart: maybe startup Renderfarm node...? -00000475: appstate.cpp:140: TRACE: thread_1: maybeStart: maybe startup Lumiera GTK GUI...? -00000476: subsystemrunner.hpp:144: INFO: thread_1: triggerStartup: Starting subsystem "Lumiera GTK GUI" -00000477: interface.c:54: TRACE: thread_1: lumiera_interface_open: lumieraorg_GuiStarterPlugin -00000483: interface.c:168: TRACE: thread_1: lumiera_interface_open_interfacenode: lumieraorg_GuiStarterPlugin 0 () - *** Ha Ha Ha - this is the GuiStarterPlugin speaking! - now, the Lumiera GUI should be spawned.... - but actually nothing happens!!!!!!!!!!!!!! -00000491: appstate.cpp:140: TRACE: thread_1: maybeStart: maybe startup Script runner...? -00000494: appstate.cpp:171: NOTICE: thread_1: maybeWait: Shutting down Lumiera... -00000495: interface.c:230: TRACE: thread_1: lumiera_interface_close: -00000498: interface.c:258: TRACE: thread_1: lumiera_interfacenode_close: lumieraorg_GuiStarterPlugin 1 () -00000634: configfacade.cpp:83: TRACE: thread_1: ~Config: config system closed. -00000635: appstate.cpp:245: TRACE: thread_1: ~AppState: shutting down basic application layer... -00000643: interface.c:230: TRACE: thread_1: lumiera_interface_close: -00000646: interface.c:258: TRACE: thread_1: lumiera_interfacenode_close: lumieraorg_interface 1 () ------- - * incidentally, running... `lumiera --help` produces the following output ------- -00000392: configfacade.cpp:74: INFO: thread_1: Config: Config system ready. -00000394: main.cpp:55: NOTICE: thread_1: main: *** Lumiera NLE for Linux *** -Lumiera, the non linear video editor. Supported parameters: - -h [ --help ] produce help message - -f [ --session ] arg session file to load (UNIMPLEMENTED) - -s [ --script ] arg execute the given LUA script (UNIMPLEMENTED) - --headless start without GUI - -p [ --port ] arg open renderfarm node at given port (UNIMPLEMENTED) - -d [ --define ] arg enter definition into config system (UNIMPLEMENTED) ------- + by requesting all other running subsystems to terminate. In the typical case, + the UI subsystem will trigger this shutdown sequence, in response to closing + the main window. + * there is an *ON_GLOBAL_SHUTDOWN* event, which can be used for normal cleanup; + In case of an emergency exit, the *ON_EMERGENCY_EXIT* event is triggered alternatively. + * the AppState destructor tears down the core systems (config-interface and pluginloader). + * we establish a policy to _prohibit any non-local and non-trivial activities_ during the + tear-down phase after leaving the `main()` function. diff --git a/doc/design/architecture/Subsystems.txt b/doc/design/architecture/Subsystems.txt index 578892709..3184ed146 100644 --- a/doc/design/architecture/Subsystems.txt +++ b/doc/design/architecture/Subsystems.txt @@ -168,6 +168,8 @@ for link:{ldoc}/technical/library/Dependencies.html[lazily initialised dependenc establish the policy that *destructors must not rely on dependencies*. In fact, they should not do any tangible work at all, beyond releasing other resources. +anchor:lifecycle[] + Lifecycle Events ~~~~~~~~~~~~~~~~ The Application as a whole conducts a well defined lifecycle; whenever transitioning to the next phase, From b002a5e0b39adf95ab9be9972e5ed546e9bc51df Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Sun, 23 Feb 2020 08:21:22 +0100 Subject: [PATCH 05/12] DOC: fill in explanations for all subsystems (closes #1145) --- doc/design/architecture/Subsystems.txt | 160 +++++++++++++++++-------- doc/technical/howto/IdeSetup.txt | 2 +- 2 files changed, 112 insertions(+), 50 deletions(-) diff --git a/doc/design/architecture/Subsystems.txt b/doc/design/architecture/Subsystems.txt index 3184ed146..67406f860 100644 --- a/doc/design/architecture/Subsystems.txt +++ b/doc/design/architecture/Subsystems.txt @@ -5,37 +5,36 @@ Layers -- Subsystems -- Lifecycle :Date: 2018 :Toc: -WARNING: under construction -- [red]#some parts to be filled in# - Terminology ----------- Layer:: - A conceptual realm within the application to group related concerns and define an ordering. - Layers are above/below each other and may depend _solely_ on lower layers. The Application - may be operated in a partial layer configuration with only some lower layers. Each layer - deals with distinct topics and has its own style. In Lumiera, we distinguish three layers + A conceptual realm within the application in order to group related topics and to build + high-level structures in terms of low-level structures. Layers are located above/below + each other and may depend _solely_ on lower layers. The Application may be operated in + a partial layer configuration with only some lower layers present. Each layer deals + with distinct topics and has its own style. In Lumiera, we distinguish three layers + * Stage Layer -> Interaction * Steam Layer -> Processing * Vault Layer -> Data manipulation Subsystem:: - A runtime entity which acts as anchor point and framework to maintain a well defined lifecycle. - While layers are conceptual realms, subsystems can actually be started and stopped and their + A runtime entity which serves as anchor point and framework to maintain a well defined lifecycle. + While layers are conceptual realms, subsystems can actually be started and stopped, and their dependencies are represented as data structure. A subsystem typically starts one or several primary components, which might spawn a dedicated thread and instantiate further components and services. Service:: - A component within some subsystem is termed _Service_ + A component within some subsystem is called a _Service_ + -- - * when it exposes an interfaces with an associated contract + * provided that it exposes an interface with an associated contract (informal rules about usage pattern and expectations) - * when it accepts invocations from arbitrary other components - without specific wiring or hard coded knowledge + * and given that it accepts invocations from arbitrary other components + without mutual interdependency or hard coded knowledge about that other part. -- + The service lifecycle is tied to the lifecycle of the related subsystem; whenever the subsystem is ``up and running'', @@ -44,14 +43,15 @@ of elaborate _service discovery_ -- rather, services are accessed *by name*, whe of the service interface. Dependency:: - A relation at implementation level and thus a local property of an individual component. A dependency - is something we need in order to complete the task at hand, yet a dependency lies beyond that task and - relates to concerns outside the scope and theme of this actual task. Which means, a dependency is not - introduced by the task or part of the task, rather the task is the reason why some entity dealing with - it needs to _pull_ the dependency, in order to be able to handle the task. So essentially, dependencies - are accessed on-demand. Dependencies might be other components or services, and typically the user - (consumer) of a dependency just relies on the corresponding interface and remains agnostic with respect - to the dependency's actual implementation or lifecycle details. + A usage relation at implementation level and thus a local prerequisite of an individual component. A + dependency is something we need in order to complete the task at hand, yet a dependency lies beyond that + task and is satisfied by means outside the scope and theme of this actual task. Consequently, a dependency + is not introduced or provided by the local task or part of the task, rather the task at hand is the reason + why some other entity dealing with it needs to _request_ or _pull_ that dependency in to accomplish the + task at hand. So essentially, dependencies are accessed on-demand. Dependencies might be satisfied by + other components or services, and typically the user (consumer) of a dependency just relies on the + corresponding interface and remains agnostic with respect to the dependency's actual implementation, + data or lifecycle details. Subsystems ---------- @@ -67,7 +67,7 @@ it depends and relies on the scheduling service of the engine. In the end, it re architecture to keep those dependency chains ordered in a way to form a one-way route: when we start the engine, it must not instantiate a component which _requires the player_ in order to be operative. Yet we can not start the player without having started the engine beforehand; if we do, its services -will throw exceptions due to missing dependencies on first use. +will throw exceptions on first use, due to missing dependencies. However, subsystems as such are not dynamically configured. This was a clear cut design decision (and the result of a heated debate within the Lumiera team at that time). We do _not expect_ to load just some plug-in @@ -78,49 +78,111 @@ parts of the application, each with its own theme, style, relations and continge Engine ~~~~~~ -_tbw_ +The Engine performs small pieces of work known as _render jobs,_ oriented towards a deadline, +without much knowledge about the purpose of those jobs, or their further interconnections. +And thus the purpose of the *Engine Subsystem* is to provide a thread pool and activate +the scheduling mechanism. Consequently, this subsystem belongs into the »Vault Layer« + +_[yellow-background]#this part of the system is barely drafted as of 2020#_ Player ~~~~~~ -_tbw_ +The *PlayOut Subsystem* is located above the Engine and belongs into the »Steam Layer« -- and contrary +to the Engie (which handles individual jobs), the player creates and organises _calculation streams._ + +_[yellow-background]#as of 2020, the actual components to form the player need to be worked out#_ + +_^however, a fair amount of the services for dispatching streams into jobs has been prototyped^_ Session ~~~~~~~ -_tbw_ +The user performs editing activities within the »Session« -- which is a data structure with associated +methods for manipulation. There is a `Session` object and a `SessionManager` to load and save session +data and conduct the _session lifecycle._ However, all of this needs to be distinguished from the +*Session Subsystem* -- which in essence is a dispatcher thread to receive, enqueue and finally +trigger the _session commands,_ as sent from the GUI or the script runner. These activities are +conducted and controlled by the `SteamDispatcher`, which also cares for triggering the _Builder,_ +whenever new commands have been dispatched. Moreover, when instantiating the `DispatcherLoop`, +also the `SessionCommand` façade is opened, which is the primary »Steam Layer« interface.footnote:[ +Note the relation between Session-the-datastructure and Session-the-subsystem is rather indirect: +the _dispatching_ of commands is blocked, unless there is also a session-datastructure loaded +and fully configured. However, a running dispatcher loop is not a prerequisite for opening +a session -- just without a running dispatcher, commands will queue up and nothing else will happen.] + +_[green]#as of 2020, this subsystem is operative and commands can be dispatched#_ + +_^...while the session data structure as such is mostly still a skeleton...^_ User Interface ~~~~~~~~~~~~~~ -_tbw_ +The Lumiera GUI is loaded as self-contained plug-in, which is the task of the *GUI Subsystem*. +As can be expected, this is a rather convoluted process, while the actual name of the UI plug-in module +to load is configured in the 'setup.ini', which has been evaluated earlier, in the application init phase. +However, as it stands, Lumiera is built with a GTK-3 interface, and within the corresponding plug-in module +`gtk_gui.lum`, the class `GtkLumiera` serves as top-level guard to carry on all further activities, +when triggered from within the subsystem lifecycle to run in a dedicated GUI thread. It will establish +the _UI backbone_ by activating the _UI-Bus_ and building the _UI Manager_ controlling the UI global context. +After these systems are established and connected, the GTK windows can be created and finally control is handed +over to the GTK (GIO) event loop. Whenever this loop terminates, be it regularly, or by exception, application +shutdown is initiated. + +The GUI Subsystem is special, insofar it not only attaches to the session interface, but also opens a +_Layer separation interface_ oriented downwards, to be used by the lower layers. This interface -- known +as GUI Notification façade -- serves to populate the UI with actual content, to mark and animate the +tangible elements visible to and manipulated by the user in turn. It is outfitted with a cross-thread +dispatcher mechanism, to forward any invocation as message onto the UI-Bus. This setup allows the +lower layers to address the tangible parts in the UI based on their ID, which previously was given +alongside with the content when populating the structures. + +_[green]#as of 2020, this backbone is established and connected in both directions#_ + +_^...while the large part of the actual widgets still remains to be built...^_ Script Runner ~~~~~~~~~~~~~ -_tbw_ +One of the most fundamental design decisions for Lumiera is that everything can be done without GUI. +Conceptually, this would allow to instantiate a script execution environment with appropriate bindings, +either to conduct operations on an existing session, or to build and render a session from scratch. +Alternatively, also a CLI-style shell-like interface is conceivable. + +_[maroon orange-background]#this is a concept without any detailed planning as of 2020#_ Net Node ~~~~~~~~ -_tbw_ +In variation to the script runner concept, it is conceivable to send instructions to a Lumiera +instance over the net. Expanding on that idea, it would be possible to define a protocol to +distribute the session definitions to slave nodes and then to launch distributed render tasks. +Since Lumiera is built as a self-contained bundle, it is well suited to run within a containerised +environment. However, in the light of current trends towards container orchestration frameworks +like Kubernetes, we should refrain from building to much process management functionality into +the application itself. + +_[aqua teal-background]#this is a mere idea, and certainly not a priority as of 2020#_ + + +.... + +.... Lifecycle --------- Dependencies and abstraction through interfaces are ways to deal with complexity getting out of hand. -When done well, we can avoid adding _accidental complexity_ -- yet essential complexity as such can not -be removed, but with the help of abstractions it can be raised to another level.footnote:[Irony tags here. +When done well, we can avoid adding _accidental complexity_ -- but essential complexity as such can not +be removed, yet with the help of abstractions it can be raised to another level.footnote:[Irony tags here. There is a lot of hostility towards abstractions, because it is quite natural to conflate the abstraction -with the essential complexity it has to deal with. It seems compelling to kill the abstraction, in the -hope to kill the complexities as well -- an attitude rather effective, in practice...]. -When components express their external needs in the form of dependency on an interface, the immediate tangling -at the code level is resolved, however, someone needs to implement that interface, and this other entity needs -to be _available_. It is now an architecture challenge to get those dependency chains ordered. A way to -circumvent this problem is to rely on a _lifecycle_ with several _phases._ -This is the idea behind the subsystems and the subsystem runner. +with the essential complexity it has to deal with. It seems compelling to kill the abstraction, in the hope +to kill the complexities as well -- a tremendously effective attitude, as it turns out, especially in practice...] +When components express their external needs by depending on an interface, the immediate tangling at the code level +is resolved. However, someone needs to implement that interface, and this other entity needs to be _available_. +The problem has been shifted, since it is now an architecture level challenge to get those dependency chains +satisfied. A clever way to circumvent this problem rather then to deal with it explicitly, is to rely on a +_lifecycle_ with several _phases._ This is the idea behind the subsystems and the subsystem runner. . First we define an ordering between the subsystems. The most basic subsystem (the Engine) is started first. . Within a subsystem, components may be mutually dependent. However, we establish a new rule, dictating that - during the _startup phase_ only local operations within a single component are allowed. The component need - to be written in a way that it does not need the help of anything ``remote'' in order to get its inner - workings up and ready. The component may rely on its members and on other services it created, _owns and - manages._ And sometimes we do need to rely on a more low-level service in another subsystem or in the + during the _startup phase_ only local operations within a single component are allowed. Each component must + to be written in such a way, not to rely on the help of anything ``remote'' in order to get its inner workings + up and ready. The component may rely on its members and on other services it _created itself,_ or which it + _owns and manages._ +. However, sometimes we _do need to rely_ on a more low-level service in another subsystem or in the application core.footnote:[A typical example would be the reliance on threading, locking or application configuration.] -- which then creates a hard dependency on _architecture level_ . Moreover, we ensure that all operational activity is generated by actual work tasks, and that such tasks @@ -136,8 +198,8 @@ The problem with emergencies ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This concept has a weak spot however: A catastrophic failure might cause any subsystem to break down immediately. The handler within the subsystem's primary component will hopefully detect the corresponding exception and signal -emergency to the subsystem runner. However, the working services of that subsystem are already gone at that point. -And even while other subsystems get the (emergency) shutdown trigger, some working parts may be already failing +emergency to the subsystem runner. Yet the working services of that subsystem are already gone at that point. +And even before other subsystems might get the (emergency) shutdown trigger, some working parts may be failing catastrophically due to their dependencies being dragged away suddenly. Lumiera is not written for exceptional resilience or high availability. Our attitude towards such failures can @@ -153,20 +215,20 @@ However, anything beyond the scope of `main()` is not meant to be used for regul initialisation, dependency management and decommissioning -- when actually necessary -- should be part of the application code proper.footnote:[this is established ``best practice'' for good reasons. The interplay of static lifespan, various translation units and even dynamically loaded libraries together with shared access -becomes intricate and insidious quite easily. And since in theory any static function could use some static +tends to becomes intricate and insidious easily. And since, in theory, any static function could use some static variable residing in another translation unit, it is always possible to construct a situation where objects are accessed after being destroyed. Typically such objects do not even look especially ``dead'', since the static storage remains in place and still holds possibly sane values. Static (global) variables, like raw pointers, allow to subvert the deterministic automatic memory management, which otherwise is one of the greatest strengths of C++. Whenever we find ourselves developing extended collaborative logic based on several statics, we should consider to transform this logic into regular objects, which are easier to -test and better to reason about. If it really can not be avoided to use such units of logic from a -static context, it should at least be packaged as a single object, plus we should ensure this logic -can only be accessed through a regular (non static) object as front-end. Packaged this way, the -most common and dangerous pitfalls with statics can be avoided.] And since Lumiera indeed allows +test and to reason about. If it really can not be avoided to use such units of logic from a static +context, it should at least be packaged as a single object, plus we should ensure this logic can +only be accessed through a regular (non static) object as front-end. Packaged this way, the most +common and dangerous pitfalls with statics can be avoided.] And since Lumiera indeed allows for link:{ldoc}/technical/library/Dependencies.html[lazily initialised dependencies], we -establish the policy that *destructors must not rely on dependencies*. In fact, they should -not do any tangible work at all, beyond releasing other resources. +establish the policy that *destructors must not rely on dependencies*. In fact, they +should not do any tangible work at all, beyond releasing other resources. anchor:lifecycle[] diff --git a/doc/technical/howto/IdeSetup.txt b/doc/technical/howto/IdeSetup.txt index 901e6e920..05741b2a4 100644 --- a/doc/technical/howto/IdeSetup.txt +++ b/doc/technical/howto/IdeSetup.txt @@ -64,7 +64,7 @@ also able to _build_ that code (even partially) from within the IDE, since the i more cross linking information. However, this is not a strict requirement -- even while `F3` often fails, the ``Open Type'' dialog is able to spot the definition in many cases non the less, and when this fails, you can still use ``brute-force'' file search. What turns out to be much more an impediment in practice is the fact -that you'll have to jump through that C++ binding layer, and you need to pick up some basic knowledge how +that you'll have to jump through that C\++ binding layer, and you need to pick up some basic knowledge how this layer works to wrap the underlying plain-C GTK entities; don't confuse the C++ _wrapper objects_ with the _gobject_ (a concept from GLib) used by GTK. From 710e35c87abba2e9514ba90d047ddb5d61dc90b5 Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Fri, 11 Dec 2020 23:48:30 +0100 Subject: [PATCH 06/12] Fix some further mentions and links to Cinelerra-CV as indicated by Igor Vladimirsky --- doc/design/gui/GuiDiscussion/ConceptProposals/Barnes.txt | 2 +- doc/devel/rfc/Manifest.txt | 6 +++--- src/lib/psplay.c | 2 +- src/lib/safeclib.h | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/design/gui/GuiDiscussion/ConceptProposals/Barnes.txt b/doc/design/gui/GuiDiscussion/ConceptProposals/Barnes.txt index e707933cb..b97a18128 100644 --- a/doc/design/gui/GuiDiscussion/ConceptProposals/Barnes.txt +++ b/doc/design/gui/GuiDiscussion/ConceptProposals/Barnes.txt @@ -14,7 +14,7 @@ He previously worked for Intrigo as a User Experience Designer while living in T *********************************************************************************************************** I’ve pulled together some drafts of my ideas for the design of the timeline portion -of the Lumiera non-linear video editor (hopefully, the successor to http://cinelerra-cv.org[Cinelerra]). +of the Lumiera non-linear video editor (hopefully, the successor to http://cinelerra-cv.wikidot.com[Cinelerra]). The un-annotated version:: image:{imgg}/Barnes.proposal.png[ diff --git a/doc/devel/rfc/Manifest.txt b/doc/devel/rfc/Manifest.txt index d3944ed62..8d27ceeb3 100644 --- a/doc/devel/rfc/Manifest.txt +++ b/doc/devel/rfc/Manifest.txt @@ -27,9 +27,9 @@ Background ~~~~~~~~~~ Cinelerra is quite an old project, there is an original version from -heroinewarrior.com and a community fork at cinelerra-cv.org. The original author -claims that there was no-one producing useable input despite their proposes -while cinelerra was in development, and indeed the Cinelerra-CV community only +heroinewarrior.com and a community fork link:https://cinelerra-cv.wikidot.com/[Cinelerra-CV]. +The original author claims that there was no-one producing useable input despite their +requests while cinelerra was in development, and indeed the Cinelerra-CV community only feeds back the source released by the original author into their SVN repository and maintains few fixes. There is not much development going on. Some people have created new functionality/features from time to time which have rarely diff --git a/src/lib/psplay.c b/src/lib/psplay.c index a6f0e32eb..5036f3b5b 100644 --- a/src/lib/psplay.c +++ b/src/lib/psplay.c @@ -3,7 +3,7 @@ Copyright (C) 2004, 2005, 2006, Christian Thaeter - Copyright (C) CinelerraCV + Copyright (C) (contributed to CinelerraCV) 2007, 2008, Christian Thaeter This program is free software; you can redistribute it and/or diff --git a/src/lib/safeclib.h b/src/lib/safeclib.h index 63ff7f356..8dd76ff8c 100644 --- a/src/lib/safeclib.h +++ b/src/lib/safeclib.h @@ -1,7 +1,7 @@ /* SAFECLIB.h - Portable and safe wrappers around some clib functions and some tools - Copyright (C) CinelerraCV + Copyright (C) 2008, Christian Thaeter This program is free software; you can redistribute it and/or From 80377eb744f316cdc5913c3352a3a17fe6a865df Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Tue, 20 Sep 2022 00:06:27 +0200 Subject: [PATCH 07/12] DOC: Draft rework of the Architecture diagram On my visit to Benny in the Black forest, we decided to concentrate on a "Playback Vertical Slice" and to announce that in the development report, using an architecture diagram... --- doc/devel/draw/Lumi.Architecture-2.svg | 1561 ++++++++++++++++++++++++ 1 file changed, 1561 insertions(+) create mode 100644 doc/devel/draw/Lumi.Architecture-2.svg diff --git a/doc/devel/draw/Lumi.Architecture-2.svg b/doc/devel/draw/Lumi.Architecture-2.svg new file mode 100644 index 000000000..4445f84da --- /dev/null +++ b/doc/devel/draw/Lumi.Architecture-2.svg @@ -0,0 +1,1561 @@ + + + + + Lumiera Architecture + + + + + + + + image/svg+xml + + Lumiera Architecture + + September 2022 + + + Ichthyostega + + + + + Lumiera.org + + + Overview Layers, Components and Relations + + + + + + + + + + + + + + + + Stage + + + + Steam + + + + Vault + + + + + + UI-Bus + + + + High-Level Model + + + + Low-Level Model + + + + + Render Engine + + + + Dispatcher + + + + Scheduler + + + + + RenderJobs + + + + Cache + + + + IO-System + + + + RenderToolkit + + + + Fixture + + + + Assets + + + + PathGuide + + + + + StreamTypes + + + + RulesSystem + + + + + + + + + + + + + + + Plug-ins + + + Defaults + + + + FactBase + + + + ConfigRules + + + + ActionLog + + + + SessionManager + + + + Player + + + + SteamAllocator + + + + SteamDispatcher + + + + AppState + + + + ExtensionManager + + + + ConfigSystem + + + + Output-Director + + + + Buffer-Manager + + + + NetNode + + + + StorageLocator + + + + TimelineController + + + + GTKLumiera + + + + Nexus + + + + UIState + + + + InteractionDirector + + + + TrackPresenter + + + + ClipPresenter + + + + ViewResolver + + + + Spot-Resolver + + + + Focus-Tracker + + + + Navigator + + + + AssetPresenter + + + + UI-Manager + + + + UI-Style + + + + Wizzard + + + + ScriptRunner + + + + Serialiser + + + + Gestures + + + + Zoom + + + + Layout + + + + LocalSetup + + + + + + Builder + + + + GTK Panels and Widgets + + + + + + + + + + Libraries + + From 42730da6121d212ca284c2391e2dcd6777abae09 Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Thu, 22 Sep 2022 23:42:08 +0200 Subject: [PATCH 08/12] DOC: Rework and complete the Architecture diagramm Especially Steam Layer was confusing; Re-arranged components to indicate relations and weight --- doc/devel/draw/Lumi.Architecture-2.svg | 1456 +++++++++++++++++++----- 1 file changed, 1151 insertions(+), 305 deletions(-) diff --git a/doc/devel/draw/Lumi.Architecture-2.svg b/doc/devel/draw/Lumi.Architecture-2.svg index 4445f84da..df204b168 100644 --- a/doc/devel/draw/Lumi.Architecture-2.svg +++ b/doc/devel/draw/Lumi.Architecture-2.svg @@ -22,7 +22,263 @@ Lumiera Architecture + id="defs2"> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + sodipodi:insensitive="true" + style="display:inline;opacity:1"> + id="g1250" + style=""> Stage + id="g1255" + transform="translate(0,2.6458404)" + style=""> + style="color:#000000;display:inline;overflow:visible;visibility:visible;opacity:1;vector-effect:none;fill:#65dcce;fill-opacity:0.16470588;fill-rule:evenodd;stroke:#25a393;stroke-width:0.35277778;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:0.35277775, 0.70555555000000003;stroke-dashoffset:0;stroke-opacity:1;marker:none;marker-start:none;marker-mid:none;marker-end:none;enable-background:accumulate" /> Steam + id="g1260" + transform="translate(0,14.284746)" + style=""> + y="186.7986" /> + transform="translate(0,20.500036)" + style="display:inline" + sodipodi:insensitive="true"> + + + Render Engine + @@ -186,146 +471,55 @@ x="44.750763" y="118.59637">UI-Bus - - - High-Level Model - + transform="translate(13.229184,76.729146)"> + y="72.104126" /> Low-Level Model - - - - - Render Engine - - - - Dispatcher - - - - Scheduler - + y="79.747688">Low-Level Model + transform="translate(13.229184,47.625028)"> + y="143.5416" /> RenderJobs + transform="translate(85.989605,107.15624)"> Cache + x="44.757977" + y="118.91266">Cache IO-System + x="44.580208" + y="119.5369">IO-System + transform="translate(46.302099,46.30209)"> RenderRenderToolkit + y="155.17398">Toolkit + transform="translate(13.229184,41.010418)"> Fixture - - - - Assets + y="104.77087">Fixture + transform="translate(13.229184,39.687496)"> Guide + transform="translate(169.33334,46.302072)"> Types RulesRulesSystem + y="86.633461">System + y="192.48955" /> + y="192.48955" /> + y="192.48955" /> + y="205.71872" /> + y="205.71872" /> + y="205.71872" /> + y="205.71872" /> Plug-ins + y="202.38936">Plug-ins Defaults + transform="translate(198.4375,33.072907)"> + y="107.82288" /> FactFactBase + x="41.725624" + y="120.05159">Base + + + + High-Level Model + + + + Assets + transform="translate(171.97916,9.2604097)"> ConfigRules ActionActionLog + y="149.89162">Log SessionManager + + + + + + + + + + + + transform="translate(-95.25,19.843743)"> Player + y="125.93754">Player SteamAllocator + id="tspan1094" + x="190.29436" + y="125.87243">Instructor + style="color:#000000;display:inline;overflow:visible;visibility:visible;opacity:1;vector-effect:none;fill:#d9d9d2;fill-opacity:1;fill-rule:evenodd;stroke:#333333;stroke-width:0.26458329;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;marker:none;marker-start:none;marker-mid:none;marker-end:none;enable-background:accumulate" /> SteamDispatcher + transform="translate(-83.343709,-41.010412)"> AppAppState + y="149.7614">State ExtensionExtensionManager + y="148.6917">Manager + transform="translate(-83.343709,-15.874997)"> ConfigConfigSystem + y="147.48764">System Output-Director + y="127.5116">Output-Director - - + + + Dispatcher + + + + Scheduler + + + + Buffer-Manager + style="color:#000000;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:4.23333311px;line-height:100%;font-family:'Bitstream Vera Sans';-inkscape-font-specification:'Bitstream Vera Sans, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-indent:0;text-align:start;text-decoration:none;text-decoration-line:none;letter-spacing:normal;word-spacing:normal;text-transform:none;writing-mode:lr-tb;direction:ltr;text-anchor:start;display:inline;overflow:visible;visibility:visible;opacity:1;vector-effect:none;fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#333333;stroke-width:0.13229167;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;marker:none;marker-start:none;marker-mid:none;marker-end:none;enable-background:accumulate" + xml:space="preserve">Buffer-Manager + NetNetNode + y="149.7614">Node + transform="translate(-142.87501,115.09374)"> StorageLocator Runner Layout LocalSetup + id="g1144" + transform="translate(0,13.229162)"> Builder + x="184.13037" + y="132.55212">Builder + y="212.3333" /> Libraries + y="215.48557">Libraries + + + + main + + + + + Timeline + + + + + + + + Sequence + + + + Clip + + + + Clip + + + + + + + + + + + + + Node + + + + + + + Node + + + + + + + + + Node + + + + + + + Node + + + + + + + Node + + + + + + + + + Node + + + + + Node + + + + + Node + + + + + + + Node + + + + + + Eventloop + + From 05afb717b81f38e7a7587a9228514a4e18bf8be8 Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Thu, 22 Sep 2022 23:55:47 +0200 Subject: [PATCH 09/12] DOC: indicate the "Playback Vertical Slice" this is just an "interpretation" of the current architecture diagram, created for inclusion into the developer report, indicating those components to be augmented and integrated to get a simple render/playback to work --- doc/devel/draw/Lumi.Architecture-2.svg | 14 +- doc/devel/draw/VerticalSlice.Playback.svg | 2558 +++++++++++++++++++++ 2 files changed, 2566 insertions(+), 6 deletions(-) create mode 100644 doc/devel/draw/VerticalSlice.Playback.svg diff --git a/doc/devel/draw/Lumi.Architecture-2.svg b/doc/devel/draw/Lumi.Architecture-2.svg index df204b168..394139824 100644 --- a/doc/devel/draw/Lumi.Architecture-2.svg +++ b/doc/devel/draw/Lumi.Architecture-2.svg @@ -286,11 +286,11 @@ borderopacity="1.0" inkscape:pageopacity="0.0" inkscape:pageshadow="2" - inkscape:zoom="1" - inkscape:cx="490.30055" + inkscape:zoom="0.5" + inkscape:cx="475" inkscape:cy="640.50626" inkscape:document-units="mm" - inkscape:current-layer="layer1" + inkscape:current-layer="l1_compo" showgrid="false" units="px" inkscape:window-width="1680" @@ -328,6 +328,8 @@ Overview Layers, Components and Relations + +envisioned Lumiera Architecture 2022 @@ -425,7 +427,7 @@ @@ -1891,7 +1893,7 @@ diff --git a/doc/devel/draw/VerticalSlice.Playback.svg b/doc/devel/draw/VerticalSlice.Playback.svg new file mode 100644 index 000000000..75f657117 --- /dev/null +++ b/doc/devel/draw/VerticalSlice.Playback.svg @@ -0,0 +1,2558 @@ + + + + + Lumiera Architecture + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + VerticalSlice : Playback + + September 2022 + + + Ichthyostega + + + + + Lumiera.org + + + Overview Layers, Components and Relations + +Indicated: scope of the planned Playback-Vertical-Slice + + + + + + + + + + + + + + + + Stage + + + + Steam + + + + Vault + + + + + + Render Engine + + + + UI-Bus + + + + Low-Level Model + + + + RenderJobs + + + + Cache + + + + IO-System + + + + RenderToolkit + + + + Fixture + + + + PathGuide + + + + + StreamTypes + + + + RulesSystem + + + + + + + + + + + + + + + Plug-ins + + + Defaults + + + + FactBase + + + + High-Level Model + + + + Assets + + + + ConfigRules + + + + ActionLog + + + + SessionManager + + + + + + + + + + + + + + + Player + + + + Instructor + + + + SteamDispatcher + + + + AppState + + + + ExtensionManager + + + + ConfigSystem + + + + Output-Director + + + + + Dispatcher + + + + Scheduler + + + + Buffer-Manager + + + + + NetNode + + + + StorageLocator + + + + TimelineController + + + + GTKLumiera + + + + Nexus + + + + UIState + + + + InteractionDirector + + + + TrackPresenter + + + + ClipPresenter + + + + ViewResolver + + + + Spot-Resolver + + + + Focus-Tracker + + + + Navigator + + + + AssetPresenter + + + + UI-Manager + + + + UI-Style + + + + Wizzard + + + + ScriptRunner + + + + Serialiser + + + + Gestures + + + + Zoom + + + + Layout + + + + LocalSetup + + + + + + Builder + + + + GTK Panels and Widgets + + + + + + + + + + Libraries + + + + + + + + + + + + + + + + + + + + + + + + + + + Builder + + + + + main + + + + + Timeline + + + + + + + + Sequence + + + + Clip + + + + Clip + + + + + + + + + + + + + Node + + + + + + + Node + + + + + + + + + Node + + + + + + + Node + + + + + + + Node + + + + + + + + + Node + + + + + Node + + + + + Node + + + + + + + Node + + + + + + Eventloop + + + + From 0b9f2e2c3139cad542013caac593cb5972a2e3f6 Mon Sep 17 00:00:00 2001 From: Ichthyostega Date: Fri, 7 Oct 2022 15:21:08 +0200 Subject: [PATCH 10/12] DOC: Eventloop -> event-loop ...small Fix in the diagram proposed by Benny, improves readability for the non-technical reader --- doc/devel/draw/Lumi.Architecture-2.svg | 9 ++++----- doc/devel/draw/VerticalSlice.Playback.svg | 8 ++++---- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/doc/devel/draw/Lumi.Architecture-2.svg b/doc/devel/draw/Lumi.Architecture-2.svg index 394139824..8c9df5fb3 100644 --- a/doc/devel/draw/Lumi.Architecture-2.svg +++ b/doc/devel/draw/Lumi.Architecture-2.svg @@ -16,7 +16,6 @@ id="svg8" inkscape:version="0.92.4 (5da689c313, 2019-01-14)" sodipodi:docname="Lumi.Architecture-2.svg" - inkscape:export-filename="/home/hiv/Desktop/DRAFT-Lumi.Architecture.png" inkscape:export-xdpi="96" inkscape:export-ydpi="96"> Eventloop</tspan></text> + y="135.74843">event-loop</tspan></text> <rect style="color:#000000;display:inline;overflow:visible;visibility:visible;opacity:1;vector-effect:none;fill:none;fill-opacity:1;fill-rule:evenodd;stroke:#999999;stroke-width:0.238125;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;marker:none;marker-start:none;marker-mid:none;marker-end:none;enable-background:accumulate" id="rect4805" - width="16.829708" + width="17.868351" height="4.927865" - x="35.90284" + x="35.383518" y="132.29166" - rx="1.653646" + rx="1.6536459" ry="1.6536459" /> </g> </g> diff --git a/doc/devel/draw/VerticalSlice.Playback.svg b/doc/devel/draw/VerticalSlice.Playback.svg index 75f657117..f463e8df0 100644 --- a/doc/devel/draw/VerticalSlice.Playback.svg +++ b/doc/devel/draw/VerticalSlice.Playback.svg @@ -2543,15 +2543,15 @@ Indicated: scope of the planned Playback-Vertical-Slice sodipodi:role="line" id="tspan4811" x="44.248707" - y="135.74843">Eventloop</tspan></text> + y="135.74843">event-loop</tspan></text> <rect style="color:#000000;display:inline;overflow:visible;visibility:visible;opacity:1;vector-effect:none;fill:none;fill-opacity:1;fill-rule:evenodd;stroke:#999999;stroke-width:0.238125;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;marker:none;marker-start:none;marker-mid:none;marker-end:none;enable-background:accumulate" id="rect4805" - width="16.829708" + width="17.868351" height="4.927865" - x="35.90284" + x="35.383518" y="132.29166" - rx="1.653646" + rx="1.6536459" ry="1.6536459" /> </g> </g> From 4c748c543b3a4581d554209bc9a6982b2d8596ce Mon Sep 17 00:00:00 2001 From: Ichthyostega <prg@ichthyostega.de> Date: Sat, 4 Feb 2023 02:08:25 +0100 Subject: [PATCH 11/12] Rfc: overrule and push the "Scripting language Lua" back to draft. During the last years, I became more and more doubtful and regretted that decision. In hindsight, the fundamental conflict was present already in the original discussion. My own experience showed me again and again: skipping the hard work of specification for sake of some kind of fluid prototyping rarely leads to anything solid. If "time to market" counts, this can be a viable strategy though... --- doc/devel/rfc/ScriptingLanguage.txt | 70 +++++++++++++++---- .../ScriptingLanguage.txt | 0 2 files changed, 58 insertions(+), 12 deletions(-) rename doc/devel/{rfc_final => rfc_pending}/ScriptingLanguage.txt (100%) diff --git a/doc/devel/rfc/ScriptingLanguage.txt b/doc/devel/rfc/ScriptingLanguage.txt index 1cb126e6d..b408eb836 100644 --- a/doc/devel/rfc/ScriptingLanguage.txt +++ b/doc/devel/rfc/ScriptingLanguage.txt @@ -1,6 +1,6 @@ [grid="all"] `------------`----------------------- -*State* _Final_ +*State* _Pending_ *Date* _2008-07-26_ *Proposed by* link:ct[] ------------------------------------- @@ -125,21 +125,67 @@ it's your decision to make, all the best, we are looking forward to alphas and betas in the future -- mytwocents - This proposal is about the ''required'' scripting language, i.e. when - accepted, Lua will be a necessary prerequisite for running Lumiera. This - doesn't rule out the ''use'' of other scripting languages. We strive at - having clean interfaces, thus it shouldn't be much of a problem to create - Python bindings. And given the popularity of Python, I guess it won't be long - until we have some Python bindings. But ''requiring'' Python would mean - having a Python runtime in memory most of the time -- for such Lua obviously - is a better choice, because it's much more lightweight and minimalistic. - -- link:Ichthyostega[] [[DateTime(2008-09-30T02:17:08Z)]] +This proposal is about the ''required'' scripting language, i.e. when +accepted, Lua will be a necessary prerequisite for running Lumiera. This +doesn't rule out the ''use'' of other scripting languages. We strive at +having clean interfaces, thus it shouldn't be much of a problem to create +Python bindings. And given the popularity of Python, I guess it won't be long +until we have some Python bindings. But ''requiring'' Python would mean +having a Python runtime in memory most of the time -- for such Lua obviously +is a better choice, because it's much more lightweight and minimalistic. + -- link:Ichthyostega[] [[DateTime(2008-09-30T02:17:08Z)]] +Many Years Later +~~~~~~~~~~~~~~~~ +(See link:https://issues.lumiera.org/ticket/134[Ticket #134]) +Hereby I *overrule* and *reject* the decision to support Lua or any +other scripting language directly; this topic shall be moved back into discussion later. + +After careful consideration, and weighting in my experience as professional developer, +I came to the conclusion that we want _scriptability of the application,_ yet turning +the application itself into a multi-language codebase, even more so using any kind of +``easy going'' dynamically typed language, is detrimental to longevity. + +Developers, as non-developers alike, tend to foster the dream of a fluid limitless +expression, a technology that just makes our intention flow into reality, be it through +the promise of new fancy languages, the ability for ad hoc extensions, the reliance on +almighty frameworks or even some kind of artificial entity able to guess what we want, +or anything to relieve us from the pain of spelling out clearly what we aspire, +with all the consequences and limitations of reality. + +Building a coherent architecture with clean and understandable interfaces is hard work. +There is no shortcut around that, and the only path towards a scriptable application is: + +- build a coherent architecture with clean interfaces and _well defined functionality_ +- build a script-runner component with the ability to actuate and control the application +- cast the abilities of this script-runner in terms of a clear self-explanatory interface +- define a binding into the object model of one or several scripting languages. +- build test coverage both for the interface and the language binding. + +This is damn hard work and the very opposite of the idea underlying the original proposal, +which to my understanding was to open up internals of the application for easy prototyping, +while just promising strict design work for later. To quote ``Things which usefulness is +doubtful can be prototyped and tried out in a afternoon rather than a week''. + +Based on my experience, this is a common anti-pattern. If something is of doubtful usefullness, +and requires a week to be built properly, you should rather spend some hours to write a clear +specification in natural language, instead of sneaking in a half-baked prototype; chances +are that this prototype will stick, since the inventor likes the basic idea, but wants +to avoid the hard thought work to turn that idea into something solid; and because a clear +specification is lacking, it is hard to test anything other than the happy path; rather +the new feature will be amended and bashed into submission, and then further new and +exciting additions will be based on it, progressively corroding the application. Any +attempt to rework confused code into something coherent becomes exponentially expensive, +the more it gets tangled with further immature code and ideas not spelled out well. + -- link:Ichthyostega[] [[DateTime(2023-02-04T01:07:51Z)]] + Conclusion ---------- -Lua is '''accepted''' as the required scripting language by October.2008 dev -meeting. +Lua was _accepted_ as the required scripting language by October.2008 dev +meeting. However, Ichthyo _questions and overrules_ this decision in Feb.2023 +and moves this proposal back into the inception stage. Back to link:/documentation/devel/rfc.html[Lumiera Design Process overview] + diff --git a/doc/devel/rfc_final/ScriptingLanguage.txt b/doc/devel/rfc_pending/ScriptingLanguage.txt similarity index 100% rename from doc/devel/rfc_final/ScriptingLanguage.txt rename to doc/devel/rfc_pending/ScriptingLanguage.txt From 2eb82d1da020f96ee93569300420d8f016e68072 Mon Sep 17 00:00:00 2001 From: Ichthyostega <prg@ichthyostega.de> Date: Thu, 9 Feb 2023 01:05:42 +0100 Subject: [PATCH 12/12] Rfc: clarify my reasons for the rejection After sleeping some nights over it, rework the wording to make my reasoning more clear and remove any possibly insulting undertone. I have seen what I describe here, happening over and over again -- and several times I myself was the one cooking up "simplifications", which caused lots of pain further down the road. --- doc/devel/rfc/ScriptingLanguage.txt | 92 +++++++++++++++++------------ 1 file changed, 53 insertions(+), 39 deletions(-) diff --git a/doc/devel/rfc/ScriptingLanguage.txt b/doc/devel/rfc/ScriptingLanguage.txt index b408eb836..f1a39c5ae 100644 --- a/doc/devel/rfc/ScriptingLanguage.txt +++ b/doc/devel/rfc/ScriptingLanguage.txt @@ -8,14 +8,14 @@ Scripting Language ------------------ -Add support for the 'Lua' scripting language in Lumiera. +Add support for the *Lua* scripting language in Lumiera. Description ~~~~~~~~~~~ We talked since the beginning about that we want to have scripting support within Lumiera. Some weeks ago we did a non formal decision on IRC to bless Lua -as 'official' scripting language. +as »official« scripting language. Tasks @@ -60,21 +60,22 @@ Comments -------- To make it more clear: the intention is to have the scripts call into well -defined Intefaces / API functions, which are accessed via the plugin system. It -is ''not'' intended to have just arbitrary scripts anywhere and everywhere, but +defined Interfaces / API functions, which are accessed via the plugin system. It +is _not_ intended to have just arbitrary scripts anywhere and everywhere, but -- on the other hand -- all important functionality which can be activated via the GUI should be similarly accessible via the scripting APIs without restrictions. So, as Python and Ruby and Perl are popular scripting language, -we'll have the neccessary bindings sooner or later. +we'll have the necessary bindings sooner or later. -Beyond that, it is possible to have some ''extension points'' where +Beyond that, it is possible to have some _extension points_ where configuration is added to the application in a well defined manner by scripts. These scripts provide for some basic customisation and can even add some of the important higher-level features. With respect to these, the idea of this -proposal is to have one ''required scripting language'', so scripts in this +proposal is to have one _required scripting language_, so scripts in this language are guaranteed to work and may be used to add essential features. I consider Lua to be an almost perfect fit for this purpose. - -- link:Ichthyostega[] [[DateTime(2008-07-27T22:36:52Z)]] + +Ichthyostega:: '2008-07-27T22:36:52Z' Well my intention is to make Lua a real first class binding where any internal interface gets exported and would be useable from scripting, that contradicts @@ -95,17 +96,20 @@ your limitation to make is only an extension language; but hold on: * of course if this is used wrong it can really damage the health of the system, but I think this is oblivious and very explicit, there are easier ways to damage it, just whack your computer with a sledgehammer for example. - * there might some lazyness to keep prototypes in Lua instead reimplement them + * there might some laziness to keep prototypes in Lua instead reimplement them properly in C/C++, well IMHO that's OK, at some point need will arise to make it proper, if the Lua implementation is insufficient, but that's arguable. - -- link:ct[] [[DateTime(2008-07-30T16:22:32Z)]] + +ct:: '2008-07-30T16:22:32Z' + I have no problems using Lua. It is proven in the industry, well supported, fast, efficient, high level and designed for this purpose. My only "complaint" is that Lua isn't my pet language (Scheme). And that really isn't a complaint at all. - -- link:PercivalTiglao[] [[DateTime(2008-07-28T19:56:25Z)]] + +PercivalTiglao:: '2008-07-28T19:56:25Z' I think Python should be reconsidered: it's given that all languages in this @@ -122,18 +126,20 @@ get this into professional production houses, then I think having a single language from OS admin the whole way through the stack is a massive gain for the types of users who will be using it. I personally prefer Ruby. Naturally it's your decision to make, all the best, we are looking forward to alphas and -betas in the future - -- mytwocents +betas in the future + + -- *mytwocents* -This proposal is about the ''required'' scripting language, i.e. when + +This proposal is about the _required_ scripting language, i.e. when accepted, Lua will be a necessary prerequisite for running Lumiera. This -doesn't rule out the ''use'' of other scripting languages. We strive at +doesn't rule out the _use_ of other scripting languages. We strive at having clean interfaces, thus it shouldn't be much of a problem to create Python bindings. And given the popularity of Python, I guess it won't be long -until we have some Python bindings. But ''requiring'' Python would mean +until we have some Python bindings. But _requiring_ Python would mean having a Python runtime in memory most of the time -- for such Lua obviously is a better choice, because it's much more lightweight and minimalistic. - -- link:Ichthyostega[] [[DateTime(2008-09-30T02:17:08Z)]] + +Ichthyostega:: '2008-09-30T02:17:08Z' Many Years Later @@ -147,38 +153,46 @@ I came to the conclusion that we want _scriptability of the application,_ yet tu the application itself into a multi-language codebase, even more so using any kind of ``easy going'' dynamically typed language, is detrimental to longevity. -Developers, as non-developers alike, tend to foster the dream of a fluid limitless -expression, a technology that just makes our intention flow into reality, be it through -the promise of new fancy languages, the ability for ad hoc extensions, the reliance on -almighty frameworks or even some kind of artificial entity able to guess what we want, -or anything to relieve us from the pain of spelling out clearly what we aspire, -with all the consequences and limitations of reality. +I can understand -- even sympathise with the original proposal. + +Developers, as non-developers alike, tend to foster the dream of a fluid and limitless +technology of expression, a technology that just makes our intention flow into reality, +be it through the promise of new fancy languages, the ability for ad hoc extensions, +the reliance on almighty frameworks or even some kind of artificial entity able to guess +what we want -- anything to relieve us from the pain of writing down and spelling out +clearly what we aspire, with all the consequences and limitations of reality. Building a coherent architecture with clean and understandable interfaces is hard work. There is no shortcut around that, and the only path towards a scriptable application is: -- build a coherent architecture with clean interfaces and _well defined functionality_ +- build a coherent architecture first, with _well defined functionality..._ - build a script-runner component with the ability to actuate and control the application - cast the abilities of this script-runner in terms of a clear self-explanatory interface - define a binding into the object model of one or several scripting languages. - build test coverage both for the interface and the language binding. -This is damn hard work and the very opposite of the idea underlying the original proposal, -which to my understanding was to open up internals of the application for easy prototyping, -while just promising strict design work for later. To quote ``Things which usefulness is -doubtful can be prototyped and tried out in a afternoon rather than a week''. +Thus, as far as _scriptability_ is concerned, we have yet a long way to go indeed. -Based on my experience, this is a common anti-pattern. If something is of doubtful usefullness, -and requires a week to be built properly, you should rather spend some hours to write a clear -specification in natural language, instead of sneaking in a half-baked prototype; chances -are that this prototype will stick, since the inventor likes the basic idea, but wants -to avoid the hard thought work to turn that idea into something solid; and because a clear -specification is lacking, it is hard to test anything other than the happy path; rather -the new feature will be amended and bashed into submission, and then further new and -exciting additions will be based on it, progressively corroding the application. Any -attempt to rework confused code into something coherent becomes exponentially expensive, -the more it gets tangled with further immature code and ideas not spelled out well. - -- link:Ichthyostega[] [[DateTime(2023-02-04T01:07:51Z)]] +However, this »scripting language« proposal seems to take quite a different, if not antithetical +approach (as confirmed by the ensuing discussion and the tickets), namely to open up internals +of the application for easy prototyping, while just promising strict design work for later. +To quote ``Things which usefulness is doubtful can be prototyped and tried out in a afternoon +rather than a week''. + +As compelling as it may sound -- based on my experience, this is a common anti-pattern: +If something is of doubtful usefulness, and requires a week to be built properly, you should +rather spend an afternoon to write a specification in plain natural language, instead of sneaking in +a half-baked prototype; if you can not write down your intentions in clear terms, using simple language +and coherent terminology, chances are that the code you write will be confused and tricky. And worse, +this kind of sketchy code has the tendency to stick; the more it needs to be amended and bashed +into submission, the higher the emotional investment. And soon further new and exciting additions +will be based on it, progressively corroding the application. + +Adding _more_ technology, like adding _yet another_ language or library to the mix, can never +be a means of simplification -- it is an investment rather, and has to pay off. When you +``move fast and break things'', you end up with lots of broken stuff -- unless you have the +actual capacity to clean up the mess and build everything from scratch, sound and solid. + +Ichthyostega:: '2023-02-04' ~<prg@ichthyostega.de>~ Conclusion