LUMIERA.clone/doc/technical/overview.txt

Lumiera: The Inner Core
=======================
:Author: Lumiera_Core_Developers
:Date: 2012 / 2025
:toc:

[abstract]
******************************************************************************
The Lumiera Developers have a distinct vision about the core of a modern NLE.
This document outlines some of the basic technical concepts and highlights
the fundamental design decisions. New Developers may find this Document useful
to get an idea about how the different components work together.
******************************************************************************


Overview
--------

Lumiera constitutes of a broad range of subsystems. These are roughly grouped
into three layers plus some extra components. This structure is mostly kept in
the source directory structures.

This three Layers are:

 The Stage Layer::
     Interaction and presentation. User interfaces are implemented as plug-ins,
     and while most commonly one will see the GTK UI, it would be possible to
     construct a entirely different user interface, a CLI interface, or even
     operate the application ``headless'', script driven.

 The Steam Layer::
     Keeps the Session, generates the rendering graphs for sequences,
     arranges what to do when and how.

 The Vault Layer::
     Manages worker pools, schedules jobs, does the memory management for the
     heavy multimedia data. Loads and delegates to external libraries for
     media processing.

The extra components are:

 Lumiera::
     The main program itself, which starts all Subsystems and controls the
     overall lifecycle.

 Common::
     Vital components and common services which must be available for pulling up
     any part of the system.

 Library::
     A lot of (largely) stateless helper functionality used throughout the code.


Coding Style
~~~~~~~~~~~~

The Lumiera team agreed on using GNU coding style with slight adaptations.
Generally speaking, we strive to keep the code base consistent and stick to
widely accepted guidelines and best practices. See our separate
link:{ldoc}/technical/code/codingGuidelines.html[Coding Guidelines] page.



Documentation
~~~~~~~~~~~~~
The central location for all design and technical documentation is the Lumiera
website you're reading right now. Besides that, a summary and introduction
for various components can be found in the file-level doxygen comments, while
details are usually explained in the class and function level comments.

==== the Development TiddlyWiki
Currently, Lumiera is still in the design- and evolution phase. Documentation
is written as an ongoing effort.
There is an embedded JavaScript wiki (TiddlyWiki) within the source tree, mostly
used as design notebook, featuring day-to-day design sketches and notes, but also
quite some more persistent planning. Finished documentation text is constantly
moved over to the documentation section(s) of the Lumiera website. +
-> access the development link:/wiki/renderengine.html[TiddlyWiki online here]


Test Driven Development
~~~~~~~~~~~~~~~~~~~~~~~

We strive to use _Test-Driven-Development:_ tests are to be written first,
defining the specification of the entity being tested. Then things get
implemented until they pass their tests. While increasing the initial
development effort significantly, this approach is known to lead to
clearly defined components and overall increases code quality.
In practice, this approach might not be suitable at times,
nevertheless we try to stick to it as far as possible
and maintain fairly complete test coverage.

Furthermore, tests are a vital form of documentation: when done right,
a test engages the functionality by accessing and combining the interfaces
of the code. For many central elements of the Lumiera Application, some tests
are written in a ``narrative style'' to explain how parts are meant to fit
together and shall be used. This kind of test does not focus on covering
all corner cases -- rather it is organised like a trail to showcase
the central ideas and basic assumptions of a component.


Releases and Development Process
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lumiera is developed in a collaborative style. Releases are done ``when ready''.
Feature development and releases are organised using the
link:code/GitBranching.html[Git-flow] branching scheme.
We (the core developers) are always open for comments, new ideas and proposals.
In fact, there is even a world-pushable ``mob'' Git repository on our server:
minor contributions can be pushed there right away, we'll notice any changes.
It might be a good idea to discuss larger changes prior to jumping into the
code though. Our Mailinglist is the main information hub. Users and developers --
everyone is welcome.

Lumiera should be usable on any contemporary Linux (or similar *NIX) system.
Porting to other platforms is OK for us, but requires Someone^TM^ to volunteer.
For development and dependency management, we try to stick close to what
Debian/stable provides. Whenever we need special or more recent libraries,
we care for a backport and publish a DEB package for use on Debian/Stable.
We also run a series of link:/debian/[Debian Repositories], and we try to build
a Lumiera package for an array of Debian-like distributions (Ubuntu, Mint).
Through these ongoing efforts, we try to keep up with the evolution of platform,
compilers, language and libraries, and we maintain our ability to
release, even while there are no public releases planned
for the foreseeable future.



Lumiera Application
-------------------
Generally speaking, the Application is comprised of several self contained
_subsystems_, which may depend on each other. Dependencies between components
are to be abstracted through interfaces. Based on the current configuration,
the application framework brings up the necessary subsystems and finally
conducts a clean shutdown. Beyond that, the application framework remains
passive, yet provides vital services commonly used.

While the core components are linked into a coherent application and may
utilise each other's services directly based on C/C++ language facilities,
several important parts of the application are loaded as plug-ins, starting
with the GUI.


Stage: The User Interface(s)
----------------------------

The purpose of the user interface(s) is to act on the _high-level data model_
contained within the Session, which belongs to the _steam layer_ below.
User interfaces are implemented as plug-ins and are pulled up on demand,
they won't contain any relevant persistent state beyond presentation.

_As of 2023, the one and only interface under active development is
the Lumiera GTK GUI,_ based on GTK-3 / gtkmm. The sources are in tree
(directory 'src/stage') and it is integrated into the standard build and
installation process. By default, running the 'lumiera' executable will
load and start this GUI as a Lumiera module from 'modules/gtk_gui.lum'



Steam Layer
-----------

High Level Model
~~~~~~~~~~~~~~~~
All the content of the Project is represented as a tree of symbolic objects,
connected together through _Placements._ There are some general conventions
how these objects are expected to be arranged; any content fitting with these
conventions will be discovered, interpreted and transformed into something
that can be rendered and thus ``performed''

_tbw_

Assets
^^^^^^
_tbw_

Placement
^^^^^^^^^
_tbw_

Scoping
^^^^^^^
_tbw_

QueryFocus
^^^^^^^^^^
_tbw_

Output Management
~~~~~~~~~~~~~~~~~
_tbw_

Stream Type System
~~~~~~~~~~~~~~~~~~
_tbw_

Command Frontend
~~~~~~~~~~~~~~~~
_tbw_

Defaults Manager
~~~~~~~~~~~~~~~~
_tbw_

Rules System
~~~~~~~~~~~~
_tbw_

Builder
~~~~~~~
_tbw_

Low Level Model
~~~~~~~~~~~~~~~
The content of the current Session (the Project) can be _performed_ by
pulling from a set of render pipelines. These are generated automatically
By the Builder and represented as a **D**irected **A**cyclic **G**raph of
_Render Nodes_, chained together, so that the output of one node becomes the
input of the next node.
Each Render Node is backed by a processing function, which is provided by
a »Media processing library« and typically loaded dynamically, as an extension
or Plug-in. The Low-level Model is optimised for rendering performance; the
user can control its structure and contents by re-arranging the objects in
the High-level Model.

_tbw_


Play/Rendering subsystem
~~~~~~~~~~~~~~~~~~~~~~~~
Within Lumiera, »Player« is the name for a subsystem responsible for organising and tracking
_ongoing playback and render processes._ The player subsystem does not perform or even manage
any render operations, nor does it handle the outputs directly. +
Yet it addresses some central concerns:

uniformity::
 All playback and render processes are on equal footing, handled in a similar way.

integration::
 The player cares for the necessary integration with the other subsystems
+
it consults the _Output Management,_ retrieves the necessary informations from the _Session_
and coordinates the forwarding of Vault-Layer calls.

time quantisation::
 The player translates continuous time values into discrete frame counts.
+
To perform this _quantisation,_ the help of the session for building a TimeGrid
for each output channel is required.


The player service
^^^^^^^^^^^^^^^^^^
Client code accesses the player (subsystem) through the play-facade (`lumiera::Play`).
The exposed service allows to _set up an output connection for playback or rendering,_
resulting in a play-controller object.

.Play::Controller
This controller frontend represents the presence of such an active output connection
and incorporates a state machine supporting the usual things you'd expect to do with
a player (Play, pause, FFwd, Rew, scrubbing, jumping, looping). This controller object
is a copyable smart-handle -- all instances act as if wired in parallel.

.time control
The play-controller frontend makes heavy use of `time::Control`. This is a mediator
to accept and forward _mutations_ on time values and time ranges, possibly involving
frame quantisation. After attaching it to a target time value, it accepts changes,
offsets and nudging, translates these into the appropriate target modifications
and notifies any attached _change listeners_.

.play process
Ongoing effort to calculate a stream of frames for playback or rendering. +
The play process is an conceptual entity linking together several activities in the vault
and the render engine. It maintains a registration entry for the process to keep track of
associated entities, resources allocated and calls dispatched as a consequence. Besides
each play process is wired to at least one play-controller acting as frontend interface
and information hub for the client code.

NOTE: the player is in no way engaged in any of the actual calculation and management tasks
necessary to make the stream of calculations actually happen. The play process code contained
within the player subsystem is largely comprised of organisational concerns and not especially
performance critical.

- the vault is responsible for dispatching the calculation stream and scheduling calculation jobs
- the render engine has the ability to carry out individual frame calculations
- the OutputSlot exposed by the output manager is responsible for accepting timed frame delivery




Vault Layer
-----------

Engine Interface
~~~~~~~~~~~~~~~~
While on itself just a thin interface and adaptation layer forwarding calls to
the primary vault facilities, the Engine Interface is the primary point of service
accessed by Steam-Layer to use the vault layer services for rendering content.

.Calculation Streams
The Engine Interface is cast in terms of an _calculation stream_ entity. This is
a stream of expected and ongoing frame calculations for multiple channels, to be
managed as a compound. The calculated frames will be delivered into an output slot
right away. No assumptions are made regarding the ordering of these individual
calculations -- they may be performed in parallel, constrained by input and
resource prerequisites solely.

.Frame Dispatcher
For the actual processing, calculation streams need to be translated into individual
calculation jobs to be scheduled. For each uniform _segment of the effective timeline,_
the typical recursive descent call characteristic for _pull processing_  results in a
Job Ticket.

.Job Ticket
This structural descriptor of the actual calculations to be performed is the base
for creating individual jobs: Already specialised for a distinct segment of the
effective timeline and tailored for the needs of a given calculation stream,
the job ticket acts as blueprint for the actual jobs to be enqueued
with the _Scheduler._


Scheduler
~~~~~~~~~
The Scheduler serves as the central hub in the implementation of the RenderEngine
and coordinates the processing resources of the application. Its purpose is to
invoke and control the dependency and time based execution of the _Render Jobs._

Regarding architecture, the Scheduler is located in the Vault-Layer and running the Scheduler
is equivalent to activating the »Vault Subsystem«. An EngineFaçade acts as entrance point,
providing high-level render services to other parts of the application: render jobs can be
activated under various timing and dependency constraints, which can be grouped
into three basic modes of control:

Deadline::
 Higher priority jobs ordered by a deadline time plus some (negative) hysteresis.
 Jobs are started when they approach their deadline. When it is foreseeable that a
 jobs will miss the deadline, it will be skipped, preferring to use resources on
 calculations expected to be successful and on time.

Best Effort::
 Background calculations will use excess resources of the system;
 a typical example would be the calculation of preview images and sound waveform
 outlines for the GUI.

Freewheeling::
 Calculation for final rendering will use all available resources and never
 skip any processing, irrespective of the time it takes to complete.

Internally, the implementation of the Scheduler is organised in two layers:

Layer-2: Coordination::
  Maintain a network of interconnected activities, track dependencies and observe
  timing constraints; coordinate a pool of _active Workers_ to dispatch the next activities.

Layer-1: Invocation::
  Operates a low-level priority scheduling mechanism for time-bound execution of activities.

Job
^^^
A _Render Job_ is prepared as a self-contained functor, with a well defined time window
of execution. A job, once triggered, will run to completion unattended and can not be aborted.
The scheduler is able to observe and maintain dependencies between render jobs, allowing
to break down an extended calculation into a chain of small steps.


Resource Management
~~~~~~~~~~~~~~~~~~~

Running Lumiera requires a lot different resources, such as CPU-Time, Threads,
IO Bandwidth, Memory, Address space and so on. Many of this resources are rather
hard limited and the system will return errors when this limits are hit, but
often one does not even reach this hard limits because performance will
degrade way before coming into the realm of this limits. The goal for Lumiera
is to find a sweet spot for operating with optimal performance. Thus we have
some facilities to monitor and adjust resource usage depending and adapting to
the system and current circumstances.





Common Services
---------------

Subsystem runner
~~~~~~~~~~~~~~~~
Conceptually, the application is structured into layers. However, as actual runtime
entities we rely on a small number of link:{ldoc}/design/architecture/Subsystems.html[Subsystems],
which can be started and stopped and have well defined dependencies:

- Engine
- Session and Builder
- Play and Render Subsystem
- Lumiera GUI
- Script runner [yellow-background small]#planned#
- Renderfarm node [yellow-background small]#planned#

Lumiera's `main()` function defines an _Application Object_, which in turn operates
the _Subsystem Runner_. Each Subsystem implements a descriptor with the necessary
hooks callbacks to be started and stopped when necessary.

Lifecycle events
~~~~~~~~~~~~~~~~
Beyond parsing the commandline and triggering start of the subsystems, the Application
Object also issues _Lifecycle Events_. Individual components can register a callback
to be invoked when this happens. Currently (2018) the following events are known
(see 'include/lifecycle.h'):

- ON_BASIC_INIT
- ON_GLOBAL_INIT
- ON_GLOBAL_SHUTDOWN
- ON_EMERGENCY_EXIT

-> see here for
link:{ldoc}/design/application/SubsystemLifecycle.html#_initialisation_and_lifecycle[explanation in detail]
[small]#(somewhat preliminary and messy, yet still valid as of [yellow-background]##2018##)# +
-> more
link:{ldoc}/design/architecture/Subsystems.html#_lifecycle[documentation]
[small]#(likewise incomplete, but we're getting there, eventually [yellow-background]##2018##)#


Plugin loader
~~~~~~~~~~~~~
_tbw_

Advice framework
~~~~~~~~~~~~~~~~
This is a ``whiteboard'' system, allowing implementation-level services to _publish_
some piece of information as _advice_, while other parts of the system may pick up
this advice just by a name token, without requiring a direct knowledge of the
original _advisor._ The _Advice System_ is a singleton service maintaining a
lookup and registration data structure. Individual _piece of advice_ elements
are stored _as value copy_. Publishing new advice requires locking, but accessing
advice is lock-free (actually there needs to be a memory barrier ``somewhere'',
otherwise the advice requesting client might not see new advice)

.Advice topics
Advice is organised into categories, based on the type of the advice item and
some additional symbolic identifiers. Actually these are syntactically represented
similar to the _atoms_ of a rules based system (``Prolog syntax''). Currently (2010)
only ground terms (completely static symbols) are supported. But the intention is to
extend the system to allow for variables in these terms. This will turn the matching
of advice provisions and requests into an unification, allowing the advice item to
be parametrised.


Rules system
~~~~~~~~~~~~
_tbw_

Serialiser
~~~~~~~~~~
_tbw_




Library
-------

The Lumiera support library contains lots of helper functionality
factored out from the internals and re-used. It is extended as we go.

practical shortcuts
~~~~~~~~~~~~~~~~~~~
The header 'lib/util.hpp' defines some shortcuts heavily used throughout
the code base. The idea is to highlight a common semantic meaning, while
hide differentiation on the technical details.

isnil:: indicate a _missing value_, irrespective if this is a NULL pointer,
  an empty string or an empty container. Several of our custom wrappers also
  support this notion.

contains:: indicate that a value is _somehow contained_ within a collection,
  irrespective if this is a set, a map, a vector or string (substring test)

sanitise:: make any string usable as identifier.



Locking
~~~~~~~
General purpose Locking is based on object monitors. Performance critical code
in the Render Engine relies on __»Atomics«__, or may use futures and further
synchronisation primitives through the wrappers of the C++ standard library.

NOTE: At the time when Lumiera project started, the standard library provided
      no portable support for concurrency and locking. As a remedy, various
      solutions were built on top of the POSIX primitives. Some of these
      ad-hoc solutions turned out to be viable and were reworked and adapted,
      as proper concurrency support became part of the language standard.
      For new code, the direct use of POSIX primitives is discouraged.

Time
~~~~
Time values are represented by a family of opaque date types
with overloaded operators. The implementation was inspired by `gavl_time_t`
and is thus based on a µ-seccond time grid, represented as 64-bit integer.
Thus, the arithmetic on time values and time spans is limited and any Time
handling and conversion is centralised in library routines.

We distinguish between time values and a _quantisation_ into a frame
or sample grid. In any case, quantisation has to be done once, explicitly
and as late as possible. See the link:{rfc}/TimeHandling.html[Time handling RfC].

.time values
The Lumiera library defines several flavours of time values. All of
these internal time values have in common that they are _opaque_ and not
directly related to any human readable or external (wall clock) time.
Moreover, most of these time values are immutable, yet there are two
mechanisms to allow for changing time values (TimeVar and Mutation).

.quantised time
Special flavours of these time values additionally carry the reference
to an (frame) alignment grid, while being time value entities in all other
respects. But _only those quantised time values_ expose functions to
convert the internal opaque time into a human readable or externally
relevant time format -- including SMPTE or frame counts.

.time (alignment) grid
Thus, any usage of time values is forced to refer to such a time alignment
grid explicitly, at least when leaving the realm of the internal opaque
time values. This is the point where the *time quantisation* is actually
performed, imposing some information loss (as any rounding operation does). +
A time alignment grid is exactly that: a set of functions to perform
this lossy conversion. Implicitly this involves the definition of an
_time origin_ (reference point where the external time is zero), and
typically this also includes the definition of a _frame rate_ (but
in the most general case, this frame rate might be variable and
change at various places of the time axis). Consequently, all time
grids are Assets and defined as part of the concrete session.


Time Code
^^^^^^^^^
Historically, Time Code was seen as the foundation of any film editing.
Similarly, the first generation of editing systems used Time Code as a
foundation. Today, we consider such a design as questionable.

Lumiera takes a different approach here: Time code is reduced to a mere
mode of presentation, i.e. a formatting of existing time values. It is
void of any substantial meaning. To the contrary, the operation of
_frame quantisation_ (see above) is considered to be fundamental,
causing a irreversible loss of information. The design of time handling
chosen within Lumiera forces you to decide on using a specific _time grid_,
prior to being able to format an internal (opaque) time value in any kind
of time code. And only as late as when _actually retrieving_ this time code
formatted value, the actual quantisation (grid alignment) happens.

In practice, establishing a time grid requires knowledge about the output
format. Thus, an (sufficiently resolved) *output designation* is required
to perform any grid alignment a and time code formatting. Typically this
happens when a timeline or similar part of the High-Level-Model is connected
to a concrete output or an global bus defining a frame rate already. The model
contents as such are _frame rate independent_.

The following time code formats are supported, both for programmatic access
and display within the GUI

- frame count
- SMPTE
- SMPTE drop-frame _[,yellow]#TODO# as of 2011_
- hours:mins:secs  _[,yellow]#TODO# as of 2011_
- fractional seconds  _[,yellow]#TODO# as of 2011_
- musical bars:beats _[,yellow]#TODO# as of 2011_

As a corollary, as any rendering is based on frame numbers, it requires an
output connection or something similar to establish a frame grid.


Errors
~~~~~~

 * As a Rule, Exceptions + RAII are to be preferred over error
   codes and manual cleanup. At external interfaces we rely on
   error states though.
 * Exceptions can happen everywhere and any time
 * Exceptions and Errors shall only be dealt with at locations where it's actually
   possible to _handle_ them (the so called ``fail not repair'' rule)
 * API functions are categorised by _error safety guarantee_
   - *EX_FREE* functions won't raise an exception or set an error state,
     unless the runtime system is corrupted.
   - *EX_STRONG* functions are known to have no tangible effect in
     case of raising an exception / error state (transactional behaviour).
   - *EX_SANE* functions might leave a partial change, but care to leave
     any involved objects in a sane state.
 * Error states are identified by pointers to static strings.
 * Error states are thread local and sticky (a new state can't be set
   unless a pending state got cleared).


Exception hierarchy
^^^^^^^^^^^^^^^^^^^
Typically, when an error situation is detected, the error will be categorised
by throwing the appropriate exception. Exceptions provide a mechanism to attach
the root cause. The classification happens according to the _relevance for the
application_ as a whole -- _not_ according to the cause.

 `lumiera::Error`:: root of error hierarchy. extends `std::exception`
 `error::Logic`::   contradiction to internal logic assumptions detected
 `error::Fatal`::   (⤷ `Logic`) unable to cope with, internal logic floundered
 `error::Config`::  execution aborted due to misconfiguration
 `error::State`::   unforeseen internal state (usually causes component restart)
 `error::Flag`::    (⤷ `State`) non-cleared error state from C code
 `error::Invalid`:: invalid input or parameters encountered
 `error::External`:: failure in external service the application relies on
 `error::Assertion`:: assertion failure

Please be sure to clear the error state whenever catching and handling an exception.


Error states
^^^^^^^^^^^^
Errors states get declared in headers with the `LUMIERA_ERROR_DECLARE(err)` macro.
A matching definition needs to reside in some translation unit, using the
`LUMIERA_ERROR_DEFINE(err, msg)` macro. There is no central registry, any component
can introduce its own error codes but must ensure that the error identifier is
unique. An error state is maintained in a thread-local variable, and set automatically
when throwing an exception. Error states can be helpful when diagnosing unexpected
failures, and can be used to ``sneak'' an error condition through a call-chain
of plain-C functions, but are no longer considered a suitable concept for error
handling in general.




Singletons and Dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Deliberately, Lumiera stays below the level of utilising a dependency injection framework.
Consequently, we access most services _by type name_, pulling up a single implementation
instance on demand. Yet some aspects of the *Singleton Pattern* are known to be problematic.
Rather than placing this singleton lifecycle logic into the individual implementation classes,
we use a _singleton factory_, to manage the instance creation and lifecycle of the singleton
or service instance separate from the singleton class itself, as a *Monostate*. Singleton
initialisation happens lazily with this approach, and thus needs to be protected in a
multithreaded environment; we use Double Checked Locking with `std::atomic` for this purpose.

Singletons need to be written in a way to cope with that somewhat nondeterministic environment.
They will be created on-demand, and be destroyed _eventually,_ in the application shutdown phase.
This results in the general policy within Lumiera that performing any ``business'' code in the
application shutdown phase (after exiting `main()`) is _strictly prohibited._ Generally speaking,
destructors _must not perform any significant work_ and are are expected to be failsafe.

.accessing the singleton instance
By convention, when clients are expected actively to access the singleton instance,
the interface class holds the singleton factory as a public static member with the
name `instance`. This allows clients to write `SomeService::instance()` to get a
reference to the implementation.

.subclasses and services
Actually, what appears as a mere ``singleton'' here is a form of _Dependency Injection,_
and can easily be extended to cover other forms of dependencies and other types of lifecycle.
Our experience indicates that this is rarely necessary, but when it happens, it's essential.
Lumiera's Dependency-Factory thus offers a configuration mechanism, which allows to switch
those modes of operation for individual types; the client consuming the ``singleton'' remains
unaware of such special setup.

.mock testing support
Moreover, the Dependency-Factory also offers a mechanism to ``push aside'' the existing singleton
or service instance and shadow it temporarily with a mock implementation. Again, it turned out that
such mocking mechanisms are used only occasionally and rarely required. Likely this is a result of
Lumiera being developed _test-driven_ -- things are being written mostly in a unit test friendly way.

-> more about link:{ldoc}/technical/library/Dependencies.html[dependency handling]


Visiting Tool
~~~~~~~~~~~~~
_tbw_

Iterators
~~~~~~~~~
Iterators serve to decouple a collection of elements from the actual data type
implementation used to manage those elements. The use of iterators is a
design pattern.
-> see link:{ldoc}/technical/library/iterator.html[detailed library documentation]

Lumiera Forward Iterator
^^^^^^^^^^^^^^^^^^^^^^^^
Within Lumiera, we don't treat _Iterator_ as a base class -- we treat it as a  _concept_
for generic programming, similar to the usage in the STL. But we use our own definition
of the iterator concept, placing the primary focus on interfaces and decoupling.
Our ``Lumiera Forward Iterator'' concept deliberately removes most of the features
known from the STL. Rather, such an iterator is just the promise for pulling values
_once_. The iterator can be disposed when _exhausted_ -- there is no way of resetting,
moving backwards or doing any kind of arithmetic with such an object. The _exhausted
state can be detected by a +bool+ conversion (contrast this with STL iterators, where
you need to compare to an +end+ iterator). Beyond that, the usage is quite similar,
even compatible to +std::for_each+.

Iterator Adapters
^^^^^^^^^^^^^^^^^
We provide a collection of pre defined adapter templates to ease building
Lumiera Forward Iterators.

- a generic solution using a _iteration control_ callback API
- the `lib::RangeIter` just wraps up a pair of iterators for ``current position''
  and ``and'' -- compatible with the STL
- there is a variant for automatically dereferencing pointers
- plus a set of adapters for STL containers, allowing to expose each value, each
  key, distinct values and so on.

Iterator Adapters are designed for ease of use, they don't conceal the underlying
implementation (and the actual type is often quite convoluted).

Iteration Sources
^^^^^^^^^^^^^^^^^
To the contrary, the `lib::IterSource<TY>` template is an abstract base class.
This allows to expose the promise to deliver values through any kind of API, without
disclosing the actual implementation. Obviously, this requires the use of virtual
functions for the actual iteration.

Again, there are pre-defined adaptors for STL containers, but the actual container
is concealed in this case.

Itertools
^^^^^^^^^
Iterators can be used to build pipelines. This technique from functional programming
allows to abstract away the actual iteration completely, focussing only on the way
individual elements are processed. To support this programming style, several support
templates are provided to build _filtering iterators, transforming iterators,_ to pick
only _unique values,_ to _take a snapshot on-the-fly_ etc. There are convenience
builder functions for those operations, figuring out the actual source and destination
types by template metaprogramming.

Pipeline Builder
^^^^^^^^^^^^^^^^
As an extension to the iterator framework, the pipeline builder is able
to wrap arbitrary source iterators, decorate them with filters and transformations
provided as Lambda, and finally package all together into a self contained new
iterator (value object). This is a rather new feature [small]#(as of 2017)# and
we still need to determine its viability, yet it proved helpful when implementing
certain kinds of search algorithms on top of an abstracted data source exposed
as iterator. -> 'lib/iter-explorer.hpp'


Front-end for boost::format
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Formatting values with `printf` is a notorious source for intricate errors.
Additionally, using (s|n)`printf` can be clunky in practice, and it doesn't
support custom defined string conversions, which are an important diagnostic
aid when working with objects. We might
link:{ldoc}/technical/howto/UsingBoost.html[use Boost], which provides the
`boost::format` library to address those problems. Unfortunately including this
header-only library solution incurs a significant overhead, both in terms of
compilation time and code size. While maybe still acceptable at the implementation
level, using boost::format is thus certainly a ``no go'' for any code residing
in headers frequently included.

To work around these problems, we provide a front-end wrapper, defined in
'lib/format-string.hpp'. This allows to keep the actual boost::format based
implementation confined to a single translation unit, while still being able
to use all primitive types as usual with boost::format or printf. Additionally,
our frontend automatically invokes a custom or built-in string conversion, if
applicable, it dereferences pointers and catches all errors encountered while
in formatting. So it's well suited for usage in error handling code.

[source,C]
------------------------------------------------------------
#include "lib/format-string.hpp"
using util::_Fmt;

double total = 22.9499;
const char * currency = "€";

cout << _Fmt("price %+5.2f %s") % total % currency << endl;
------------------------------------------------------------

WARNING: `boost::format` is known to be about 10 times slower than `printf` --
         best to avoid it for performance critical code.


Text Template Engine
~~~~~~~~~~~~~~~~~~~~
While there are a lot of text templating languages and feature-rich
libraries available, none fulfilled our requirements to be minimalist,
and support generic structured data bindings, without enforcing some
specific _property tree_ or ``object'' data type. The solution provided
in `text-template.hpp` supports substitution with the `${name}` syntax,
also with conditionals and repeated instantiation. Data is provided
through a type parameter, which has to fulfil a data binding protocol.
A ready-to-use implementation is available for `std::map<string>`,
key-value bindings given in a string (for the purpose of easy testing)
and for the Lumiera xref:_external_tree_description[»ETD«] framework
(i.e. structured data encapsulated as `GenNode`).

LL Parser
~~~~~~~~~
A similar situation was encountered regarding the requirement to parse
nested syntax structures. There is an abundance of library solutions
available, but none fulfilled the requirements of simplicity and
generic data binding. Some solutions are extremely heavyweight,
attempt to mimic an EBNF syntax notation, or deliver the result
model packaged into yet-another object or property tree data format.
The library solution provided in `parse.hpp` is based on the
technique of parser combinators, but uses template metaprogramming
to generate a functor to accept and produce that specific data
structure needed for each individual use case. The accepted syntax
is defined with a builder notation. Instead of a lexer,
regular expressions are used as terminal symbols.

CSV Data Tables
~~~~~~~~~~~~~~~
Template Metaprogramming also allows to build a succinct solution
for tabular data with generic typing, which was developed to support
performance testing and statistical evaluations. Data is maintained
in a vector per column, using a generic column definition scheme.
Data can be persisted and loaded from text files with CSV syntax.
The header `data.hpp` is complemented by `statistic.hpp` with some
basic evaluation functions and is used, for example, to generate
Gnuplot diagrams from timing observations, also relying on
the Text Template Engine to parametrise a Gnuplot script.

External Tree Description
~~~~~~~~~~~~~~~~~~~~~~~~~
It might seem convenient, yet is in fact dangerous to build a larger
application around a common shared data model. Such a central model
exhibits a fatal tendency to become an ubiquitous source of truth,
which would create an ill-guided incentive towards strong coupling.
According to the principle of subsidiarity, in Lumiera we prefer
treating a data model as an local detail of the implementation.

However, parts of the application need a way to collaborate by
exchanging chunks of tree-shaped information. A schematic and
formalised notation is used for that purpose, designated as
**E**xternal **T**ree **D**escription, which is structured
similar to JSON, but uses binary data. The implementation
is based on a recursive algebraic data type, but the variant
branches are kept opaque, so that processing can not proceed
by switch-on-selector and thus either requires knowledge about
the expected data pattern in advance, or must be rely on a visitor
implementation otherwise.

This structured data notation is used at various places, notably
to pass command invocations over the UI-Bus, or to send model changes
in the form of a »Diff message«.


Wrappers and Opaque Holders
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.smart handle
This pervasively used handle type is based on the reference counting
smart-pointer type of C\++ (`boost::shared_ptr` and C++11). Typically
this also includes a pointer to some kind of implementation service. +
Yet still, handles based on `lib::Handle<TY>` should not be confused with
smart pointers. Rather, we use the ref-counting mechanism to invoke a custom
clean-up callback when the last handle goes out of scope. Typically, the
implementation service is kept entirely opaque, while the copyable handle
objects also implement a front-end interface for client access.

.unified holder for value/ptr/reference
_tbw_

.ownership managing collection
When implementing services, frequently we encountered the situation that
a manager object creates and owns some component elements or sub-services.
The library provides two special collections, which also take ownership
of their contents and care for automatic clean-up. Especially, contrary
to the STL containers, those custom containers support use of
_non-copyable object_ (as a rule, all objects with reference semantics
are defined non-copyable in Lumiera).

- the `ScopedPtrVect` is a vector taking ownership of heap allocated objects
- the `ScopedCollection` is a fixed-size collection, holding all the child
  objects within a single (heap allocated) storage block


.opaque holder
There is a family of holder objects, all based on placement-new of the
contained object into an embedded buffer. The purpose is to ``piggyback''
an object inline, without the need for heap allocated storage. Frequently
the motivation for this usage pattern is *type erasure*: the detailed knowledge
context used to build some kind of object is discarded prior to further use,
relying on generic information and the hidden parametrisation henceforth.

.polymorphic values
The C++ language has direct support for _value semantics_ and allows to build
value objects to be treated as first class citizens. Unfortunately this doesn't
fit well with the chosen approach to object orientation, where polymorphism relies
on reference semantics. Thus, most of the fundamental design patterns drive us into
having an object manager somewhere hidden within the implementation level, to
manage the memory for maintaining the subclass instances to be concealed
at the usage site.

To avoid this dilemma, we utilise the technique of the opaque holder to provide
objects with value semantics, while actually placing the instance of a subclass
into the inline buffer. Clients access this embedded object by automatic type
conversion to the interface type, which gives us polymorphism. While the
definition of such a beast is quite involved, the runtime overhead is
surprisingly low. When compared with standard polymorphism, creating
objects and invoking operations has zero overhead, while copying
and assignment incur the cost of an additional virtual call,
assuming the target objects cooperate by mixing in a
copy management interface.


Unique Identifiers
~~~~~~~~~~~~~~~~~~

LUID
^^^^
Generating 128 bit non cryptographic strong unique identifiers.

 - having an alternative representation to store a pointer
 - may be extended for a strong (slow) and a fast (weak) variant in future

EntryID
^^^^^^^
Combines an user readable ID, a (compile time) type tag and a hash-ID.
The latter is based on the symbolic ID and the type tag, which is discarded
at runtime (type erasure)

Typed Lookup
^^^^^^^^^^^^
_[,yellow]#planned#_ a system of per-type lookup tables, based on `EntryID`, together
with an type specific access functor. Together, this allows to translate
transparently and typesafe from symbolic ID to object instance, which
is an prerequisite for integrating a rules based system. Besides, these
tables allow unique IDs per type +
-> more details about this concept link:/wiki/renderengine.html#EntryID%20TypedID%20TypedLookup[in the TiddlyWiki]


Allocators
~~~~~~~~~~
Lumiera utilises several custom allocators, each tailored to a specific purpose.
All these allocator-_frontends_ share a common pooling allocation backend

WARNING: currently (as of 2011) the low-level pooled allocator within the
         backend isn't implemented; instead we do just heap allocations.
         See Ticket #231

.Allocation Cluster
This allocation scheme is used within the context of the Builder and the
Low-Level-Model. The predominant usage trend in this realm is to create
and wire a family of small objects right after each other, within a build
process. These objects are intended to work together and will be discarded
all at once, after hot-swapping a new version of that model segment.

.Typed Allocation Manager
This allocation framework is used at various places when a large number of
similar objects is expected to be coming and going. New objects are
placement-constructed into the allocated space and immediately wrapped
with a ref-counting smart-ptr to manage ownership.

.BlockFlow (Epoch Allocator)
A custom memory management scheme for the Scheduler in the Render Engine.
The scheduler produces a _ongoing flow_ of _Render-Activity_ records, which
are added and wired, and then activated once based on temporal ordering.
For each frame calculation job and thus also for each supporting calculation
a _temporal deadline_ can be defined. This observation is exploited to order
the allocations into _extents based on deadline._ Once the deadline for such
an »epoch« has passed, all data records can be discarded without any further
checks or clean-up work.




Template Metaprogramming
~~~~~~~~~~~~~~~~~~~~~~~~

Typelists
^^^^^^^^^
_tbw_

Tuples
^^^^^^
_tbw_

Functor Utils
^^^^^^^^^^^^^
_tbw_

Duck Typing
^^^^^^^^^^^
_tbw_



Preprocessor Metaprogramming
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

'ppmpl.h'


Algorithms & Datastructures
~~~~~~~~~~~~~~~~~~~~~~~~~~~


Intrusive Linked List
^^^^^^^^^^^^^^^^^^^^^
The template `LinkedElements` allows to build intrusive single-linked
data structures, that can be iterated. They can be configured with a
_policy baseclass_ to control if the container takes ownership and to
integrate a custom allocator. The typical usage for such a setup is in
low-level performance-critical data structures for rendering and playback,
where data is already arranged with good locality and shall be
made traversable in some way.

Split-Splice
^^^^^^^^^^^^
Highly configurable generic algorithm for handling a segmentation
of an ordered data axis into seamlessly arranged intervals in
ascending order. The segmentation can be refined with new intervals,
handling all possible cases of relationship to the existing interval
structure.

Several Elements
^^^^^^^^^^^^^^^^
A generic, abstracted data container with random access, which plays
a crucial role for the mass allocation of small objects in the Render Engine.
The template `lib::Several<TY>` provides array-like access to a sequence of
elements marked as type `TY`, without disclosing the actual data layout or
data type. Notably this allows to place size-constrained polymorphic data
objects into a common memory block. The data structure is created with
the helper template `lib::SeveralBuilder` and can be configured with
policies to control aspects of allocation, initialisation and destruction.
In combination with the `AllocationCluster` custom allocator, it is even
possible to build arbitrary-sized collections, adding and growing the
structure within very strict limits. The typical, intended usage pattern
is to build-up a data structure dynamically, driven by a discovery or
compilation process. The data structure is then finalised and knowledge
regarding the detail data type can be discarded (``type erasure'').
Data can be used subsequently with array-style subscript access and is
finally discarded as a whole, possibly even without invoking any destructor.

//Undocumented
//symbol.hpp
//cmdline.cpp
//cmdline.hpp
//del-stash.hpp
//diagnostic-context.hpp
//element-tracker.hpp
//functor-util.hpp
//hash-indexed.hpp
//iter-adapter-stl.hpp
//iter-adapter.hpp
//iter-source.hpp
//itertools.hpp
//lifecycle.cpp
//lifecycleregistry.hpp
//lumitime-fmt.hpp
//lumitime.cpp
//null-value.hpp
//observable-list.hpp
//opaque-holder.hpp
//query.cpp
//query.hpp
//result.hpp
//scoped-holder.hpp
//streamtype.cpp
//typed-counter.hpp
//variant.hpp
//visitor.hpp
//wrapper.hpp