Lumiera: The Inner Core
=======================

[abstract]
******************************************************************************
The Lumiera Developers have a vision about a modern core for a NLE. Here are
some of the design decisions and rationales in a rather sketchy way explained.
New Developers may find this Document useful to get an idea about how the
different components work together.
******************************************************************************


Overview
--------

Lumiera constitutes of a broad range of subsystems. These are roughly grouped
into three layers plus some extra components. This structure is mostly kept in
the source directory structures.

This three Layers are:
 The User Interface::
     User interfaces are implemented as plugins, most commonly one will see
     the default GUI. But also scripting interfaces or specialized GUI's are
     possible.

 The Processing layer::
     Keeps the Session, generates the rendering graphs for sequences.

 The IO and System interface Backend::
     Manages thread-queues, schedules jobs, does the memory management for the
     heavy multimedia data.

The extra components are:
 Lumiera::
     The main program itself, basically acts only as loader to pull the rest up.

 Common::
     Vital components which must be available for pulling the system up, part
     of the main program.

 Library::
     A lot of (mostly) stateless helper functionality which is used by all the
     rest.


Coding Style
~~~~~~~~~~~~

The Lumiera team agreed on using GNU coding style with some exceptions (no
tabs, line wrap must not always be enforced). Otherwise we are a bit pedantic
to be consistent to make the codebase uniform. Function nameing conventions
and other details are described in sereral RFCs.

Documentation
~~~~~~~~~~~~~

Every public function should be documented with doxygen comments. The design
is documented in this text, in various RFC's. Bootstrapping the Lumiera design
used serveral other places, some tiddlywikis, an uml model and cehteh's
private wiki, most of this information is asciidoced meanwhile and in progress
to be integrated in a now documentation hierarchy.


Test Driven Development
~~~~~~~~~~~~~~~~~~~~~~~

We strive to use _Test-Driven-Development_ this means tests are written first,
defining the specification of the entity being tested. Then things get
implemented until they pass their tests. Practice shows that this approach
isn't always suiteable, nevertheless we try to follow it closely and maintain
rigid testsuites.


Lumiera Main
------------


Common
------

Config System
~~~~~~~~~~~~~

Plugin loader and interfaces
~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Lua Scripting
~~~~~~~~~~~~~

Library
-------

Meant to be extended as we go

Locking
~~~~~~~

mutex, condition vars, rwlocks. intentionally no semaphores.

macros to put scoped around critical sections

//sync-classlock.hpp
//sync.hpp

Time
~~~~
time handling and conversion at one central point


Errors
~~~~~~

 * Errors are identified by pointers to static strings.
 * Errors are sticky (you cant set a new error unless the pending one got
   cleared).

.General definition and declarations of errors
Errors get declared in headers with the `LUMIERA_ERROR_DECLARE(err)` macro.

Then the implementaton file uses the macros `LUMIERA_ERROR_DEFINE(err, msg)`
to instance the error. There is no central registry, Any component can
introduce its own errorcodes but must ensure that the error identifier is
unique.

.Error handling in C
There are two helper macro forms for setting up error conditions, one is
`LUMIERA_ERROR_SET..(flag, err, extra)` and the other one is
`LUMIERA_ERROR_GOTO..(flag, err, extra)`. Each for differnt logging levels.
The `SET` form just logs an error and sets it, the `GOTO` form also jumps to
an error handler. Both take a NoBug flag used for logging and a optional
`extra` c-string.


[source,C]
--------------------------------------------------------------------------------
const char*
mayfail()
{
  const char* ret = foo();
  if (!ret)
    LUMIERA_ERROR_GOTO (flag, FOO_FAILED, 0);

  if (!bar(ret))
    LUMIERA_ERROR_GOTO (flag, BAR_FAILED, 1,
                        lumiera_tmpbuf_snprintf (256, "foo was %s", ret));

  return "everything ok";

  /* cleanup in reverse setup order */
 BAR_FAILED1:
  lumiera_free (ret);
 FOO_FAILED0:
  return NULL;
}
--------------------------------------------------------------------------------


.C++ exceptions



Memory Pools
~~~~~~~~~~~~

Fast memory pools for moderately small static sized allocations in highly
dynamic situations.

 * optimized for cache locality
 * supporting a destructor callback to free all objects



Polymorphic Programming in C
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Just a macro for simplyfying vtable function calls
 VCALL(Handle, function, arguments...)
translates to
 Handle->vtable->function (Handle, arguments...)

The user is responsible for setting up a `vtable` member in his datastructures
this macro does some NoBug checks that self and function are initialized.


Unique Identifiers
~~~~~~~~~~~~~~~~~~

Generating 128 bit non cryptographic strong unique identifiers.

 * having an alternative representation to store a pointer

 * may be extended for a strong (slow) and a fast (weak) variant in future


CLib wrappers
~~~~~~~~~~~~~

Some wrapers for the C memory management functions malloc, calloc, realloc and
free which never fail. In case of an error the resourcecollector in the
backend is invoked to free resources or doing an emergency shutdown.

Safe wrapers for some string functions from the C-library which also never
fail. NULL strings are propagated to "" empty strings.


Temporary Buffers
~~~~~~~~~~~~~~~~~

Provides a small number of round robin buffers which need not to be freed.
Must only be used locally when no deep recursion may use tmpbufs. Using these
wrong (recursively) will result in corrupted data.

Very fast and efficient from smallest too hugest allocations. No need to care
for 'free()'



Preprocessor Metaprogramming
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

ppmpl.h


Algorithms & Datastructures
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Probabilistic Splay Tree
^^^^^^^^^^^^^^^^^^^^^^^^

Self optimizing splay tree

Advantage:
 * can be iterated
 * very fast in situations where few common elements are queried most often

Disadvantages
 * provides no own locking, needs to be locked from outside
 * (almost) every access is a mutation and needs an exclusive lock, bad for concurrency


BTree
^^^^^

Generic B+/B* Tree implementation. Details are provided by a vtable at an
actual implementation.

 * Fine grained (block level) locking with different modes
 * supports cursors to iterate over the data, in both directions
 * exact and inexact searched (whats close before/after something)


Cuckoo Hashing
^^^^^^^^^^^^^^

Currently defunct, to be revieved someday.


Hash functions
^^^^^^^^^^^^^^

planned


Linked Lists
^^^^^^^^^^^^

.llist

Cyclic double linked intrusive list.

.slist

Single linked variant.


Most Recent used Cachelists
^^^^^^^^^^^^^^^^^^^^^^^^^^^

A list where old entries are recycled from the tail, used things are removed
from the list (ownership acquired) and released back to the head.

Items not used propagate towards the tail where they will be reused.


//Undocumented
//access-casted.hpp
//advice.hpp symbol-impl.cpp
//allocationcluster.cpp symbol.hpp
//bool-checkable.hpp
//cmdline.cpp
//cmdline.hpp
//del-stash.hpp
//diagnostic-context.hpp
//element-tracker.hpp
//external
//factory.hpp
//format.hpp
//frameid.hpp
//functor-util.hpp
//handle.hpp
//hash-indexed.hpp
//iter-adapter-stl.hpp
//iter-adapter.hpp
//iter-source.hpp
//itertools.hpp
//lifecycle.cpp
//lifecycleregistry.hpp
//lumitime-fmt.hpp
//lumitime.cpp
//multifact-arg.hpp
//multifact.hpp
//nobug-init.cpp <<why here and not in common?
//nobug-init.hpp
//null-value.hpp
//observable-list.hpp
//opaque-holder.hpp
//p.hpp
//query.cpp
//query.hpp
//ref-array-impl.hpp
//ref-array.hpp
//result.hpp
//scoped-holder.hpp
//scoped-ptrvect.hpp
//scopedholdertransfer.hpp
//singleton-ref.hpp
//singleton-subclass.hpp
//singleton.hpp
//singletonfactory.hpp
//singletonpolicies.hpp
//singletonpreconfigure.hpp
//streamtype.cpp
//test
//thread-local.hpp
//tree.hpp
//typed-allocation-manager.hpp
//typed-counter.hpp
//util-foreach.hpp
//util.cpp
//util.hpp
//variant.hpp
//visitor-dispatcher.hpp
//visitor-policies.hpp
//visitor.hpp
//wrapper.hpp
//wrapperptr.hpp



User Interfaces
---------------

User interfaces are stateless and act only as manipulators on the Session
which is kept in the processing layer below. User interfaces are implemented
as plugins and are pulled up on demand.


Processing Layer
----------------

High Level Model
~~~~~~~~~~~~~~~~

Rules System
~~~~~~~~~~~~

Low Level Model
~~~~~~~~~~~~~~~



Backend
-------

I/O Subsystem
~~~~~~~~~~~~~

.OS Filehandles

 * as mru cache, round robin reused

.Files

Lumiera has its own abstract file handles which store the state and name of a
file. The associated filehandle doesn't need to be kept open and will be
reopened on demand. Hardlinked files are recognized and opened only once.

.Memory Mapping

All file access is done by memory mapping to reduce data copies between
userland and kernel. Moreover the kernel becomes responsible to schedule
paging (which will be augmented by lumiera) to make the best use of available
resources. Memory is mapped in biggier possibly overlapping windows of
resonable sized chunks. Requests asking for a contingous set of data from the
file in  memory.


.Indexing

.Frameprovider


Threadpools
~~~~~~~~~~~

Manages serveral classes of threads in pools. The threadpool is reasonable
dumb. Higher level management will be done by the Schedulers and Jobs.


Schedulers
~~~~~~~~~~

Scheduling Queues for different purposes:

.Deadline
Higher priority jobs ordered by a deadline time plus some (negative) hystersis. Jobs are
started when they approach their deadline. Jobs who miss their deadline are
never scheduled here.

.Background
Background jobs scheduled by priority and timeout.


.Realtime
Timer driven queue which starts jobs at defined absolute times. Timer might be
also an external synchronization entity.


Job
^^^

a job can be part of multiple queues, the queue which picks them first runs
them. When other queues hit a running job they either just drop it or promote
its priority (to be decided).


Resource Management
~~~~~~~~~~~~~~~~~~~

Running Lumiera requires a lot different resources, such as CPU-Time, Threads,
IO Bandwidth, Memory, Address space and so on. Many of this resources are rather
hard limited and the system will return errors when this limits are hit, but
often one does not even reach this hard limits because performance will
degrade way before coming into the realm of this limits. The goal for Lumiera
is to find a sweet spot for operating with optimal performance. Thus we have
some facilities to monitor and adjust resource usage depending and adapting to
the system and current circumstances.


Profiler
^^^^^^^^

Collects statistic about resource load, helps to decide if job constraints can
be fulfilled.

Things to watch:
 * cpu utilization
 * memory usage (swapping, paging)
 * I/O load, latency


Budget Manager
^^^^^^^^^^^^^^

resources need to be distributed among a lot subsystems and jobs. Each of this
component can become part of a budgeting system which accounts resource usage
and helps to distribute it. Resource usage is only voluntary managed.


Resourcecollector
^^^^^^^^^^^^^^^^^

Handles system errors related to resource shortage. There are serveral classes
of resources defined. Other subsystems can hook in functions to free
resources. Has multiple policies about how aggressive resources should be freed.

If no one cares it does a final abort(). So all systems should hook better
recovery here in!






