Lumiera (as seen) from Outer Space
==================================
:Author: Christian Thäter ct@pipapo.org
:Date: Summer 2010


[abstract]
******************************************************************************
The Lumiera Community creates a non linear video editing and compositing FOSS
application for Linux/Unix/Posix Operating Systems, suitable for professional
and quality oriented work, building on common open source video, sound and GUI
toolkits and libraries, providing flexibility and a high degree of
configurability and full control of all parameters, but at the same time a
smooth workflow which scales well to larger and more complicated editing
projects. This Document outlines the Design from some distance,
helping people to understand the Ideas behind Lumiera and understand the tools
they get to work with. It is aimed for workflow designers any anyone who wants
to know how the program works in general.
******************************************************************************


About this Document
-------------------

// all things starting with '//' are asciidoc comments and drafts/notes while
// working on this document

This document is meant to be read electronically, it contains a lot
hyper-links between explanations denoted by an arrow ->. Lumiera is still in
development, we describe here planned features without explicitly tagging them;
some things are not worked out in detail yet. Although this document is heavily
cross-linked we try to start with a broad overview and work out more detailed
things towards the end.


Vision
------

// objective and goals of the project

Lumiera claims to be `professional', this is quite a vague term and needs
some explanation what it means to us:

 Reliability::
        Whatever happens, your work must be safe, protected against software
        glitches and incompatibilities. Ideally Lumiera should be very stable and
        never crash, in practice even crashes or power outages should not
        result in lost work.

 Productivity::
        Professionals want to get things done, in time, but optionally with control
        over every aspect. Balancing these goals should be the central concern for 
        workflow design and usability.

 Quality::
        If you work with high quality, cinema grade digital video material you
        want to be sure that you can deliver this crisp quality without
        compromise throughout you workflow to your final product. All rendering
        must be reproducible to the bit.

 Scalability::
        Projects and budgets differ, hardware advances, Lumiera must scale
        in different dimensions and use the available resources as best as it
        can. From small Laptops to multicore Computers and Renderfarms.

 Future Proofness::
        Soft and Hardware advances at a fast pace. We must not lock into the
        current state of technology but being flexible to extend the System
        without breaking compatibility. Projects you create nowadays with
        Lumiera should be usable in foreseeable future, at least there needs
        to be a guaranteed upgrade path.
<

Fundamental Forces
------------------

// the basic ideas which drive the lumiera design

The Lumiera design is guided by a small number of basic principles. Keeping
these in mind will help to understand how actually more interesting things can
be built up on that foundation.


Open ended combining of Building Blocks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Lumiera is not so much defined in terms of _features_ -- rather it allows to
combine basic _building blocks._ These basic modules, entities or objects each
have a distinct _type_ explicitly limiting the connections. Within these
limits, any conceivable combination shall be supported without further hidden
limitations.

Lumiera is neither a set of Lego bricks, nor is it the business application
driven by finite usage stories.


Medium level Abstraction and Project specific Conventions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

These building blocks within Lumiera create a moderate level of abstraction; a
user may, if desired, directly manipulate through the GUI clips, individual
effects, masks, and even the placements xref:placement[->] used to stitch the
objects together, which is comparatively low-level. On the other hand, these
abstractions shield the user from the actual technical details like format
conversions and the accessing of individual channels.

To complement this approach, Lumiera does _not_ rely on hard wired, global
conventions -- rather we allow to build up project specific conventions and
rules xref:rules[->] to fit the given requirements and preferred working
style. To help getting started, Lumiera will ship with a fairly conventional
project template and default configuration.


[[graphs]]
Rendering is Graph Processing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Processing of Video (and audio) data can be generalized as graph processing
(more precisely ``directed acyclic graphs''). Data flows on the edges of these
graphs and is processed in the nodes.

image:lumiera_big_graph.svg[Example for a graph]

When we look at this model we discover that we only need to build
xref:builder[->] such graphs, the nodes themselves can be seen as black boxes
and will be implemented by plugins xref:plugins[->]. Moreover one can
preconfigure subgraphs and handle them as single entity xref:pluginstack[->].

In Lumiera everything will be translated into such a graph. Your footage will
be demultiplexed xref:demultiplexer[->] at a first node, down to the encoding
xref:encoder[->] and multiplexer xref:multiplexer[->] which assembles the
final video.


Pulling not Pushing
~~~~~~~~~~~~~~~~~~~

On a first glance, it looks fairly natural to set up the graphs xref:graphs[->]
as described above and then push data into the system through the input nodes
whereas the final result can then be seen soon on the output node. Several
multimedia frameworks use this approach. But it has a lot of shortcomings
which make it inappropriate for non-linear video editing.

Lumiera instead pulls data though the pipe, that is a request starts at the
output node and makes it way up to the inputs. This has certain advantages
xref:pull[->], explained later.


Don't waste work
~~~~~~~~~~~~~~~~

Rendering A/V Data can be quite CPU intensive, to ensure that we do not waste
CPU power by rendering things twice, or the worse, have to throw results away
because they couldn't be rendered in time, we use sophisticated caching
xref:caching[->] and profiling xref:profiling[->].


The visible Universe
--------------------

// coarse overview whats seen on the gui, details later

Now its time to take a look at the prelimary Lumiera GUI:

image:lumiera_screenshot.png[Screenshot of Lumiera]

The GUI is a plugin by itself and only one way to work Lumiera, it will become
possible to create special-purpose GUIs or control Lumiera in different ways,
like a headless rendernode xref:rendernode[->] or frameserver
xref:frameserver[->]. Completely script driven interfaces for automated
processing are also planned.

The GUI screenshot you see above is faily default as when you start Lumiera up
for the first time (the plan is to add a 2nd Viewer to the default
configuration). While we support a much more sophisticated screen concept
xref:screenconcept[->] to adapt to different workplaces and workflows.


Viewer
~~~~~~

// only one viewer type used for everything
// how is audio integrated in the viewer
// effects may add overlays (masking/rotoscoping, information for example)
//  these may be manipulateable in the viewer, but not part of the rendered
//  video. Maybe effects can add widgets to the viewer too (how, where?)
// one can open as many viewers he needs
// these can be attached everyhere in the processing graph (pre/post effect)
// have bus in front to adapt output format
// detachable window, fullscreen, external screen



Transport Controls
~~~~~~~~~~~~~~~~~~

// current gui is not final (transport controls attached to the timeline)
// It is not finally decided where transport controls will be integrated
// possibly as its own gui element

// This are devices either controlled by widgets or by some input device (midi
// etc) so their gui may look differently.
// Either way they connect to a Play Controler xref.. in the core which
// manages playing and cursor positioning.

// thus there will be some gui facility to attach Transport controls to Play
// Controllers. Transport controls are ganged when they attach to the same
// Play Controler.

// just playing some footage for preview creates a simple internal timeline,
// no magic here.

// TODO: bit unrelated, think about how ganging controls in general should
// work, also for faders, masks and so on
// Note by Ichthyo: the connection to a fader is handled through the placements,
// which allows to inherit such a control connection. IMHO together with the
// tree-like tracks this removes 80% of the need to gang faders.


Timeline View
~~~~~~~~~~~~~

// hierarchical tracks, not just a stack

// Format Independent Timeline, one can put anything on the timeline.
// the busses constrain what kind of data is pulled out and in turn the
// builder creates a processing graph which does the necessary conversions and
// stuff.
// Q: how to handle interaction, for example when some conversion can only be
//    done in a lossy way and some conversion node may or may not be inserted
//    (i mean gui wise)?
// A: this usually is detected at build time, which means the incriminating
//    object and exit node is just in scope when the problem is detected.
//    My intention was to have a problem flag with accompanying information
//    attached to this object, so the GUI can highlight the problem location
//    and give a general alert.

// TBD: Cursors .. discussion, handling, gui representation
// Note by Ichthyo: we shouldn't focus on cursors, but rather on selections.
//                  IMHO a playhead or edit marker or similar cursor is just
//                  a special case of a selection.


// Busses
// ~~~~~~
// How will the gui define busses?


Asset View
~~~~~~~~~~

// Manages all assets available in one project.
//  * source media/footage/soundfiles
//  * prepared clips, known subprojects
//  * all available effects and transitions
//  * internal artefacts like sequences and automation data sets

// First this will be simply implemented showing data loaded into the session
// and all available plugins/effects
// The user may build custom effect collections ("effect palette")

// (much) Later it is planed to make this a database driven interface, where
// the tabs showing things are basically just database queries. Then it
// becomes possible to create/extend this by customized queries and augment
// assets with all kinds of metadata which can be queried

// Actually, the same underlying data structure is used to implement the
// asset view with folders, clip bins and effect palettes, and the timeline
// view with tracks, clips and attached effects. Technically, there is no
// difference between a track or a clip bin -- just the presentation varies.
// Timeline contents can be viewed like assets for bookkeeping purposes, and
// the contents of a clip bin can be played like a storyboard



Dark Matter
-----------

// coarse overview about things the user does not see but have some contact
// with, details later...

Now lets take a look under the hood. Lumiera


Session storage
~~~~~~~~~~~~~~~

// not generateable data

// its the timeline mostly
// session storage
// benefits, unlimited undo, selective undo
// export/import plugins
// everything is stored in the session


[[placement]]
Placements
~~~~~~~~~~

Generic mechanism to stitch together media objects. Any placement may contain
a list of conditions how to locate the placed object, examples being
time-absolute/relative, relative to another object, or relative to some
specific source media frame.

All of the session model contents are attached by placement, forming a large
tree. Placements are to be _resolved_ to find out the actual position, output
and further locational properties of an object. Missing placement information
is _inherited_ from parent placements in the session tree. This causes a lot
of relational and locational properties to be inherited from more global
settings, unless defined locally at a given object: time reference point,
output destination, layering, fade control, audio pan,...


Rendering Engine
~~~~~~~~~~~~~~~~
// rendering

[[builder]]

// rules system


I/O Subsystem
~~~~~~~~~~~~~
// file handling
//  vault, work, cache
//  repositories
// explain details later


Configuration
~~~~~~~~~~~~~
// configuration system
// serves defaults, actual data are stored in the session


Plugins/Interfaces
~~~~~~~~~~~~~~~~~~
// explain whats it is
// portability
// versioning



Rendering Video
---------------

// describe the flow of data to render a frame

// viewer
// final


Pulling a Frame
~~~~~~~~~~~~~~~

// special cases,
// case studies,
// constrain viewer

// proxy
// viewer circuit
// render circuit





//Example Plugins
//---------------

// show some case-studies that someone gets a feel how plugins work

[[pluginstack]]

//Audio




// TODO Following things need to be integrated into the document above


* [[plugins]]
* [[timeline]]
* [[demultiplexer]]
* [[multiplexer]]
* [[encoder]]
* [[pull]]
bus defines rendering format
* [[caching]]
* [[profiling]]
* [[rendernode]]
* [[frameserver]]
* [[screenconcept]]
* [[busses]]

// gui/screen concepts
//  perspectives

// Automation




// 3 layered model


such as that we only need to pull exactly what we need,
posibly down to the pixel, this also allows efficient caching xref:caching[->]
for intermediate data, to be reused later.





Glossary
--------

The above outline of the design uses a lot of special terms and common termes
used with specific meaning within Lumiera. To ease the understanding, we've
collected a link:Glossary.html[Glossary of common terms].

