LUMIERA.clone/doc/user/intro/intro.txt

Lumiera (as seen) from Outer Space
==================================
:Author: Lumiera_Core_Developers
:Date: Winter 2012
 

******************************************************************************
.abstract
The Lumiera Community is in the process of making a non-linear video editing 
and compositing FOSS application for Linux/Unix/Posix operating systems.  The
application is geared towards professional, high-quality work; but
it is equally suitable for low-end users, due to its in-design scalability.
Lumiera builds on common open source video, sound and GUI toolkits and
libraries, being highly flexible, configurable -- user-control over a broad
spectrum of configurable parameters -- and with smooth workflows that scale well
to larger more intricate projects.

This document outlines the design from a more general perspective,
providing potential users with sufficient insight into the tools and technology
behind Lumiera to start working with Lumiera quickly.
******************************************************************************

// all things starting with '//' are asciidoc comments
// and drafts/notes while
// working on this document

.About this Document
Lumiera is innovative video editing software, still under active development
by an open-source community. This document acts as introduction. We try to start
with a broad overview, then explain some core concepts towards the end.
We describe planned features here without explicitly tagging them as such and we
omit technical details.
There is a link:Glossary.html[Glossary of common terms]; readers curious about
the inner workings might want to continue their path of exploration into
link:{ldoc}/technical/overview.html[The Inner Core]




// ################################################
// #########  Part I: Vision and Concepts #########
// ################################################



Vision
------

Lumiera strives to be a *professional non-linear video editing system*. It is
important to note that by ``professional'' we do not imply ``commercial'' nor
``industrial''. We do, however, wish to convey an attitude or frame of mind
in the approach made to such work; not its goal or purpose.

Professionalism implies _doing work with sincerity and being subject to any
kind of wider goal, demand, or purpose._ When it comes to editing film, this
wider goal, demand, or purpose might be artistry, a narration or have meaning to
convey, a political message or commercial revenue -- it might be something to
reveal to your audience, to let them see, feel and perceive.

Bearing this idea of professionalism in mind, we can identify a number of key
aspects film production tools will have to exhibit to aspire to be labelled
'professional':

 Reliability::
        All work must be secure and safe against any software glitches,
        incompatibility or instability.  Lumiera should be reliable,
        very stable and resistant to software crashes.  Even
        power failures should not result in data or work loss.

 Quality::
        High-quality, cinema grade digital video material
        requires crisp-quality results with no concessions made throughout the
        entire workflow that might compromise quality of the final work.
        A common requirement is the ability to reproduce all rendering
        calculations down to the last bit. 

 Performance and Productivity::
        Professionals want to get things done on time, but
        with control over all detail. The fine balance of these goals is a
        central goal of workflow design and usability.

 Scalability and Adaptability::
        Projects and budgets differ, hardware advances; and Lumiera must scale
        in many ways and make optimum use of the resources that are
        available. From small Laptops to multi-core computers up to render farms,
        Lumiera must be adept in optimum use of the hardware at hand.

 Durability::
        The rapid pace at which software and hardware rampage forward is surely 
        a warning to new software projects and the dangers of locking into any
        current technological ``fashion'' to achieve a ``cheap'' goal, feature
        or performance boost.  Once the fad fades, the software woes begin.
        The software must be able to engage new technological developments
        without any compromise to functionality or backward compatibility.

	

Fundamental Forces
------------------

The Lumiera design is guided by a small number of basic principles. Keeping
these principles in mind will help you understand how more interesting things can
be built up from these fundamental principles.


Open Ended Combination of Building Blocks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Lumiera is not so much defined in terms of _features_.
Instead it can be better seen as a workshop in which the emphasis is on
individual _basic building-blocks_ that can be combined together in interesting
ways to form more complex structures. These basic modules, entities or objects
each have a distinct _type_ which explicitly identifies the possible connections between
modules. Within these constraints, any combination is possible without
hidden restrictions.

Lumiera is not a set of Lego bricks, nor is it a business application
driven by finite usage stories.


Medium Level Abstraction and Project Specific Conventions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

These building blocks within Lumiera create a moderate level of abstraction; a
user may use the GUI to manipulate several elements directly -- not only the
clips and effects, but also individual effect parameter automation, masks, layering,
and even the placements xref:placement[<-] used to stitch the objects together,
which is comparatively low-level. Such moderate abstraction encourages the user to
understand the inner workings, while shielding the user from merely technical
details such as header format conversion or access to individual channels.

To complement this approach, Lumiera does _not_ rely on hard-wired, global
conventions -- rather we allow the building up of project specific conventions and
rules xref:rules[<-] to fit a given requirement and preferred working style. 

Lumiera will be supplied with a conventional template and a default
configuration to ease the learning curve for new users.


[[graphs]]
Rendering is Graph Processing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Processing video (and audio) data can be conceived as a graph (more precisely
a _directed acyclic graph_).  In this model of video processing, data flows
along the edges of the graph and is processed at the nodes.

The following figure depicts manipulating video data as a graph.  The nodes at
the top of the graph are where data input occurs. From there, the data moves on
to the next nodes: the direct siblings of the root.  Here, the video data is
pre-processed.  All other operations on the data can be represented by
nodes, and data flows from one operation to the next along the nodes of the
graph.


image:{imgd}/lumiera_big_graph.png[Example for a graph]

When we look at this model we discover that we only need to build
xref:builder[<-] such graphs, the nodes themselves can be seen as black boxes
and will be implemented by plugins xref:plugins[<-]. Moreover one can
pre-configure subgraphs and handle them as a single entity xref:pluginstack[<-].

In Lumiera everything will be translated into such a graph. Footage will
be demultiplexed xref:demultiplexer[<-] at the first node, thereafter passed
to the actual processing nodes for calculating effects, masks and layering,
and finally be forwarded to the encoding stage xref:encoder[<-] and on to
the multiplexer xref:multiplexer[<-] which assembles the final video.


Pulling not Pushing
~~~~~~~~~~~~~~~~~~~

At a first glance, it looks fairly natural to set up the graphs xref:graphs[<-]
as described above.  Data can then be pushed into the system through the input
nodes and the final result can be seen at the output node. Several multimedia
frameworks use this approach. However this scheme exhibits a number of
shortcomings which make it inappropriate for non-linear video editing. 

Lumiera takes a different approach.  Data is pulled through a pipe: any rendering
request starts at the output node and makes its way back up through the graph to
the inputs. This scheme offers a number of advantages over the naive scheme.



Don't waste work
~~~~~~~~~~~~~~~~

Rendering A/V data can be computationally intensive.  To avoid duplicated or wasted work and 
ensure it is possible to render on time, 
Lumiera employs a sophisticated means of using
cache xref:caching[<-] and profiling xref:profiling[<-].






// ################################################
// #########  Part II: The Visible        #########
// ################################################

''''''''


The visible Universe
--------------------

Lumiera consists of three major parts:

- Graphical User Interface (GUI)
- Processing layer (``Proc-layer'')
- Backend for storage and access

In this section, we discuss the user's perspective when working with the GUI.


Although Lumiera will initially ship with a standard, default GUI, we do not presume that this GUI will be suitable for all uses.
We expect there to be multiple, different GUIs, each designed for different kinds of tasks.  We expect that some of these will be written by users, and the proc-layer is designed to make this easy.

Indeed, Lumiera can even work
satisfactorily without a GUI, for example, for special purposes
(e.g. as headless render-node xref:rendernode[<-] or frameserver
xref:frameserver[<-]).  Scripts will eventually be written
to facilitate automated processing.


Now it is time to take a look at the preliminary Lumiera GUI:

image:{l}/images/lumiera_gui_small.png[Current Lumiera GUI Screenshot]

We expect this GUI to change once we are at the point of having feedback from actual users.

The screenshot of the GUI presented above is more or less the standard GUI when
it is started for the first time without any user configuration etc.  A second
viewer is planned for later to be added to the default. We support a much more
sophisticated screen concept xref:screenconcept[<-] to adapt to different
workplaces and workflows. 
// TODO
// The above paragrph isn't all it should be, but I can't tell what it should
// be without seeing it in the context of the other changes I'm making.
// I suspect it's better to discuss how rewritable the GUI is rather than
// the specifics ot the GUI presented above. -- hendrik

At a first glance, the GUI might seem somewhat overwhelming, something similar
to the cockpit of a jumbo jet.  However, nearly all the bells and whistles can
be subdivided into groups according to their functionality.  The following
sub-sections discusses the more prominent groupings.


Viewer
~~~~~~

The viewer is an area where footage and the assembled edit can be played back and investigated.
Obviously, support for audio will also be provided.  As there are many sources that
can be displayed, a viewer is attached to a source via the viewer switch board.
Timelines, probe points, wire taps and individual clips are examples of sources
that can be attached to a viewer. Moreover, the number of viewers open at any one
time is only limited by the hardware, and each viewer can be collapsed or hooked
up to a beamer or monitor.


Transport Controls
~~~~~~~~~~~~~~~~~~

The preliminary layout in the current GUI is rather provisional -- a final decision
has yet to be taken on where the transport controls will be located.  In later
versions of the standard GUI, the transport controls will change.  There is even
a possibility that the transport controls will be allocated their own GUI element.
// TODO:
// A final decision about where the transport controls are located?
// Isn't the whole point of enabling multiple GUIs that there won't be
// final  decisions?  Only provisional ones?


Transport controls are devices that are controlled by widgets or by some input device
(MIDI, control surface, etc) so the look-and-feel of the transport control
widget may vary, depending on what the transport controls have been assigned
to.  Irrespective of their look-and-feel, they are connected to a play
controller.

A _play controller_ coordinates playback, cueing and rewinding.  Transport
controls are ganged when they attach to the same play controller.


Timeline View
~~~~~~~~~~~~~

A timeline is a container that provides a time axis and an output. There may be
connections to various output designations, each with a different configuration.
A more elaborate production setup will use several timelines, one for each distinct
output configuration. A timeline does not temporally arrange material, this is
performed by a sequence, which can be connected (snapped) to a timeline.

A typical film will define many sequences, but will only have a few timelines. A
sequence contains a number of tracks which are ordered in a hierarchy. Tracks do
not have any format associated with them and more or less anything can be put
into a track. Consequently, audio and video material can equally be assigned to
a track, there is no discrimination between audio and video in the Lumiera
concept of a track.

A timeline must be assigned to viewer and transport control if playback viewing is desired.
In most cases, these connections are created automatically, on demand: Just playing some
footage for preview creates a simple internal timeline.



Global Pipes
~~~~~~~~~~~~

The GUI provides a separate _bus view_ which displays the _global pipes_. _Global
pipes_ are master buses for each type of output--which includes both sound and
image--and collect the outputs of subgroups together in a manner similar to an
audio mixing desk. Any _pipe_ is just a means of collecting, mixing or
overlaying output produced by other pipes. A simple (linear) chain of effects
can be applied within each pipe.

Most pipes are managed automatically, e.g. the pipes corresponding to
individual clips, or the pipes collecting output from transitions, from nested
sequences and from groups of tracks. At some point, at the timeline level, all 
processed data is collected within the aforementioned global pipes to form the
small number of output streams produced by rendering and playback. Each timeline
uses a separate set of global pipes. 


Asset View
~~~~~~~~~~
An Asset View can be conceived as the timeline's book keeper: it manages the
various constituents in the timeline.  Moreover, in addition to managing timeline
constituents, raw material, clips, bins (folders) and effects are managed by the
Asset View, i.e., typical management operations including, deleting, adding,
naming, tagging, grouping into bins, etc. all occur here.

There are various kinds of assets available in any project:

  * source media/footage/soundfiles
  * all available effects and transitions
  * user defined ``effect stacks'' and effect collections ("effect palette")
  * internal artefacts like sequences and automation data sets
  * markers, labels, tags and similar kinds of meta data

Actually, the same underlying data structure is used to implement the
asset view with folders, clip bins and effect palettes, and the timeline
view with tracks, clips and attached effects. Technically, there is no
difference between a track or a clip bin -- just the presentation appears
different. Timeline contents can be viewed, just like assets, for bookkeeping
purposes, and the contents of a clip bin can be played like a storyboard.



// ################################################
// #########  Part III: The Invisible     #########
// ################################################

''''''''


Dark Matter
-----------

The material in this section provides a cursory view of features not required by
a typical user, but of more importance to people looking under the hood
(advanced users, software developers)

Most of the material in this section is to be found in the Proc-layer and in the
Backend.



Session Storage
~~~~~~~~~~~~~~~

The current representation of a project resident in memory is internally known 
as a session, whereas from a GUI user perspective, this is known as a project.
In this section, we will use the term ``session'' as we are discussing the internals
of Lumiera here. 

Everything is stored in a session.  If a user saves a session, then there should
be no difference to the user whether they continue working or throw all their
work away and load the session that has just been saved. In other words
``everything is stored in a session!''

This ambitious goal has a number of advantages.  If all steps are to be stored,
then all steps must be available as an object to be stored. We can perform
operations on these objects.  For example, unlimited ability to undo previous
steps, selective undo of previous steps or the possibility of merging various
steps might even be a possibility.  On a practical note, work on a project at
the office and work on the same project at home can be merged each morning and
evening. 

Session storage is envisaged to operate similar to a database or journalling
file system. Any operation will be logged prior to execution. This protects
the session contents against corruption and allows for automated recovery
in case a crash occurs. The actual storage itself is delegated to several backends.
One of these backends implements a _binary storage format_ for performance
reasons, while another exposes all session contents in textual,
human readable form. The storage format is designed in a way to ensure
a certain degree of compatibility while the application evolves.



[[placement]]
Placements
~~~~~~~~~~

// Can placements place things in space as well as in time??

Generic mechanism to stitch together media objects. Any placement may contain
a list of conditions on how to locate the placed object, examples being
time-absolute/relative, relative to another object, or relative to some
specific source media frame.

All of the session model contents are attached by placement, forming a large
tree. Placements are to be _resolved_ to find out the actual position, output
and further locational properties of an object. Missing placement information
is _inherited_ from parent placements in the session tree. This causes a lot
of relational and locational properties to be inherited from more global
settings, unless defined locally at a given object: time reference point,
output destination, layering, fade control, audio pan,...


Rendering Engine
~~~~~~~~~~~~~~~~
[red]#to be written#
rendering...

[[builder]]
builder
^^^^^^^
[red]#to be written#

rules system
^^^^^^^^^^^^
[red]#to be written#



Input-Output Subsystem
~~~~~~~~~~~~~~~~~~~~~~

Lumiera will process large quantities of data and it is of critical importance
to perform this efficiently.  The input and output subsystem are all processed
in the backend, in fact, this is one very important function provided by the
back end.

The typical Lumiera user will have many clips, in various configurations located
at various places.  All this data will have to be stored by the backend,
moreover all this data will have to be rapidly retrieved from storage and
provided to the user. The demands on memory are high: huge chunks of data,
that can be quickly stored and equally quickly fetched, even if stored over
longer periods of time. Moreover, due to the scalability requirement of Lumiera,
this process will have to perform adequately on lower-end hardware, and perform
efficiently on higher-end hardware.

Lumiera will break down processes that need to be processed into smaller units
called _frame job_. Typically, there will be many hundreds of jobs waiting for
processing at any one time. These jobs are queued for processing and the order
in which this is performed is managed by the _scheduler_.
This is all done in the backend.

Apart from memory, the backend will be responsible for accessing and saving
files. It is essential to do this efficiently. This will be carried out
in the backend using low-level mechanisms.

// file handling
//  vault, work, cache
//  repositories
// explain details later


Configuration
~~~~~~~~~~~~~
[red]#to be written#

// configuration system
// serves defaults, actual data are stored in the session


Plugins/Interfaces
~~~~~~~~~~~~~~~~~~


What are Plugins?
^^^^^^^^^^^^^^^^^

A Plug-in is a kind of generalisation of a library.

All applications use, to varying degrees of intensity, libraries. A programmer
will not reinvent the wheel each time he sits down to programme an
application. A programmer will typically borrow and use features and
functionality from other programmers---or even borrow from himself, stuff
written long ago in the past. Such features are collected together in
libraries. 

A library is used in an application by _linking_ the library into the
application. (There are other things to be done, but we'll call these 'details',
which wont concern us here.) There are different ways to _link_ a library
into an application: statically linking and dynamically linking.

_Statically Linking_ is done while the application is being built, or
compiled. It is performed by the linker. The linker can perform some checks
(mostly checks on syntax) and warn the user that some particular feature is
being used incorrectly. The user can then correct the offending code, and
recompile. 
There are a number of disadvantages associated with static linking. Features and
libraries are being constantly improved. If the application wants to use new
features, it will have to be recompiled with the new library which provides the
new features.

_Dynamic Linking_ helps avoid the necessity of having to recompile. If a
new, improved library becomes available, all the user has to do is to install
the new library onto the operating system, restart the application and the new
features can be used by the application. The features provided by a dynamic
library are loaded when the application starts to run.

However both methods exhibit a number of shortcomings. Wouldn't it be better if
all features could be loaded only when needed? If features could be loaded only
when needed, then they could also be unloaded when not required, thus saving
memory and possibly increasing performance. This scheme of making features
available to an application is known as run-time linking, aka plug-ins.
Plug-ins offer other benefits: the application can continue to use both the old
features and the new features together, side-by-side, by using the version
number associated with the plug-in. This saves the application from the havoc
of library incompatibility that is 
associated with other linking methods.

Most modern applications use plug-ins, some are heavily dependent on plug-ins
and only provide limited functionality without any plug-ins.
Lumiera will not reinvent the wheel. One major goal is to provide considerable
functionality via well-designed, external code supplied to Lumiera by plug-ins.


How are Plugins Implemented?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We can view Lumiera itself as being a slender application with many of the bells
and whistles being supplied __directly_ by external FOSS tools.  Such tools will
be made available to the Lumiera code base via the Lumiera Plugin Interface (LPI).

As LPI will be used throughout most layers of Lumiera--from the low-level
backend, right up to the GUI--the interface will have to be compatible to both C
and C++. In effect, this means that LPI will be implemented in C.

Using LPI a plugin interface can be defined for each external library.  The
Lumiera code base can then use the external library by including the 
plugin interface for that particular library.

LPI defines a uniform framework of how to manage plugins within Lumiera. It
declares a number of macros that allow details of a plugin to be defined:
    - name of the plugin
    - version of the plugin (include major and minor version numbers)
    - name each call-back function defined by the plugin and exported by the
      external library 
    - methods to initialise and destroy the plugin

Defining some plugin interface then entails using LPI to specify the details of
the plugin.


Integration with external libraries, frameworks and applications
----------------------------------------------------------------
[red]#to be written#

____
 .
 .
 .
____

''''


Further reading
---------------

Glossary
~~~~~~~~

The above outline of the design uses a lot of special terms and common terms
used with specific meaning within Lumiera. To ease understanding, we've
collected together a link:Glossary.html[glossary of common terms].

Tutorials and User Manual
~~~~~~~~~~~~~~~~~~~~~~~~~
For most users, the next step would be to skim through the user manual and to
install the application. But we are not this far yet, unfortunately. As of
1/2013, Lumiera isn't usable in any way. Later, when we approach the alpha and
beta development phase, we'll devote some attention to a user manual and provide
a sections on tutorials.

The Inner Core
~~~~~~~~~~~~~~
Developers and Readers curious about the inner workings of the application
might want to have a look into the overview document of our technical
documentation section. Learn more about 
link:{ldoc}/technical/overview.html[The Inner Core]...