418 lines
14 KiB
Text
418 lines
14 KiB
Text
Lumiera (as seen) from Outer Space
|
|
==================================
|
|
:Author: Christian Thäter ct@pipapo.org
|
|
:Date: Summer 2010
|
|
|
|
|
|
[abstract]
|
|
******************************************************************************
|
|
The Lumiera Community creates a non linear video editing and compositing FOSS
|
|
application for Linux/Unix/Posix Operating Systems, suitable for professional
|
|
and quality oriented work, building on common open source video, sound and GUI
|
|
toolkits and libraries, providing flexibility and a high degree of
|
|
configurability and full control of all parameters, but at the same time a
|
|
smooth workflow which scales well to larger and more complicated editing
|
|
projects. This Document outlines the Design from some distance,
|
|
helping people to understand the Ideas behind Lumiera and understand the tools
|
|
they get to work with. It is aimed for workflow designers any anyone who wants
|
|
to know how the program works in general.
|
|
******************************************************************************
|
|
|
|
|
|
About this Document
|
|
-------------------
|
|
|
|
// all things starting with '//' are asciidoc comments and drafts/notes while
|
|
// working on this document
|
|
|
|
This document is meant to be read electronically, it contains a lot
|
|
hyper-links between explanations denoted by an arrow ->. Lumiera is still in
|
|
development, we describe here planned features without explicitly tagging them;
|
|
some things are not worked out in detail yet. Although this document is heavily
|
|
cross-linked we try to start with a broad overview and work out more detailed
|
|
things towards the end.
|
|
|
|
|
|
Vision
|
|
------
|
|
|
|
// objective and goals of the project
|
|
|
|
Lumiera claims to be `professional', this is quite a vague term and needs
|
|
some explanation what it means to us:
|
|
|
|
Reliability::
|
|
Whatever happens, your work must be safe, protected against software
|
|
glitches and incompatibilities. Ideally Lumiera should be very stable and
|
|
never crash, in practice even crashes or power outages should not
|
|
result in lost work.
|
|
|
|
Productivity::
|
|
Professionals want to get things done, in time, but optionally with control
|
|
over every aspect. Balancing these goals should be the central concern for
|
|
workflow design and usability.
|
|
|
|
Quality::
|
|
If you work with high quality, cinema grade digital video material you
|
|
want to be sure that you can deliver this crisp quality without
|
|
compromise throughout you workflow to your final product. All rendering
|
|
must be reproducible to the bit.
|
|
|
|
Scalability::
|
|
Projects and budgets differ, hardware advances, Lumiera must scale
|
|
in different dimensions and use the available resources as best as it
|
|
can. From small Laptops to multicore Computers and Renderfarms.
|
|
|
|
Future Proofness::
|
|
Soft and Hardware advances at a fast pace. We must not lock into the
|
|
current state of technology but being flexible to extend the System
|
|
without breaking compatibility. Projects you create nowadays with
|
|
Lumiera should be usable in foreseeable future, at least there needs
|
|
to be a guaranteed upgrade path.
|
|
<
|
|
|
|
Fundamental Forces
|
|
------------------
|
|
|
|
// the basic ideas which drive the lumiera design
|
|
|
|
The Lumiera design is guided by a small number of basic principles. Keeping
|
|
these in mind will help to understand how actually more interesting things can
|
|
be built up on that foundation.
|
|
|
|
|
|
Open ended combining of Building Blocks
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Lumiera is not so much defined in terms of _features_ -- rather it allows to
|
|
combine basic _building blocks._ These basic modules, entities or objects each
|
|
have a distinct _type_ explicitly limiting the connections. Within these
|
|
limits, any conceivable combination shall be supported without further hidden
|
|
limitations.
|
|
|
|
Lumiera is neither a set of Lego bricks, nor is it the business application
|
|
driven by finite usage stories.
|
|
|
|
|
|
Medium level Abstraction and Project specific Conventions
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
These building blocks within Lumiera create a moderate level of abstraction; a
|
|
user may, if desired, directly manipulate through the GUI clips, individual
|
|
effects, masks, and even the placements xref:placement[->] used to stitch the
|
|
objects together, which is comparatively low-level. On the other hand, these
|
|
abstractions shield the user from the actual technical details like format
|
|
conversions and the accessing of individual channels.
|
|
|
|
To complement this approach, Lumiera does _not_ rely on hard wired, global
|
|
conventions -- rather we allow to build up project specific conventions and
|
|
rules xref:rules[->] to fit the given requirements and preferred working
|
|
style. To help getting started, Lumiera will ship with a fairly conventional
|
|
project template and default configuration.
|
|
|
|
|
|
[[graphs]]
|
|
Rendering is Graph Processing
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Processing of Video (and audio) data can be generalized as graph processing
|
|
(more precisely ``directed acyclic graphs''). Data flows on the edges of these
|
|
graphs and is processed in the nodes.
|
|
|
|
image:{imgd}/lumiera_big_graph.png[Example for a graph]
|
|
|
|
When we look at this model we discover that we only need to build
|
|
xref:builder[->] such graphs, the nodes themselves can be seen as black boxes
|
|
and will be implemented by plugins xref:plugins[->]. Moreover one can
|
|
preconfigure subgraphs and handle them as single entity xref:pluginstack[->].
|
|
|
|
In Lumiera everything will be translated into such a graph. Your footage will
|
|
be demultiplexed xref:demultiplexer[->] at a first node, down to the encoding
|
|
xref:encoder[->] and multiplexer xref:multiplexer[->] which assembles the
|
|
final video.
|
|
|
|
|
|
Pulling not Pushing
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
On a first glance, it looks fairly natural to set up the graphs xref:graphs[->]
|
|
as described above and then push data into the system through the input nodes
|
|
whereas the final result can then be seen soon on the output node. Several
|
|
multimedia frameworks use this approach. But it has a lot of shortcomings
|
|
which make it inappropriate for non-linear video editing.
|
|
|
|
Lumiera instead pulls data though the pipe, that is a request starts at the
|
|
output node and makes it way up to the inputs. This has certain advantages
|
|
xref:pull[->], explained later.
|
|
|
|
|
|
Don't waste work
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
Rendering A/V Data can be quite CPU intensive, to ensure that we do not waste
|
|
CPU power by rendering things twice, or the worse, have to throw results away
|
|
because they couldn't be rendered in time, we use sophisticated caching
|
|
xref:caching[->] and profiling xref:profiling[->].
|
|
|
|
|
|
The visible Universe
|
|
--------------------
|
|
|
|
// coarse overview whats seen on the gui, details later
|
|
|
|
Now its time to take a look at the prelimary Lumiera GUI:
|
|
|
|
image:{l}/images/lumiera_gui_small.png[Current Lumiera GUI Screenshot]
|
|
|
|
The GUI is a plugin by itself and only one way to work Lumiera, it will become
|
|
possible to create special-purpose GUIs or control Lumiera in different ways,
|
|
like a headless rendernode xref:rendernode[->] or frameserver
|
|
xref:frameserver[->]. Completely script driven interfaces for automated
|
|
processing are also planned.
|
|
|
|
The GUI screenshot you see above is faily default as when you start Lumiera up
|
|
for the first time (the plan is to add a 2nd Viewer to the default
|
|
configuration). While we support a much more sophisticated screen concept
|
|
xref:screenconcept[->] to adapt to different workplaces and workflows.
|
|
|
|
|
|
Viewer
|
|
~~~~~~
|
|
[red]#to be written#
|
|
|
|
// only one viewer type used for everything
|
|
// how is audio integrated in the viewer
|
|
// effects may add overlays (masking/rotoscoping, information for example)
|
|
// these may be manipulateable in the viewer, but not part of the rendered
|
|
// video. Maybe effects can add widgets to the viewer too (how, where?)
|
|
// one can open as many viewers he needs
|
|
// these can be attached everyhere in the processing graph (pre/post effect)
|
|
// have bus in front to adapt output format
|
|
// detachable window, fullscreen, external screen
|
|
|
|
|
|
|
|
Transport Controls
|
|
~~~~~~~~~~~~~~~~~~
|
|
|
|
The layout in current gui is rather preliminary -- it is not finally decided
|
|
where transport controls will be integrated; possibly as its own gui element
|
|
|
|
This are devices either controlled by widgets or by some input device
|
|
(MIDI, control surface, etc) so their gui may look differently.
|
|
Either way they connect to a Play Controler xref.. in the core which
|
|
manages playing and cursor positioning.
|
|
|
|
thus there will be some gui facility to attach Transport controls to Play
|
|
Controllers. Transport controls are ganged when they attach to the same
|
|
Play Controler.
|
|
|
|
just playing some footage for preview creates a simple internal timeline,
|
|
no magic here.
|
|
|
|
// TODO: bit unrelated, think about how ganging controls in general should
|
|
// work, also for faders, masks and so on
|
|
// Note by Ichthyo: the connection to a fader is handled through the placements,
|
|
// which allows to inherit such a control connection. IMHO together with the
|
|
// tree-like tracks this removes 80% of the need to gang faders.
|
|
|
|
|
|
Timeline View
|
|
~~~~~~~~~~~~~
|
|
|
|
hierarchical tracks, not just a list
|
|
|
|
Format Independent Timeline, one can put anything on the timeline.
|
|
the busses constrain what kind of data is pulled out and in turn the
|
|
builder creates a processing graph which does the necessary conversions and
|
|
stuff.
|
|
|
|
// Q: how to handle interaction, for example when some conversion can only be
|
|
// done in a lossy way and some conversion node may or may not be inserted
|
|
// (i mean gui wise)?
|
|
// A: this usually is detected at build time, which means the incriminating
|
|
// object and exit node is just in scope when the problem is detected.
|
|
// My intention was to have a problem flag with accompanying information
|
|
// attached to this object, so the GUI can highlight the problem location
|
|
// and give a general alert.
|
|
|
|
// TBD: Cursors .. discussion, handling, gui representation
|
|
// Note by Ichthyo: we shouldn't focus on cursors, but rather on selections.
|
|
// IMHO a playhead or edit marker or similar cursor is just
|
|
// a special case of a selection.
|
|
|
|
|
|
Busses
|
|
~~~~~~
|
|
The GUI provides a separate _bus view_, showing the master busses (subgroups)
|
|
in a manner similar to an audio mixing desk. Any bus is just a means to collect
|
|
and sum up the output of a specific kind of media (video, audio, number of channels),
|
|
produced by various processing elements and other busses. Conceptionally, these global
|
|
busses are considered a part of the timeline
|
|
|
|
|
|
Asset View
|
|
~~~~~~~~~~
|
|
|
|
Manages all assets available in one project.
|
|
* source media/footage/soundfiles
|
|
* prepared clips, known subprojects
|
|
* all available effects and transitions
|
|
* internal artefacts like sequences and automation data sets
|
|
|
|
// First this will be simply implemented showing data loaded into the session
|
|
// and all available plugins/effects
|
|
// The user may build custom effect collections ("effect palette")
|
|
|
|
// (much) Later it is planed to make this a database driven interface, where
|
|
// the tabs showing things are basically just database queries. Then it
|
|
// becomes possible to create/extend this by customized queries and augment
|
|
// assets with all kinds of metadata which can be queried
|
|
|
|
// Actually, the same underlying data structure is used to implement the
|
|
// asset view with folders, clip bins and effect palettes, and the timeline
|
|
// view with tracks, clips and attached effects. Technically, there is no
|
|
// difference between a track or a clip bin -- just the presentation varies.
|
|
// Timeline contents can be viewed like assets for bookkeeping purposes, and
|
|
// the contents of a clip bin can be played like a storyboard
|
|
|
|
|
|
''''''''
|
|
|
|
|
|
Dark Matter
|
|
-----------
|
|
|
|
[red]#to be written#
|
|
coarse overview about things the user does not see but have some contact
|
|
with, details later...
|
|
|
|
|
|
Now lets take a look under the hood.
|
|
|
|
|
|
Session storage
|
|
~~~~~~~~~~~~~~~
|
|
[red]#to be written#
|
|
|
|
// not generateable data
|
|
|
|
// its the timeline mostly
|
|
// session storage
|
|
// benefits, unlimited undo, selective undo
|
|
// export/import plugins
|
|
// everything is stored in the session
|
|
|
|
|
|
[[placement]]
|
|
Placements
|
|
~~~~~~~~~~
|
|
|
|
Generic mechanism to stitch together media objects. Any placement may contain
|
|
a list of conditions how to locate the placed object, examples being
|
|
time-absolute/relative, relative to another object, or relative to some
|
|
specific source media frame.
|
|
|
|
All of the session model contents are attached by placement, forming a large
|
|
tree. Placements are to be _resolved_ to find out the actual position, output
|
|
and further locational properties of an object. Missing placement information
|
|
is _inherited_ from parent placements in the session tree. This causes a lot
|
|
of relational and locational properties to be inherited from more global
|
|
settings, unless defined locally at a given object: time reference point,
|
|
output destination, layering, fade control, audio pan,...
|
|
|
|
|
|
Rendering Engine
|
|
~~~~~~~~~~~~~~~~
|
|
[red]#to be written#
|
|
rendering...
|
|
|
|
[[builder]]
|
|
[red]#to be written#
|
|
|
|
rules system
|
|
[red]#to be written#
|
|
|
|
I/O Subsystem
|
|
~~~~~~~~~~~~~
|
|
[red]#to be written#
|
|
|
|
// file handling
|
|
// vault, work, cache
|
|
// repositories
|
|
// explain details later
|
|
|
|
|
|
Configuration
|
|
~~~~~~~~~~~~~
|
|
[red]#to be written#
|
|
|
|
// configuration system
|
|
// serves defaults, actual data are stored in the session
|
|
|
|
|
|
Plugins/Interfaces
|
|
~~~~~~~~~~~~~~~~~~
|
|
[red]#to be written#
|
|
|
|
// explain whats it is
|
|
// portability
|
|
// versioning
|
|
|
|
|
|
|
|
Rendering Video
|
|
---------------
|
|
[red]#to be written#
|
|
|
|
// describe the flow of data to render a frame
|
|
|
|
// viewer
|
|
// final
|
|
|
|
|
|
Pulling a Frame
|
|
~~~~~~~~~~~~~~~
|
|
[red]#to be written#
|
|
|
|
// special cases,
|
|
// case studies,
|
|
// constrain viewer
|
|
|
|
// proxy
|
|
// viewer circuit
|
|
// render circuit
|
|
|
|
|
|
|
|
|
|
|
|
//Example Plugins
|
|
//---------------
|
|
|
|
// show some case-studies that someone gets a feel how plugins work
|
|
|
|
|
|
|
|
[red]#TODO# Consider integrating the following things into the document above
|
|
|
|
* plugins
|
|
* timeline
|
|
* pull
|
|
* bus defines rendering format
|
|
* caching
|
|
* frameserver
|
|
* screenconcept / perspectives
|
|
* automation
|
|
* 3 layered model
|
|
|
|
|
|
|
|
|
|
Glossary
|
|
--------
|
|
|
|
The above outline of the design uses a lot of special terms and common termes
|
|
used with specific meaning within Lumiera. To ease the understanding, we've
|
|
collected a link:Glossary.html[Glossary of common terms].
|
|
|