Lumiera (as seen) from Outer Space ================================== :Author: Lumiera_Core_Developers :Date: Summer 2010 [abstract] ****************************************************************************** The Lumiera Community is in the process of making a non-linear video editing and compositing FOSS application for Linux/Unix/Posix operating systems. The application is geared towards professional, high-quality work; but it is equally suitable for low-end users, due to its in-design scalability. Lumiera builds on common open source video, sound and GUI toolkits and libraries, being highly flexibile, configurable---user-control over a broad spectrum of configurable parameters---and with smooth workflows that scale well to larger more intricate projects and smaller projects. This document outlines the design from a more general perspective, providing potential users with sufficient insight into the tools and technology behind Lumiera to start working with Lumiera quickly. ****************************************************************************** // all things starting with '//' are asciidoc comments and drafts/notes while // working on this document .About this Document // It contains many hyper-links to explanations which are denoted by an arrow ->. Lumiera is still under active development. Here we describe planned features without explicitly tagging them; some points have still to be worked out in detail. Although this document is heavily cross-linked, we try to start with a broad overview, then develop details towards the end. Vision ------ // objective and goals of the project Lumiera strives towards being a _professional non-linear video editor_. It is important to note that by ``professional'' we do not necessarily imply ``commercial'' or even ``industrial''. We do, however, wish to suggest an attitude or frame of mind in how this work is approached, but not the goal, requirement or purpose of this work. The concept of professionality in film editing can indicate something of an artistic nature, suggest that there is a narration, meaning or political messages revealed by the work; or, simply, that the work attempts to convey something to an audience. Anyhow, for the tools, the editing software used to this end, we can identify several properties and requirements, to be labeled ``professional'': With this perspective in mind, we can identify a number of key properties film production tools should have and be professional: Reliability:: All work must be secure and be safe against any software glitches, incompatibility or instability. Ideally, Lumiera should be reliable, very stable and not susceptible to software crashes, i.e., crashes or power failure should ideally not result in data or work loss. Quality:: The demands placed on high-quality, cinema grade digital video material requires crisp-quality results with no concessions made throughout the entire workflow that might compromise quality of the final work. It will be necessary to be able to reconstruct all rendering down to a single digit. Performance and Productivity:: Professionals want to get things done in time and content, but, ideally, with control over all detail. The fine balance of these goals is a central goal of workflow design and usability. Scalability and Adaptability:: Projects and budgets differ, hardware advances; and Lumiera must scale in different dimensions and make optimum use of the resources that are available. From small Laptops to multi-core computers up to render farms, Lumiera must be adept in optimum use of the hardware at hand. Durability:: The rapid pace at which software and hardware rampage forward is surely a warning to new software projects and the dangers of locking into any current technological ``fashion'' to achieve a ``cheap'' goal, feature or performance boost. Once the fad fades, the software woes begin. The software must be able to engage new technological developments without any compromise to functionality or backward compatibility. Fundamental Forces ------------------ // the basic ideas which drive the Lumiera design The Lumiera design is guided by a small number of basic principles. Keeping these principles in mind will help you understand how more interesting things can be built up from these fundamental principles. Open Ended Combination of Building Blocks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Lumiera is not so much defined in terms of _features_. Instead it can be better envisaged as a workshop in which the emphasis is on individual _basic building-blocks_ that can be combined together in interesting ways to form more complex structures. These basic modules, entities or objects each have a distinct _type_ which explicitly limits the connections between the modules. Within these limits, any conceivable combination is possible without additional hidden restrictions. Lumiera is not a set of Lego bricks, nor is it a business application driven by finite usage stories. Medium Level Abstraction and Project Specific Conventions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ These building blocks within Lumiera create a moderate level of abstraction; a user may, if desired, directly manipulate -- using, for example, GUI clips -- individual effects, masks, and even the placements xref:placement[->] used to stitch the objects together, which is comparatively low-level. Such abstraction protects the user from technical details such as format conversion or access to individual channels. To complement this approach, Lumiera does _not_ rely on hard-wired, global conventions -- rather we allow the building up of project specific conventions and rules xref:rules[->] to fit a given requirement and preferred working style. Lumiera will be supplied with a conventional template and a default configuration to ease the learning curve #for new users. [[graphs]] Rendering is Graph Processing ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Processing video (and audio) data can be conceived as a graph (more precisely ``directed acyclic graphs''). In this model of video processing, data flows along the edges of the graph and is processed at the nodes. The following figure depicts manipulating video data as a graph. The nodes of the root of the graph is where data input occurs. From there, the data moves on to the next nodes: the direct syblings of the root. Here, the video data pre-processing occurs. All other operations on the data can be represented by nodes, and data flows from one operation to the next along the nodes of the graph. image:{imgd}/lumiera_big_graph.png[Example for a graph] When we look at this model we discover that we only need to build xref:builder[->] such graphs, the nodes themselves can be seen as black boxes and will be implemented by plugins xref:plugins[->]. Moreover one can pre-configure subgraphs and handle them as a single entity xref:pluginstack[->]. In Lumiera everything will be translated into such a graph. Footage will be demultiplexed xref:demultiplexer[->] at the first node, thereafter to the encoding stage xref:encoder[->] and on to the multiplexer xref:multiplexer[->] which assembles the final video. Pulling not Pushing ~~~~~~~~~~~~~~~~~~~ At a first glance, it looks fairly natural to set up the graphs xref:graphs[->] as described above. Data can then be pushed into the system through the input nodes and the final result can be seen at the output node. Several multimedia frameworks use this approach. However this scheme exhibits a number of shortcomings which make it inappropriate for non-linear video editing. Lumiera takes a different approach. Data is pulled through a pipe, i.e., a request starts at the output node and makes its way back up through the graph to the inputs. This scheme offers a number of advantages over the naive scheme xref:pull[->], beu we will defer the reasons until later. Don't waste work ~~~~~~~~~~~~~~~~ Rendering A/V data can be CPU intensive. To relieve the CPU workload by not rendering material twice or avoiding having to throw away material because it could not be rendered in time, Lumiera employs a sophisticated means of using cache xref:caching[->] and profiling xref:profiling[->]. The visible Universe -------------------- // coarse overview whats seen on the gui, details later Now its time to take a look at the prelimary Lumiera GUI: image:{l}/images/lumiera_gui_small.png[Current Lumiera GUI Screenshot] Lumiera consists of three major parts: the GUI, proc and backend. The initial GUI Lumiera will ship with is not the only GUI possible, indeed Lumiera can work satisfactorily without a GUI, for example, for specail purposes. headless rendernode xref:rendernode[->] or frameserver xref:frameserver[->]. Later scripts will be written for automated processing. Special purpose GUIs are also envisaged later. The GUI screenshot you above is more or less the default you see when when you start Lumiera up for the first time. A 2nd viewer is planned for later to be added to the default. We support a much more sophisticated screen concept xref:screenconcept[->] to adapt to different workplaces and workflows. Viewer ~~~~~~ The viewer is an area where material can be displayed, i.e., ``play-back'', which also supports audio playback connections. As there are many sources that can be displayed, a viewer is attached to a source via the viewer switch board. Timelines, probepoints, wiretaps and individual clips are examlpes of sources that can be atached to a viewer. Moreover, the number of viewers open at any one time is only limited by the hardware, and each viewer can be collapsed, hooked up to a beamer or monitor. Transport Controls ~~~~~~~~~~~~~~~~~~ The layout in current gui is rather preliminary -- it is not finally decided where transport controls will be integrated; possibly as its own gui element This are devices either controlled by widgets or by some input device (MIDI, control surface, etc) so their gui may look differently. Either way they connect to a Play Controler xref.. in the core which manages playing and cursor positioning. thus there will be some gui facility to attach Transport controls to Play Controllers. Transport controls are ganged when they attach to the same Play Controler. just playing some footage for preview creates a simple internal timeline, no magic here. // TODO: bit unrelated, think about how ganging controls in general should // work, also for faders, masks and so on // Note by Ichthyo: the connection to a fader is handled through the placements, // which allows to inherit such a control connection. IMHO together with the // tree-like tracks this removes 80% of the need to gang faders. Timeline View ~~~~~~~~~~~~~ A timeline is a container that provides a time axis and an output. The output can de derived from various sources and have different configurations. An output can have various configurations, for each output configuration, there will be one timeline. A timeline does not temporally arrange material, this is performed by a sequence, which can be snapped to a timeline. A typical film will define many sequences, but only a few timelines. A sequence contains a number of tracks which are ordered in a hierarchy. Tracs do not have any format associated with them and more or less anything can be put into a track. Consequently, audio and video material can be equally assigned to a track, there is no discrimination between audio and video in the Lumiera concept of a track. A timeline must be assigned to viewer if playback viewing is desired. //Format Independent Timeline, one can put anything on the timeline. //the busses constrain what kind of data is pulled out and in turn the //builder creates a processing graph which does the necessary conversions and //stuff. // // Q: how to handle interaction, for example when some conversion can only be // done in a lossy way and some conversion node may or may not be inserted // (i mean gui wise)? // A: this usually is detected at build time, which means the incriminating // object and exit node is just in scope when the problem is detected. // My intention was to have a problem flag with accompanying information // attached to this object, so the GUI can highlight the problem location // and give a general alert. // TBD: Cursors .. discussion, handling, gui representation // Note by Ichthyo: we shouldn't focus on cursors, but rather on selections. // IMHO a playhead or edit marker or similar cursor is just // a special case of a selection. Busses ~~~~~~ The GUI provides a separate _bus view_, showing the master busses (subgroups) in a manner similar to an audio mixing desk. Any bus is just a means of collecting and and adding (aka overlaying) together the output of various kinds of media (video, audio, number of channels), produced by various processing elements and from other busses. These global busses can be concieved as being part of the timeline. Asset View ~~~~~~~~~~ We can conceive the Asset View as the timeline's book keeper: it manages the various constituent in the timeline. Moreover, in addition to managing timeline constituents, raw material, clips, bins (folders) are managed by the Asset View, i.e., typical management operations including, deleting, adding, naming, tagging, groupping into bins, etc. all occur here. Plugins are also managed in the Asset View. Manages all assets available in one project. * source media/footage/soundfiles * all available effects and transitions * internal artefacts like sequences and automation data sets // First this will be simply implemented showing data loaded into the session // and all available plugins/effects // The user may build custom effect collections ("effect palette") // (much) Later it is planed to make this a database driven interface, where // the tabs showing things are basically just database queries. Then it // becomes possible to create/extend this by customized queries and augment // assets with all kinds of metadata which can be queried // Actually, the same underlying data structure is used to implement the // asset view with folders, clip bins and effect palettes, and the timeline // view with tracks, clips and attached effects. Technically, there is no // difference between a track or a clip bin -- just the presentation varies. // Timeline contents can be viewed like assets for bookkeeping purposes, and // the contents of a clip bin can be played like a storyboard '''''''' Dark Matter ----------- The material in this section provides a cursory view of features not required by a typical user, but of more importance to people loking under the hud, i.e., programers, etc. Most of the material in this section is to be found in the proc layer and in the backend. Session storage ~~~~~~~~~~~~~~~ [red]#to be written# //databank with logging, no data loss. // not generateable data // its the timeline mostly // session storage // benefits, unlimited undo, selective undo // export/import plugins // everything is stored in the session [[placement]] Placements ~~~~~~~~~~ Generic mechanism to stitch together media objects. Any placement may contain a list of conditions how to locate the placed object, examples being time-absolute/relative, relative to another object, or relative to some specific source media frame. All of the session model contents are attached by placement, forming a large tree. Placements are to be _resolved_ to find out the actual position, output and further locational properties of an object. Missing placement information is _inherited_ from parent placements in the session tree. This causes a lot of relational and locational properties to be inherited from more global settings, unless defined locally at a given object: time reference point, output destination, layering, fade control, audio pan,... Rendering Engine ~~~~~~~~~~~~~~~~ [red]#to be written# rendering... [[builder]] [red]#to be written# rules system [red]#to be written# Input-Output Subsystem ~~~~~~~~~~~~~~~~~~~~~~~ Lumiera will process large quantities of data and it is of critical imnportance to perform this efficiently. The input and output subsystem are all processed in the backend, in fact, this is one very important function provided by the back end. The typical Lumiera user will have many clips, in various configurations located at various places. All this data will have to be stored by the backend, moreover all this data will have to be rapidly retrieved from storage and provided to the user. The demands on memory are high: huge chunks of data, that can be quickly stored and equally quickly fetched, even if stored over longer periods of time. Moreover, due to the scalability requirement of Lumiera, this process will have to perform adequately on lower-end hardware, and perform efficiently on higher-end hardware. Lumiera will break down processes that need to be processed into smaller units called _tasks_. Typically, there will be many hundreds of tasks waiting for processing at any one time. These tasks are qued for processing and the order in which this is performed is managed by the _scheduler_. This is all done in the back end. Apart from memory, the backend will be responsible for accessing and saving files. It will be of the utmost to do this efficiently. This will be carried out in the backend using low-level mechanisms. // file handling // vault, work, cache // repositories // explain details later Configuration ~~~~~~~~~~~~~ [red]#to be written# // configuration system // serves defaults, actual data are stored in the session Plugins/Interfaces ~~~~~~~~~~~~~~~~~~ What are Plugins? ^^^^^^^^^^^^^^^^^ A Plug-in is a kind of generalisation of a library. All applications use, to varying degrees of intensity, libraries. A programmer will not reinvent the wheel each time he sits down to programme an application. A programmer will typically borrow and use features and functionality from other programmers---or even borrow from himself, stuff written long ago in the past. Such features are collected together in libraries. A library is used in an application by _linking_ the library into the application. (There are other things to be done, but we'll call these 'details', which wont concern us here.) There are different ways to _link_ a library into an application: statically linking and dynamically linking. _Staticall Linking_ is done while the application is being built, or compiled. It is performed by the linker. The linker can perform some checks (mostly checks on syntax) and warn the user that some particular feature is being used incorrectly. The user can then correct the offending code, and recompile. There are a number of disadvantages associated with static linking. Features and libraries are being constantly improved. If the application wants to use new features, it will have to be recompiled with the new library which provides the new features. _Dynamic Linking_ helps rectify the necessity of having to recompile. If a new, improved library becomes available, all the user has to do is to install the new library onto the operating system, restart the application and the new features can be used by the application. The features provided by a dynamic library are loaded when the application starts to run. However both methods exibit a number of shortcomings. Wouldn't it be better if all features could be loaded only when needed? If features could be loaded only when needed, then they could also be unloaded when not required, thus saving memory and possibly increasing performance. This scheme of making features available to an application is known as run-time linking, aka plug-ins. Plug-ins offer other benifits: the application can continue to use both the old features and the new features together, side-by-side, by using the version number associated with the plug-in. This saves the application from considerable headaches associated with other linking methods, havocked library version incompatibility. Most modern applications use plug-ins, some are heavily dependent on plug-ins and only provide limited functionality without any plug-ins. Lumiera will not reinvent the wheel. One major goal is to provide considerable functionality via well-designed, external code supplied to Lumiera by plug-ins. How are Plugins Implemented? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Rendering Video --------------- [red]#to be written# // describe the flow of data to render a frame // viewer // final Pulling a Frame ~~~~~~~~~~~~~~~ [red]#to be written# // special cases, // case studies, // constrain viewer // proxy // viewer circuit // render circuit //Example Plugins //--------------- // show some case-studies that someone gets a feel how plugins work [red]#TODO# Consider integrating the following things into the document above * plugins * timeline * pull * bus defines rendering format * caching * frameserver * screenconcept / perspectives * automation * 3 layered model Glossary -------- The above outline of the design uses a lot of special terms and common termes used with specific meaning within Lumiera. To ease the understanding, we've collected a link:Glossary.html[Glossary of common terms].