ichthyo/LUMIERA.clone
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

continue drafting the implementation

Browse source 
lots of details still to work out...
This commit is contained in:
Fischlurch 2007-12-10 07:37:56 +01:00
parent c905d06002
commit 5703451ee0
2 changed files with 94 additions and 19 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

BIN
wiki/draw/Proc.builder1.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 12 KiB

113
wiki/renderengine.html
View file

@ -570,7 +570,7 @@ For every Asset we generate a __Ident tuple__ and a long ID (hash) derived from
[img[how to implement Automation|uml/fig129669.png]]
</pre>
</div>
<div title="BasicBuildingOperations" modifier="Ichthyostega" modified="200712090611" created="200712040334" tags="design dynamic def Builder" changecount="15">
<div title="BasicBuildingOperations" modifier="Ichthyostega" modified="200712100627" created="200712040334" tags="design dynamic def Builder" changecount="20">
<pre>Starting out from the concepts of Objects, Placement to Tracks, Ports and connection properties (&amp;rarr; see [[here|TrackPortEDL]]) within the EDL, we can identify the elementary operations occuring within the Builder. Overall, the Builder is organized as application of //visiting tools// to a collection of objects, so finally we have to consider some object kind appearing in the working function of the given builder tool, which holds at this moment some //context//. The job now is to organize this context such as to create a predictable build process from this //event driven// approach.
!Builder working Situations
@ -587,7 +587,7 @@ For every Asset we generate a __Ident tuple__ and a long ID (hash) derived from
##* and regarding the chaining order
## next we have to assess the [[ports|Port]] to which the clip has been placed
## producing a [[wiring request|WiringRequest]] for every pair {{{(chainEndpoint, port)}}}
# attaching an ''Effect'' is actually always an //insertion operation// which is done by //prepending// to the previously built nodes
# [&gt;img[draw/Proc.builder1.png]] attaching an ''Effect'' is actually always an //insertion operation// which is done by //prepending// to the previously built nodes. Effects may be placed as attached to clips and ports, which causes them to be included in the processing chain at the given location. Effects may as well be placed at an absolute time, which means they are to be applied to every clip that happens to be at this time &amp;mdash; but this usecase will be reolved when creating the Fixture, causing the effect to be attached to the clips in question. The same holds true for Effects put on tracks.
# treating an ''wiring request'' means
## detecting possible and impossible connections
## deriving additional possible &quot;placement dimensions&quot; generated by executing such an connection (e.g. connecting a mono source to a spatial sound system bus creates panning possibilities)
@ -600,7 +600,7 @@ For every Asset we generate a __Ident tuple__ and a long ID (hash) derived from
After consuming all input objects and satisfying all wiring requests, the result is a set of [[exit nodes|ExitNode]] ready for pulling data. We call the network reachable from such an exit node a [[Processor]], together all processors of all segments and output data types comprise the render engine.
!!!dependencies
Ports need to be there first, as everything else will be placed to a port at some point. The same holds true for tracks. But, on the other hand, both are optional. We can have EDLs with ~MObjects without configuring ports (but won't be able to build any render processor of course). And we could have an EDL without any track, if we place every ~MObject within this EDL directly to some port.
Ports need to be there first, as everything else will be placed to a port at some point. The same holds true for tracks. But, on the other hand, both are optional. We can have ~EDLs with ~MObjects without configuring ports (but won't be able to build any render processor of course). And we could have an EDL without any track, if we place every ~MObject within this EDL directly to some port.
Effects can be attached only to already existing pipelines, starting out at some port or clip. Besides that, all further parts can be built in any order and independent of each other. This is made possible by using [[wiring requests|WiringRequest]], which can be resolved later on. So, as long as we start out with the tracks (to resolve any port they are placed to), and further, if we manage to get any effect placed to some clip-MO //after// treating the clip, we are fine and can do the building quasi event driven.
@ -720,7 +720,7 @@ config.macros.timeline.handler = function(place,macroName,params,wikifier,paramS
}
//}}}</pre>
</div>
<div title="BuildProcess" modifier="Ichthyostega" modified="200712090625" created="200706190658" tags="dynamic Builder" changecount="26">
<div title="BuildProcess" modifier="Ichthyostega" modified="200712100440" created="200706190658" tags="dynamic Builder" changecount="27">
<pre>All decisions on //how // the RenderProcess has to be carried out are concentrated in this rather complicated Builder Subsystem. The benefit of this approach is, besides decoupling of subsystems, to keep the actual performance-intensive video processing code as simple and transparent as possible. The price, in terms of increased complexity &amp;mdash; to pay in the Builder &amp;mdash; can be handled by making the Build Process generic to a large degree. Using a Design By Contract approach we can decompose the various decisions into small decision modules without having to trace the actual workings of the Build Process as a whole.
[&gt;img[Outline of the Build Process|uml/fig129413.png]]
@ -731,13 +731,14 @@ Note, //we still have to work out how exactly building, rendering and playback w
&amp;rarr;see also: BasicBuildingOperations
&amp;rarr;see also: BuilderStructures
&amp;rarr;see also: PlanningBuildFixture
&amp;rarr;see also: PlanningSegementationTool
&amp;rarr;see also: PlanningNodeCreatorTool
[img[Colaborations in the Build Process|uml/fig128517.png]]
</pre>
</div>
<div title="Builder" modifier="Ichthyostega" modified="200712090634" created="200706220317" tags="def overview" changecount="23">
<div title="Builder" modifier="Ichthyostega" modified="200712100631" created="200706220317" tags="def overview" changecount="24">
<pre>The Builder takes some MObject/[[Placement]] information (called Timeline) and generates out of this a Render Engine configuration able to render this Objects. It does all decisions and retrieves the current configuration of all objects and plugins, so the Render Engine can just process them stright forward.
The Builder is the central part of the [[Builder Pattern|http://en.wikipedia.org/wiki/Builder_pattern]]
@ -745,7 +746,7 @@ The Builder is the central part of the [[Builder Pattern|http://en.wikipedia.org
As the builder has to create a render node network implementing most of the features and wiring possible with the various MObject kinds and placement types, it is a rather complicated piece of software. In order to keep it manageable, it is broken down into several specialized sub components:
* clients access builder functionality via the BuilderFacade
* the [[Proc-Layer-Controller|Controller]] initiates the BuildProcess and does the overall coordination of scheduling edit operations, rebuilding the fixture and triggering the Builder
* to carry out the building, we use several tools (SegementationTool, NodeCreaterTool,...), which are supplied by the [[tool factory|BuilderToolFactory]]
* to carry out the building, we use several tools (SegmentationTool, NodeCreaterTool,...), which are supplied by the [[tool factory|BuilderToolFactory]]
* the actual building (i.e. the application of those tools to the timeline) is done by the [[Assembler|BuilderAssembler]], which is basically a collection of functions (but has a small amount of global configuration state)
* any non-trivial wiring of render nodes, tracks, ports and automation is done by the services of the [[connection manager|ConManager]]
</pre>
@ -826,12 +827,16 @@ The main tool used to implement this separation is the [[Builder Pattern|http://
Another pertinent theme is to make the basic building blocks simpler, while on the other hand gaining much more flexibility for combining these building blocks. For example we try to unfold any &quot;internal-multi&quot; effects into separate instances (e.g. the possibility of having an arbitrary number of single masks at any point of the pipeline instead of having one special masking facility encompassing multiple sub-masks. Similarly, we treat the Objects in the EDL in a more uniform manner and gain the possibility to [[place|Placement]] them in various ways.
</pre>
</div>
<div title="EDL" modifier="Ichthyostega" modified="200709272059" created="200706210610" tags="def" changecount="6">
<div title="EDL" modifier="Ichthyostega" modified="200712100537" created="200706210610" tags="def" changecount="8">
<pre>''EDL'' is a short-hand for __E__dit __D__ecision __L__ist. The use of this term can be confusing; for the usual meaning see the definition in [[Wikipedia|http://en.wikipedia.org/wiki/Edit_decision_list]]
Cinelerra uses this term in a related manner but with a somewhat shifted focus (and we just stick to this usage here): In Cinelerra the EDL is comprised of the whole set of clips and other media objects parametrized and placed onto the tracks by the user. It is the result of the user's //editing efforts.//
In this usage, the EDL in most cases will be almost synonymous to the [[Session|SessionOverview]], just the latter emphasizes more the state aspect, as it can be thought as the current EDL contents contained in a file or data structure together with additional Option values and settings for the GUI. The Session is what you save and load, while the EDL rather denotes a structured collection of Objects placed in time.
The EDL within the Session is just a //logical grouping//, allowing for lots of flexibility. Actually, we can have several ~EDLs within one Session, and these ~EDLs can be linked together or not, they may be in sequence or may constitue a logical grouping of clips used simultanously in compositional work etc. Multiple ~EDLs can use the same or different tracks, and tracks as well are only an organisational (grouping) device. But at any time, we have exactly one [[Fixture]], derived automatically from all ~EDLs and containing the content actually to be rendered.
&amp;rarr; see considerations about [[the role of Tracks and Ports in conjunction with the EDL|TrackPortEDL]]
</pre>
</div>
<div title="EditingOperations" modifier="Ichthyostega" modified="200710111508" created="200709251610" tags="design decision" changecount="5">
@ -952,15 +957,14 @@ Some further details
* and, as the created smart-pointer is a template parameter, such a custom Functor can create all sorts of Proxies, wrappers and the like
</pre>
</div>
<div title="Fixture" modifier="Ichthyostega" modified="200709270236" created="200706220324" tags="def" changecount="3">
<div title="Fixture" modifier="Ichthyostega" modified="200712100439" created="200706220324" tags="def" changecount="4">
<pre>a specially configured EDL
* all MObjects have their position, length and configuration set up ready for rendering.
* compound objects (e.g. multichannel clips) have been resolved to single non-compound basic objects.
* every MObject is associated with an ExplicitPlacement, which declares a fixed position (Time, Track)
* this ~ExplicitPlacements are contained in a ordered List called the Timeline
As the builder and thus render engine //only consults the fixture,// while all editing operations finally propagate to the fixture as well, we get an isolation layer between the high level part of the Proc layer (editing, object manipulation) and the render engine
</pre>
As the builder and thus render engine //only consults the fixture,// while all editing operations finally propagate to the fixture as well, we get an isolation layer between the high level part of the Proc layer (editing, object manipulation) and the render engine. Creating the Fixture can be seen as a preprocessing step to simplify the build process. For this reason, the process of [[(re)building the fixture|PlanningBuildFixture]] has been designed together with the [[Builder]]</pre>
</div>
<div title="Frame" modifier="Ichthyostega" modified="200706220333" created="200706220332" tags="def" changecount="2">
<pre>This term has //two meanings, //so care has to be taken for not confusing them.
@ -1421,10 +1425,10 @@ From experiences with other middle scale projects, I prefer having the test code
[img[Example: Interfaces/Namespaces of the ~Session-Subsystems|uml/fig130053.png]]
</pre>
</div>
<div title="Loading Media" modifier="Ichthyostega" modified="200709240049" created="200709220005" tags="design spec" changecount="2">
<div title="Loading Media" modifier="Ichthyostega" modified="200712100630" created="200709220005" tags="design spec" changecount="4">
<pre>Opening and accessing media files on disk poses several problems, most of which belong to the domain of cinelerra's data backend. Here, we focus on the questions related to making media data available to the EDL and the render engine. Each media will be represented by an MediaAsset object, which indeed could be a compound object (in case of MultichannelMedia). Building this asset object thus includes getting informations from the real file on disk. For delegating this to the backend, we use the following query interface:
* {{{queryFile(char* name)}}} requests accessing the file and yields some (opaque) handle when successful.
* {{{queryChannel(FHandle, int)}}} will then be issued in sequence with ascending index numbers, until it returns {{{NULL}}}.
* {{{queryChannel(fHandle, int)}}} will then be issued in sequence with ascending index numbers, until it returns {{{NULL}}}.
* the returned struct (pointer) will provide the following information:
** some identifier which can be used to create a name for the corresponding media (channel) asset
** some identifier characterizing the access method (codec) needed to get at the media data. This should be rather a high level description of the media stream type, e.g. &quot;H264&quot;
@ -1557,6 +1561,9 @@ So, when creating a clip out of such a compound media asset, the clip has to be
* we distinguish elementay (non-compound) Clips from compound clips by concrete subtype. The builder can only handle elementary clips, because he needs to build a separate pipeline for every output channel. So the work of splitting common effect stacks for clips with several channels needs to be done when calculating the Fixture for the current EDL. The Builder expects to be able to build the render nodes corresponding to each entity found in the Fixture one by one.
* the Builder gets at the ProcPatt (descriptor) of the underlying media for each clip and uses this description as a template to build the render pipeline. That is, the ProcPatt specifies the codec asset and maybe some additional effect assets (deinterlace, scale) necessary for feeding media data corresponding to this clip/media into the render nodes network.</pre>
</div>
<div title="NodeCreaterTool" modifier="Ichthyostega" created="200712100626" tags="def" changecount="1">
<pre>NodeCreaterTool is a [[visiting tool|VisitorUse]] used as second step in the [[Builder]]. Starting out from a [[Fixture]], the builder first [[divides the Timeline into segments|SegmentationTool]] and then processes each segment with the NodeCreaterTool to build a render nodes network (Render Engine) for this part of the timeline. While visiting individual Objects and Placements, the NodeCreaterTool creates and wires the necessary [[nodes|ProcNode]]</pre>
</div>
<div title="ObjectCreation" modifier="Ichthyostega" modified="200711130113" created="200709030139" tags="impl design" changecount="17">
<pre>We have to consider carefully how to handle the Creation of new class instances. Because, when done naively, it can defeat all efforts of separating subsystems, or &amp;mdash; the other extreme &amp;mdash; lead to a //switch-on-typeID// programming style. We strive at a solution somewhere in the middle by utilizing __Abstract Factories__ on Interface or key abstraction classes, but providing specialized overloads for the different use cases. So in each use case we have to decide if we want to create a instance of some general concept (Interface), or if we have a direct collaboration and thus need the Factory to provide a more specific sub-Interface or even a concrete type.
@ -2186,10 +2193,59 @@ Placements have //value semantics,// i.e. we don't stress the identity of a plac
</pre>
</div>
<div title="PlanningNodeCreaterTool" modifier="Ichthyostega" modified="200712100258" created="200712090659" tags="impl Builder" changecount="5">
<div title="PlanningBuildFixture" modifier="Ichthyostega" modified="200712100633" created="200712100445" tags="impl Builder" changecount="10">
<pre>//This page is a scrapbook for working out the implementation of how to (re)build the [[Fixture]]//
Structurally, (re)building the Fixture rather belongs to [[Session]]/[[EDLs|EDL]], but it is implemented very similar to the render engine build process: by treating all ~MObjects found in the various ~EDLs with a common [[visiting tool|VisitorUse]], this tool collects a simplified view with everyting made explicit, which can be pulled of as Fixture, i.e. (special kind of) EDL
afterwards.
* there is a //gathering phase// and a //solving phase//, the gathering is done by visiting.
* during the gathering phase, there ''need to be a lock'' preventing any other edit operation.
* the solving is delegated to the individual ~Placements. It is effectively a {{{const}}} operation creating a ExplicitPlacement (copy)
* thus the Fixture contains these newly created ~ExplicitPlacements, refering to ~MObjects shared with the original Placements within the ~EDLs
!!!prerequisites
* Session and ~EDLs exist.
* Ports exist and are configured
!!!postconditions
* the Fixture contains one sorted timeline of ExplicitPlacement instances
* Anything in this list is actually to be rendered
* {{red{TODO: how to store and group the effects?}}}
* any meta-clips or other funny things have been resolved to normal clips with placement
* any multichannel clips has been broken down to elementary clips {{red{TODO: what is &quot;elementary&quot;. e.g. stereo sound streams?}}}
* any globally or otherwise strangely placed effects have been attached either to a clip or to some port
* we have one unified list of tracks
&lt;&lt;tasksum start&gt;&gt;
&lt;&lt;taskadder below&gt;&gt;
&lt;&lt;task &gt;&gt; work out how to get the processing of effects chained to some clip right
&lt;&lt;task &gt;&gt; work out how to handle multichannel audio (and stereo video)
!gathering phase
!!preparing
&lt;&lt;task&gt;&gt;what data collections to build?
!!treating a Track
&lt;&lt;task&gt;&gt;work out how to refer to ports and do other config
&lt;&lt;task&gt;&gt;get some uniqe identifier and get relevant properties
!!treating a {{{Placement&lt;Clip&gt;}}}
&lt;&lt;task&gt;&gt;check the direct enablement status
&lt;&lt;task&gt;&gt;asses the compound status, maybe process recursively
!!treating an {{{Placement&lt;Effect&gt;}}}
&lt;&lt;task&gt;&gt;find out the application point {{red{really?}}}
!solving phase
&lt;&lt;task&gt;&gt;trigger solving on all placements
&lt;&lt;task&gt;&gt;sort the resulting ~ExplicitPlacements
&lt;&lt;tasksum end&gt;&gt;
</pre>
</div>
<div title="PlanningNodeCreaterTool" modifier="Ichthyostega" modified="200712100626" created="200712090659" tags="impl Builder" changecount="6">
<pre>//This page is a scrapbook for working out the implementation of the builder//
* NodeCraterTool is a [[visiting tool|VisitorUse]]
* NodeCreaterTool is a [[visiting tool|VisitorUse]]
* the render engine to be built is contained as state within this tool object while it is passed around
!!!prerequisites
* Session and ~EDLs exist.
@ -2577,15 +2633,34 @@ The system is ''open'' inasmuch every part mirrors the structure of correspondin
* see also the [[Entities involved in Rendering|RenderEntities]]
</pre>
</div>
<div title="SessionOverview" modifier="Ichthyostega" created="200709272105" changecount="1">
<pre>The Session (sometimes also called //Project//) contains all informations and objects to be edited by the User. It can be saved and loaded. The individual Objects within the Session, i.e. Clips, Media, Effects, are contained in one (or several) collections within the Session, which we call [[EDL (Edit Decision List)|EDL]]. Moreover, the sesion contains references to all the Media files used, and it contains various default or user defined configuration. At any given time, there is //only one current session// opened within the application.
<div title="Session" modifier="Ichthyostega" modified="200712100526" created="200712100525" tags="def" changecount="3">
<pre>The Session contains all informations, state and objects to be edited by the User. From a users view, the Session is synonymous to the //current Project//. It can be saved and loaded. The individual Objects within the Session, i.e. Clips, Media, Effects, are contained in one (or several) collections within the Session, which we call [[EDL (Edit Decision List)|EDL]]. &amp;rarr; [[Session design overview|SessionOverview]]
!Session structure
The Session object is a singleton &amp;mdash; actually it is a »~PImpl«-Facade object (because the actual implementation object can be swapped for (re)loading Sessions).&lt;br/&gt;The Session is comprised of
* a collection of ''EDL'' objects
* a ''current EDL'', which can be switched and accessed {{red{TODO not sure if I keep this design...}}}
* a collection of ''global Ports''
* the ''Fixture'' with a possibility to [[(re)build it|PlanningBuildFixture]]</pre>
</div>
<div title="SessionOverview" modifier="Ichthyostega" modified="200712100615" created="200709272105" tags="design" changecount="13">
<pre>The [[Session]] (sometimes also called //Project//) contains all informations and objects to be edited by the User. It can be saved and loaded. The individual Objects within the Session, i.e. Clips, Media, Effects, are contained in one (or several) collections within the Session, which we call [[EDL (Edit Decision List)|EDL]]. Moreover, the sesion contains references to all the Media files used, and it contains various default or user defined configuration. At any given time, there is //only one current session// opened within the application.
!!!larger projects
For larger editing projects the simple structure of a session containing &quot;the&quot; timeline is not sufficient. Rather, we have several timelines, e.g. one for each scene. Or we could have several layered or nested timelines (compositional work, multimedia productions). To support these cases without making the default case more complicated, Cinelerra-3 introduces a //focus// for selecting the //current EDL,// which will receive all editing operations.
!!!the definitive state
With all the structural complexities possible within such a session, we need an isolation layer to provide __one__ definitive state where all configuration has been made explicit. Thus the session manages one special object list, the [[Fixture]], which can be seen as all currently active object placed onto a single timeline.
</pre>
!!!organisational devices
The possibility of having multiple ~EDLs helps organizing larger projects. Each [[EDL]] is just a logical grouping; because all effective properties of any MObject within this EDL are defined by the ~MObject itself and the [[Placement]], by which the object is anchored to some time point, some track, can be connected to some port, or linked to another object. In a similar manner, Tracks are just another organisational aid for grouping objects, disabling them and defining common output ports.
!!!global ports
Any session should contain a number of global [[(destination) ports|Port]], typically video out and audio out. The goal is, to get any content producing or transforming object in some way connected to one of these outputs, either //by [[placing|Placement]] it directly// to some port, or by //placing it to a track// and having the track refer to some port. Besides the global destination ports, we can use internal ports to form busses or subgroups, either on a global (session) level, or attached to individual tracks. Normally, ports just gather and mix data, but of course any port can have an attached effect chain. (&amp;rarr; see [[more on Tracks and Ports within the EDL|TrackPortEDL]])
!!!default configuration
[&gt;img[draw/Proc.builder1.png]]While all these possibilities may seem daunting, there is a simple default configuration loaded into any pristine new session:
It will contain a global video and audio outport, just one EDL with one track; this track will have a internal video and audio port (bus) configured with one fading device sending to the global output ports. So, by adding some clip with a simple absolute placement to this track and some time position, the clip gets connected and rendered, after [[(re)building|PlanningBuildFixture]] the [[Fixture]] and passing the result to the [[Builder]] &amp;mdash; and using the resulting render nodes network (Render Engine).</pre>
</div>
<div title="SideBarOptions" modifier="CehTeh" created="200706200048" changecount="1">
<pre>&lt;&lt;search&gt;&gt;&lt;&lt;closeAll&gt;&gt;&lt;&lt;permaview&gt;&gt;&lt;&lt;newTiddler&gt;&gt;&lt;&lt;saveChanges&gt;&gt;&lt;&lt;slider chkSliderOptionsPanel OptionsPanel &quot;options »&quot; &quot;Change TiddlyWiki advanced options&quot;&gt;&gt;</pre>
@ -3856,7 +3931,7 @@ function addKeyDownHandlers(e)
</pre>
</div>
<div title="TrackPortEDL" modifier="Ichthyostega" modified="200712090611" created="200711300405" tags="design discuss def decision Builder" changecount="12">
<div title="TrackPortEDL" modifier="Ichthyostega" modified="200712100628" created="200711300405" tags="design discuss def decision Builder" changecount="21">
<pre>''towards a definition of »Track«''. We don't want to tie ourself to some naive and overly simplistic definition, just because it is convenient. For classical (analogue) media, tracks are physical entities dictated by the nature of the process by which the media works. Especially, Tape machines have read/writing heads, which creates fixed tracks to which to route the signals. This is a practical geometric necessity. For digital media, there is no such necessity. We are bound primarily by the editor's habits of working.
!!!Assessment of Properties
@ -3870,7 +3945,7 @@ This is to say we have //several degrees of freedom// within this organisational
Starting with the assumption &quot;everything is just connected processing nodes&quot;, Tracks may seem superfluous. The problem with this approach is: it doesn't scale well. While it is fine to be able to connect clips and effects as we see fit (indeed, we want to build such a system), it is clearly not feasible to wire every clip manually to the output ports or add a panner effect to each and every audio sample. Because while editing, most of the time things are done in a fairly regular and systematic manner. Preferably we use the tracks as //preconfigured group setup// and just //place media onto them;// any such [[Placement]] can do the necessary wiring semi-automatic (rule-based).
!!!the constant part
there seems to be some non time-varying part in each EDL, that doesn't fit well with the simple model &quot;objects on a timeline&quot;. Tracks seen as an organisational grid fall into this category: they are a global property of the given EDL. They could be associated to the Session as a whole, but effectively this would subvert the concept of having several EDLs. On the other hand,
there seems to be some non time-varying part in each EDL, that doesn't fit well with the simple model &quot;objects on a timeline&quot;. Tracks seen as an organisational grid fall into this category: they are a global property of the given EDL. They could be associated to the Session as a whole, but effectively this would subvert the concept of having [[several EDLs|SessionOverview]]. On the other hand,
[[ports|Port]] for Video and Sound output are obviously a global property of the Session. There can be several global ports forming a matrix of subgroup busses. We could add ports to tracks as well, but we don't do this, because, again, this would run counter to our attempt of treating tracks as merely organisational entities. We have special [[source ports|ClipSourcePort]] on individual clips though, and we will have ports on meta-clips too.
!Design