Documentation: integrate the time quantisation concept pages

These pages from the TiddlyWiky feature a complete glossary
of terms relevant for time and timecode handling, plus the
architecural decisions related to this topic

doc/design/architecture/TimeQuant.txt

@@ -0,0 +1,212 @@
+Time values and Time quantisation
+=================================
+:Author:	Hermann Voßeler
+:Email:	<Ichthyostega@web.de>
+:Date:      Spring 2011
+
+//Menu: label Time Quantisation
+
+.Definitions
+The term &raquo;Time&laquo; spans a variety of vastly different entities. +
+Within a NLE we get to deal with various _flavours of time values._
+
+continuous time::
+  without any additional assumptions, *points in time* can be specified with
+  arbitrary precision. The time values are just numbers; the point of reference
+  and the meaning is implicit. Within Lumiera, time is encoded as integral
+  number of _micro ticks,_ practically continuous
+  
+time distance::
+  a range of time, a *distance* on the time axis, measured with the same
+  arbitrary precision as time points. Distances can be determined by
+  _subtracting_ two time points, consequently they are _signed numbers._
+
+offset::
+  a distance can be used to adjust (offset) a time or a duration: this means
+  applying a relative change. The _target_ of an offset operation is a time or
+  duration, while it's _argument_ is a distance (synonymous to offset).
+
+duration::
+  the length of a time range yields a *time metric*.
+  +
+  the duration can be defined as the _absolute value_ of the offset between
+  start and endpoint of the time range. A duration always abstracts from the
+  time _when_ this duration or distance happens, the relation to any time scale
+  remains implicit
+
+time span::
+  contrary to a mere duration, a *time interval* or time span is actually
+  _anchored_ at a specific point in time. It can be seen as a _special kind of
+  duration,_ which explicitly states the information _when_ this time span takes
+  place.
+
+changing time::
+  Time values are _immutable,_ like numbers.
+  +
+  Only a *time variable* can be changed -- yet some of the special time entities
+  can recieve link:TimeMutation[mutation messages], allowing e.g. for
+  adjustments to a time interval selection from the GUI
+
+''''
+internal time::
+  While the basic continuous time values don't imply any commitment regarding
+  the time scale and origin to be used, actually, within the implementation of
+  the application, the meaning of time values is uniform and free of
+  contradictions. Thus effectively there is an *implementation time scale* --
+  but its scope of validity is _strictly limited to the implementation level of
+  a single application instance._ It is never exposed and never persisted. It
+  might not be reproducible over multiple instantiations of the application. The
+  implementation reserves the right to recalibrate this internal scale. Later,
+  when Lumiera gains the capability to run within a network of render nodes,
+  these instance connections will include a negotiation about the internal time
+  scale, which remains completely opaque to the outer world. This explains, why
+  `lumiera::Time` instances lack the ability to show their time value beyond
+  debugging purposes. This is to avoid confusion and to stress their opaque
+  nature.
+
+wall clock and system time::
+  The core property of any external real world time is that it is _running_ --
+  we have to synchronise to an external time source. This implies the presence
+  of a _running synchronisation process,_ with the authority to adjust the time
+  base;
+  +
+  contrast this to the internal time, which is static and unconnected -- 
+
+quantised time::
+  The *act of quantisation* transforms a continuous property into a *discrete*
+  structure. Prominent examples can be found in the domain of micro physics and
+  with digital information processing. In a broader sense, any measurement or
+  _quantification_ also encompasses a quantisation. Regarding time and time
+  measurement, quantisation means alignment to a predefined *time grid*.
+  Quantisation necessarily is an _irreversible process_ -- possible
+  additional information is discarded.
+  +
+  Note that quantisation introduces a *time origin* and a *reference scale*
+
+frame count::
+  within the context of film and media editing, the specification of a *frame number*
+  is an especially important instance of quantisation.
+  +
+  all the properties of quantisation apply indeed to this special case: it is a
+  time measurement or specification, where the values are aligned to a grid, and
+  there is a reference time point where the counting starts (origin) and a
+  reference scale (frames per second). Handling of quantised time values in
+  Lumiera is defined such as to ensure the presence of all those bits of
+  information. Without such precautions, operating with bare frame numbers leads
+  itself to all kinds of confusions, mismatches, quantisation errors and
+  unnecessary limitations of functionality.
+
+timecode::
+  Quantisation also is the foundation of all kinds of formalised time specifications
+  actually even a frame count is some kind of (informal) timecode -- other timecodes
+  employ a standardised format. //Every// presentation of time values and every
+  persistent storage and exchange of such values is based on time codes.
+  Yet quantisation and time code aren't identical: a given quantised time value
+  typically can be cast into multiple timecode formats.
+
+
+
+Patterns for handling quantised time
+------------------------------------
+
+When it comes to actually handling quantised time values, several patterns are
+conceivable for dealing with the quantisation operation and representing
+quantised data. As guideline for judging these patterns, the general properties
+of time quantisation, as detailed above, should be taken into account.
+Quantising a time value means both _discarding information,_ while at the same
+time _adding explicit information_ pertaining the assumptions of the context.
+
+.casual handling
+this is rather an frequently encountered *anti pattern*. When reading such
+code, the most striking observation is the sense of general unawareness of the
+problem, which is then ``discovered'' on a per case base, which leads to
+numerous repetitions of the same basic undertakings, but done with individual
+treatment of each instance (not so much copy-n-paste). +
+Typical code smells:
+
+  * the rounding, modulo and subtract-base operations pertinent with scale
+    handling are seemingly inserted as bugfix
+  * local code path forks to circumvent or compensate for otherwise hard wired
+    calculations based on specific ways to invoke a function
+  * playing strikingly clever tricks or employing heuristics to "figure out" the
+    missing scale information from accessible context after the fact
+  * advertising support for some of the conceivable cases as special feature, or
+    adding it as plugin or extension module with limited scope
+  * linking parts of the necessary additional information to completely
+    unrelated other structures, thus causing code tangling and scattering
+  * result or behaviour of calculations depends on the way things are set up in
+    a seemingly contingent way, forcing users to stick to very specific procedures
+    and ordered steps.
+
+.static typing
+an analysis of the cases to be expected establishes common patterns and some
+base cases, which are then represented by distinct types with well established
+conversions. This can be combined with generic programming for the common parts.
+Close to the data input, a factory establishes these statically typed values.
+
+.Summary
+****************************************************************************
+Lumiera uses (fixed) static typing for the plain _time values_, with a class
+wrapping a simple 64bit integer as common denominator. The strict typing is
+used to enforce a sane conversion path to quantised time values. For these
+grid alinged values, and especially for the common time codes, the principle
+of _delayed quantisation_ is preferred; the implementation relies on runtime
+typing with type erasure, but provides conversion to statically tagged values
+for the most important special formats
+
+image:{imgd}/uml.time-entities.png["Time and Time Quantisation in Lumiera"]
+****************************************************************************
+
+.tagged values
+quantised values are explicitly created out of continuous values by a quantiser
+entity. These quantised data values contain a copy of the original data,
+adjusted to be exactly aligned with respect to the underlying time grid. In
+addition, they carry a tag or ID to denote the respective scale, grid or
+timecode system. This tag can be used later on to assess compatibility or to
+recast values into another timecode system.
+
+.delayed quantisation
+with this approach, the information loss is delayed as long as possible.
+Quantised time values are rather treated as promise for quantisation, while the
+actual time data remains unaltered. Additionally, they carry a tag, or even a
+direct link to the responsible quantiser instance. Effectively, these are
+specialised time values, instances of a sub-concept, able to stand-in for
+general time values, but exposing additional accessors to get a quantised value.
+
+
+discussion
+~~~~~~~~~~
+For Lumiera, the static typing approach is of limited value -- it excels when
+values belonging to different scales are actually treated differently. There are
+such cases, but rather on the data handling level, e.g. sound samples are always
+handled block wise. But regarding time values, the unifying aspect is more
+important, which leads to prefering a dynamic (run time typed) approach, while
+_erasing_ the special differences most of the time. Yet the dynamic and open
+nature of the Lumiera high-level model favours the delayed quantisation pattern;
+the same values may require different quantisation depending on the larger model
+context an object is encountered in. This solution might be too general and
+heavy weight at times though. Thus, for important special cases, the accessors
+should return tagged values, preferably even with differing static type. Time
+codes can be integrated this way, but most notably the *frame numbers* used
+for addressing throughout the backend, can be implemented as such specifically
+typed tagged values; the tag here denotes the quantiser and thus the underlying
+grid -- it should be implemented as hash-ID for smooth integration with code
+written in plain C.
+
+At the level of individual timecode formats, we're lacking a common denominator;
+thus it is preferrable to work with different concrete timecode classes through
+_generic programming._ This way, each timecode format can expose operations
+specific only to the given format. Especially, different timecode formats expose
+different _component fields,_ modelled by the generic *Digxel* concept.
+There is a common baseclass `TCode` though, which can be used as marker
+or for _type erasure._
+
+-> more on link:TimeUsage.html[usage situations]
+////
+// TODO integrate those cross links
++
+-> Timecode link:TimecodeFormat[format and quantisation]
++
+-> Quantiser link:QuantiserImpl[implementation details]
+////
+

doc/design/architecture/TimeUsage.txt

@@ -0,0 +1,186 @@
+Time Usage Situations
+=====================
+:Author:	Hermann Voßeler
+:Email:	<Ichthyostega@web.de>
+:Date:      Spring 2011
+
+//Menu: label Time Usage
+
+the following collection of usage situations helps to shape the details of the
+time values and time quantisation design. +
+-> see also link:TimeQuant.html[more about time quantisation]
+
+time position of an object::
+  indeed the term ``time position'' encompasses two quite different questions
+  +
+  * a time or timing specification within the object
+  * determining the time point in reference to an existing scale
+
+time and length of an object::
+  basically the same situation, +
+  but the length of a time span (duration) could be treated in two ways for quantisation
+  +
+  * having a precise specification and then quantise the start and endpoint
+  * quantise the start position and then establish an (independently quantised length)
+
+moving and resizing an object::
+  this can in itself be done in two different ways, and each of them can be
+  applied in a quantised flavour, which sums up to 8 possible combinations,
+  considering that position and length are 2 degrees of freedom.
+  +
+  * a variable can be _changed_ by an offset
+  * a variable can be _defined_ to a new value
+  * another (hidden) degree of freedom lies in how to apply an quantised offset
+    to an unquantised value (and reversed), because this operation might be done
+    both in the quantised or non-quantised domain, and also the result might be
+    (un)quantised
+
+updating the playback position::
+  this can be seen as a practical application of the above; basically we can
+  choose to show the wall clock time or we can advance the playback position in
+  frame increments, thus denoting the frame currently in display. For video,
+  these distinctions may look moot, but they are indeed relevant for precise
+  audio editing, especially when combined with loop playback (recall that audio
+  is processed block wise, but the individual sample frames and thus the
+  possible loop positions are way finer than the processing block size)
+
+dispatching individual frames for calculation::
+  when a link:PlayProcess[render or playback process] is created, at some point
+  we need to translate this logical unit (``calculation stream'') into individual
+  frame job entries. This requires to break continuous time into individual
+  frames, and then ennumerating these frames.
+
+displaying time intervals::
+  for display, time intervals get _re-quantised_ into display array
+  coordinates. While evidently the display coordinates are themselves quantised
+  and we obviously don't want to cancel out the effect of an quantisation of the
+  values or intervals to be displayed (which means, we get two quantisations
+  chained up after each other), there remains the question if the display array
+  coordinates should be aligned to the grid of the _elements to be displayed,_
+  and especially if the allowed zoom factors should be limited. This decision
+  isn't an easy one, as it has an immediate and tangible effect on what can be
+  showed, how reversible and reproducible a view is and (especially note this!)
+  on the actual values which can be set and changed through the GUI.
+
+time value arithmetic::
+  Client code as well as the internal implementation of time handling needs to
+  do arithmetic operations with time values. Time values are additive and
+  totally ordered. Distance, as calculated by subtraction, can be made into a
+  metric. Another and quite different question is to what extent a quantised
+  variant of this arithmetics is required.
+
+relative placement::
+  because of the divergence between quantised and unquantised values the
+  question arises, if placement relative to another object refers to the raw
+  position or the already quantised position. Basically all the variations
+  discussed for _time and length of an object_ also do apply here.
+
+
+notable issues
+--------------
+
+*Direct quantisation of length is not possible*. This is due to the non-linear
+nature of all but the most trivial time grids: Already such a simple addition
+like a start offset destroys linearity, and this still the more is true within a
+compound grid where the grid spacing changes at some point. Thus, the length has
+to be re-established at the target position of an time interval after each
+change involving quantisation. Regarding the _strategy_ to apply when
+re-establishing the length, it seems more appropriate to treat the object as an
+entity which is moved, which means to do quantisation in two steps, first the
+position, then the endpoint (the second option in the description above). But it
+seems advisable not to hard wire that strategy -- better put it into the
+quantiser.
+
+We should note that the problems regarding quantised durations also carry over
+to _offsets:_ it is difficult to *define the semantics of a quantised offset*.
+Seemingly the only viable approach is to have a _intended offset,_ and then
+to apply a re-quantisation to the target after applying the (raw) offset.
+
+*When to materialise a quantisation*. Because of the basic intention to retain
+information, we prefer to delay the actual applicatio of the quantisation operation
+to the values stored internally as much as possible. But not materialising
+immediately at quantisation has the downside of possibly accumulating off-grid
+values without that being evident. Most notably, if we apply the raw offsets
+produced by GUI interactions, the object's positions and lengthes are bound to
+accumulate spurious information never intended by the user.
+
+Thus, an especially important instance of that problem is *how to deal with
+updates in a quantised environment*. If we handle quantisation stictly as a
+view employed on output, we run into the problems with accumulating spurious
+information. On the other hand, allowing for quantised changes inevitably pulls
+in all the complexity of mixing quantised and non-quantised values. It would be
+desirable somehow to move these distinctions out of the scope of this design and
+offload them onto the client (code using these time classes).
+
+Another closely related problem is *when to allow mutations*, if at all (-> see
+link:TimeMutation[more here...]). We can't completely do away with mutations,
+simply because we don't have a pure functional language at our disposal. The
+whole concept of _reference semantics_ doesn't play so well with immutable
+objects. The Lumiera high-level (session) model certainly relies on objects
+intended to be _manipulated._ Thus we need a re-settable length field in
+`MObject` and we need a time variable for position calculations. Yet we
+could make any _derived objects_ into immutable descriptor records, which
+certainly helps with parallelism.
+
+The *problem with playback position* is -- that indeed it's an attempt to
+conceptualise a non-existing entity. There is no such thing like ``the'' playback
+position. Yet most applications I'm aware off _do_ employ this concept. Likely
+they got trapped by the metaphor of the tape head, again. We should do away with
+that. On playback, we should show a _projection of wall-clock time onto the
+expected playback range_ -- not more, not less. It should be acknowledged that
+there is _no direct link to the ongoing playback processes,_ besides the fact
+that they're assumed to sync to wall-clock time as well. Recall, typically there
+are multiple playback processes going on in compound, and each might run on a
+different update rate. If we really want a _visual out-of-sync indicator,_ we
+should treat that as a separate reporting facility and display it apart of the
+playback cursor.
+
+An interesting point to note for the *frame dispatch step* is the fact that
+in this case quantised values and quantisation are approached in the reverse
+direction, compared with the other uses. Here, after establishing a start point
+on the time scale, we proceed with ennumerating distinct frames and later on
+need to access the corresponding raw time, especially to find out about the
+link:Segmentation[timeline segment] to address, or for retrieving parameter
+automation. -> link:FrameDispatcher[see frame dispatching].
+
+Note that the *display window might be treated as just an independent instance
+of quantisation*. This is similar to the approach taken above for modifying
+quantised time span values. When following this line of thought, we should
+provide a special kind of time grid, the display coordinates. The origin of
+these is always defined to the left (lower) side of the interval to be
+displayed, and they are gauged in screen units (pixels or similar, as used by
+the GUI toolkit set). The rest is handled by the general quantisation
+mechanisms. The problem of aligning the display should be transformed into a
+general facility to align grids, and solved for the general case. Doing so
+solves the remaining problems with quantised value changes and with *specifying
+relative placements* as well: If we choose to represent them as quantised
+values, we might (or might not) also choose to apply this _grid-alignment function._
+
+
+the complete time value usage cycle
+-----------------------------------
+
+The way time value and quantisation handling is designed in Lumiera creates a
+typical usage path, which actually is a one-way route. We might start out with a
+textual representation according to a specific *timecode* format. Assumed we
+know the implicit underlying *time grid* (coordinate system, framerate), this
+timecode string may be parsed. This brings us (back) to the very origin, which
+is a raw `TimeValue` (*internal time* value). Now, this time value might be
+manipulated, compared to other values, combined into a *time span* (time point
+and duration -- the most basic notion of an _object_ to be manipulated in the
+Session). Anyway, at some point these time values need to be related to some
+*time scale* again, leading to *quantised* time values, which -- finally --
+can be cast into a timecode format for external representation again, thus
+closing the circle.
+
+
+substantial problems to be solved
+---------------------------------
+* how to link:TimeGridAlignment[align multiple grids]
+* how to integrate link:TimeMutation[modifications of quantised values]. ([green]#✔ solved#)
+* how to isolate the Time/Quantisation part from the grid MetaAsset in the
+  session -> we use the link:Advice[Advice] system ([green]#✔ solved#)
+* how to design the relation of Timecode, Timecode formatting and Quantisation
+  -> link:TimecodeFormat[more here] [yellow-background]#WIP#
+
+

wiki/renderengine.html

@@ -7225,7 +7225,7 @@ Thus the possibly mutalble time entities get an {{{accept(time::Mutation&)}}
 Based on this time::Mutation design, we provide a specialised element for dealing with //running time values:// When attached to a target time entity, a "life" connection is established. From then on, continuous changes and mutations can be fed to the target by invoking a functor interface. Besides, a change notification signal (callback) can be installed, which will be invoked on each change. This {{{time::Control}}} element is the foundation for implementing all kinds of running time display widgets, spin buttons, timeline selections, playheads, loop playback and similar.
 </pre>
 </div>
-<div title="TimeQuant" modifier="Ichthyostega" modified="201112222249" created="201012181753" tags="Concepts Player spec img discuss draft" changecount="53">
+<div title="TimeQuant" modifier="Ichthyostega" modified="201112222249" created="201012181753" tags="Concepts Player spec img discuss draft" changecount="59">
 <pre>The term &amp;raquo;Time&amp;laquo; spans a variety of vastly different entities. Within a NLE we get to deal with various //flavours of time values.//
 ;continuous time
 :without any additional assumptions, ''points in time'' can be specified with arbitrary precision.
@@ -7249,14 +7249,14 @@ Based on this time::Mutation design, we provide a specialised element for dealin
 :yet some of the special time entities can recieve [[mutation messages|TimeMutation]], allowing e.g. for adjustments to a time interval selection from the GUI
 
 ;internal time
-:While the basic continuous time values don't imply any provision regarding the time scale and origin to be used, actually, within the implementation of the application, the meaning of time values is uniform and free of contradictions. Thus effectively there is an ''implementation time scale'' -- but its scope of validity is //strictly limited to the implementation level of a single application instance.// It is never exposed and never persisted. It might not be reproducible over multiple instantiations of the application. The implementation reserves the right to recalibrate this internal scale. Later, when Lumiera gains the capability to run within a network of render nodes, these instance connections will include a negotiation about the internal time scale, which remains completely opaque to the outer world. This explains, why {{{lumiera::Time}}} instances lack the ability to show their time value beyond debugging purposes. This is to avoid confusion and to stress their opaque nature.
+:While the basic continuous time values don't imply any commitment regarding the time scale and origin to be used, actually, within the implementation of the application, the meaning of time values is uniform and free of contradictions. Thus effectively there is an ''implementation time scale'' -- but its scope of validity is //strictly limited to the implementation level of a single application instance.// It is never exposed and never persisted. It might not be reproducible over multiple instantiations of the application. The implementation reserves the right to recalibrate this internal scale. Later, when Lumiera gains the capability to run within a network of render nodes, these instance connections will include a negotiation about the internal time scale, which remains completely opaque to the outer world. This explains, why {{{lumiera::Time}}} instances lack the ability to show their time value beyond debugging purposes. This is to avoid confusion and to stress their opaque nature.
 ;wall clock and system time
 :The core property of any external real world time is that it is //running// -- we have to synchronise to an external time source.
 :This implies the presence of a //running synchronisation process,// with the authority to adjust the time base;
 :contrast this to the internal time, which is static and unconnected -- 
 ;quantised time
 :The ''act of quantisation'' transforms a continuous property into a ''discrete'' structure. Prominent examples can be found in the domain of micro physics and with digital information processing. In a broader sense, any measurement or //quantification// also encompasses a quantisation. Regarding time and time measurement, quantisation means alignment to a predefined ''time grid''. Quantisation necessarily is an //irreversible process// -- possible additional information is discarded.
-:Note that quantisation introduces an ''time origin'' and a ''reference scale''
+:Note that quantisation introduces a ''time origin'' and a ''reference scale''
 ;frame count
 :within the context of film and media editing, the specification of a ''frame number'' is an especially important instance of quantisation.
 :all the properties of quantisation apply indeed to this special case: it is a time measurement or specification, where the values are aligned to a grid, and there is a reference time point where the counting starts (origin) and a reference scale (frames per second). Handling of quantised time values in Lumiera is defined such as to ensure the presence of all those bits of information. Without such precautions, operating with bare frame numbers leads itself to all kinds of confusions, mismatches, quantisation errors and unnecessary limitations of functionality.