LUMIERA.clone/doc/design/architecture/time/TimeUsage.txt

Time Usage Situations
=====================
:Author:	Hermann Voßeler
:Email:	<Ichthyostega@web.de>
:Date:      Spring 2011

//Menu: label Time Usage

the following collection of usage situations helps to shape the details of the
time values and time quantisation design. +
-> see also link:/x/TimeQuant.html[more about time quantisation]

time position of an object::
  indeed the term ``time position'' encompasses two quite different questions
  +
  * a time or timing specification within the object
  * determining the time point in reference to an existing scale

time and length of an object::
  basically the same situation, +
  but the length of a time span (duration) could be treated in two ways for quantisation
  +
  * having a precise specification and then quantise the start and endpoint
  * quantise the start position and then establish an (independently quantised length)

moving and resizing an object::
  this can in itself be done in two different ways, and each of them can be
  applied in a quantised flavour, which sums up to 8 possible combinations,
  considering that position and length are 2 degrees of freedom.
  +
  * a variable can be _changed_ by an offset
  * a variable can be _defined_ to a new value
  * another (hidden) degree of freedom lies in how to apply an quantised offset
    to an unquantised value (and reversed), because this operation might be done
    both in the quantised or non-quantised domain, and also the result might be
    (un)quantised

updating the playback position::
  this can be seen as a practical application of the above; basically we can
  choose to show the wall clock time or we can advance the playback position in
  frame increments, thus denoting the frame currently in display. For video,
  these distinctions may look moot, but they are indeed relevant for precise
  audio editing, especially when combined with loop playback (recall that audio
  is processed block wise, but the individual sample frames and thus the
  possible loop positions are way finer than the processing block size)

dispatching individual frames for calculation::
  when a link:/x/PlayProcess.html[render or playback process] is created, at some point
  we need to translate this logical unit (``calculation stream'') into individual
  frame job entries. This requires to break continuous time into individual
  frames, and then ennumerating these frames.

displaying time intervals::
  for display, time intervals get _re-quantised_ into display array
  coordinates. While evidently the display coordinates are themselves quantised
  and we obviously don't want to cancel out the effect of an quantisation of the
  values or intervals to be displayed (which means, we get two quantisations
  chained up after each other), there remains the question if the display array
  coordinates should be aligned to the grid of the _elements to be displayed,_
  and especially if the allowed zoom factors should be limited. This decision
  isn't an easy one, as it has an immediate and tangible effect on what can be
  showed, how reversible and reproducible a view is and (especially note this!)
  on the actual values which can be set and changed through the GUI.

time value arithmetic::
  Client code as well as the internal implementation of time handling needs to
  do arithmetic operations with time values. Time values are additive and
  totally ordered. Distance, as calculated by subtraction, can be made into a
  metric. Another and quite different question is to what extent a quantised
  variant of this arithmetics is required.

relative placement::
  because of the divergence between quantised and unquantised values the
  question arises, if placement relative to another object refers to the raw
  position or the already quantised position. Basically all the variations
  discussed for _time and length of an object_ also do apply here.


notable issues
--------------

*Direct quantisation of length is not possible*. This is due to the non-linear
nature of all but the most trivial time grids: Already such a simple addition
like a start offset destroys linearity, and this still the more is true within a
compound grid where the grid spacing changes at some point. Thus, the length has
to be re-established at the target position of an time interval after each
change involving quantisation. Regarding the _strategy_ to apply when
re-establishing the length, it seems more appropriate to treat the object as an
entity which is moved, which means to do quantisation in two steps, first the
position, then the endpoint (the second option in the description above). But it
seems advisable not to hard wire that strategy -- better put it into the
quantiser.

We should note that the problems regarding quantised durations also carry over
to _offsets:_ it is difficult to *define the semantics of a quantised offset*.
Seemingly the only viable approach is to have a _intended offset,_ and then
to apply a re-quantisation to the target after applying the (raw) offset.

*When to materialise a quantisation*. Because of the basic intention to retain
information, we prefer to delay the actual applicatio of the quantisation operation
to the values stored internally as much as possible. But not materialising
immediately at quantisation has the downside of possibly accumulating off-grid
values without that being evident. Most notably, if we apply the raw offsets
produced by GUI interactions, the object's positions and lengthes are bound to
accumulate spurious information never intended by the user.

Thus, an especially important instance of that problem is *how to deal with
updates in a quantised environment*. If we handle quantisation stictly as a
view employed on output, we run into the problems with accumulating spurious
information. On the other hand, allowing for quantised changes inevitably pulls
in all the complexity of mixing quantised and non-quantised values. It would be
desirable somehow to move these distinctions out of the scope of this design and
offload them onto the client (code using these time classes).

Another closely related problem is *when to allow mutations*, if at all (-> see
link:/x/TimeMutation.html[more here...]). We can't completely do away with mutations,
simply because we don't have a pure functional language at our disposal. The
whole concept of _reference semantics_ doesn't play so well with immutable
objects. The Lumiera high-level (session) model certainly relies on objects
intended to be _manipulated._ Thus we need a re-settable length field in
`MObject` and we need a time variable for position calculations. Yet we
could make any _derived objects_ into immutable descriptor records, which
certainly helps with parallelism.

The *problem with playback position* is -- that indeed it's an attempt to
conceptualise a non-existing entity. There is no such thing like ``the'' playback
position. Yet most applications I'm aware off _do_ employ this concept. Likely
they got trapped by the metaphor of the tape head, again. We should do away with
that. On playback, we should show a _projection of wall-clock time onto the
expected playback range_ -- not more, not less. It should be acknowledged that
there is _no direct link to the ongoing playback processes,_ besides the fact
that they're assumed to sync to wall-clock time as well. Recall, typically there
are multiple playback processes going on in compound, and each might run on a
different update rate. If we really want a _visual out-of-sync indicator,_ we
should treat that as a separate reporting facility and display it apart of the
playback cursor.

An interesting point to note for the *frame dispatch step* is the fact that
in this case quantised values and quantisation are approached in the reverse
direction, compared with the other uses. Here, after establishing a start point
on the time scale, we proceed with ennumerating distinct frames and later on
need to access the corresponding raw time, especially to find out about the
link:/x/Segmentation.html[timeline segment] to address, or for retrieving parameter
automation. -> link:/x/FrameDispatcher.html[see frame dispatching].

Note that the *display window might be treated as just an independent instance
of quantisation*. This is similar to the approach taken above for modifying
quantised time span values. When following this line of thought, we should
provide a special kind of time grid, the display coordinates. The origin of
these is always defined to the left (lower) side of the interval to be
displayed, and they are gauged in screen units (pixels or similar, as used by
the GUI toolkit set). The rest is handled by the general quantisation
mechanisms. The problem of aligning the display should be transformed into a
general facility to align grids, and solved for the general case. Doing so
solves the remaining problems with quantised value changes and with *specifying
relative placements* as well: If we choose to represent them as quantised
values, we might (or might not) also choose to apply this _grid-alignment function._


the complete time value usage cycle
-----------------------------------

The way time value and quantisation handling is designed in Lumiera creates a
typical usage path, which actually is a one-way route. We might start out with a
textual representation according to a specific *timecode* format. Assumed we
know the implicit underlying *time grid* (coordinate system, framerate), this
timecode string may be parsed. This brings us (back) to the very origin, which
is a raw `TimeValue` (*internal time* value). Now, this time value might be
manipulated, compared to other values, combined into a *time span* (time point
and duration -- the most basic notion of an _object_ to be manipulated in the
Session). Anyway, at some point these time values need to be related to some
*time scale* again, leading to *quantised* time values, which -- finally --
can be cast into a timecode format for external representation again, thus
closing the circle.


substantial problems to be solved
---------------------------------
* how to align multiple grids -- what grid to use for playback?
* how to integrate link:/x/TimeMutation.html[modifications of quantised values]. ([green]#✔ solved#)
* how to isolate the Time/Quantisation part from the grid MetaAsset in the
  session -> we use the link:/x/Advice.html[Advice] system ([green]#✔ solved#)
* how to design the relation of Timecode, Timecode formatting and Quantisation
  -> link:/x/TimeCode.html[more here] [yellow-background]#WIP#