DOC: assumptions and definitions for diff handling
...plus a started draft of the list diff algorithm
This commit is contained in:
parent
4c562e76d9
commit
42f69b6cb5
3 changed files with 189 additions and 23 deletions
141
doc/technical/library/DiffFramework.txt
Normal file
141
doc/technical/library/DiffFramework.txt
Normal file
|
|
@ -0,0 +1,141 @@
|
|||
Diff Handling Framework
|
||||
=======================
|
||||
|
||||
Within the support library, in the namespace `lib::diff`, there is a collection of loosely coupled of tools
|
||||
known as »the diff framework«. It revolves around generic representation and handling of structural differences.
|
||||
Beyond some rather general assumptions, to avoid stipulating the usage of specific data elements or containers,
|
||||
the framework is kept _generic_, cast in terms of *elements*, *sequences* and *strategies*
|
||||
for access, indexing and traversal.
|
||||
|
||||
.motivation
|
||||
**********************
|
||||
Diff handling is multi purpose within Lumiera:
|
||||
Diff representation is seen as a meta language and abstraction mechanism;
|
||||
it enables tight collaboration without the need to tie and tangle the involved implementation data structures.
|
||||
Used this way, diff representation reduces coupling and helps to cut down overall complexity — so to justify
|
||||
the considerable amount of complexity seen within the diff framework implementation.
|
||||
**********************
|
||||
|
||||
Definitions
|
||||
-----------
|
||||
element::
|
||||
the atomic unit treated in diff detection, representation and application. +
|
||||
Elements are considered to be
|
||||
|
||||
- lightweight copyable values
|
||||
- equality comparable
|
||||
- bearing distinct identity
|
||||
- unique _as far as concerned_
|
||||
|
||||
sequence::
|
||||
data is delivered in the form of a sequence, which might or might not be _ordered,_
|
||||
but in any case will be traversed once only.
|
||||
|
||||
diff::
|
||||
the changes necessary to transform an input sequence (``old sequence'') into a target sequence (``new sequence'')
|
||||
|
||||
diff language::
|
||||
differences are spelled out in linearised form: as a sequence of constant-size diff actions, called »diff verbs«
|
||||
|
||||
diff verb::
|
||||
a single element within a diff. Diff verbs are conceived as operations, which,
|
||||
when applied consuming the input sequence, will produce the target sequence of the diff.
|
||||
|
||||
diff application::
|
||||
the process of consuming a diff (sequence of diff verbs), with the goal to produce some effect at the
|
||||
_target_ of diff application. Typically we want to apply a diff to a data sequence, to mutate it
|
||||
into a new shape, conforming with the shape of the diff's ``target sequence''
|
||||
|
||||
diff generator::
|
||||
a facility producing a diff, which is a sequence of diff verbs.
|
||||
Typically, a diff generator works lazily, demand driven.
|
||||
|
||||
diff detector::
|
||||
special kind of diff generator, which takes two data sequences as input:
|
||||
an ``old sequence'' and a ``new sequence''. The diff detector traverses and compares
|
||||
these sequences to produce a diff, which describes the steps necessary to transform
|
||||
the ``old'' shape into the ``new'' shape of the data.
|
||||
|
||||
|
||||
List Diff Algorithm
|
||||
-------------------
|
||||
While in general this is a well studied subject, in Lumiera we'll confine ourselves to a very
|
||||
specific flavour of diff handling: we rely on _elementary atomic units_ with well established
|
||||
object identity. And in addition, within the scope of one coherent diff handling task,
|
||||
we require those elements to be 'unique'. The purpose of this design decision is to segregate
|
||||
the notorious matching problem and treat diff handling in isolation.
|
||||
|
||||
Effectively this means that, for any given element, there can be at most _one_ matching
|
||||
counterpart in the other sequence, and the presence of such can be detected by using an *index*.
|
||||
In fact, we retrieve an index for every sequence involved into the diff detection task;
|
||||
this is our trade-off for simplicity in the diff detection algorithm.footnote:[traditionally,
|
||||
diff detection schemes, especially those geared at text diff detection, engage into great lengths
|
||||
of producing an ``optimal'' diff, which effectively means to build specifically tuned pattern
|
||||
or decision tables, from which the final diff can then be pulled or interpreted.
|
||||
We acknowledge that in our case building a lookup table index can be O(n log n); we might
|
||||
well be able to do better, but certainly for the price of an algorithm more mentally challenging.]
|
||||
In case this turns out as a performance problem, we might consider integrating the index
|
||||
maintenance into the data structure to be diffed, which shifts the additional impact of
|
||||
indexing onto the data population phase.footnote:[in the general tree diff case this is far
|
||||
from trivial, since we need an self-contained element index for every node, and we need the
|
||||
ability to take a snapshot of the ``old'' state before mutating a node into ``new'' shape]
|
||||
|
||||
Element classification
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
By using the indices of the old and the new sequence, we are able to _classify_ each element:
|
||||
|
||||
- elements only present in the new sequence are treated as *inserts*
|
||||
- elements only present in the old sequence are treated as *deletes*
|
||||
- elements present in both sequences form the *permutation*
|
||||
|
||||
Processing pattern
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
We _consume both the old and the new sequence synchronously, while emitting the diff sequence_.
|
||||
|
||||
The diff describes a sequence of operations, which, when applied, consume a sequence congruent
|
||||
to the old sequence, while emitting a sequence congruent to the new sequence. We use the
|
||||
following *list diff language* here:
|
||||
|
||||
verb `ins(elm)`::
|
||||
insert the given argument element `elm` at the _current processing position_
|
||||
into the target sequence. This operation allows to inject new data
|
||||
verb `del(elm)`::
|
||||
delete the _next_ element `elm` at _current position._
|
||||
For sake of verification, the element to be deleted is also included as argument (redundancy).
|
||||
verb `pick(elm)`::
|
||||
accepts the _next_ element at _current position_ into the resulting altered sequence.
|
||||
The element is given redundantly as argument.
|
||||
verb `push(elm)`::
|
||||
effect a re-ordering of the target list contents. This verb requires to take
|
||||
the _next_ element, which happens to sit at _current processing position_ and
|
||||
_push it back_ further into the list, to be placed at a position _behind_ the
|
||||
_anchor element_ `elm` given as argument.
|
||||
|
||||
Since _inserts_ and _deletes_ can be detected and emitted right at the processing frontier,
|
||||
for the rest of this theoretical discussion, we consider the insert / delete part filtered
|
||||
away conceptually, and concentrate on generating the permutation part.
|
||||
|
||||
Handling sequence permutation
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
This paragraph describes how to consume two permutations of the same sequence simultaneously,
|
||||
while emitting `push` and `pick` verbs to describe the re-ordering. Consider the sequences
|
||||
split into an already-processed part, and a part still-to-be-processed.
|
||||
|
||||
.Invariant
|
||||
Matters are arranged such, that, in the to-be-processed part, each element appearing at the
|
||||
front of the ``new'' sequence _can be picked right away_.
|
||||
|
||||
Now, to arrive at that invariant, we have especially to deal with the case that a different
|
||||
(not matching) element appears at the front of the ``old'' list. We have to emit additional
|
||||
`push` verbs to get rid of non-matching elements in the ``old'' order, until we get into a state
|
||||
where the invariant is re-established (and we're able to `pick` to consume the same element
|
||||
from the existing sequence and the target sequence). Obviously, the tricky part is how to
|
||||
determine the *anchor element* for this `push` directrive...
|
||||
|
||||
- we need to be sure the anchor _is indeed present_ in the current shape of the sequence in processing.
|
||||
- the anchor must be in the right place, so to conform to the target sequence at the point of picking it.
|
||||
- it is desirable to emit at most one `push` directive for any given element; we want it to settle at the
|
||||
right place with a single shot
|
||||
|
||||
|
||||
|
||||
|
|
@ -11,7 +11,7 @@ Lumiera Forward Iterator
|
|||
While most modern languages provide some kind of _Iterator,_ the actual semantics
|
||||
and the fine points of the implementation vary greatly from language to language.
|
||||
Unfortunately the C++ standard library uses a very elaborate and rather low-level
|
||||
notion of iterators, which doesn't mix well with the task of building clean interfaces.
|
||||
notion of iterators, which does not mix well with the task of building clean interfaces.
|
||||
|
||||
Thus, within the Lumiera core application, we're using our own Iterator concept,
|
||||
initially defined as link:{ldoc}/devel/rfc/LumieraForwardIterator.html[RfC],
|
||||
|
|
@ -22,13 +22,13 @@ readability and simplicity for (lesser) performance.
|
|||
An Iterator is a self-contained token value,
|
||||
representing the promise to pull a sequence of data
|
||||
|
||||
- rather then deriving from an specific interface, anything behaving
|
||||
appropriately _is a Lumiera Forward Iterator._ (``duck typing'')
|
||||
- rather then deriving from an specific interface, anything which behaves
|
||||
appropriately _is a Lumiera Forward Iterator._ (``duck typing'' or ``a Concept'')
|
||||
- the client finds a typedef at a suitable, nearby location. Objects of this
|
||||
type can be created, copied and compared.
|
||||
- any Lumiera forward iterator can be in _exhausted_ (invalid) state, which
|
||||
can be checked by +bool+ conversion.
|
||||
- especially, default constructed iterators are fixed to that state.
|
||||
can be checked through +bool+ conversion.
|
||||
- notably, any default constructed iterator is fixed to that exhausted state.
|
||||
Non-exhausted iterators may only be obtained by API call.
|
||||
- the exhausted state is final and can't be reset, meaning that any iterator
|
||||
is a disposable one-way-off object.
|
||||
|
|
@ -41,34 +41,35 @@ Motivation
|
|||
~~~~~~~~~~
|
||||
The Lumiera Forward Iterator concept is a blend of the STL iterators and
|
||||
iterator concepts found in Java, C#, Python and Ruby. The chosen syntax should
|
||||
look familiar to C++ programmers and indeed is compatible to STL containers and
|
||||
ranges. To the contrary, while a STL iterator can be thought of as being just
|
||||
a disguised pointer, the semantics of Lumiera Forward Iterators is deliberately
|
||||
reduced to a single, one-way-off forward iteration, they can't be reset,
|
||||
manipulated by any arithmetic, and the result of assigning to an dereferenced
|
||||
iterator is unspecified, as is the meaning of post-increment and stored copies
|
||||
in general. You _should not think of an iterator as denoting a position_ --
|
||||
just a one-way off promise to yield data.
|
||||
look fairly familiar to C++ programmers and indeed is compatible to STL containers
|
||||
and ranges. Yet while a STL iterator can be thought of as being just a pointer
|
||||
in disguise, the semantics of Lumiera Forward Iterators is deliberately more
|
||||
high-level, reduced to a single, one-way-off forward iteration. Lumiera Forward
|
||||
Iterators can't be reset, they can not be manipulated by any arithmetic, and the
|
||||
result of assigning to an dereferenced iterator is unspecified, as is the meaning
|
||||
of post-increment and stored copies in general.
|
||||
You _should not think of an iterator as denoting a position_ --
|
||||
just treat it as an one-way off promise to yield data.
|
||||
|
||||
Another notable difference to the STL iterators is the default ctor and the
|
||||
+bool+ conversion. The latter allows using iterators painlessly within +for+
|
||||
and +while+ loops; a default constructed iterator is equivalent to the STL
|
||||
container's +end()+ value -- indeed any _container-like_ object exposing
|
||||
Lumiera Forward Iteration is encouraged to provide such an `end()`-function,
|
||||
additionally enabling iteration by `std::for_each` (or Lumiera's even more
|
||||
convenient `util::for_each()`).
|
||||
which additionally enables iteration by `std::for_each` (or Lumiera's even more
|
||||
convenient `util::for_each()`), and use in ``range for''-loops.
|
||||
|
||||
Implementation
|
||||
~~~~~~~~~~~~~~
|
||||
As pointed out above, within Lumiera the notion of ``Iterator'' is a concept
|
||||
(generic programming) and doesn't mean a supertype (object orientation). Any
|
||||
(generic programming) and does not mean a supertype (object orientation). Any
|
||||
object providing a suitable set of operations can be used for iteration.
|
||||
|
||||
- must be default constructible to _exhausted state_
|
||||
- must be a copyable value object
|
||||
- must provide a `bool` conversion to detect _exhausted state_
|
||||
- must provide a pre-increment operator (`++i`)
|
||||
- must allow dreferentiation (`*i`) to yield the current object
|
||||
- must allow dereferentiation (`*i`) to yield the current object
|
||||
- must throw on any usage in _exhausted state_.
|
||||
|
||||
But, typically you wouldn't write all those operations again and again.
|
||||
|
|
|
|||
|
|
@ -7703,25 +7703,47 @@ Using transitions is a very basic task and thus needs viable support by the GUI.
|
|||
Because of this experience, ichthyo wants to support a more general case of transitions, which have N output connections, behave similar to their "simple" counterpart, but leave out the mixing step. As a plus, such transitions can be inserted at the source ports of N clips or between any intermediary or final output pipes as well. Any transition processor capable of handling this situation should provide some flag, in order to decide if he can be placed in such a manner. (wichin the builder, encountering a inconsistently placed transition is just an [[building error|BuildingError]])
|
||||
</pre>
|
||||
</div>
|
||||
<div title="TreeDiffFundamentals" creator="Ichthyostega" modifier="Ichthyostega" created="201411070156" modified="201412010102" tags="Concepts design" changecount="11">
|
||||
<pre>At some points within the implementation we encounter problems of structural difference computation, like finding the effective changes in a tree data structure, or using a diff represetation as notification message format. Pointing out some observations and relating our approach(es) to the problem as generally known might be in place.
|
||||
<div title="TreeDiffFundamentals" creator="Ichthyostega" modifier="Ichthyostega" created="201411070156" modified="201412210005" tags="Concepts design" changecount="17">
|
||||
<pre>At various points within the implementation we encounter problems of structural difference computation, like finding the effective changes in a tree data structure, or using a diff represetation as notification message format. It might be in place to point out some observations, especially relating our approach(es) to the problem as generally known .
|
||||
|
||||
!classical diff problem
|
||||
There is very wide spread use of a quite special flavour of diff calculations: the classical diff of textual data. Here, the treated data has a completely uniform linear structure -- it is either a sequence of lines or a sequence of bytes ("binary diff"). The individual data cells are built from a finite alphabet of comparable values (characters). Since a given character or word may be present several times in the same document, we need to establish a //matching// as foundation of any difference calculation. We need to establish the identical parts to find the differences. This matching is not unique, and largely determines the differences found, so it is hard to determine if a given diff is //correct.// Thus, the core idea to attack and solve this classical diff problem was to look for an ''optimal diff''. Some variations include
|
||||
There is very wide spread use of a very specific flavour of diff calculations: the classical diff of textual data. Here, the treated data has a completely uniform linear structure -- it is either a sequence of lines or a sequence of bytes ("binary diff"). The individual data cells are built from a finite alphabet of comparable values (characters). Since a given character or word may be present several times in the same document, we need to establish a //matching// as foundation of any difference calculation. We need to establish the identical parts to find the differences. This matching is not unique, and largely determines the differences found, so it is hard to determine if a given diff is //correct.// Thus, the core idea to attack and solve this classical diff problem was to look for an ''optimal diff''. Some variations include
|
||||
* longest common substrings
|
||||
* largest common subsequence
|
||||
* minimal edit script to perform the diff
|
||||
|
||||
!challenge of the diff problem
|
||||
While, from a mathematical point of view, the above optimisation problem can be considered solved, in practice the available solutions are far from perfect. The fundamental assumption of a linear sequence to base the diff turns out as an oversimplification -- real world data carries meaning and can be judged by imposing additional structure. There can be structure violating diffs and there can be nonsensical and misguiding diffs. In the light of this notion, every diff is in fact a structural diff.
|
||||
While, from a mathematical point of view, the above optimisation problem can be considered as solved, in practice the available solutions are far from perfect. The fundamental assumption of a linear sequence to base the diff turns out as an oversimplification -- real world data carries meaning and can be judged by imposing additional structure. There can be structure violating diffs and there can be nonsensical and misguiding diffs. In the light of this notion, every diff is in fact a structural diff.
|
||||
Unfortunately, attempts to amend the beautiful mathematical solution by incorporating this additional structure turns out to be incredible hard. Either, the general case problem can be proven to be ~NP-hard, or, at least, exploiting some special structural properties successfully renders the meaning of an "optimal" diff solution more or less arbitrary. The quest for a generic diff problem and an universal solution turns out to be a dead end.
|
||||
|
||||
!fundamentals of diff handling
|
||||
The most fundamental distinction is the difference between //finding// a diff and //representing// a diff. The former is concerned with uncovering structural relations, while the latter deals with knowledge about structural relations and thus is more general. It is possible to capture structural relations while they emerge -- this way describing a process of transformation. A likewise fundamental distinction is between //reordering// elements and //mutating// them. This is related to the notion of //identity,// which in turn implies an underlying model of the elements and entities to be considered for diffing. If, as an example, we model the tokens, functions and classes of program code as mere characters, we will be happily matching curly braces and can not expect meaningful diffs. So first we need a theory about what can possibly happen, and we use this as a foundation to establish the representation of such possible processes. The choices taken here can make all the difference towards efficient and usable methods. The //matching problem// should be viewed from here: a matching is actually a hypothesis about possible processes of transformation -- this observation explains why the matching strategy is at the core of any diff algorithm. If we use some arbitrary notation of optimality, we have to consider endless combinations and end up with a lot of accidental complexity. Yet if we manage to base the calculation on a representation well aligned with the nature of the entities in question, the matching can be as simple as retrieving a given entry by ID. The actual work is reduced to extracting the raw changes in data.
|
||||
Before we can consider a diffing technique, we need to clarify the primitive operations used as foundation. These primitives form //a language.// This incurs a problem and choice of context. We might base the representation language on the situation where the diff is applied, we may care for the conciseness of the representation, or the effort and space required to apply it. We should consider if the focus is on reading and understanding the diff, if we intend to derive something directly from the diff representation -- say, if and what and when something is changed -- and we should consider how the context where the diff is established relates to the context where it is applied.
|
||||
|
||||
&rarr; [[Implementation considerations|TreeDiffImplementation]]
|
||||
</pre>
|
||||
</div>
|
||||
<div title="TreeDiffModel" creator="Ichthyostega" modifier="Ichthyostega" created="201410270313" modified="201412120312" tags="Model GuiPattern spec draft" changecount="46">
|
||||
<div title="TreeDiffImplementation" creator="Ichthyostega" modifier="Ichthyostega" created="201412210015" modified="201412210150" tags="Model GuiPattern design draft" changecount="8">
|
||||
<pre>//This page details decisions taken for implementation of Lumiera's diff handling framework//
|
||||
This topic is rather abstract, since diff handling is multi purpose within Lumiera: Diff representation is seen as a meta language and abstraction mechanism; it enables tight collaboration without the need to tie and tangle the involved implementation data structures. Used this way, diff representation reduces coupling and helps to cut down overall complexity -- so to justify the considerable amount of complexity seen within the diff framework implementation.
|
||||
|
||||
!outline of the diff handling framework
|
||||
Provided as a loosely coupled collection of tools in the namespace {{{lib::diff}}}, the diff framework revolves around generic representation and handling of structural differences. Beyond some rather general assumptions, to avoid stipulating the usage of specific data elements or containers, the framework is cast in terms of ''elements'', ''sequences'' and ''strategies'' for access, indexing and traversal.
|
||||
;elements
|
||||
:the atomic units treated in diff detection, representation and application are considered to be
|
||||
:*lightweight copyable values
|
||||
:*equality comparable
|
||||
:*bearing distinct identity
|
||||
:*unique //as far as concerned//
|
||||
;data sequence
|
||||
:any collection of data is delivered in the form of a sequence, which might or might not be //ordered,// but in any case will be traversed once only.
|
||||
;diff language
|
||||
:a diff represents the changes necessary to transform an input sequence ("old sequence") into a target sequence ("new sequence")
|
||||
:differences are spelled out in ''linearised form'': as a sequence of constant-size diff actions (called &raquo;diff verbs&laquo;)
|
||||
:they are conceived as operations, which, when applied consuming the input sequence, will produce the target sequence of the diff.
|
||||
</pre>
|
||||
</div>
|
||||
<div title="TreeDiffModel" creator="Ichthyostega" modifier="Ichthyostega" created="201410270313" modified="201412210002" tags="Model GuiPattern spec draft" changecount="47">
|
||||
<pre>for the purpose of handling updates in the GUI timeline display efficiently, we need to determine and represent //structural differences//
|
||||
We build a slightly abstracted representation of tree changes and use this to propagate //change notifications// to the actual widgets. To keep the whole process space efficient, a demand-driven, stateless implementation approach is chosen. This reduces the problem into several layered stages.
|
||||
* our model is a heterogeneous tree &rArr; use demand-driven recursion
|
||||
|
|
@ -7729,7 +7751,9 @@ We build a slightly abstracted representation of tree changes and use this to pr
|
|||
* find changes in ordered collections of children &rArr; symbolic list diffing algorithm
|
||||
* problems with identity and state &rArr; encapsulate state and have a airtight object identity scheme
|
||||
|
||||
Doubtless we're dealing with a highly specific application here. &rarr; see [[discussion of diffing solutions|TreeDiffFundamentals]]
|
||||
Doubtless we're dealing with a highly specific application here.
|
||||
&rarr; see [[discussion of diffing solutions|TreeDiffFundamentals]]
|
||||
&rarr; see [[diff handling implementation technique|TreeDiffImplementation]]
|
||||
|
||||
!list diffing algorithm
|
||||
| !source data|!|!desired result |
|
||||
|
|
|
|||
Loading…
Reference in a new issue