Ichthyostega
051cb31e28
The RfC documents were written to complement discussions of the Lumiera developers; yet since the time where ''Ichthyo'' is working basically alone on the project, this kind of discussions have ceased. During the following years, some ideas promoted by the existing RfC documents became rather detached from the actual state of development in the code base. Many of the existing RfC documents require some commentary to place them into context, and some of the decisions taken in the early stage of the project should be **re-assessed**. This includes the decision to reject some proposals, which initially might have seemed desirable, yet could not be reconciled with the understanding of the matter and topic in question, as was gained through the ongoing analysis and development.
190 lines
8.3 KiB
Text
190 lines
8.3 KiB
Text
SystematicMetadata
|
|
==================
|
|
|
|
// please don't remove the //word: comments
|
|
|
|
[options="autowidth"]
|
|
|====================================
|
|
|*State* | _Idea_
|
|
|*Date* | _Mo 08 Okt 2012 04:39:16 CEST_
|
|
|*Proposed by* | Ichthyostega <prg@ichthyostega.de>
|
|
|====================================
|
|
|
|
********************************************************************************
|
|
.Abstract
|
|
[red]#TODO# _give a short summary of this proposal_
|
|
********************************************************************************
|
|
|
|
Lumiera is a metadata processing application: the actual _Data_ processed
|
|
is the _media data_, used for rendering and playback, while everything
|
|
else is _metadata_. Since our basic decision is to rely on existing libraries for
|
|
handling the data, the ``metadata part'' is what _actually happens_ within the
|
|
Lumiera application.
|
|
|
|
This RfC describes a fundamental approach towards metadata handling.
|
|
|
|
|
|
Description
|
|
-----------
|
|
//description: add a detailed description:
|
|
Metadata is conceived as a huge uniform tree. This tree is conceptual -- it is
|
|
never represented as a whole. In the implemented system, we only ever see parts
|
|
of this virtual tree as being cast into concrete data representations. These parts
|
|
are like islands of explicitly defined and typed structure, yet they never need
|
|
to span the whole virtual model, and thus there never needs to be an universal
|
|
model data structure definition. Data structure becomes implementation detail.
|
|
|
|
Parts of the system talk to each other by _describing_ some subtree of metadata.
|
|
This description is transferred _in the form of a tree diff:_ the receiver pulls
|
|
a sequence of diff verbs from a diff iterator, and interpreting these verbs will
|
|
walk him down and expand or transform the tree in question. As part of this
|
|
description of model changes, sub-scopes are ``opened'' and populated, similar
|
|
to populating a filesystem. It is up to the receiver to assemble these information
|
|
into a suitable representation. Some receiver might invoke an object factory,
|
|
while another serialises data into an external textual or binary representation.
|
|
|
|
|
|
Abstract Metadata Model
|
|
~~~~~~~~~~~~~~~~~~~~~~~
|
|
The conceptual model for metadata is close to what the JSON format uses: +
|
|
There are primitive values as +null+, string, number and boolean. Compund values
|
|
can be arrays or records, the latter being a sub-scope populated with key-value pairs.
|
|
|
|
We might consider some extensions
|
|
|
|
* having data values similar to BSON of MongoDB: integrals, floats, timestamps
|
|
* introducing two _special magic keys_ for records: `"type"` and `"id"`
|
|
|
|
|
|
Sources and Overlays
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
Metadata is delivered from _sources_, which can be _layered_. Similarly, on the
|
|
receiving side, there can be multiple _writeable layers_, with a routing strategy
|
|
to decide which writeable layer receives a given metadata element. This routing
|
|
is implemented within a pipeline connecting sender and receiver; if the default
|
|
routing strategy isn't sufficient, we can control the routing by introducing a
|
|
a meta-tree in some separate branch, this way making the metadata self-referential.
|
|
|
|
|
|
Some points to note
|
|
~~~~~~~~~~~~~~~~~~~
|
|
- this concept doesn't say anything about the actual meaning of the metadata elements,
|
|
since that is always determined by the receiver, based on the current context.
|
|
- likewise, this concept doesn't state anything about the actual interactions, the
|
|
involved parts and how the interaction is initiated and configured; this is considered
|
|
an external topic, which needs to be solved within the applicable context (e.g. the
|
|
session has a specific protocol how to retrieve a persisted session snapshot)
|
|
- there is no separate _system configuration_ -- configuration appears just as a
|
|
local record of key-value pairs, which is interpreted according to the context.
|
|
- in a similar vein, this concept deliberately doesn't state anything regarding the
|
|
handling of _defaults_, since these are so highly dependent on the actual context.
|
|
|
|
|
|
Tasks
|
|
~~~~~
|
|
// List what needs to be done to implement this Proposal:
|
|
// * first step ([green]#✔ done#)
|
|
* define the interaction API [yellow-background]#WIP#
|
|
* scrutinise this concept to find the pitfalls [yellow-background]#WIP#
|
|
* build a demonstration prototype, where the receiver fabricates an object [red yellow-background]#TBD#
|
|
** the unit tests related to the _Diff System_ could be counted as such a demonstration +
|
|
Ichthyostega:: '2025-09-13'
|
|
|
|
Discussion
|
|
~~~~~~~~~~
|
|
|
|
Pros
|
|
^^^^
|
|
- the basic implementation of a meta data structure (a _property tree_) and the
|
|
representation of a ``Diff'' is comparatively simple -- much simpler than the
|
|
task to define and build a global data model, and get it properly connected
|
|
to the various implementation facilities to operate on this data.
|
|
- parts can be combined and can evolve in an open ended way, without requiring
|
|
a finalised concept up-front,
|
|
- even complex routing and overlaying strategies remain manageable, since routing
|
|
and data fusion can be treated in isolation, local for a given scope and apart
|
|
from the storage representation;
|
|
- library implementations for textual representations can be integrated.
|
|
|
|
|
|
|
|
Cons
|
|
^^^^
|
|
- the theoretical view is challenging and rather uncommon
|
|
- a naive implementation holds the whole data tree in memory twice
|
|
- how the coherent ``islands'' are combined is only a matter of invocation order
|
|
and thus dangerously flexible
|
|
|
|
|
|
|
|
|
|
Alternatives
|
|
^^^^^^^^^^^^
|
|
//alternatives: explain alternatives and tell why they are not viable:
|
|
The classical alternative is to define a common core data structure, which
|
|
needs to be finalised quickly. Isolated functional modules will then be written
|
|
to work on that common data set, which leads to a high degree of coupling.
|
|
Since this approach effectively doesn't scale well, what happens in practice is
|
|
that several independent storage and exchange systems start to exist in parallel,
|
|
e.g. system configuration, persisted object model, plug-in parameters, presentation
|
|
state.
|
|
|
|
|
|
|
|
Rationale
|
|
---------
|
|
//rationale: Give a concise summary why it should be done *this* way:
|
|
Basically common (meta) data could take on a lot of shapes between two extremes:
|
|
|
|
- the _precise typed structure_, which also is a contract
|
|
- the _open dynamic structure_, which leaves the contract implicit
|
|
|
|
The concept explained in this RfC tries to reconcile those extremes by avoiding
|
|
a global concrete representation; instead, several interactions, each local in
|
|
scope, can coexist on top of the abstract structure as backdrop. Each of these
|
|
actual interactions can have its own specialised standard, which actually is
|
|
a collaboration protocol, and is based on a common understanding of the
|
|
_structure,_ instead of sharing actual data implementation.
|
|
|
|
|
|
//Conclusion
|
|
//----------
|
|
//conclusion: When approved (this proposal becomes a Final)
|
|
// write some conclusions about its process:
|
|
|
|
|
|
|
|
|
|
Comments
|
|
--------
|
|
//comments: append below
|
|
|
|
This RfC seems to be more like a vision statement; it contains some interesting
|
|
ideas, but not much of an actual proposal. At that time, presumably I had hoped
|
|
to spur further discussion or provoke some objection, in order to clarify what
|
|
we should be aiming at.
|
|
|
|
_During the following years,_ many of the ideas spelled out first in this text
|
|
found their way into the *Diff System*, now used as a foundation for connecting the
|
|
GUI to the Session core in Steam Layer. Especially the discussion of ``Alternatives''
|
|
seem to indicate that an essential motivation for this RfC was to find a viable
|
|
alternative to building the whole Application around a _central data model_ --
|
|
which is the naive solution and tends to become a major impediment to
|
|
hinder the further evolution within larger software projects.
|
|
|
|
- -> see the design page regarding link:{ldoc}/design/architecture/ETD.html[»External Tree Description«]
|
|
- -> and the corresponding technical description
|
|
regarding the link:{ldoc}/technical/library/DiffFramework.html[Diff Framework]
|
|
|
|
[underline]#Bottom line#: not sure what to do with this RfC; concepts explained therein
|
|
seem still highly relevant and central to what Lumiera is intended to become -- and they
|
|
have already been implemented; hoever, this text does not fit into the format of an RfC,
|
|
nor is there a community of developers to discuss such a design vision appropriately.
|
|
|
|
Ichthyostega:: '2025-09-13'
|
|
|
|
|
|
//endof_comments:
|
|
|
|
''''
|
|
Back to link:/x/DesignProcess.html[Lumiera Design Process overview]
|