At various places, concepts and drafts from the early stage of the
Lumiera Project are still reflected in the online documentation pages.
During the last months, development focussed on the Render Engine,
causing a shift in some parts of the design, and obsoleting other
parts altogether (notably we consider to use IO_URING for async IO)
As follow-up to the rework of thread-handling, likewise also
the implementation base for locking was switched over from direct
usage of POSIX primitives to the portable wrappers available in
the C++ standard library. All usages have been reviewed and
modernised to prefer λ-functions where possible.
With this series of changes, the old threadpool implementation
and a lot of further low-level support facilities are not used
any more and can be dismantled. Due to the integration efforts
spurred by the »Playback Vertical Slice«, several questions of
architecture could be decided over the last months. The design
of the Scheduler and Engine turned out different than previously
anticipated; notably the Scheduler now covers a wider array of
functionality, including some asynchronous messaging. This has
ramifications for the organisation of work tasks and threads,
and leads to a more deterministic memory management. Resource
management will be done on a higher level, partially superseding
some of the concepts from the early phase of the Lumiera project.
The EventLog seems to provide all the building blocks, but we need
some higher level special matchers (and maybe we also want to hide
some of the basic EventLog matchers). A soulution might be to wrap
the EventMatcher and delegate all follow-up builder calls.
This seems adequate, since the EventLog-Matcher is basically used as black box,
building up more elaborate matchers from the provided basic matchers...
Spent some time again to understand how EventLog matching works.
My feelings towards this piece of code are always the same: it is
somewhat too "tricky", but I am not aware of any other technique
to get this degree of elaborate chained matching on structured records,
short of building a dedicated matching engine from scratch.
The other alternative would be to use a flat textual log (instead of
the structured log records from EventLog), but then we'd have to
generate quite intricate regular expressions from the builder,
and I'm really doubtful it would be easier and clearer....
- introduce a new entity: RenderDrive
- it supersedes the CalcPlanCalculation, but is managed by CalcStream
- moreover, the RenderDrive will house a IterTreeExplorer-Pipeline
- define the concerns and relationships more clearly (see Drawing)
- prerequisite to disentangle the Job-planning "mechanics"
- decision: the Monad-style iteration framework will be abandoned
- the job-planning will be recast in terms of the iter-tree-explorer
- job-planning and frame dispatch will be disentangled
- the Scheduler will deliberately offer a high-level interface
- on this high-level, Scheduler will support dependency management
- the low-level implementation of the Scheduler will be based on Activity verbs
The drawing code extracts style information from some "virtual"
widgets, which serve as logical placeholder for the actual nested
structure of tracks.
For sake of demonstration, I used rather obvious colours and
also all kinds of margin and padding; a screenshot was added
with annotations to indicate where some specific style settings
are utilised from the drawing code
- pick up all relevant values from CSS
- also control the width of the StaveBracket
- observe the given overall height
Moreover, complete documentation drawing in Inkscape
and add a page to the TiddlyWiki, describing the principles
underlying this design and construction.
.timeline__head : The complete header container on the left side
.timeline__navi : navigation control at top
.timeline__pbay : container holding the patchbay on the left side
.fork__head : each individual TrackHeadWidget (possibly nested)
.fork__control : container for the control components for each track scope
.fork__bracket : the StaveBracket drawing to indicate the nesting structure
Identify the elements of the construction geometry in the "Sketch"
object in the FreeCAD document and paste the corresponding coordinate
values into the SVG drawing prepared for documentation.
The arc segment parameters seemingly are given in radians;
and while FreeCAD uses the common mathematical right-handed orientation,
the orientation in SVG is applied clockwise rather.
...since the construction is determined now (and was worked out in FreeCAD),
the SVG will serve to document the construction; thus the drawing
primitives are rearranged to use the unscaled reference coordinates
to be extracted from the FreeCAD document; all scaling and placement
in the SVG document will be applied through common groups.
My idea was to use the brackets from musical notation as inspiration;
if you know some principles of typography, it is rather straight-forward
to come up with a pleasing design of such a bracket, using a
cascade of golden ratio relationships.
BUT ... all of this is geometry, and translating that into a symbolic
or numerical calculation is excessively complicated. Thus I looked
for ways to use some geometry or CAD software to build such a construction.
The geometry software I tired was woefully inadequate for this task.
Using the Constraint system in FreeCAD, building the construction went
smooth and straight forward, but then I was unable to export that drawing
in a way indicating the construction clearly.
So in the end, I'll have to hand-pick the resulting numerical coordinates
from the FreeCAD XML document and integrate them directly into Cairo
drawing code...
After sleeping some nights over it, rework the wording to make
my reasoning more clear and remove any possibly insulting undertone.
I have seen what I describe here, happening over and over again --
and several times I myself was the one cooking up "simplifications",
which caused lots of pain further down the road.
During the last years, I became more and more doubtful and regretted
that decision. In hindsight, the fundamental conflict was present
already in the original discussion.
My own experience showed me again and again: skipping the hard work
of specification for sake of some kind of fluid prototyping rarely
leads to anything solid. If "time to market" counts, this can be
a viable strategy though...
this is just an "interpretation" of the current architecture diagram,
created for inclusion into the developer report, indicating those components
to be augmented and integrated to get a simple render/playback to work
We are using buttons now, but the standard theme introduces a lot of padding arount button's contents.
Thus we need to consider ways to address the compound of widgets forming an ElementBox; moreover,
this is the classical situation where the BEM notation helps to clarify the intention....
The problem leading to custom styling here is the padding within buttons;
the default stylesheet seemingly adds a min-width and min-height setting,
and some padding within the Button; based on systematic CSS class names,
it is possible to remove these settings specifically for buttons
within the IDLabel in general (no need to treat only the case of an EventBoxLabel
-- IDLabel could become a custom widget on its own
As we continue with building the backbone of the UI,
and abundance of detail information regaring Layout and styling
will be encountered -- it is tantamount to have a place to
write those findings down....
Further extended GTK code survey to clarify the role of the minimum_size,
it is indeed ignored by most standard containers, but it is actually
used by Gtk::Layout as starting point for the query sequence. Thus
it does not make sense to treat minimum and natural size differently;
both queries should be responded by returning our size constraint.
Unless we define additional borders and margins in the CSS, we can be sure
that GTK will base the size allocation on the exact values returned
from the get_required_* functions.
These functions will be invoked only from within the Event Loop
and after the ctor is finished, but before the first "draw".
They will be re-invoked on each "size change" event and on each
focus change (since a focus change may change the style and thus
the actual extension).
On my visit to Benny in the Black forest,
we decided to concentrate on a "Playback Vertical Slice"
and to announce that in the development report, using an
architecture diagram...
Complete the investigation and turn the solution into a generic
mix-in-template, which can be used in flexible ways to support
this qualifier notation.
Moreover, recapitulate requirements for the ElementBoxWidget
- move construct into the buffer
- directly invoke the payload constructor through PlantingHandle
- reconsider type signature and size constraint
- extend the unit test
- document a corner case of c++ "perfect forwarding",
which caused me some grief here
In 2017, I did a first design draft, followed by a design critique,
which partially obsoleted some ideas regarding command binding.
Mostly, the reason to abandon parts of that initial design was
due to the fact, that to many actual construction details of the
UI framework were not worked out at that time.
Thus I rather focussed on (re)-building a backbone for the timeline display,
in order to support that kind of flexibility aspired within the session model.
Now, when re-visiting the topic of an UI gesture (using simple dragging
of a clip in the timeline as an example for a first draft), I picked up
some of those planned structures, but tend to bind them together in
a slightly different way -- more akin to a state machine and less
in the way of an LR-parser.
This chagneset updates the relevant part within the TiddlyWiki
and the corresponding UML drawing to better reflect my actual thinking.
...in an attempt to clarify why numerous cross links are not generated.
In the end, this attempt was not very successful, yet I could find some breadcrumbs...
- file comments generally seem to have a problem with auto link generation;
only fully qualified names seem to work reliably
- cross links to entities within a namespace do not work,
if the corresponding namespace is not documented in Doxygen
- documentation for entities within anonymous namespaces
must be explicitly enabled. Of course this makes only sense
for detailed documentation (but we do generate detailed
documentation here, including implementation notes)
- and the notorious problem: each file needs a valid @file comment
- the hierarchy of Markdown headings must be consistent within each
documentation section. This entails also to individual documented
entities. Basically, there must be a level-one heading (prefix "#"),
otherwise all headings will just disappear...
- sometimes the doc/devel/doxygen-warnings.txt gives further clues
No new information added, rather removed lots of technical details,
which do not belong into design documentation. And try to present
the existing information more comprehensively
..at that time, that page was hastily written do document an
somewhat controversial implementation draft, which later on evolved
into the Application-main object Lumiera relied on since then.
Recently, a dedicated page dealing with subsystems and lifecycle
has been added to the Architecture section; so this page here
should be rewritten to focus on the "Application realm" as such.
The population message is just made up, in order to create more interesting structures
in the UI and so to further the development of the timeline display.
For the actual structure I choose to mirror my example drawing in draw/UI-TimelineLayout-1.png
which is also used in the TiddlyWiki, on the #GuiTimelineView tiddler
https://lumiera.org/wiki/renderengine.html#GuiTimelineView
Mostly, std::regexp can be used as a drop-in replacement.
Note: unfortunately ECMA regexps do not support lookbehind assertions.
This lookbehind is necesary here because we want to allow parsing values
from strings with additional content, which means we need explicitly to
exclude mismatches due to invalid syntax.
We can work around that issue like "either line start, or *not* one of these characters.
Alternatively we could consider to make the match more rigid,
i.e we would require the string to conain *only* the timecode spec to be parsed.
Even while EveryoneElese indulges in cool "flat" UI graphics,
we still think that a plausible 3D structure of UI widgets supports intuitive user interaction
As an asside, this commit fixes a mistake with the licenses of several of these documentation drawings.
I am the author of all these SVGs and thus can fix such a license glitch without much ado.
These drawing shall be licensed in accordance to the general rule for Lumiera Documentation,
which is to use a Libre-style license, here CC-by-sa (which does *not* limit commercial use)
- most notably the NOBUG logging flags have been renamed now
- but for the configuration, I'll stick to "GUI" for now,
since "Stage" would be bewildering for an occasional user
- in a similar vein, most documentation continues to refer to the GUI
Considering this since some time, since it more and more occurred to me
the existing conventional names are a misfit. And they are dull and clumsy.
This fall, I mentioned it to Benny, and he seemed to be rather favourable towards that idea,
which encourages me just to go ahead. Unfortunately, I am alone on the coding frontier
right now, which has several downsides, but at least it gives me the ability
to pull off radical moves.
The goal is to run the build_website.sh script in the doc/ dir
(for local testing purposes only, not on the server)
added to doc/.gitignore because:
/broken_links
the linkchecker generates it
/documentation
symlink to ./ to fix links
/css
/js
/page.conf
/images
symlink to some local copy of the
lumiera-website checked out
/menu.html
fill with (to create a fake empty page):
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<TITLE>no menu</TITLE>
.*
*~
ignore hidden and backup files
Then from the 'doc' dir:
python -m SimpleHTTPServer 8010 &
echo http://localhost:8010 >.baseurl
/path/to/build_website.sh
to generate the documentation. Other build_website.sh options work as well.
bottom line is to do most autmatically, and to establish a slave-relation
navigation-area -> timeline-ruler
header-pane-content -> corresponding track-body
this can be accomplished mostly by connecting the aproprieate signals,
thus these widgets will live within the Layout-Manager, which consequently
is renamed into TimelineLayout
...just to decide not to follow-up too much on that topic right now.
As it turns out, GTK seems to be lacking in that respect. I have plotted
some ideas how we could work around that discrepancy in future...
And for this simple DemoGuiRoundtrip, we'll just use direct styling,
but we'll store a table of bookmarks for the error entries, allowing
us to add further features later on top
Paragraph ordering was somewhat confusing: you talk about
transport controls before defining what they actually are. Better
to define them first so that the reader knows what you are talking
about.
The important page on contacting Lumiera was buried below
somewhere in Contributing to Lumiera. This information must
be provided top-level and easily accessible. So I've added it,
maybe not terribly suitably, to the tutorials page. However its
position there is saved and justified by placing it under the
tutorial on contributing to Lumiera.
Maybe someone has a better idea.
In the end, I decided against building a generic service here,
since it pretty much looks like a one-time problem.
Preferrably UI content will be pushed or pulled on demand,
rather than actively coding content from within the UI-Layer
- activation signal is a facility offered and used solely by Gtk::Application
- we do not need nor want an Gtk::Application, we deal with our own application
concerns as we see fit.
Static initialisation and shutdown can be intricate; but in fact they
work quite precise and deterministic, once you understand the rules
of the game.
In the actual case at hand the ClassLock was already destroyed, and
it must be destroyed at that point, according to the standard. Simply
because it is created on-demand, *after* the initialisation of the
static DependencyFactory, which uses this lock, and so its destructor
must be called befor the dtor of DependencyFactory -- which is precisely
what happens.
So there is no need to establish a special secure "base runtime system",
and this whole idea is ill-guided. I'll thus close ticket #1133 as wontfix
Conflicts:
src/lib/dependable-base.hpp
all these tests are ported by drop-in replacement
and should work afterwards exactly as before (and they do indeed)
A minor twist was spotted though (nice to have more unit tests indeed!):
Sometimes we want to pass a custom constructor *not* as modern-style lambda,
but rather as direct function reference, function pointer or even member
function pointer. However, we can not store those types into the closure
for later lazy invocation. This is basically the same twist I run into
yesterday, when modernising the thread-wrapper. And the solution is
similar. Our traits class _Fun<FUN> has a new typedef Functor
with a suitable functor type to be instantiated and copied. In case of
the Lambda this is the (anonymous) lamda class itself, but in case of
a function reference or pointer it is a std::function.
- polish the text in the TiddlyWiki
- integrate some new pages in the published documentation
Still mostly placeholder text with some indications
- fill in the relevant sections in the overview document
- adjust, expand and update the Doxygen comments
TODO: could convert the TiddlyWiki page to Asciidoc and
publish it mostly as-is. Especially the nice benchmarks
from yesterday :-D
As it turned out, we had two bugs luring in the code base,
with the happy result of one cancelling out the adverse effects of the other
:-D
- a mistake in the invocation of the Itertools (transform, filter,...)
caused them to move and consume any input passed by forwarding, instead
of consuming only the RValue references.
- but util::join did an extraneous copy on its data source, meaning that
in all relevant cases where a *copy* got passed into the Itertools,
only that spurious temporary was consumed by Bug #1.
(Note that most usages of Itertools rely on RValues anyway, since the whole
point of Itertools is to write concise in-line transformation pipelines...)
*** Added additional testcode to prove util::stringify() behaves correct
now in all cases.
surprising behaviour encountered while covering more cases
...obviously the return type of ExpandFunctor::operator()
was inferred as value, even while the invoked functor, from which
this type was deduced, clearly returns a reference.
Solution is simple not to rely on inference, moreover since we know
the exact type in the enclosing scope, thanks to the refactoring which
made this ExpandFunctor a nested class
NOTE:
as it turned out, this is not a compiler bug,
but works as defined by the language:
on return type inference, the detected type is decayed,
which usually helps to prevent returning a reference to a temporary
..this collection of ideas, terms and conclusions has been shaped
since some time within the TiddlyWiki. Since I've now started even
some supporting implementation regarding these concepts, its time
to publish them in the design documentation section of the Website
this becomes more relevant now, since the actual MutationMessage iterators
are implemented in terms of a shared_ptr to IterSource. Thus, when building
processing pipelines, we most definitively want to move that smart-ptr into
the destination, since this avoids touching the shared count and thus avoids
generating unnecessary memory barriers.
obsoleted by C++11
* in most cases, it can be replaced by an explicit conversion operator
* especially for the Lumiera Forward Iterators, we need an implicit conversion
This changeset fixes a huge pile of problems, as indicated in the
error log of the Doxygen run after merging all the recent Doxygen improvements
unfortunately, auto-linking does still not work at various places.
There is no clear indication what might be the problem.
Possibly the rather unstable Sqlite support in this Doxygen version
is the cause. Anyway, needs to be investigated further.
After some consideration, it became clear that this service implementation
is closely tied to the DispatcherLoop -- which will consequently be
responsible to run and expose this service implementation
...because of questionable utility
And it turned out to be quite a liability when it comes to implementing
the building blocks for binding tree implementation data structures
this is a tricky problem and a tough decision.
After quite some pondering I choose to enforce mandatory fields
through the ctor, and not to allow myself cheating my way around it
BoUML was close sourced by its author and consequently dropped
from Debian. We use UML as a drawing and documentation tool,
and thus we'll just start over with the /obvious/ choice,
which is the Umbrello tool.
not sure yet if any of this works, because the
technicalities of dealing with variadic types are
quite different to our LISP-style typelist processing.
The good news is that with variadic templates it is
indeed possible, to supply dynamically picked arguments
to another function taking arbitrary arguments.
This all relies on the feature to unpack argument packs,
and, more specifically, about the possiblity to "wrap"
this unpacking around interspersed function call syntax
template<size_t...i>
Xyz
do_something(MyTuple myTuple)
{
return Xyz (std::get<i> (myTuple) ... );
}
Here the '...' will be applied to the i... and then
the whole std::get-construct will be wrapped around
each element. Mind bogging, but very powerful
this is a bit trickty, since we need to install gcc-4.9 from a PPA.
Mint only provides the libstdc++ 4.8, which is known to break when
compiling in C++14 mode
NOTE: we have the policy to always support current Debian/stable
amd at least one Ubuntu LTS release, unless hard dependency problems prevent that.
Currently, Ubuntu/Trusty is already a bit dated, but the only problematic dependency
could be libboost (1.54 in Trusty, 1.55 in Jessie).
GCC-4.8 can be replaced by GCC-4.9 in Trusty without problems
It is always a bit tricky to find out the precise lower boundary,
so we try to upgrade these requirements as our platform progresses.
For now we have used the level available on Ubuntu/Trusty to set
the lower constraints for most libraries
...this will be the third preview release
Lumiera is still in pre-alpha stage, and thus
there are no proper releases, just preview snapshots.
Again this version will be built and packaged
on several supported Linux platforms
This page gives the rationale for the way our diff framework is built.
This reasoning might *reduce* the relevance of any decisions
regarding the implementation data structure and thus lead to
far reaching consequences for the whole architecture.
This means we have rather tight compiler requirements now.
Beyond that, we expect no serious impact; the most notable
C++14 feature we're likely to use soon is type inference
on lambda arguments.
Introduce the new term "Fork" at various relevant places
within the documentation. We do not entirely purge the
term "track" though; rather we
- make clear that "Fork" is the entity to build tracks
- use "fork" also synonymous to the "tree of tracks"
This switches the Lumiera UI from GTK-2 to GTK-3
Unfortunately, this move breaks two crucial features, which have been
disabled for now: the display of video and our custom timeline widget.
Since both of these require some reworking, which in fact has already
started, we prefer to do the library and framework switch right away.
This too was a long-standing issue. While these practices
basically can be considered "common knowledge", experience
showed those topics are frequently unknown even to practised
programmers.
So now we have a single page dealing with all those issues of
code bloat, dependency poliferation, binary dependency resolution
and issues of transitive and circular library dependencies
the corresponding requirements are already reflected in the
SCons build, see Platform.py
NOTE: the current debian package is still based on the preview
Release 0.2.pre from last year. It will be upgraded probably after
the transition to Jessie as reference system
This piece of documentation describes an insideous special case, which
some time ago prevented us from switching to --as-needed linking.
We treat this as a special case (and it is way easier to do so
now, after the reorganisation of our test suite).
deliberately, I've left #948 open to nudge me about writing this doc
This is very arkane, hard to find knowledge about some intricacies
of the dynamic library resolution. Very relevant for Lumiera,
since we use a resolution scheme relative to the location
of the executable. Documenting this stuff was a long-standing issue
a long standing TODO to document the actual start-up sequence, which
is implemented this way since a long time now. There was an unwritten
section in the "Linking and Application Structure", which seems the
apropriate place for this kind of intricate techincal details.
Last week, Benny Lyons was here on visit in munich and he was pondering
the idea of an experimental secondary build system, as a way to learn
more about the source structure of Lumiera. This reminded me to fill
some missing parts of the documentation. Possibly this is also the
right moment to land the GTK-3 transition?
There is a long-standing RfC which basically describes the
same idea on a much wider, conceptual scope. Indeed I consider
this approach used here for solving the problem with GUI uptades
also as a proof of concept, to be expanded to a much wider scope
in case it works out well.
The new insight here is, that, by transferring a diff in pull mode,
we can circumvent the architectural problems with typing, which
showed up quite clearly in earlier design studies towards this
concept. The change from push to pull is by far not so fundamental
as it looks, since the sender still may initiate the exchange
by sending a message offering the diff iterator for the receiver
to pull. This way, we get a handshake and still sustain the
crucial part, which is to decouple the data representation
and give the receiver full control over the interpretation
of the exchanged data.
This is the first step towards a generic backbone to connect
any GUI elements to the session within Proc-Layer.
It is based on a spefic understanding of Model-View-Controller,
which turns the Model-Controller interactions into messages.
Remove some orphaned diagrams and PNG images not actually used
in the TiddlyWiki. Add a page with some hints regarding Bouml
See also #960 -- Bouml has been discontinued and is closed source now
not sure how to proceed with this
This means to discontinue any research into emitting an optimal
diff verb sequence for now and just to document the possible path
I see to reach at such a optimal solution later, when it turns out
to be really necessary performance wise.
Personal note: I arrived at this conclusion while whatching the
New Year fireworks 2014/2015 at the banks of the Isar river in
the centre of the city.
Too sad that 2014 didn't bring us World War III
