- reorganise the navigation tab structure
- place all further index lists into a common "Index" tab
- include a list of concepts
- remove the various "member" sub-tabs, these are not helpful
- the "modules" tab is now called "topics" (since C++ has now Modules!)
- generally re-sync DoxygenLayout.xml with a current pristine template
- comb though the Doxygen Warnings and fix a lot of small problems
The »Outer Space« and »Inner Core« documents present a comprehensive
overview of the vision, architecture and essential parts of the implementation.
In the light of changes to policy, design and implementation approaches,
some updates were necessary to align these crucial texts with the current
state of planning and implementation. Notably I have added a recent Screenshot
of the UI, showing a nested track structure pushed up by Diff from the core.
Over the last weeks, I conducted an extended analysis of the inception time
of the Lumiera project, which gave me a sharpened understanding of the goals,
going forward. I have also written an in-depth essay "Complexity and Flexibility".
See the corresponding commits in the Website repository.
The RfC documents were written to complement discussions of the Lumiera developers;
yet since the time where ''Ichthyo'' is working basically alone on the project,
this kind of discussions have ceased. During the following years, some ideas
promoted by the existing RfC documents became rather detached from the
actual state of development in the code base.
Many of the existing RfC documents require some commentary to place them
into context, and some of the decisions taken in the early stage of the
project should be **re-assessed**. This includes the decision to reject
some proposals, which initially might have seemed desirable, yet could not
be reconciled with the understanding of the matter and topic in question,
as was gained through the ongoing analysis and development.
The purpose of RfCs is to channel and document discussions
among a group of developers regarding questions of design.
While there is unfortunately no longer a group of developers
discussing matters and working together on the code base,
some questions can be considered more or less settled,
by being implemented and validated.
- the Scheduler
- structure of the Developer Documentation
- Application installation as relocatable bundle
Some sections of the Lumiera website document meeting minutes,
discussion protocols and design proposals from the early days
of the project; these pages were initially authored in the
»Moin Moin Wiki« operated by Cehteh on pipapo.org at that time;
this wiki backed the first publications of the »Cinelerra-3«
initiative, which turned into the Lumiera project eventually.
Some years later, those pages were transliterated into Asciidoc
semi-automatically, resulting in a lot of broken markup and links.
This is a long standing maintenance problem problem plaguing the
Lumiera website, since those breakages cause a lot of warnings
and flood the logs of any linkchecker run.
- indicate clearly that a ''past discussion'' is documented here
- point out what is the primary focus for UI design ''currently''
- arrange some links better
- use cross-links / Linkfarm; especially cross-link to
the discussion regarding Wouter's Workflow proposals (2025)
Create a new subcategory "design/architecture/time"
and rearrange several pages related to time handling and time codes.
NOTE: starting with this changeset, a ''Link-Farm'' is required for cross-links;
since we don't have an automatic solution for this task yet, I have created
the necessary forwarding pages manually in the website repository.
...fine-tuned a few small bits here and there in the
current chapter of the workflow document
===========================================================
Remark from Ichthyo(committer):
Converted from PDF to raw text with
pdftotext wouter250819v3.pdf WorkflowProposals.txt
Images extracted with:
pdfimages -png -j wouter250819v3.pdf wouter/wouter250819v3
Again adding the images separately to the Website-repository,
in order to save storage space in the main repository....
- add markup to match formatting from PDF
- link to the images extracted into the website Git-repository
- adjust image sizes to fit into the text
- add some cross references
(incidentally: TimelineDiscussion.txt -- store image locally)
Chapter 2 (about the timeline) is now complete.
- extend discussion regarding trackless
- add section about Tools / Modes / Views
- section: Adding clips to the timeline
- section: Selecting clips
- section: Arranging clips
- section: Trimming clips
- section: Splitting and merging clips
- section: Removing clips
- add subchapter regarding sections of the timeline
- Adding and editing transitions
- Changing timeline clip properties
===========================================================
Remark from Ichthyo(committer):
Converted from PDF to raw text with
pdftotext wouter250629v2.pdf WorkflowProposals.txt
Images extracted with
pdfimages -png -j wouter250629v2.pdf wouter/wouter250629v2
Images required no cropping;
the white background images were omitted.
Again adding the images separately to the Website-repository,
in order to save storage space in the main repository....
This is the version from 4.Apr
- add a subchapter about the current NLE landscape
- add a subchapter on tracks vs. trackless
- add a bit more information in the navigation subchapter.
===========================================================
Remark from Ichthyo(committer):
Converted from PDF to raw text with
pdftotext wouter250404v1.pdf WorkflowProposals.txt
Images extracted with
pdfimages -png wouter250404v1.pdf
...and then cropped in Gimp ("crop to content");
images 06 and 11 needed to be split in two parts
cropped in Gimp ("crop to content")
NOTE: Adding the images separately to the Website-repository,
in order to save storage space in the main repository,
because content, once added, can not be removed from Git....
- included the list of "Personas", quoting from Wouter's document
- add some details from the ensuing discussion
- close the document with a section "Conclusions" to list the
most relevant open points
Topic: proposals for Lumiera Workflow
Present:
- Wouter Verweijlen
- Benny Lyons
- Hermann Voßeler
Note: This commit creates a new subsection
for the discussion related to Wouter's »Lumiera Workflow Proposals«....
Starting with ''preview release'' `v0.pre.04`, branch and version tags
will be handled in accordance to the **Git-flow** naming scheme.
Notably this implies that from now on the version in-tree will indicate
the ''next expected release,'' adorned by a suffix to mark the preview.
To accommodate this transition to Git-flow
- the new branch `integration` will be introduced
- the version number will once (and the last time for this release)
be adjusted ''before'' forking the release branch
- branch `master` will transition to reflect the latest released state
- several existing branches will be discontinued, notably
`gui`, `steam`, `vault`, `release`, `play`
Starting with the upcoming ''preview release'', branches, branch names and tags
will be rearranged to follow the Git-flow pattern instead of the existing
ad-hoc organisation with a release branch.
The documentation provided here defines the actual naming conventions
and some fine points regarding the version number upgrades
and placement of release tags.
Furthermore, two helper-scripts are provided to automate version number updates
- `buildVersion.py` : extract current version from git tag and allow to bump version
- `setVersion` : manipulate all relevant files with `sed` to update the version info
This changeset removes various heuristics and marker-traits
by a constraint to tuple_like types. Furthermore, several usages
of `apply` can thereby be generalised to work on any tuple_like.
This generalisation is essential for the passing generic data blocks
via `FeedManifold` into the node invocation
This resolves an intricate problem related to metaprogramming with
variadic templates and function signatures. Due to exceptional complexity,
a direct solution was blocked for several years, and required a better
organisation of the support code involved; several workarounds were
developed, gradually leading to a transition path, which could now
be completed in an focused clean-up effort over the last week.
Metaprogramming with sequences of types is organised into three layers:
- simple tasks can be solved with the standard facilities of the language,
using pattern match with variadic template specialisations
- the ''type-sequence'' construct `Types<T...>` takes the centre stage
for the explicit definition of collections of types; it can be re-bound
to other variadic templates and supports simple direct manipulation
- for more elaborate and advanced processing tasks, a ''Loki-style type list''
can be obtained from a type-sequence, allowing to perform recursive
list processing task with a technique similar to LISP.
Indeed — this change set is kind of sad.
Because I still admire the design of the GAVL library,
and would love to use it for processing of raw video.
However, up to now, we never got to the point of actually
doing so. For the future, I am not sure if there remains
room to rely on lib-GAVL, since FFmpeg roughly covers
a similar ground (and a lot beyond that). And providing
a plug-in for FFmpeg is unavoidable, practically speaking.
So I still retain the nominal dependency on lib-GAVL
in the Build system (since it is still packaged in Debian).
But it is pointless to rely on this library just for an
external type-def `gavl_time_t`. We owe much to this
inspiration, but it can be expected that we'll wrap
these raw time-values into a dedicated marker type
soon, and we certainly won't be exposing any C-style
interface for time calculations in future, since
we do not want anyone to side-step the Lumiera
time handling framework in favour of working
„just with plain numbers“
NOTE: lib-GAVL hompage has moved to Github:
https://github.com/bplaum/gavl
Since C++17 we can use the std::filesystem instead (and we ''do use it'' indeed)
- relocate the `/lib/file.hpp` header
- adapt the self-discovery of the executable to using std::filesystem
Furthermore, some recherche regarding XVideo and Video Output
- remove obsolete configuration settings
- walk through all settings according to the documentation
https://www.doxygen.nl/manual/config.html
- now try to use the new feature to rely on Clang for C++ parsing
- walk through the doxygen-warnings.txt and fix some obvious misspellings
and structural problems in the documentation comments.
With Debian-Trixie, we are now using Doxygen 1.9.8 —
which produces massively better results in various fine points.
However, there are still problems with automatic cross links,
especially from implementation to the corresponding test classes.
Some pre C++11 features are marked deprecated and will be rejected with C++20
Notably the old marker inferfaces for unary (and binary) functions are no longer needed, since function-like objects can be detected by traits or concepts nowadays
Moreover we can get rid of some boost(bind) usages and use a λ
BuilderDoxygen and BuilderGCH are external plug-ins,
not developed in this project and probably unmaintained.
TODO: decide how to fix or replace them...
With the ability to invoke a Render Node graph,
the development on branch `play` reached some kind of milestone
regarding the »Playback Vertical Slice«.
This is a good opportunity to update the reference platform
and upgrade the preview releases and packaging setup accordingly.
This will include adjustments to compile on recent compilers and
upgrade the build system to support Python-3.
Unfortunately, there are some common syntactic structures, which can not easily be dissected by regular expressions alone, since they entail nested subexpressions. While it is possible to get beyond those fundamental limitations with some trickery, doing so remains precisely that, ''trickery.''
After fighting some inner conflicts, since ''I do know how to write a parser'' —
in the end I have brought myself to just do it.
And indeed, as you'd might expect, I have looked into existing library solutions,
and I would not like to have any one of them as part of the project.
* I do not want a ''parser engine'' or ''parser generator''
* I want the directness of recursive-descent, but combined with Regular Expressions as terminal
* I want to see the structure of the used grammar at the definition site of the custom parser function
* I want deep integration of ''model bindings'' into the parse process, i.e. binding-λ
* I do not want to write model-dissecting or pattern-matching code after the parse
* I do not want to expose ''Monads'' as an interface, since they tend to spread unhealthy structure to surrounding code
* I do not want to leak technicalities of the parse mechanics into the using code
* I do not want to impose hard to remember specific conventions onto the user
Thus I've set the following aims:
* The usage should require only a single header include (ideally header-only)
* The entrance point should be a small number of DSL-starter functions
* The parser shall be implemented by recursive-descent, using the parser-combinator technique
* But I want that wrapped into a DSL, to be able to control what is (not) provided or exposed.
* I want a stateful, applicative logic, since parsing, by its very nature, is stateful!
* I want complete compile-time typing, visible to the optimiser, without a virtual »Parser« interface
And last but not least, ''I do not want to create a ticket, since I do not know if those goals can be achieved...''
Remove left-overs from the preceding prototypical implementation,
which is now obliterated by the change to a flexibly configured `FeedManifold`
with structured, typed storage for buffers and for parameter data.
The Render Node invocation sequence, as rearranged and reworked for the »Playback Vertical Slice«, now seems reasonably clear and settled.
Adding extensive documentation to describe the conventions and structures worked out thus far;
moreover, start makeover of old documentation in the !TiddlyWiki to remove concepts obviously obsoleted now...
The overall goal is eventually to arrive at something akin to a ''»Dummy Media-processing Library«''
* this will offer some „Functionality“
* it will work on different ''kinds'' or ''flavours'' of data
* it should provide operations that can be packaged into ''Nodes''
However — at the moment I have no clue how to get there...
And thus I'll start out with some rather obvious basic data manipulation functions,
and then try to give them meaningful names and descriptors. This in turn
will allow to build some multi-step processing netwaorks — which actually
is the near-term goal for the ''main effort'' (which is after all, to get
the Render Node code into some sufficient state of completion)...
There is an insidious problem when the Transformer takes references to internal state
within upstream iterators or state core. This problem only manifests when
a invariant based filtering or grouping operation is added after the Transformer,
because such an operation (notably Filter) will typically attempt to establish
the invariant from the constructor (to avoid dangling state). Unfortunately
doing so involves pulling data ''before the overall pipeline is moved into final location''
A workaround is to make the Transformer ''disengage'' on copy, so to provoke
a refresh and new pull in the new location after the copy / move / swap.
This only works if the transformer function as such is idempotent.
Indeed the solution worked out yesterday could be extracted and turned generic.
Some in-depth testing is necessary though, and possibly some qualifications to allow pass-through of references...
Moreover, last days I started collecting notes regarding problem solving patterns,
which I tend to use frequently, but which might not be obvious and thus can easily
be forgotten. In fact, I had encountered several cases, where I did invent some
roughly similar solution repeatedly, having forgotten about already settled matters.
Hopefully the habit of collecting notes and hints at a central location serves to remedy
Basically I am sick of writing for-loops in those cases
where the actual iteration is based on one or several data sources,
and I just need some damn index counter. Nothing against for-loops
in general — they have their valid uses — sometimes a for-loop is KISS
But in these typical cases, an iterator-based solution would be a
one-liner, when also exploiting the structured bindings of C++17
''I must admit that I want this for a loooooong time —''
...but always got intimidated again when thinking through the fine points.
Basically it „should be dead simple“ — as they say
Well — — it ''is'' simple, after getting the nasty aspects of tuple binding
and reference data types out of the way. Yesterday, while writing those
`TestFrame` test cases (which are again an example where you want to iterate
over two word sequences simultaneously and just compare them), I noticed that
last year I learned about the `std::apply`-to-fold-expression trick, and
that this solution pattern could be adapted to construct a tuple directly,
thereby circumventing most of the problems related to ''perfect forwarding''
So now we have a new util function `mapEach` (defined in `tuple-helper.hpp`)
and I have learned how to make this application completely generic.
As a second step, I implemented a proof-of-concept in `IterZip_test`,
which indeed was not really challenging, because the `IterExplorer`
is so very sophisticated by now and handles most cases with transparent
type-driven adaptors. A lot of work went into `IterExplorer` over the years,
and this pays off now.
The solution works as follows:
* apply the `lib::explore()` constructor function to the varargs
* package the resulting `IterExplorer` instantiations into a tuple
* build a »state core« implementation which just lifts out the three
iterator primitives onto this ''product type'' (i.e. the tuple)
* wrap it in yet another `IterExplorer`
* add a transformer function on top to extract a value-tuple for each ''yield'
As expected, works out-of-the-box, with all conceivable variants and wild
mixes of iterators, const, pointers, references, you name it....
PS: I changed the rendering of unsigned types in diagnostic output
to use the short notation, e.g. `uint` instead of `unsigned int`.
This dramatically improves the legibility of verification strings.
I changed the rendering of unsigned types in diagnostic output
to use the short notation, e.g. `uint` instead of `unsigned int`.
This dramatically improves the legibility of verification strings.
Moreover, I took the opportunigy to look through the existing page
with codeing style guides to explicitly write down some conventions
formed over years of usage.
I did not just »make up« those light heartedly, rather these conventions
are the result of a craftsman's ''attentive observation and self-reflection.''
* based on reproducible data in `TestFrame`
* using Murmur64A hash-chaining to »mark« with a parameter
This emulates the simplest case of 1:1 processing and can also be applied ''in-place''
- the starting point is the idea to build a dedicated ''turnout system''
- `StateAdapter`, `BuffTable` ⟶ `FeedManifold` and _Invocation_ will be fused
- actually, the `TurnoutSystem` will be ''pulled'' and orchestrate the invocation
- the structure is assumed to be recursive
The essence of the Node-Invocation, as developed 2009 / 2011 remains intact,
yet it will be organised along a clearer structure
The behaviour seems consistent and the schedule breaks at the expected point.
At first sight, concurrency seems slightly to low; detailed investigation
however shows that this is due to the structure of the load graph,
and in fact the run time comes close to optimal values.
