The detailed analysis of corner cases indicated a lot of hidden dangers
when stretching the `IterExplorer` design beyond its limits.
By deliberate choice, this design was kept open so that advanced cohesive
implementation structures can be built. It is even possible ''and tolerated''
to rely on ''references'' to some degree -- assuming however that the
referred memory location remains stable.
As it turned out, these basic assumptions can be broken by a combination
of read-ahead processing (as is necessary for grouping and filtering),
together with adaptor objects embedded as value, especially when
the resulting pipeline object is then moved or copied during evaluation.
While my general attitude towards such extremely clever implementation
constructs is "don't shoot yourself into your foot unless you need to" --
it seems prudent to clean-up some crappy code from the first attempts
towards an iterator pipeline -- notably some of the »Itertools«.
This is a feature I wasn’t sure for a long time how best to tackle.
It’s worth noting that an initial attempt at `IterExplorer` was based
on the monad pattern, which turned out to be a mistake, since monads
are more of an anti-pattern, than being genuinely useful.
The actual use-case that made me reconsider such a feature it
to build an seamlessly chained sequence from several iterators.
It would be simple to develop a suitable utility as a stand-alone helper,
yet it seemed worth the effort to integrate this into the `IterExplorer`
ecosystem, due to the flexible automatic adaptation to iterable source
pipelines provided therein.
Going this route incurred some cost however, unveiling several defects
related to handling nested iterable types in the generic processing framework.
After some investigation, I seem to understand the problem better now.
This builds upon the Introduction of our `ValueTraits´ two years ago:
e176e54004
The root cause to these ongoing sporadic problems with type deduction
seems to be that we encounter a **clash of concepts**: What Lumiera calls
an »Iterator« is essentially different to the order of terms used by the STL.
In the STL, an iterator can be seen as an ''abstracted pointer'', and is
thus only meaningful in conjunction with a backing container. It is also
this backing STL container that provides the value-type definitions.
In Lumiera however, an ''Iterator is a promise to retrieve a sequence of evaluations''.
For that reason, the Iterator itself provides value-type definitions,
and the notion of "iterable" can largely be conflated with "Lumiera Iterator".
Now, the `lib::RangeIter` can act as a bridge between both usage patterns,
and thus it is possible to create a `RangeIter` both from a Lumiera Iterator
or from a STL container. (As an aside, this is a similar idea as promoted
by the "Range TS" in C++20 -- yet the Lumiera iterator concept is much more
abstract and allows evaluations like tree expansion and search algorithms,
which would be never possible with the STL based notion of iteration)...
Thus the value type binding in `RangeIter` is the only correct place
where to mitigate this ''clash of concepts:''
* if the `IT` parameter is a Lumiera iterator, we need to retrieve
the type bindings defined therein
* otherwise, we can assume that `IT` is an STL iterator and thus
essentially a pointer, and we must strip that pointer and work from there.
...to emplace some object into the buffer space.
I wouldn't expect that to be necessary for the real usage,
but it comes in handy for tests, where we often want to place
some sentinel value into a buffer.
- can now throw away the old convoluted `TrackingInMemoryBlockSequence`
- instead use a simplistic implementation of the Connection interface
- do not perform any kind of validation, allocate new buffer for each frame
- use the `NaiveBufferSetup` (`BufferProvider`) as implementation backend.
This will be the first ''reality test'' for the new `OutputSlot` internal design:
Is it possible to implement a tracking test implementation in a clean
and straight-forward way, avoiding all the tangling and indirection
that plagued the old prototype implementation from 2012?
After resolving the inner tensions in the old prototype from 2012,
it was now possible to resolve all issues in a single strike
* `DataSink` becomes a functor, yet embeds a shared ref-count
* in the λ-closure a ''specific Buffer Descriptor'' is prepared for each feed
* building upon the prototyping work, a dedicated `OutputBufferProxy`
exposes a `BuffHandle` to the client, with `emit()` and `release()` calls
* the implementation ''delegates directly'' to the `OutputSlot::Connection`
* an this is where all actual output technology is encapsulated
...this is the gordian λ
Note
- embedding a copy of the shared-ptr in the closure
- pre-configure a buffer descriptor (buffer type)
* with the buffer size as retrieved from the connection
* and pass the connection itself marked as LocalTag
Thus, on invocation of the DataSink (functor), it suffices
to invoke the lock() function on the BufferDescriptor,
using the new additional argument introduced with the
preceding commit
Reaping the benefits of the preceding refactorings, most of the complex
wiring within `OutputBufferProxy` can be omitted, and the proxy calls
can be delegated directly to the actual `Connection`
There is a catch however: we need to create an additional pathway
to feed the frame-number information into the `BufferStore` backend.
The solution adopted here is the result of extensive reflection.
Taken literally, or naively, the reworked architecture would imply
to route every call through a delegation-chain that is dispatched
in the state-managing "PImpl" (`AllocState<CON>`).
However, in keeping with the idea of an opaque functor embedded into
`DataSink`, direct and explicitly typed calls can be bound in the
associated closure. Doing so will strengthen the role of the
interface `Connection` — which defines the boundary to the
actual domain of output technology.
This changeset attempts to breach into the implementation design of OutputSlot
(while largely retaining the API and the »Output Slot Protocol«).
While the actual problem that brought me to this point can be traced back to some
rather obvious mistake, namely to conflate the `BufferProvider` used as
backing implementation with a `BufferProvider` used as front-end API,
creating the new connections to use an `OutputBufferProxy` unveils some
further long-standing structural mismatch; notably this leads to the
difficulties to get the lifecylce management of `OutputSlot` and `DataSink`
under control.
So I remove one entire level from the implementation,
in a bold move to realign the design with the prospective usage pattern.
Obviously, this breaks a lot of further code, and I need to rewire
the Allocation implementation altogether, and provide a new functor based
implementation for `DataSink`
After a lot of pondering over various solution schemes,
it seems that `BufferProxyAdaptor` is too rigid for the task at hand.
It could be retained as a technology demonstration, and used as blueprint
for a custom-made implementation. In a nutshell, we need a `BufferStore`
that directly delegates to the respective `Connection` — which also implies
to use the `OutputSlot::AllocState` as an intermediate to hendle that delegation.
...which still looks like a nasty task
* the good part is, we have a working `BufferProxyAdaptor` now
* yet this setup does not seem to fit into the `OutputSlot::AllocState`
As an aside, several test definitions need to be relocated to the Vault,
since the whole compound around `BufferProvider` migrated down
This is a nasty new requirement, discovered as an aside while looking
into ways to resolve the ''crucial mismatch'' between the `OutputSlot`
and `BufferProvider`, using the proxy-implementation of `BufferProvider`
(thereby expanding on the recent draft with the `BufferProxyAdaptor`)
The point is, if the BufferMetadata will be used beyond the internal coordination
within BufferProvider, then it will be used as part of actual buffer storage.
And this implies that the actual size of the allocation may differ (be greater)
than the nominally required size (since we'll very likely employ some kind
of memory tiling, for reasons of performance).
This leads again to an analysis of `BufferMetadata` usage,
with the conclusion to add the actual storage size directly into the API,
but treat it as an ''extra'' and not integrated into the ''buffer type''
Phew ... quite complex alltogether
This might have seemed like a good idea, adding an additional ''consistency check''...
However — we ''do have'' a generic check, insofar `BufferMetadata` is a **state engine**.
As long as we're able to prevent, reliably, that the same buffer address
ends up being classified as a different buffer type, this should be enough
to prevent double-use of any buffer (letting aside that `BufferProvider` is
backed by an implementing memory allocator, which in itself should prevent
any double-use.
Removing this feature resolves some ambiguity in the overall design,
and allows to turn `BufferMetadata::lock(type, address)` into a private
implementation function.
In the old code, there were some external access paths directly entering
into that implementation function, which caused a lot of confusion (at least for me...)
This refacotring prepares for the planned changes in `OutputSlot`
* introduce a new compound parameter (tuple) BufferAlloc
* wire this through all calls involved with storage handling
__Background__:
* `BufferMetadata` deemed important for the implementation of buffer storage
* ''new requirement detected'': must be able to handle an actual storage block
that is ''larger'' than the requested buffer size (⟼ tiling)
* need compact and uniform info, also for the `BufferProxyAdaptor` callbacks
There was an old UML diagram created at the time of the first draft.
It is integrated into the »OutputSlot« page in TiddlyWiki. This description
and the diagram are highly relevant for understanding the design in this case.
Unfortunately, Umbrello was "improved" with a lot of features since I used it
for the last time, but the **quality** of the software has been **degraded**,
it is now buggy and the layout is imprecise and gets messed up easily.
__Warning__: the old diagrams look ''slighly off alignment'',
this is very likely due to some change in font handling.
I have set the default font now to 'DejaVu Sans Condensed' 9pt
The old implementation relied on two layered APIs:
one for the client and one for the implementer of
the Allocation state. This is logically sound, yet
clumsy in terms of design, since the internal API
turns out to be void of actual meaning.
It seems possible to eliminate this confusing structure
by a clear separation of ''implementation realms''
* the ''mechanics'' of connection state management
* the actual implementation of the connection
The first can be provided as generic template subclass,
while the latter should be confined entirely into the
implementation of the `Connection` interface.
Doing so requires to keep each part of the implementation
strictly focused onto its purpose — something I obviously
was not able to see at the time when writing this old
implementation, notably because I was completely guided
and driven by the test/dummy implementation without
considering how an actual implementation of
low-level data output would work.
The old implementation of `DiagnosticOutputSlot` is now disabled
and will be re-written from scratch, and no longer tangled to `BufferProvider`
- `OutputManager` remains in Steam-Layer for now
- the `Timings` and `OutputSlot` are now a self-contained
One of the goals of this extended clean-up and refactoring is to allow
access to the data buffers of a `DataSink` through a `BuffHandle` as front-end.
Which essentially means that the ''Buffer Provider Protocol'' and the
''Output Slot Protocol'' mesh together well.
After the clean-up of the `BufferProvider` implementation hierarchy,
the viability of this idea was demonstrated with the `BufferProxyAdapter` draft;
so it is now roughly clear what must be changed within the existing implementation
of `OutputSlot`. More specifically, the only actual implementation ever completed
was a dummy implementation for test (`DiagnosticOutputSlot`), which hooked
confusingly into the corresponding dummy implementation of `BufferProvider`,
that was written one year earlier (and was also without actual implementation
at that time).
So the second half of the task ahead seems very similar to the first part:
- the base implementation of the ''Output Slot Protocol'' seems roughly acceptable
- but the actual dummy implementation was somewhat confused and should be
re-written from scratch, without ''any attempt to cut corners by re-using other dummy code''
__Note__ as part of this analysis I discovered some gap in the design
regarding timeout handling ⟶ Ticket #1417
building upon all the preparations done during the last days,
the compound of code connected to the `BufferProvide` can now
be migrated down into the vault:
- `BuffHandle`
- `BufferProvider`
- `BufferMetadata`
- `BufferProviderSetup`
- the simple standard implementations
- `DiagnosticBufferProvider`
- all directly related unit tests
These are generic vehicles to verify computation structures.
Since they are in no way related to our models, they belong into »Vault« layer
Furthermore this reorganisation of test support components is prerequisite
to also move further generic facilities down into the Vault
This resolves a long-standing issue:
we could not use proper include paths for test-helper code that was defined alongside the tests.
Since some test-helper code must now be relocated into the »Vault«-Layer,
I took the opportunity to finally resolve this annoyance. The solution is two-fold
* introduce a dedicated test-support library
* remove the quirks that were necessary to support some plain-C tests
Together, this allows to raise the include-directory by one level,
and then to adjust all folder names so that they match the namespace
This is something I wanted to clean-up since a very long time indeed,
but always postponed it, since it requires a lot of nasty clean-up and disentangling....
Several parts had been stashed away into `src/lib/test`, which are not really related:
* the testrunner and testsuite builder
* generic helper utilities
* specialised test-support facilities, e.g. the EventLog
This changeset introduces a new top-level test folder and a namespace `::test`
Code from this folder is linked into a shared library `libtestsupport.so`,
which resides in the linkage hierarchy between »Vessel« and »Vault«,
yet is no longer part of the Lumiera application and only linked into the test-suite.
The next step with the `BufferProvider` and `OutputSlot` refactoring
will require to create new definition headers and to move those self-contained
services down into the »Vault« layer. In preparation, I decided deliberately to
abandon that old idea of a ''minimal bootstrap'' of arbitrary components in
favour of building the application as a coherent whole, with well-defined
boundaries and a backbone structure, marked as such explicitly by name
and arrangement of the code.
The changes were documented and applied to the integration branch first,
and now it's time to join that effort with the ongoing work on `BufferProvider`.
During the last months, a lot of gradual changes to policies and guidelines
had been finalised and documented. Moreover, recent clarifications regarding
memory management in the render engine will lead to a clearer definition
of the »Vault« Layer's role, with some ''engine code'' expected to move down.
On this occasion, I reworked relevant sections in the documentation to
confirm the role that now falls to central parts of the overall architecture.
Notably I decided to rename those parts related to the application backbone
as »Vessel« — to underpin the more active and defining role they assumed.
Explored further aspects of this new design;
as an aside, I tried to build a Concept to describe the policy,
which howerer turned out to be much more challenging than expected.
Lastly, I could now complete the prototype of `BufferProxyAdaptor`
While it is trivially simple to define some struct with callback functions,
such a definition turns out to be obnoxiously complex and hard to read in pracice,
thereby nullifying any benefit from the shorthand λ-based notation.
This kind of problem with the readability at client-side is quite common
when using callbacks for a policy-based design. To overcome this difficutly,
it would be necessary to define the policy and the callbacks in one
succinct builder expression. Unfortunately such a builder syntax
typically requires to write a lot of boilerplate template code.
After several failed experiments, I have found a bold new way to
exploit features of C++20 in conjunction with a macro, allowing
to generate a suitable data record directly form an C++ expression.
The idea is to use a "transient" Setup helper class, which lives
only during the invocation of the BufferProviderSetup-constructor.
This transient frame allows to accept a policy class from the client,
and move it through this transient Setup helper and finally into
the BufferStore backend implementation, where it is actually needed.
Now I have a working setup for a `BufferProxyAdaptor`.
Unfortunately this is still confusingly complex.
This is due to my intention to build a generic template,
that can be configured with a policy to invoke arbitrary callbacks,
whenever one of the passed-through buffers transitions through its lifecycle.
* we need to configure the `BufferProviderSetup` to use our specific implementation classes
* and the second one of these, our `BufferStore` implementation needs to invoke the callbacks
This closes a ''conceptual gap.''
There was a `lib::TestFrame::reseed()` function, which should be invoked
whenever a clean / deterministinc seed is required for computations involving
`TestFrame` data.
On occasion I noticed that most actual usages of `TestFrame`, which became quite common
lately, were missing this `reseed()` call, while the enclosing test
had indeed a proper Test::seedRand() call.
So my conclusion is: if even ''I myself'' miss that fine point frequently,
it is justified to automated this setup, even while `TestFrame` is right now
the only client of such a setup-facility.
There is a twist though.
`TestFrame` is part of the ''Engine'' test coverage, i.e. it is linked
into `libtest-core.so`. While the support for random numbers and reseeding
is obviously a very basic library feature, and linked into `liblumiera.so`
**Oh well**.
So now we have ''yet another'' static registration service.
I kept everything as simple as possible, which means, we continue
to use the ''common, global random generator'' from the library,
and there is no protection against some other code using those
library functions in a non-deterministic way.
`lib/random-reseed.cpp` is just a glorified hard-wired setup, beware!
This is an implementation experiment to validate the improvements
and internal clean-up. Previously, I was unable to implement such a
forwarding proxy buffer provider, as would be required for the
intended harmonisation of the `OutputSlot` interface with `BufferProvider`
With the reworked verison of the BufferProvider standard implementation
it seems straight-forward to...
* define a »Buffer Type« that marks any of the prepared buffers to pass-through
* use the resulting Buffer Descriptor together with a rigged storage backend,
that just returns the ''local tag'' as buffer adress
The first round of the [ticket:1410 BufferProvider clean-up] is roughly complete.
* a much cleaner diagnostic interface has been defined and implemented
* this allowed to modernise the `DiagnosticBufferProvider_test`
and the `BufferProviderProtocol_test`
* the implementation scheme has been split in two realms
* the old heap-memory based implementation has been re-arranged,
so that the aforementioned tests are **green** again.
Now is the point to draft a solution to get the interfaces of
`BufferProvider` and `OutputSlot` aligned, so that the latter
can expose its lifecycle through the former.
Today, I have analysed various aspects and developed a pathway
to get into that task, and next I will reshape the previously
failed `OutputProxyProvider_test` into a prototype to
validate this idea
* write down some first considerations regarding requirements
of the actual production-strength implementation of buffer management
* consider relevance of the part of the API that was retained
* write a new series of test cases to cover handling of embedded objects,
using the test support for tracking object creation
Some weeks ago, at the time where I started to dig into this topic,
I extended the BufferProviderProtocol_test to perform a buffer usage pattern
that mimics the usage in the real Render Engine, as implemented last year.
For this, I used the `TestFrame` and the chained-hash dummy computations.
Now, after reordering and clean-up of the `BufferProvider` implementation interfaces,
it can be showed with this computation scheme that
- all buffers are indeed allocated and handed out as expected
- the metadata handling is correct and points to the processed buffers
- the complete memory used in this test is uncorruped and holds correct numbers
It covers now roughly the same procedures as before,
yet the scheme how to do verification is quite different.
Furthermore, we allocate large random-sized buffers now
and implant `TestFrame` instances, and so also the verification
covers a significant storage area.
* can now replace old verification code by new diagnostic functions
* test case runs without failure
''UNRESOLVED:'' failure in verification of memory contents
The drafted setup for capturing tracking information was indeed way to elaborate.
Notably I wanted to capture the actual `BuffHandle` -- and that caused a host
of visibility and access problems, since that handle was meant to be used
as a front-end to the BufferTracker, and only that.
Now we just capture the hash-IDs and the storage size,
and we do not even keep relations or record a state-ENUM;
we just smash a copy of that data into a registry vector
for each state separately, which allows to do a lookup,
count the number of handled allocations and access the memory.
...this turns out to be surprisingly difficult,
due to various subclass / protected visibility problems.
Not sure if the component design within `DiagnosticBufferProvider`
is sane yet...
Use a generic helper function in the `BufferProviderSetup` base class
to allow layering of a decorator implementation on top of an already
instantiated implementation class.
After the preceding internal reorganisation, it suffices
for the `DiagnosticBufferProvider` to instrument the `BufferStage` interface,
in order to capture and record all relevant buffer state transitions
In the end I refrained from applying most of the changes I had intended.
I am aware that `BufferProvider` is still quite a massive structure,
yet with the implementation refactorings done thus far, it seems
to be reasonably clear and self-contained now.
It would be possible to totally overthrow the grown structure and
expose only the two handles (BuffDescr and BuffHandle), together with
a factory function, but I cannot see how this would help to make the
code clearer and more approachable. Maybe it would make the usage
more obvious from the client's perspective — yet, to be honest,
there are not ''that much clients'' to justify such a move.
Furthermore, the implementation is far from finished: for both of
the back-end lines, a fully production-strength implementation
must be created, that is able to operate in a highly concurrent
environment; very likely some lock-free messaging and hand-over
between threads will become necessary, adding to the complexity.
Thus it seems better to note the ''design critique'', but to
postpone any further refactorings to a point where we can watch
a fully developed Render Engine under production load.
doing so requires to trace a new call path from front to back,
into the `BufferMetadata` implementation: there is now an explicit check
that the buffer is in `LOCKED` state, but not yet `EMITTED` and als that
it was not yet `released()`
I think the additional effort is justified
* in the regular / currently intended usage in the Render Engine,
this predicate is not required, since compliance to the ''Buffer Provider Protocol''
as such is sufficient to ensure that
* but since all of this is getting more generic and polished now,
I could imagine future usages beyond the current confines; and then,
this semantics is what you'd expect from such a `bool()` check:
»I am allowed to use it«
These were defined in a confusing and implementation-oriented way.
Since we have now mostly abstracted away the metadata handling from the
`BufferProvider` front-end, a more succinct definition was chosen
* a ''Descriptor'' is valid when the lookup succeeds
* a ''Handle'' is likewise valid when the underlying
state-Key can be resolved, and the memory address is given
Some aspects regarding the state remain unresolved though.
Ideally we'd want a `BuffHandle` to be valid preciselc when
it ''is in valid state to be accessed''
...while the implementation provided herein is trivial,
it represents the entrance point into the process covered by the
''Buffer Provider Protocol''; thus I went again through all the
stages of the implementation, and confirmed my analyiss from 2024:
9a23aa773b
Today I'd even go so far as to call this a **crietrion of consistency**:
The path leading to the LOCK-transition must be reproducible.
The existing implementation, now re-cast into a new API,
ensures this by requiring an explicitly given Buffer Descriptor
at that point, and ''does not offer any one-step shortcut.''
__The point to note is__: the system used for implementing those type-IDs,
which is a tree of chained hashes, is deliberately open and extensible.
''In theory'' it would be possible thus to arrive at different hash-IDs
for the same criteria, if these are added gradually, in several steps,
as a refinement. This leeway is acceptable, as long as it can not happen
that two competing allocation processes want to lock the same
memory address, but tracked through different parent type-IDs.
The `BufferProvider` is both an interface and a facade, with
standard implementations to handle the state transitions and
further steps of the ''Buffer Provider Protocol''.
These implementations have gotten somewhat out of hand, since
they directly mixed memory allocation with very elaborate usage
of the `BufferMetadata`. The goal for this series of refactorings
is to introduce two ''implementation interfaces'' -- thereby
raising the abstraction level in this »code behind«
Now able to remove some obsoleted part of the old implementation.
The existing implementation of the ''Buffer Provider Protocol'' is
quite old and dated meanwhile; it was more like an explorative prototype
and mixed up implementation and test code.
At that time, notably the assumption was that the core render engine
will be implemented in plain-C and will push around `void*` all over the place.
The `BufferProvider` was meant as an adaptor, while questions of typing
were added as an afterthought. Especially some functionality to use
the `BuffHandle` in the way of a smart-ptr are completely obsoleted
by the reworked design of the Render Engine (2024).
Since the BufferStore implementation component is now supposed
to handle only the memory allocation, a complete end-to end data flow
must be ensured, so that allocation / deallocation functions get all
the relevant information. Notably, the `LocalTag` is now used as an
implementation-defined conduit for further (opaque) marker data.
Since all data passed over this API are simple values (even `LocalTag`
is effectively a 64bit integer), we can now avoid relying on output
parameters or similar side-effect tricks.
This change largely resolves the problems with the storage API,
so that the implementation can now be split in two disjoint segments.
One operation, `mark_emitted()`, is still misplaced on that API, yet
must be retained until the refactoring on the metadata side is complete
(this latter refactoring can not be started sensibly, before the old
»tracking« implementation is cleaned up to care for allocation solely).
With this change, the existing implementation, which combines the
tracking functionality and the allocation, has been squeezed down
into the heap-memory implementation class, yet the tracking functionality
is already detached. The remaining code down there is a huge pile of mess,
that can hopefully be dismantled soon. But at least all of this code
passes the compiler again, and so it should be possible now to focus
on getting the `DiagnosticBufferProvider` up and running, thereby
providing a new clean re-implementation of the diagnostic functions.
This is an important step to prepare a split of the existing implementation APIs
into a type-management part and a memory-allocation part; generally speaking,
a memory allocator needs to know the allocation size, preferably also for
releasing memory. Up to now, this information could be retrieved locally,
with a metadata lookup through the hash-ID; now we ensure that the
size is always passed alongside through the implementation API.
Incidentally, it looks like we might be able to form a common allocation-descriptor,
to represent this internal API in a more concise way. Note further that
all these changes are considered not relevant for the performance,
since work buffers are large and media computation dominates.
...just enough to be able to write the assertions in the first test case
(..and the test case fails still, due to the pending work on sorting out
the implementation APIs between `BufferStage` und `BufferStore`)
To push ahead with the refactoring,
I will build the test/diagnostics API completely anew,
and attach it solely to an instance of `DiagnosticBufferProvider`.
Unfortunately this ''leads us into chainsaw surgery...''
* it uncovers a host of conceptual contradictions
* actually, the `HeapMemProvider` was more an implementation
of diagnostic tracking, than it was an actual `BufferProvider`
implementation (hence the old name: `TrackingHeapBlockProivder`)
* and this lured me into using these tracking functions directly,
while the `DiagnosticBufferProvider` remained in unfinished state.
So basically I have now **disabled** all verification from the Unit-tests.
I'll have to rebuild them from scratch, thereby reproducing the logic/intention
More or less forcibly squeeze all the existing implementation
into the new basic structure with two separate »pillars«
**Breakages**
- existing code expected the `BufferStore` to transition to ''locked'' state
- existing code expected `provideLockedBuffer(typeID)` to tail-call the
`buildHandle()` function of the enclosing `BufferProvider` base implementation
- various diagnostic usages bypassed the official APIs and called directly
into the implementation. This is no longer possible; rather, `DiagnosticBufferProvider`
needs to provide additional diagnostic API to suit those purposes
As a consequence of preceding experiments..
* I'll stick to the structure of the BufferProvider front-end interface
* also the »code behind« will remain there; a purely abstract API has no merit here
* however, the ''setup'' of the actual implementation will move into a derived class
* this derived scope class, `BufferProviderSetup` will house two implementation classes
* these two, now named `Stage` and `Store` provide the base implementation
for the two "pillars" of the implementation
Admitted, this structure is now much more elaborate, but it's the only path forward
I'm currently able to come up in order to disentangle the existing implementation mess...
The outward interface has stood the test of time quite well.
But the back-side towards the implementation is a tangled mess,
presumably due to being developed from prototyping code: the
actual implementation is completely overgrown by diagnostics
and test instrumentation, since running in unit test setup
was the only actual usage of this code ever since.
My first attempt with decomposing into a decorator-chain failed.
Thus the new line of attack is to introduce two implementation
pillars right from top-level
- `BufferStage` : the state and metadata handling
- `BufferStore` : the storage backend
These are defined as nested interfaces, and the further refactoring
starts with subclassing everything, nested as it is.
The intention is to build the tracking functionality from scratch,
while wrapping the old implementation as delegate. Hopefully this allows
to reduce the complexity of the existing implementation step by step,
until I'm ''able to grasp what this implementation does''.
This old implementation works, but is highly problematic code.
The Git history shows that I had already problems to understand this code
while building the DiagnosticOutputSlot, one year later.
To my defense, in both cases (2011 and 2012), I had no clear idea yet
how the processing within the engine would work, and thus I considered
that Buffer Type registry a mere implementation detail, that can be
lumped into the »code behind«.
The prototype implementations of BuferProvider and OutputSlot are confusing;
in-depth analysis shows that this is related to intermingling use
of test instrumentation and features of a prototypical default.
To get anywhere with the existing code, it seems we'd need to split off
the test instrumentation (i.e. `DiagnosticBufferProvider`) altogether,
even if this causes duplication of implementation code. Many years ago,
when first implementing this prototype, I had no clear idea what the usage
in the Render Engine entails, yet I noticed that some basic buffer »tracking«
could be used as implementation primitive to simplify the prototyping work.
This might be one of the reasons, why the "code behind" in the protocol
implementations both in `BufferProvider` and `OutputSlot` became so strongly
tangled, so that I way unable to make both protocols orthogonally composable.
`EngineCtx` is considered merely an access front-end now;
After some analysis of existing code, it seems the RenderEnvironment
is the most appropriate framework to configure the global engine services
and thus to ''boot the Render Engine''.
The access to the `EngineCtx` is tricky, because of the requirements
* it will be dependency-injected into the internals of the Render Nodes
* excessive indirection should be avioded; preferrably the actual
code in the render pipeline sees a pointer (coded as language reference)
which directly refers to the buffer and cache manager.
* but on the other hand, we want configurability; simple tests shall
access "the buffer manager" without much ado, while the actual
render engine obviously has a life cycle and needs an administrative
access to the various global implementation facilities
* this both together implies some kind of ''virtual constructor''
* and since such a pattern can not be implemented in C++ with simple
inheritance (because the base constructor does not "see" the actual
implementation type at the time when it's invoked), the solution
is to rely on a second indirection through our dependency-injection
framework: thus the setup of the actual implementation is what
can be configured flexibly, and overridden by mock implementation
for some test cases
It seems evident that the the problems with the existing BufferProvider implementation
are rooted in a lack of understanding of the actual usage situation at that time(2011).
This caused a »hexagonal design«, i.e. something entirely driven by uint testing
for the sake of unit testing.
So to identify the unsuitable decisions, some degree of deductive work is necessary,
based on our much better understanding of the Render Engine.
A crucial challenge is that buffer management must perform well
within a massively concurrent environment, and do so without excessive latency.
This picks up the work for [ticket:1395 resolve integration of BufferProvider with DataSink (#1395)]:
Almost a year ago, before the long hiatus due to important general clean-up
and maintenance work, and the research regarding Video output —
a discrepancy was discovered between the »Buffer Provider Protocol«
and the »Output Slot Protocol«. This mismatch prevents us from building
a simple preliminary `BufferProvider` to be used for the engine integration
as part of the [ticket:1228 »Playback Vertical Slice«].
After re-establisihing the work context, I have read through the new
Render Node code built last year, and distilled the sequence of
buffer management operations that actually happen during a recursive
`pull()` call. With this kowledge, I am now able to fill in the
''standard case for rendering'' — which I was unable to formulate
during the first attempt of building the Render Engine in 2011.
__Next Steps__: Redesign of the standard implementation base of the
`BufferProvider` to prepare for a massively concurrent usage.
...some notes to my personal mindmap.
- Webserver OS upgrade
- Clean-up of services and cronjobs
- Apache configuration updated and fine-tuned
- deployed Anubis to protect against overload by AI-scraper bots
Hereby I introduce a ''Development Status'' page, where the
current phase of the release cycle can be marked and similar
status information is published. This page also includes
the Changelog (from the file NEWS). Since updates must
be checked into Git and then pushed to our website,
such status marks are documented in the history.
Some notes regarding the phases of the Release cycle
were also added to the Release checklist.
+ some additional links resources and documentation tweaks
- comb through the Website, starting at the frontpage
- add a **news** entry to confirm this major upgrade step (C++23)
- improve the wording in various overview pages
- adapt the ''release checklist'' to align it with **git-flow**
- reorganise the image folder(s) on the website
- the animated beating heart is back ;-)
On the Website, there is a set of interconnected pages related
to compiling from source, the Debian package and our DEB depot.
Although these pages have always been superficially kept up-to-date,
the overall presentation feels dated and generally not well organized.
During the last weeks, I have almost entirely rewritten all those pages,
added information about the (now reworked) DEB package, explained the
alternate ways to build from a Debian source package (notably the
`mk-build-deps` for handling the build dependencies, which effectively
the only method which always worked during the last years, since our
DEB package was totally outdated)
Furthermore, I have tested most of the build steps with the current
source trees, repositories and package definition, and finally I
have also updated and polished the front-page of our DEB-Depot
(-> see commit on the depot-Branch)
Wheew ... quite some work done!
Many versions enforced with this changeset are chosen such
as to support Ubuntu/Noble (24.04) and otherwise use versions
reasonably close to Debian-Trixie (≙reference-platform)
Since we now require a fairly modern compiler for C++23,
I have added now an explicit version check, which however
is performed only if the defined compiler is named `g++*`
Furthermore, I combed through all of our build tutorials and documentation pages
and updated a lot of information regarding dependencies and build practices...
Many years ago, I integrated the IconSvgRenderer (written by Joel Holdsworth)
directly into the SCons build by means of a custom builder with an ''emitter function''.
The build as such works fine and automatically determines which icons can be
generated from a given SVG source. However, those SVG icons tend to be rebuilt
very frequently, even while none of the SVG sources has changed.
Basically this was more of an annoyance, since we have now about 15 icons
and the rendering is really fast ... it is just ugly, especially on
incremental builds (and it could become a problem once we have a
massive amount of graphics to process.
So I took the opportunity to take close look now, since I am doing
uninspiring clean-up work since several weeks now. But that problem
turned out to be quite insidious and hard to spot.
First, AI set me off into a completely wrong angle, since it is ''not''
caused by a missing string representation of the custom Action class.
However, from there I went to watching the target checks with the debugger,
and this eventually got me to realise, that SCons mistakenly detects
a change in the Executor / the Action class.
The root cause is, that we invoke Python code from an external Module,
IconSvgRenderer.py, and this is »materialised« by SCons automatically
into a string representation, which includes memory addresses of functions
in that module. And that yields a signature, that is, quite obviously,
not stable, even while you end up sometimes with loading the code to
the same memory location.
As a solution / workaround, we now subclass the standard implementation
from SCons and override the signature function; instead of fingerprinting
the binary code, we just compute a MD5 over the python source file,
which can be easily achieved with the help of the File-Node from SCons.
Essential resources:
https://scons.org/doc/4.8.0/HTML/scons-user.html#chap-builders-writing
...and the Reference / Manpage of SCons
https://scons.org/doc/4.8.0/HTML/scons-man.html#action_objects
SCons/Action.py
SCons/Node/__init__.py
* Simplify some constructs from the Python-3 migration
* update copyright, since I did maintain the SCons build continuosly
* remove unused method from Buildhelper
Furthermore, the documentation page for our build system
https://lumiera.org/documentation/technical/build/SCons.html
has been reworked, to add a synopsis, some further background
information about the internal structure of SCons and about
the specific conventions and definitions used for Lumiera
Lumiera is built such that it can be invoked from a ''bundle directory''
without the need to install it into the system; this requires to perform
a relative lookup of shared modules loaded as dependencies.
Since Lumiera has a layered architecture, and each layer is built into a shared object,
we get transitive dependencies and thus transitive relative lookup.
There was a bug in binutils 2.23, which caused that transitive relative lookup to fail.
* as of 2025, Debian/Trixie is defined as ''Reference Platform''
* in Trixie the binutils version is v2.44-3
* after removing the workaround, Lumiera and all tests can be launched
* thus ''considering this workaround as obsolete now''
This is a Debian policy, which collides with the default behaviour
of GCC rsp. the plattform linker. The latter creates all ELF files as
executable, possibly because some platforms require that, and also
because some libraries provide a main() function for diagnostics.
The argument by the Debian people is that most libraries don't provide
such a main() and that this is an esoteric feature which should not be
supported by default.
We can fix that in our SCons build, since we use a specialised Builder
to also define the install targets systematically; it suffices to add
a post-action to these install targets for shared objects.
Note: currently Debian/Trixie provides SCons 4.8, but v4.10 will provide
a shorthand notation with the env.Chmod Action factory.
Some recherche regarding the requirements and conditions of the GPL
made it clear that we are obliged to have at least an summary statement
of all authors and the years of their contributions embedded somewhere
in the source itself. The Git history alone is not sufficient to fulfil
this requirement, since a code base could be packaged as Tarball
and passed-on outside of any Git repository.
Thus I propose the following policy
- add yourself to the copyright header whenever you made
some ''significant'' contribution to a code file
- maintain a list of relevant authors at top-level,
indicating a time range of their contribution
- use the same information also in the debian/copyright (DEB package)
- clarify in the LICENSE that authorship can be extracted from Git
__Remark__: Anton Yakovlev made a single code contribution in 2009 (87e528bd),
which was never used anywhere and removed as part of a general clean-up in 2023.
Thus I do not list him as a code author, but he is mentioned in credits.txt
- use a current copy of the GPL2 text, downloaded from fsf.org
- reformat the AUTHORS and get rid of the table formatting
- Stackoverflow has upgraded its Subscriber Content license to CC-By-SA 4.0
A handwritten changelog is considered important and
helps to single out the overall relevant achievements.
Following a proposal from the Debian policy, this summary
of development steps will now be maintained in the NEWS file.
It will be marked with the version number of Lumiera.
...and split-out the description of preview-releases,
since that rather belongs into some kind of changelog (NEWS).
Furthermore, check the links to the build dependencies
See corresponding commit in the website repository.
- Verified all authors via Git
- simplify the list and group by core/contributors
- list only significant contributors here
__Note__: the page "project/contributors" on the website
contains a much more extensive list with all helpers,
casual contributors and concept discussion participants
The XDG Desktop spec is accepted universally, and thus
the old Debian-specific 'menu' system is deprecated and
no longer engaged automatically.
See: https://lists.debian.org/debian-devel-announce/2015/09/msg00000.html
Thus we'll now provide a Desktop file and install that already from out
SCons build system, together with a suitable variation of the Lumiera icon.
TODO: not sure if everything was done the correct way
* do we need to ''register'' the new file in some way (preinst script?)
* the menu entry shows up, but not the icon
* but if we put an absolute path for the lumiera.svg into the desktop file, it shows up
Remark: in later experiments with package building,
the menu entry and the icon showed up in the menu.
Not sure if this requires a reboot or some similar
trigger (like restart of the destkop)
...funny enough I never noticed this obvious mistake,
since I never install software directly into my system,
while the DEB-build does not use absolute paths...
Solution:
* SCons has this speical convention that a path prefixed with '#'
is resolved relative to the root of the build (where the SConstruct resides)
* now we apply this automatically to the two relevant settings
** INSTALLDIR
** TARGDIR
* but only (conditionally) if the configured path is relative, not absolute
As a consequence, most other hard-coded usages of the '#'-prefix can then be dropped
Yes, I know...
Programmers absolutely LOVE to sneak-in various nifty toggles via environment.
Yet this is an anti-pattern!
And, by extension of that verdict, a "build interface" which relies
on the implicit convention that some magical variables are set,
is **not a proper interface** but a hack.
Thus we abandon that bad practice and handle the build in a clean and explicit way.
* the DEB-Build in `debian/rules` now invokes SCons explicitly, passing arguments
* the Lumiera Build-System is FSH aware and knows the proper installation locations
* the setup of the application uses a setup configuration, shipped with the package
* there is no need to ''compile-in any configuration''
** `LUMIERA_PLUGIN_PATH` is obsolete and unused since several years now
** `LUMIERA_CONFIG_PATH` was never used at all
** consequently, we also do not need `PKGLIBDIR` and `PKGDATADIR`
...this bug was spotted as the test failed reproducibly
when built with `-fstack-protector-strong` — which adds
additional "canary" markers to some kinds of problematic
stack storage; this seemed to have the effect that now
the second test uses exactly the same location as the
preceding test, and thus finds valid data in the newly
created work buffers.
There is no reason for using uninitialised storage in this
test (I can recall that I wanted to build a generic helper
and intended to use that as ''virtual overlay'' over existing
memory — but that usage never took place, and the `struct Buffer`
is not a general-purpose tool now, but only made for this
specific test. Thus memory can be easily zero-initialised.
Turns out that in practice there will be (at least temporarily)
some version-tags including a suffix: the RC-versions!
Now there is the special twist, that Git does not allow '~' in Tag names,
and thus `git-buildpackage` introduced an additional layer of translation.
So we are forced to revert that translation, which is possible,
since the basic Debian version syntax disallows '_' in version numbers
(because '_' is used to separate the package name from the version number).
It seems prudent to implement that as an preprocessing step
and thus keep it out of the regular version number syntax.
Furthermore we need the ability to handle existing suffixes,
which (as we know now) can be picked up from the Git history.
* my decision is to allow both pass-through and suppressing them
* use `--suffix=False` to suppress / remove any existing suffix
* the latter now allows us also to automate the setting of
the final Release version
Based on the additional insight gained through the FrOSCon talk,
an attempt was made to investigate and fix the garbled display of the
docking panel header directly above our VideoDisplayWidget.
However, that turned out to be a problem with insufficeint support
of the XVideo standard with my (very old, 2011) NVidia graphic card.
Indeed, the hardware+software stack needs to support this Auto-Keying
extension of XVideo, so that the video display can be integrated
seamlessly into the rest of the UI. In my case, the driver seemingly
just fills the complete XWindow with the key-marker and thus the
decorations of the docking panel, which happens to use the same XWindow,
will just not be flushed to the display.
This unsuccessful research indicates that upholding support for XVideo
could become problematic...
- reorganise the navigation tab structure
- place all further index lists into a common "Index" tab
- include a list of concepts
- remove the various "member" sub-tabs, these are not helpful
- the "modules" tab is now called "topics" (since C++ has now Modules!)
- generally re-sync DoxygenLayout.xml with a current pristine template
- comb though the Doxygen Warnings and fix a lot of small problems
The »Outer Space« and »Inner Core« documents present a comprehensive
overview of the vision, architecture and essential parts of the implementation.
In the light of changes to policy, design and implementation approaches,
some updates were necessary to align these crucial texts with the current
state of planning and implementation. Notably I have added a recent Screenshot
of the UI, showing a nested track structure pushed up by Diff from the core.
Over the last weeks, I conducted an extended analysis of the inception time
of the Lumiera project, which gave me a sharpened understanding of the goals,
going forward. I have also written an in-depth essay "Complexity and Flexibility".
See the corresponding commits in the Website repository.
Expand on this problem with the consequences of rule changes.
Need to consider how recording this information is related
to the data structures of the Event-Log...
__Remark__: this topic belongs into the ''next planned'' Vertical Slice »LoadClip«
Allowing to change rules and facts seems to be in direct contradiction
to using an Event-Log for deterministic and reproducible behaviour.
Notably, the facts can be a representation of the environment,
like e.g. the sound setup in a Studio vs. working on a Laptop.
Since a long time, I used the formula to separate decisions
from the processing logic in the code. An analysis of the kinds
of decisions relevant for the situation with editing a media arrangement
reveals that a dangerous conflict might ensue when the foundation
of a past rule application changes, possibly by undo or replay
of the Event-Log.
A solution could be not only to apply rules, but also capture
and materialise the fact that a rule has been applied and
especially to record a connection between the setting and
the rule it was based on. This amounts to documenting
the reasoning why some decision came about. If done correct,
such a data model would allow to re-evaluate past decisions
quickly in case the facts or the rules are changed.
While I am currently deeply entrenched in the history of Lumiera
and related general questions of dealing with complexity, some
ramifications became clear for the next topics to address,
notably related to ''Event Sourcing''
The RfC documents were written to complement discussions of the Lumiera developers;
yet since the time where ''Ichthyo'' is working basically alone on the project,
this kind of discussions have ceased. During the following years, some ideas
promoted by the existing RfC documents became rather detached from the
actual state of development in the code base.
Many of the existing RfC documents require some commentary to place them
into context, and some of the decisions taken in the early stage of the
project should be **re-assessed**. This includes the decision to reject
some proposals, which initially might have seemed desirable, yet could not
be reconciled with the understanding of the matter and topic in question,
as was gained through the ongoing analysis and development.
Some sections of the Lumiera website document meeting minutes,
discussion protocols and design proposals from the early days
of the project; these pages were initially authored in the
»Moin Moin Wiki« operated by Cehteh on pipapo.org at that time;
this wiki backed the first publications of the »Cinelerra-3«
initiative, which turned into the Lumiera project eventually.
Some years later, those pages were transliterated into Asciidoc
semi-automatically, resulting in a lot of broken markup and links.
This is a long standing maintenance problem problem plaguing the
Lumiera website, since those breakages cause a lot of warnings
and flood the logs of any linkchecker run.
- indicate clearly that a ''past discussion'' is documented here
- point out what is the primary focus for UI design ''currently''
- arrange some links better
- use cross-links / Linkfarm; especially cross-link to
the discussion regarding Wouter's Workflow proposals (2025)
Create a new subcategory "design/architecture/time"
and rearrange several pages related to time handling and time codes.
NOTE: starting with this changeset, a ''Link-Farm'' is required for cross-links;
since we don't have an automatic solution for this task yet, I have created
the necessary forwarding pages manually in the website repository.
Christian started a series of reworks in 2018;
the result was considered experimental and was parked on website-staging ever since.
As part of the current release- and clean-up activities I now reviewed
and integrated those changes into the current website as far as applicable.
The topic of this talk was ''Video output from a Linux desktop application''
after extended research we built a demo application in four flavours
(XVideo, SDL, OpenGL legacy / modern).
This research project was set off by our immediate needs for a setup
to show computed video pixel data in a viewer window in the GUI; the
investigated technologies should work at present and in the near term
future (yet leaving out Vulkan and Wayland)
