...this turns out to be surprisingly difficult,
due to various subclass / protected visibility problems.
Not sure if the component design within `DiagnosticBufferProvider`
is sane yet...
Use a generic helper function in the `BufferProviderSetup` base class
to allow layering of a decorator implementation on top of an already
instantiated implementation class.
After the preceding internal reorganisation, it suffices
for the `DiagnosticBufferProvider` to instrument the `BufferStage` interface,
in order to capture and record all relevant buffer state transitions
This is clearly implementation code and can now be removed from the interfaces;
`SimpleBufferStateRegistry` can thus delegated to a private metadata registry
In the end I refrained from applying most of the changes I had intended.
I am aware that `BufferProvider` is still quite a massive structure,
yet with the implementation refactorings done thus far, it seems
to be reasonably clear and self-contained now.
It would be possible to totally overthrow the grown structure and
expose only the two handles (BuffDescr and BuffHandle), together with
a factory function, but I cannot see how this would help to make the
code clearer and more approachable. Maybe it would make the usage
more obvious from the client's perspective — yet, to be honest,
there are not ''that much clients'' to justify such a move.
Furthermore, the implementation is far from finished: for both of
the back-end lines, a fully production-strength implementation
must be created, that is able to operate in a highly concurrent
environment; very likely some lock-free messaging and hand-over
between threads will become necessary, adding to the complexity.
Thus it seems better to note the ''design critique'', but to
postpone any further refactorings to a point where we can watch
a fully developed Render Engine under production load.
doing so requires to trace a new call path from front to back,
into the `BufferMetadata` implementation: there is now an explicit check
that the buffer is in `LOCKED` state, but not yet `EMITTED` and als that
it was not yet `released()`
I think the additional effort is justified
* in the regular / currently intended usage in the Render Engine,
this predicate is not required, since compliance to the ''Buffer Provider Protocol''
as such is sufficient to ensure that
* but since all of this is getting more generic and polished now,
I could imagine future usages beyond the current confines; and then,
this semantics is what you'd expect from such a `bool()` check:
»I am allowed to use it«
These were defined in a confusing and implementation-oriented way.
Since we have now mostly abstracted away the metadata handling from the
`BufferProvider` front-end, a more succinct definition was chosen
* a ''Descriptor'' is valid when the lookup succeeds
* a ''Handle'' is likewise valid when the underlying
state-Key can be resolved, and the memory address is given
Some aspects regarding the state remain unresolved though.
Ideally we'd want a `BuffHandle` to be valid preciselc when
it ''is in valid state to be accessed''
In the past we allowed a lot of leeway when referring to our
web-interface to the Lumiera Git repositories (git.lumiera.org/gitweb)
Unfortunately this caused the AI scraper bots »out there« to capture
millions of URLs for all the different flavours of spelling them out.
And now they haunt our website and try to scrap that content again and again;
even while we effectively prevent that kind of abuse, rejecting those
ongoing attempts at content-scraping causes a dangerously high amount of traffic.
With this changeset, all cross references in the documentation section
are rewritten to the only remaining valid form of that Gitweb URL.
...while the implementation provided herein is trivial,
it represents the entrance point into the process covered by the
''Buffer Provider Protocol''; thus I went again through all the
stages of the implementation, and confirmed my analyiss from 2024:
9a23aa773b
Today I'd even go so far as to call this a **crietrion of consistency**:
The path leading to the LOCK-transition must be reproducible.
The existing implementation, now re-cast into a new API,
ensures this by requiring an explicitly given Buffer Descriptor
at that point, and ''does not offer any one-step shortcut.''
__The point to note is__: the system used for implementing those type-IDs,
which is a tree of chained hashes, is deliberately open and extensible.
''In theory'' it would be possible thus to arrive at different hash-IDs
for the same criteria, if these are added gradually, in several steps,
as a refinement. This leeway is acceptable, as long as it can not happen
that two competing allocation processes want to lock the same
memory address, but tracked through different parent type-IDs.
The `BufferProvider` is both an interface and a facade, with
standard implementations to handle the state transitions and
further steps of the ''Buffer Provider Protocol''.
These implementations have gotten somewhat out of hand, since
they directly mixed memory allocation with very elaborate usage
of the `BufferMetadata`. The goal for this series of refactorings
is to introduce two ''implementation interfaces'' -- thereby
raising the abstraction level in this »code behind«
This whole exercise is an attempt to make the whole implementation
clearer and more approachable — especially regarding the future extension
to a concurrency-hardened variant. The implementation is still backed
by a `BufferMetadata` table, and the rearranged code can be inlined.
In the old version of `BufferProvider`, there was a single
backend implementation, which was assumed to call-back into
`buildHandle(typeID, storage, localTag)` after allocating memory.
This implies, the state transition to `LOCKED` happened there.
My next goal is to clarify and abstract also the state transition API,
and this refactoring helps a lot to see the underlying call structure.
Now able to remove some obsoleted part of the old implementation.
The existing implementation of the ''Buffer Provider Protocol'' is
quite old and dated meanwhile; it was more like an explorative prototype
and mixed up implementation and test code.
At that time, notably the assumption was that the core render engine
will be implemented in plain-C and will push around `void*` all over the place.
The `BufferProvider` was meant as an adaptor, while questions of typing
were added as an afterthought. Especially some functionality to use
the `BuffHandle` in the way of a smart-ptr are completely obsoleted
by the reworked design of the Render Engine (2024).
Since the BufferStore implementation component is now supposed
to handle only the memory allocation, a complete end-to end data flow
must be ensured, so that allocation / deallocation functions get all
the relevant information. Notably, the `LocalTag` is now used as an
implementation-defined conduit for further (opaque) marker data.
Since all data passed over this API are simple values (even `LocalTag`
is effectively a 64bit integer), we can now avoid relying on output
parameters or similar side-effect tricks.
This change largely resolves the problems with the storage API,
so that the implementation can now be split in two disjoint segments.
One operation, `mark_emitted()`, is still misplaced on that API, yet
must be retained until the refactoring on the metadata side is complete
(this latter refactoring can not be started sensibly, before the old
»tracking« implementation is cleaned up to care for allocation solely).
With this change, the existing implementation, which combines the
tracking functionality and the allocation, has been squeezed down
into the heap-memory implementation class, yet the tracking functionality
is already detached. The remaining code down there is a huge pile of mess,
that can hopefully be dismantled soon. But at least all of this code
passes the compiler again, and so it should be possible now to focus
on getting the `DiagnosticBufferProvider` up and running, thereby
providing a new clean re-implementation of the diagnostic functions.
This is an important step to prepare a split of the existing implementation APIs
into a type-management part and a memory-allocation part; generally speaking,
a memory allocator needs to know the allocation size, preferably also for
releasing memory. Up to now, this information could be retrieved locally,
with a metadata lookup through the hash-ID; now we ensure that the
size is always passed alongside through the implementation API.
Incidentally, it looks like we might be able to form a common allocation-descriptor,
to represent this internal API in a more concise way. Note further that
all these changes are considered not relevant for the performance,
since work buffers are large and media computation dominates.
...just enough to be able to write the assertions in the first test case
(..and the test case fails still, due to the pending work on sorting out
the implementation APIs between `BufferStage` und `BufferStore`)
To push ahead with the refactoring,
I will build the test/diagnostics API completely anew,
and attach it solely to an instance of `DiagnosticBufferProvider`.
Unfortunately this ''leads us into chainsaw surgery...''
* it uncovers a host of conceptual contradictions
* actually, the `HeapMemProvider` was more an implementation
of diagnostic tracking, than it was an actual `BufferProvider`
implementation (hence the old name: `TrackingHeapBlockProivder`)
* and this lured me into using these tracking functions directly,
while the `DiagnosticBufferProvider` remained in unfinished state.
So basically I have now **disabled** all verification from the Unit-tests.
I'll have to rebuild them from scratch, thereby reproducing the logic/intention
- write some documentation
- identify some add-on API that must be retracted
- consider further how to disentangle ''diagnostics'' and ''implementation''
More or less forcibly squeeze all the existing implementation
into the new basic structure with two separate »pillars«
**Breakages**
- existing code expected the `BufferStore` to transition to ''locked'' state
- existing code expected `provideLockedBuffer(typeID)` to tail-call the
`buildHandle()` function of the enclosing `BufferProvider` base implementation
- various diagnostic usages bypassed the official APIs and called directly
into the implementation. This is no longer possible; rather, `DiagnosticBufferProvider`
needs to provide additional diagnostic API to suit those purposes
As a consequence of preceding experiments..
* I'll stick to the structure of the BufferProvider front-end interface
* also the »code behind« will remain there; a purely abstract API has no merit here
* however, the ''setup'' of the actual implementation will move into a derived class
* this derived scope class, `BufferProviderSetup` will house two implementation classes
* these two, now named `Stage` and `Store` provide the base implementation
for the two "pillars" of the implementation
Admitted, this structure is now much more elaborate, but it's the only path forward
I'm currently able to come up in order to disentangle the existing implementation mess...
The outward interface has stood the test of time quite well.
But the back-side towards the implementation is a tangled mess,
presumably due to being developed from prototyping code: the
actual implementation is completely overgrown by diagnostics
and test instrumentation, since running in unit test setup
was the only actual usage of this code ever since.
My first attempt with decomposing into a decorator-chain failed.
Thus the new line of attack is to introduce two implementation
pillars right from top-level
- `BufferStage` : the state and metadata handling
- `BufferStore` : the storage backend
These are defined as nested interfaces, and the further refactoring
starts with subclassing everything, nested as it is.
The intention is to build the tracking functionality from scratch,
while wrapping the old implementation as delegate. Hopefully this allows
to reduce the complexity of the existing implementation step by step,
until I'm ''able to grasp what this implementation does''.
This old implementation works, but is highly problematic code.
The Git history shows that I had already problems to understand this code
while building the DiagnosticOutputSlot, one year later.
To my defense, in both cases (2011 and 2012), I had no clear idea yet
how the processing within the engine would work, and thus I considered
that Buffer Type registry a mere implementation detail, that can be
lumped into the »code behind«.
The prototype implementations of BuferProvider and OutputSlot are confusing;
in-depth analysis shows that this is related to intermingling use
of test instrumentation and features of a prototypical default.
To get anywhere with the existing code, it seems we'd need to split off
the test instrumentation (i.e. `DiagnosticBufferProvider`) altogether,
even if this causes duplication of implementation code. Many years ago,
when first implementing this prototype, I had no clear idea what the usage
in the Render Engine entails, yet I noticed that some basic buffer »tracking«
could be used as implementation primitive to simplify the prototyping work.
This might be one of the reasons, why the "code behind" in the protocol
implementations both in `BufferProvider` and `OutputSlot` became so strongly
tangled, so that I way unable to make both protocols orthogonally composable.
The implementation code for `DiagnosticOutputProvider` and `TrackingHeapBlockProvider`
look very confusing and tangled. It seems, instrumentation for tests
got mixed up with the effort to provide a simplistic prototype implementation.
What we need foremost here is the latter; most tests that import the
DiagnosticOutputProvider do not even use any instrumentation....
`EngineCtx` is considered merely an access front-end now;
After some analysis of existing code, it seems the RenderEnvironment
is the most appropriate framework to configure the global engine services
and thus to ''boot the Render Engine''.
The access to the `EngineCtx` is tricky, because of the requirements
* it will be dependency-injected into the internals of the Render Nodes
* excessive indirection should be avioded; preferrably the actual
code in the render pipeline sees a pointer (coded as language reference)
which directly refers to the buffer and cache manager.
* but on the other hand, we want configurability; simple tests shall
access "the buffer manager" without much ado, while the actual
render engine obviously has a life cycle and needs an administrative
access to the various global implementation facilities
* this both together implies some kind of ''virtual constructor''
* and since such a pattern can not be implemented in C++ with simple
inheritance (because the base constructor does not "see" the actual
implementation type at the time when it's invoked), the solution
is to rely on a second indirection through our dependency-injection
framework: thus the setup of the actual implementation is what
can be configured flexibly, and overridden by mock implementation
for some test cases
While this topic is only tangentially related to the problems
with the »Buffer Provider Protocol«, the configuration of service instances
is relevant, insofar it provides a configuration point for unit testing.
The structure of the `EngineCtx` was created as a placeholder, to get those
issues out of the way while drafting the render node code; in its current
form it is clearly both to complicated and inflexible.
My intention is to merge the `EngineCtx::Facilities`, and to prefer
configuration of more elaborate setup by subclasses.
It seems evident that the the problems with the existing BufferProvider implementation
are rooted in a lack of understanding of the actual usage situation at that time(2011).
This caused a »hexagonal design«, i.e. something entirely driven by uint testing
for the sake of unit testing.
So to identify the unsuitable decisions, some degree of deductive work is necessary,
based on our much better understanding of the Render Engine.
A crucial challenge is that buffer management must perform well
within a massively concurrent environment, and do so without excessive latency.
This picks up the work for [ticket:1395 resolve integration of BufferProvider with DataSink (#1395)]:
Almost a year ago, before the long hiatus due to important general clean-up
and maintenance work, and the research regarding Video output —
a discrepancy was discovered between the »Buffer Provider Protocol«
and the »Output Slot Protocol«. This mismatch prevents us from building
a simple preliminary `BufferProvider` to be used for the engine integration
as part of the [ticket:1228 »Playback Vertical Slice«].
After re-establisihing the work context, I have read through the new
Render Node code built last year, and distilled the sequence of
buffer management operations that actually happen during a recursive
`pull()` call. With this kowledge, I am now able to fill in the
''standard case for rendering'' — which I was unable to formulate
during the first attempt of building the Render Engine in 2011.
__Next Steps__: Redesign of the standard implementation base of the
`BufferProvider` to prepare for a massively concurrent usage.
...some notes to my personal mindmap.
- Webserver OS upgrade
- Clean-up of services and cronjobs
- Apache configuration updated and fine-tuned
- deployed Anubis to protect against overload by AI-scraper bots
The paragraph that describes the pull processing seems slightly confusing,
notably the follow-up processing is done by child nodes, not siblings.
I attempted to use a more descriptive language
This was a classical ''collaborative f***up''.
- in 2018, `cehteh` wrote a infrastructure config summary
- late in 2025, `ichthyo` merged various documentation content
and placed that piece of information into the technical docu.
Unfortunately, the Baidu search engine has picked that info up,
and in the last week of December 2025, some stray bots started
to »visit« the staging area.
Hereby I introduce a ''Development Status'' page, where the
current phase of the release cycle can be marked and similar
status information is published. This page also includes
the Changelog (from the file NEWS). Since updates must
be checked into Git and then pushed to our website,
such status marks are documented in the history.
Some notes regarding the phases of the Release cycle
were also added to the Release checklist.
+ some additional links resources and documentation tweaks
This is a **preview release** — Lumiera is not usable yet.
Publication was again motivated by a series of major upgrades and clean-up.
Compilation now requires C++23 and Debian/Trixie is considered the reference
platform. The GUI is connected to the core over an asynchronous messaging
bus and the timeline display has been reshaped to accommodate flexibly
nested structures of media content, instructed by the session through
a series of _diff messages_. A scheduler has been implemented to drive
the Render Engine, but the connection to the player and UI for output
is yet unfinished.
- comb through the Website, starting at the frontpage
- add a **news** entry to confirm this major upgrade step (C++23)
- improve the wording in various overview pages
- adapt the ''release checklist'' to align it with **git-flow**
- reorganise the image folder(s) on the website
- the animated beating heart is back ;-)
On the Website, there is a set of interconnected pages related
to compiling from source, the Debian package and our DEB depot.
Although these pages have always been superficially kept up-to-date,
the overall presentation feels dated and generally not well organized.
During the last weeks, I have almost entirely rewritten all those pages,
added information about the (now reworked) DEB package, explained the
alternate ways to build from a Debian source package (notably the
`mk-build-deps` for handling the build dependencies, which effectively
the only method which always worked during the last years, since our
DEB package was totally outdated)
Furthermore, I have tested most of the build steps with the current
source trees, repositories and package definition, and finally I
have also updated and polished the front-page of our DEB-Depot
(-> see commit on the depot-Branch)
Wheew ... quite some work done!
Many versions enforced with this changeset are chosen such
as to support Ubuntu/Noble (24.04) and otherwise use versions
reasonably close to Debian-Trixie (≙reference-platform)
Since we now require a fairly modern compiler for C++23,
I have added now an explicit version check, which however
is performed only if the defined compiler is named `g++*`
Furthermore, I combed through all of our build tutorials and documentation pages
and updated a lot of information regarding dependencies and build practices...
Many years ago, I integrated the IconSvgRenderer (written by Joel Holdsworth)
directly into the SCons build by means of a custom builder with an ''emitter function''.
The build as such works fine and automatically determines which icons can be
generated from a given SVG source. However, those SVG icons tend to be rebuilt
very frequently, even while none of the SVG sources has changed.
Basically this was more of an annoyance, since we have now about 15 icons
and the rendering is really fast ... it is just ugly, especially on
incremental builds (and it could become a problem once we have a
massive amount of graphics to process.
So I took the opportunity to take close look now, since I am doing
uninspiring clean-up work since several weeks now. But that problem
turned out to be quite insidious and hard to spot.
First, AI set me off into a completely wrong angle, since it is ''not''
caused by a missing string representation of the custom Action class.
However, from there I went to watching the target checks with the debugger,
and this eventually got me to realise, that SCons mistakenly detects
a change in the Executor / the Action class.
The root cause is, that we invoke Python code from an external Module,
IconSvgRenderer.py, and this is »materialised« by SCons automatically
into a string representation, which includes memory addresses of functions
in that module. And that yields a signature, that is, quite obviously,
not stable, even while you end up sometimes with loading the code to
the same memory location.
As a solution / workaround, we now subclass the standard implementation
from SCons and override the signature function; instead of fingerprinting
the binary code, we just compute a MD5 over the python source file,
which can be easily achieved with the help of the File-Node from SCons.
Essential resources:
https://scons.org/doc/4.8.0/HTML/scons-user.html#chap-builders-writing
...and the Reference / Manpage of SCons
https://scons.org/doc/4.8.0/HTML/scons-man.html#action_objects
SCons/Action.py
SCons/Node/__init__.py
* Simplify some constructs from the Python-3 migration
* update copyright, since I did maintain the SCons build continuosly
* remove unused method from Buildhelper
Furthermore, the documentation page for our build system
https://lumiera.org/documentation/technical/build/SCons.html
has been reworked, to add a synopsis, some further background
information about the internal structure of SCons and about
the specific conventions and definitions used for Lumiera
Lumiera is built such that it can be invoked from a ''bundle directory''
without the need to install it into the system; this requires to perform
a relative lookup of shared modules loaded as dependencies.
Since Lumiera has a layered architecture, and each layer is built into a shared object,
we get transitive dependencies and thus transitive relative lookup.
There was a bug in binutils 2.23, which caused that transitive relative lookup to fail.
* as of 2025, Debian/Trixie is defined as ''Reference Platform''
* in Trixie the binutils version is v2.44-3
* after removing the workaround, Lumiera and all tests can be launched
* thus ''considering this workaround as obsolete now''
This is a Debian policy, which collides with the default behaviour
of GCC rsp. the plattform linker. The latter creates all ELF files as
executable, possibly because some platforms require that, and also
because some libraries provide a main() function for diagnostics.
The argument by the Debian people is that most libraries don't provide
such a main() and that this is an esoteric feature which should not be
supported by default.
We can fix that in our SCons build, since we use a specialised Builder
to also define the install targets systematically; it suffices to add
a post-action to these install targets for shared objects.
Note: currently Debian/Trixie provides SCons 4.8, but v4.10 will provide
a shorthand notation with the env.Chmod Action factory.
Some recherche regarding the requirements and conditions of the GPL
made it clear that we are obliged to have at least an summary statement
of all authors and the years of their contributions embedded somewhere
in the source itself. The Git history alone is not sufficient to fulfil
this requirement, since a code base could be packaged as Tarball
and passed-on outside of any Git repository.
Thus I propose the following policy
- add yourself to the copyright header whenever you made
some ''significant'' contribution to a code file
- maintain a list of relevant authors at top-level,
indicating a time range of their contribution
- use the same information also in the debian/copyright (DEB package)
- clarify in the LICENSE that authorship can be extracted from Git
__Remark__: Anton Yakovlev made a single code contribution in 2009 (87e528bd),
which was never used anywhere and removed as part of a general clean-up in 2023.
Thus I do not list him as a code author, but he is mentioned in credits.txt
- use a current copy of the GPL2 text, downloaded from fsf.org
- reformat the AUTHORS and get rid of the table formatting
- Stackoverflow has upgraded its Subscriber Content license to CC-By-SA 4.0
A handwritten changelog is considered important and
helps to single out the overall relevant achievements.
Following a proposal from the Debian policy, this summary
of development steps will now be maintained in the NEWS file.
It will be marked with the version number of Lumiera.
...and split-out the description of preview-releases,
since that rather belongs into some kind of changelog (NEWS).
Furthermore, check the links to the build dependencies
