The next step with the `BufferProvider` and `OutputSlot` refactoring
will require to create new definition headers and to move those self-contained
services down into the »Vault« layer. In preparation, I decided deliberately to
abandon that old idea of a ''minimal bootstrap'' of arbitrary components in
favour of building the application as a coherent whole, with well-defined
boundaries and a backbone structure, marked as such explicitly by name
and arrangement of the code.
The changes were documented and applied to the integration branch first,
and now it's time to join that effort with the ongoing work on `BufferProvider`.
During the last months, a lot of gradual changes to policies and guidelines
had been finalised and documented. Moreover, recent clarifications regarding
memory management in the render engine will lead to a clearer definition
of the »Vault« Layer's role, with some ''engine code'' expected to move down.
On this occasion, I reworked relevant sections in the documentation to
confirm the role that now falls to central parts of the overall architecture.
Notably I decided to rename those parts related to the application backbone
as »Vessel« — to underpin the more active and defining role they assumed.
Explored further aspects of this new design;
as an aside, I tried to build a Concept to describe the policy,
which howerer turned out to be much more challenging than expected.
Lastly, I could now complete the prototype of `BufferProxyAdaptor`
While it is trivially simple to define some struct with callback functions,
such a definition turns out to be obnoxiously complex and hard to read in pracice,
thereby nullifying any benefit from the shorthand λ-based notation.
This kind of problem with the readability at client-side is quite common
when using callbacks for a policy-based design. To overcome this difficutly,
it would be necessary to define the policy and the callbacks in one
succinct builder expression. Unfortunately such a builder syntax
typically requires to write a lot of boilerplate template code.
After several failed experiments, I have found a bold new way to
exploit features of C++20 in conjunction with a macro, allowing
to generate a suitable data record directly form an C++ expression.
Now I have a working setup for a `BufferProxyAdaptor`.
Unfortunately this is still confusingly complex.
This is due to my intention to build a generic template,
that can be configured with a policy to invoke arbitrary callbacks,
whenever one of the passed-through buffers transitions through its lifecycle.
* we need to configure the `BufferProviderSetup` to use our specific implementation classes
* and the second one of these, our `BufferStore` implementation needs to invoke the callbacks
This closes a ''conceptual gap.''
There was a `lib::TestFrame::reseed()` function, which should be invoked
whenever a clean / deterministinc seed is required for computations involving
`TestFrame` data.
On occasion I noticed that most actual usages of `TestFrame`, which became quite common
lately, were missing this `reseed()` call, while the enclosing test
had indeed a proper Test::seedRand() call.
So my conclusion is: if even ''I myself'' miss that fine point frequently,
it is justified to automated this setup, even while `TestFrame` is right now
the only client of such a setup-facility.
There is a twist though.
`TestFrame` is part of the ''Engine'' test coverage, i.e. it is linked
into `libtest-core.so`. While the support for random numbers and reseeding
is obviously a very basic library feature, and linked into `liblumiera.so`
**Oh well**.
So now we have ''yet another'' static registration service.
I kept everything as simple as possible, which means, we continue
to use the ''common, global random generator'' from the library,
and there is no protection against some other code using those
library functions in a non-deterministic way.
`lib/random-reseed.cpp` is just a glorified hard-wired setup, beware!
This is an implementation experiment to validate the improvements
and internal clean-up. Previously, I was unable to implement such a
forwarding proxy buffer provider, as would be required for the
intended harmonisation of the `OutputSlot` interface with `BufferProvider`
With the reworked verison of the BufferProvider standard implementation
it seems straight-forward to...
* define a »Buffer Type« that marks any of the prepared buffers to pass-through
* use the resulting Buffer Descriptor together with a rigged storage backend,
that just returns the ''local tag'' as buffer adress
The first round of the [ticket:1410 BufferProvider clean-up] is roughly complete.
* a much cleaner diagnostic interface has been defined and implemented
* this allowed to modernise the `DiagnosticBufferProvider_test`
and the `BufferProviderProtocol_test`
* the implementation scheme has been split in two realms
* the old heap-memory based implementation has been re-arranged,
so that the aforementioned tests are **green** again.
Now is the point to draft a solution to get the interfaces of
`BufferProvider` and `OutputSlot` aligned, so that the latter
can expose its lifecycle through the former.
Today, I have analysed various aspects and developed a pathway
to get into that task, and next I will reshape the previously
failed `OutputProxyProvider_test` into a prototype to
validate this idea
* write down some first considerations regarding requirements
of the actual production-strength implementation of buffer management
* consider relevance of the part of the API that was retained
* write a new series of test cases to cover handling of embedded objects,
using the test support for tracking object creation
Some weeks ago, at the time where I started to dig into this topic,
I extended the BufferProviderProtocol_test to perform a buffer usage pattern
that mimics the usage in the real Render Engine, as implemented last year.
For this, I used the `TestFrame` and the chained-hash dummy computations.
Now, after reordering and clean-up of the `BufferProvider` implementation interfaces,
it can be showed with this computation scheme that
- all buffers are indeed allocated and handed out as expected
- the metadata handling is correct and points to the processed buffers
- the complete memory used in this test is uncorruped and holds correct numbers
It covers now roughly the same procedures as before,
yet the scheme how to do verification is quite different.
Furthermore, we allocate large random-sized buffers now
and implant `TestFrame` instances, and so also the verification
covers a significant storage area.
* can now replace old verification code by new diagnostic functions
* test case runs without failure
''UNRESOLVED:'' failure in verification of memory contents
In the end I refrained from applying most of the changes I had intended.
I am aware that `BufferProvider` is still quite a massive structure,
yet with the implementation refactorings done thus far, it seems
to be reasonably clear and self-contained now.
It would be possible to totally overthrow the grown structure and
expose only the two handles (BuffDescr and BuffHandle), together with
a factory function, but I cannot see how this would help to make the
code clearer and more approachable. Maybe it would make the usage
more obvious from the client's perspective — yet, to be honest,
there are not ''that much clients'' to justify such a move.
Furthermore, the implementation is far from finished: for both of
the back-end lines, a fully production-strength implementation
must be created, that is able to operate in a highly concurrent
environment; very likely some lock-free messaging and hand-over
between threads will become necessary, adding to the complexity.
Thus it seems better to note the ''design critique'', but to
postpone any further refactorings to a point where we can watch
a fully developed Render Engine under production load.
These were defined in a confusing and implementation-oriented way.
Since we have now mostly abstracted away the metadata handling from the
`BufferProvider` front-end, a more succinct definition was chosen
* a ''Descriptor'' is valid when the lookup succeeds
* a ''Handle'' is likewise valid when the underlying
state-Key can be resolved, and the memory address is given
Some aspects regarding the state remain unresolved though.
Ideally we'd want a `BuffHandle` to be valid preciselc when
it ''is in valid state to be accessed''
Now able to remove some obsoleted part of the old implementation.
The existing implementation of the ''Buffer Provider Protocol'' is
quite old and dated meanwhile; it was more like an explorative prototype
and mixed up implementation and test code.
At that time, notably the assumption was that the core render engine
will be implemented in plain-C and will push around `void*` all over the place.
The `BufferProvider` was meant as an adaptor, while questions of typing
were added as an afterthought. Especially some functionality to use
the `BuffHandle` in the way of a smart-ptr are completely obsoleted
by the reworked design of the Render Engine (2024).
Since the BufferStore implementation component is now supposed
to handle only the memory allocation, a complete end-to end data flow
must be ensured, so that allocation / deallocation functions get all
the relevant information. Notably, the `LocalTag` is now used as an
implementation-defined conduit for further (opaque) marker data.
Since all data passed over this API are simple values (even `LocalTag`
is effectively a 64bit integer), we can now avoid relying on output
parameters or similar side-effect tricks.
...just enough to be able to write the assertions in the first test case
(..and the test case fails still, due to the pending work on sorting out
the implementation APIs between `BufferStage` und `BufferStore`)
To push ahead with the refactoring,
I will build the test/diagnostics API completely anew,
and attach it solely to an instance of `DiagnosticBufferProvider`.
Unfortunately this ''leads us into chainsaw surgery...''
* it uncovers a host of conceptual contradictions
* actually, the `HeapMemProvider` was more an implementation
of diagnostic tracking, than it was an actual `BufferProvider`
implementation (hence the old name: `TrackingHeapBlockProivder`)
* and this lured me into using these tracking functions directly,
while the `DiagnosticBufferProvider` remained in unfinished state.
So basically I have now **disabled** all verification from the Unit-tests.
I'll have to rebuild them from scratch, thereby reproducing the logic/intention
- write some documentation
- identify some add-on API that must be retracted
- consider further how to disentangle ''diagnostics'' and ''implementation''
More or less forcibly squeeze all the existing implementation
into the new basic structure with two separate »pillars«
**Breakages**
- existing code expected the `BufferStore` to transition to ''locked'' state
- existing code expected `provideLockedBuffer(typeID)` to tail-call the
`buildHandle()` function of the enclosing `BufferProvider` base implementation
- various diagnostic usages bypassed the official APIs and called directly
into the implementation. This is no longer possible; rather, `DiagnosticBufferProvider`
needs to provide additional diagnostic API to suit those purposes
The outward interface has stood the test of time quite well.
But the back-side towards the implementation is a tangled mess,
presumably due to being developed from prototyping code: the
actual implementation is completely overgrown by diagnostics
and test instrumentation, since running in unit test setup
was the only actual usage of this code ever since.
My first attempt with decomposing into a decorator-chain failed.
Thus the new line of attack is to introduce two implementation
pillars right from top-level
- `BufferStage` : the state and metadata handling
- `BufferStore` : the storage backend
These are defined as nested interfaces, and the further refactoring
starts with subclassing everything, nested as it is.
The intention is to build the tracking functionality from scratch,
while wrapping the old implementation as delegate. Hopefully this allows
to reduce the complexity of the existing implementation step by step,
until I'm ''able to grasp what this implementation does''.
This old implementation works, but is highly problematic code.
The Git history shows that I had already problems to understand this code
while building the DiagnosticOutputSlot, one year later.
To my defense, in both cases (2011 and 2012), I had no clear idea yet
how the processing within the engine would work, and thus I considered
that Buffer Type registry a mere implementation detail, that can be
lumped into the »code behind«.
The prototype implementations of BuferProvider and OutputSlot are confusing;
in-depth analysis shows that this is related to intermingling use
of test instrumentation and features of a prototypical default.
To get anywhere with the existing code, it seems we'd need to split off
the test instrumentation (i.e. `DiagnosticBufferProvider`) altogether,
even if this causes duplication of implementation code. Many years ago,
when first implementing this prototype, I had no clear idea what the usage
in the Render Engine entails, yet I noticed that some basic buffer »tracking«
could be used as implementation primitive to simplify the prototyping work.
This might be one of the reasons, why the "code behind" in the protocol
implementations both in `BufferProvider` and `OutputSlot` became so strongly
tangled, so that I way unable to make both protocols orthogonally composable.
The implementation code for `DiagnosticOutputProvider` and `TrackingHeapBlockProvider`
look very confusing and tangled. It seems, instrumentation for tests
got mixed up with the effort to provide a simplistic prototype implementation.
What we need foremost here is the latter; most tests that import the
DiagnosticOutputProvider do not even use any instrumentation....
It seems evident that the the problems with the existing BufferProvider implementation
are rooted in a lack of understanding of the actual usage situation at that time(2011).
This caused a »hexagonal design«, i.e. something entirely driven by uint testing
for the sake of unit testing.
So to identify the unsuitable decisions, some degree of deductive work is necessary,
based on our much better understanding of the Render Engine.
A crucial challenge is that buffer management must perform well
within a massively concurrent environment, and do so without excessive latency.
This picks up the work for [ticket:1395 resolve integration of BufferProvider with DataSink (#1395)]:
Almost a year ago, before the long hiatus due to important general clean-up
and maintenance work, and the research regarding Video output —
a discrepancy was discovered between the »Buffer Provider Protocol«
and the »Output Slot Protocol«. This mismatch prevents us from building
a simple preliminary `BufferProvider` to be used for the engine integration
as part of the [ticket:1228 »Playback Vertical Slice«].
After re-establisihing the work context, I have read through the new
Render Node code built last year, and distilled the sequence of
buffer management operations that actually happen during a recursive
`pull()` call. With this kowledge, I am now able to fill in the
''standard case for rendering'' — which I was unable to formulate
during the first attempt of building the Render Engine in 2011.
__Next Steps__: Redesign of the standard implementation base of the
`BufferProvider` to prepare for a massively concurrent usage.
* Simplify some constructs from the Python-3 migration
* update copyright, since I did maintain the SCons build continuosly
* remove unused method from Buildhelper
Furthermore, the documentation page for our build system
https://lumiera.org/documentation/technical/build/SCons.html
has been reworked, to add a synopsis, some further background
information about the internal structure of SCons and about
the specific conventions and definitions used for Lumiera
...funny enough I never noticed this obvious mistake,
since I never install software directly into my system,
while the DEB-build does not use absolute paths...
Solution:
* SCons has this speical convention that a path prefixed with '#'
is resolved relative to the root of the build (where the SConstruct resides)
* now we apply this automatically to the two relevant settings
** INSTALLDIR
** TARGDIR
* but only (conditionally) if the configured path is relative, not absolute
As a consequence, most other hard-coded usages of the '#'-prefix can then be dropped
A long-standing unnecessary mutual dependency caused rebuild
of most tests in the standard target, because the ''valgrind suppression''
executable was classified as »tool«, while also depending on the libraies
with the test code.
...this bug was spotted as the test failed reproducibly
when built with `-fstack-protector-strong` — which adds
additional "canary" markers to some kinds of problematic
stack storage; this seemed to have the effect that now
the second test uses exactly the same location as the
preceding test, and thus finds valid data in the newly
created work buffers.
There is no reason for using uninitialised storage in this
test (I can recall that I wanted to build a generic helper
and intended to use that as ''virtual overlay'' over existing
memory — but that usage never took place, and the `struct Buffer`
is not a general-purpose tool now, but only made for this
specific test. Thus memory can be easily zero-initialised.
- reorganise the navigation tab structure
- place all further index lists into a common "Index" tab
- include a list of concepts
- remove the various "member" sub-tabs, these are not helpful
- the "modules" tab is now called "topics" (since C++ has now Modules!)
- generally re-sync DoxygenLayout.xml with a current pristine template
- comb though the Doxygen Warnings and fix a lot of small problems
Notably some links on the **Gnome documentation** pages are problematic,
since there was seemingly an infrastructure change -- which unfortunately
leads to loosing several deep-links into the GTK-3 related documentation
and tutorials.
While the differences to GTK-4 are often rather minimal, I would still prefer
to link to those documentation pages used as reference at implementation time
of our related library and GUI functionality.
In several cases I have thus looked up the old URLs at Archive.org
Yet another chainsaw massacre.
One of the most obnoxious annoyances with C++ metaprogramming
is the need to insert `typename` and `template` qualifiers into
most definitions, to help the compiler to cope with the syntax,
which is not context-free.
The recent standards adds several clarifications, so that most
of these qualifiers are redundant now, at least at places where
it is unambiguously clear that only a type can be given.
GCC already supports most of these relaxing rules
(Clang unfortunately lags way behind with support of newer language features...)
Now able to remove most complicated comparison operators and most usages of boost::operators...
In most cases it is sufficient just to define one ''spaceship operator'',
and often even that one can be synthesised.
However — we still use boost::operators for arithmetic types,
notably the `lib::time::TimeValue`, which is addable and mutipliable
Only minor rearrangements necessary to make that possible with C++20
And while at this change (which requires a full rebuild of Lumiera)
- simplify the defined comparison operators, as C++20 can infer most variations
- also mark various usages of `const char*` either as Literal or CStr
Remark: regarding copyright, up to now this is entirely my work,
with two major creation steps in 2008 (conception) and
in 2017 (introduction of a symbol table)
This changeset removes various heuristics and marker-traits
by a constraint to tuple_like types. Furthermore, several usages
of `apply` can thereby be generalised to work on any tuple_like.
This generalisation is essential for the passing generic data blocks
via `FeedManifold` into the node invocation
- integrate the concept definition into tuple-helper.hpp
- use it to replace the `is_Structured` traits check
- do not need `enable_if_TupleProtocol` any more
Integrate test coverage of the concept metafunctions
and the generalised get accessor
''This changeset was made at LAC 2025 in Lyon, France''
...to rely on the new formulation and the extended template `WithIdxSeq`
This is in preparation to use this new iteration scheme also from the tuple_like concept
Motivated by the difficulties encountered with `std::apply` —
which basically forced us to define our own alternative with
conceptually more adequate limitations....
...so these are the first attempts towards building a C++20 concept.
The Lumiera »Reference Platform« is now upgraded to Debian/Buster, which provides GCC-14 and Clang-20.
Thus the compiler support for C++20 language features seems solid enough, and C++23,
while still in ''experimental stage'' can be seen as a complement and addendum.
This changeset
* upgrades the compile switches for the build system
* provides all the necessary adjustments to keep the code base compilable
Notable changes:
* λ-capture by value now requires explicit qualification how to handle `this`
* comparison operators are now handled transparently by the core language,
largely obsoleting boost::operators. This change incurs several changes
to implicit handling rules and causes lots of ambiguities — which typically
pinpoint some long standing design issues, especially related to MObjects
and the ''time entities''. Most tweaks done here can be ''considered preliminary''
* unfortunately the upgraded standard ''fails'' to handle **tuple-like** entities
in a satisfactory way — rather an ''exposition-only'' concept is introduced,
which applies solely to some containers from the STL, thereby breaking some
very crucial code in the render entities, which was built upon the notion of
''tuple-like'' entities and the ''tuple protocol''. The solution is to
abandon the STL in this respect and **provide an alternative implementation**
of the `apply` function and related elements.
This resolves an intricate problem related to metaprogramming with
variadic templates and function signatures. Due to exceptional complexity,
a direct solution was blocked for several years, and required a better
organisation of the support code involved; several workarounds were
developed, gradually leading to a transition path, which could now
be completed in an focused clean-up effort over the last week.
Metaprogramming with sequences of types is organised into three layers:
- simple tasks can be solved with the standard facilities of the language,
using pattern match with variadic template specialisations
- the ''type-sequence'' construct `Types<T...>` takes the centre stage
for the explicit definition of collections of types; it can be re-bound
to other variadic templates and supports simple direct manipulation
- for more elaborate and advanced processing tasks, a ''Loki-style type list''
can be obtained from a type-sequence, allowing to perform recursive
list processing task with a technique similar to LISP.
after all the relevant library components do support both kinds of
type sequences transparently, any usages in core code can now be
switched over to the new, variadic type sequences.
A very performance relevant shortcoming of the existing implementation
of partial function closure is that the result is always wrapped into a
std::function, which typically causes a heap allocation when more than
a single pre-bound argument must be stored — which is annoying,
since the underlying Binder provides inline storage and thus
could be handled directly as a value object.
However, returning the Binder directly is also problematic, since
this object is outfitted with several overloaded function call operators,
which defeats most techniques to detect a function signature. Notably,
relevant down-stream metaprogramming code, like the tuple-closure used
in the `NodeBuilder` would break when being confronted directly with
a binder object.
An investigation shows that there is no direct remedy, short of
wrapping the binder into another functor. This can be accomplished
with a helper template, that generates a wrapper; however, this
wrapper builder must be supplied with explicit type information
regarding the function arguments (precisely because this type
signature can not be picked up from the Binder object itself)
This is a rather intricate and technical change, but allows in the end
to switch back all usages to a main implementation patch, which is now
based on `func::BindToArgument` — so this could become the final
implementation core and replace the old `PApply` template eventually...
Largely, these changes are related to allow for ''perfect forwarding''
of the functor and the argument values to be closed; these will be
copied into the ''Binder object'' created by `std::bind`.
Notably the `TupleConstructor` was changed to perfect-forward its »source«
into the specialised `ElmMapper`; this is possible since the latter
already receives a `SRC` template parameter, which can be supplied
with whatever base type the `std::forward` invocation will expose.
In the specialisation relevant here, template `PartiallyInitTuple`,
now an ''universal reference'' is stored and passed to `std::get`,
so that (depending on the input used), either a LValue or an
RValue reference is used for the extracted data elements.
After these changes, all existing usages of `applyFirst()` or `applyLast()`
can be replaced by this modernised implementation back-end, thus obsoleting
the various hard-coded workaround added during the last years.
A lot of repetitive, pre C++11 metaprogramming code can now be removed,
and several helper constructs, which were needed to handle generic
function application and passing a tuple of values to create a binder.
Note however, the highly complex and technical core of this header
still remains intact; which is to create a ''partial closure'' over
some arguments of a function, while keeping the remaining arguments
open as parameters for invocation.
TODO: Even in the remaining code there is a lot of redundancy
and helper construct which are no longer necessary
...because swapping in the new standards-based implementation
leads to compile failures on tests to cover out-of-bounds cases.
Under the (wrong) assumption, that some mistake must be hidden in
the Splice-metafunction, I first provided a complete test coverage;
while the actual problem was right below my nose, and quite obvious...
The old implementation, being based on a case distinction over the argument count,
simply was not able even to notice excess arguments; other the new implementation,
based on variadics and `std::apply`, which is fully generic and thus
passes excess arguments to `std::bind` when a position beyond the actual
argument list is specified to be closed.
The old behaviour was to silently ignore such an out-of-bounds spec,
and this can be reinstated by explicitly capping the prepared tuple
of binders and actual arguments passed to `std::bind`
Another question of course is, if being tolerant here is a good idea.
And beyond that, function-closure.hpp is still terrifyingly complex,
unorganised and use-case driven, to start with....
...can now be formulated in a single function,
based on the apply-to-λ technique invented by David Vandervoorde.
WARNING: the rewritten version of BindToArgument<...>::reduced()
does not compile in the out-of-bounds case, revealing a possibly
long standing defect in the typelist-metafunction Splice
