This is a feature I wasn’t sure for a long time how best to tackle.
It’s worth noting that an initial attempt at `IterExplorer` was based
on the monad pattern, which turned out to be a mistake, since monads
are more of an anti-pattern, than being genuinely useful.
The actual use-case that made me reconsider such a feature it
to build an seamlessly chained sequence from several iterators.
It would be simple to develop a suitable utility as a stand-alone helper,
yet it seemed worth the effort to integrate this into the `IterExplorer`
ecosystem, due to the flexible automatic adaptation to iterable source
pipelines provided therein.
Going this route incurred some cost however, unveiling several defects
related to handling nested iterable types in the generic processing framework.
...to emplace some object into the buffer space.
I wouldn't expect that to be necessary for the real usage,
but it comes in handy for tests, where we often want to place
some sentinel value into a buffer.
- visible Signature of TestFrame changed due to being relocated into the Vault
- attempt to make the WorkForce_test more robust;
frequently fails on testsuite run after full build;
so better ''wait'' for the workers to have started working,
instead of using a fixed delay (that must be extremely short
and is hard to predict)
After resolving the inner tensions in the old prototype from 2012,
it was now possible to resolve all issues in a single strike
* `DataSink` becomes a functor, yet embeds a shared ref-count
* in the λ-closure a ''specific Buffer Descriptor'' is prepared for each feed
* building upon the prototyping work, a dedicated `OutputBufferProxy`
exposes a `BuffHandle` to the client, with `emit()` and `release()` calls
* the implementation ''delegates directly'' to the `OutputSlot::Connection`
* an this is where all actual output technology is encapsulated
...this is the gordian λ
Note
- embedding a copy of the shared-ptr in the closure
- pre-configure a buffer descriptor (buffer type)
* with the buffer size as retrieved from the connection
* and pass the connection itself marked as LocalTag
Thus, on invocation of the DataSink (functor), it suffices
to invoke the lock() function on the BufferDescriptor,
using the new additional argument introduced with the
preceding commit
Reaping the benefits of the preceding refactorings, most of the complex
wiring within `OutputBufferProxy` can be omitted, and the proxy calls
can be delegated directly to the actual `Connection`
There is a catch however: we need to create an additional pathway
to feed the frame-number information into the `BufferStore` backend.
This changeset attempts to breach into the implementation design of OutputSlot
(while largely retaining the API and the »Output Slot Protocol«).
While the actual problem that brought me to this point can be traced back to some
rather obvious mistake, namely to conflate the `BufferProvider` used as
backing implementation with a `BufferProvider` used as front-end API,
creating the new connections to use an `OutputBufferProxy` unveils some
further long-standing structural mismatch; notably this leads to the
difficulties to get the lifecylce management of `OutputSlot` and `DataSink`
under control.
So I remove one entire level from the implementation,
in a bold move to realign the design with the prospective usage pattern.
Obviously, this breaks a lot of further code, and I need to rewire
the Allocation implementation altogether, and provide a new functor based
implementation for `DataSink`
...which still looks like a nasty task
* the good part is, we have a working `BufferProxyAdaptor` now
* yet this setup does not seem to fit into the `OutputSlot::AllocState`
As an aside, several test definitions need to be relocated to the Vault,
since the whole compound around `BufferProvider` migrated down
This is a nasty new requirement, discovered as an aside while looking
into ways to resolve the ''crucial mismatch'' between the `OutputSlot`
and `BufferProvider`, using the proxy-implementation of `BufferProvider`
(thereby expanding on the recent draft with the `BufferProxyAdaptor`)
The point is, if the BufferMetadata will be used beyond the internal coordination
within BufferProvider, then it will be used as part of actual buffer storage.
And this implies that the actual size of the allocation may differ (be greater)
than the nominally required size (since we'll very likely employ some kind
of memory tiling, for reasons of performance).
This leads again to an analysis of `BufferMetadata` usage,
with the conclusion to add the actual storage size directly into the API,
but treat it as an ''extra'' and not integrated into the ''buffer type''
Phew ... quite complex alltogether
This might have seemed like a good idea, adding an additional ''consistency check''...
However — we ''do have'' a generic check, insofar `BufferMetadata` is a **state engine**.
As long as we're able to prevent, reliably, that the same buffer address
ends up being classified as a different buffer type, this should be enough
to prevent double-use of any buffer (letting aside that `BufferProvider` is
backed by an implementing memory allocator, which in itself should prevent
any double-use.
Removing this feature resolves some ambiguity in the overall design,
and allows to turn `BufferMetadata::lock(type, address)` into a private
implementation function.
In the old code, there were some external access paths directly entering
into that implementation function, which caused a lot of confusion (at least for me...)
This refacotring prepares for the planned changes in `OutputSlot`
* introduce a new compound parameter (tuple) BufferAlloc
* wire this through all calls involved with storage handling
__Background__:
* `BufferMetadata` deemed important for the implementation of buffer storage
* ''new requirement detected'': must be able to handle an actual storage block
that is ''larger'' than the requested buffer size (⟼ tiling)
* need compact and uniform info, also for the `BufferProxyAdaptor` callbacks
The old implementation relied on two layered APIs:
one for the client and one for the implementer of
the Allocation state. This is logically sound, yet
clumsy in terms of design, since the internal API
turns out to be void of actual meaning.
It seems possible to eliminate this confusing structure
by a clear separation of ''implementation realms''
* the ''mechanics'' of connection state management
* the actual implementation of the connection
The first can be provided as generic template subclass,
while the latter should be confined entirely into the
implementation of the `Connection` interface.
Doing so requires to keep each part of the implementation
strictly focused onto its purpose — something I obviously
was not able to see at the time when writing this old
implementation, notably because I was completely guided
and driven by the test/dummy implementation without
considering how an actual implementation of
low-level data output would work.
The old implementation of `DiagnosticOutputSlot` is now disabled
and will be re-written from scratch, and no longer tangled to `BufferProvider`
- `OutputManager` remains in Steam-Layer for now
- the `Timings` and `OutputSlot` are now a self-contained
One of the goals of this extended clean-up and refactoring is to allow
access to the data buffers of a `DataSink` through a `BuffHandle` as front-end.
Which essentially means that the ''Buffer Provider Protocol'' and the
''Output Slot Protocol'' mesh together well.
After the clean-up of the `BufferProvider` implementation hierarchy,
the viability of this idea was demonstrated with the `BufferProxyAdapter` draft;
so it is now roughly clear what must be changed within the existing implementation
of `OutputSlot`. More specifically, the only actual implementation ever completed
was a dummy implementation for test (`DiagnosticOutputSlot`), which hooked
confusingly into the corresponding dummy implementation of `BufferProvider`,
that was written one year earlier (and was also without actual implementation
at that time).
So the second half of the task ahead seems very similar to the first part:
- the base implementation of the ''Output Slot Protocol'' seems roughly acceptable
- but the actual dummy implementation was somewhat confused and should be
re-written from scratch, without ''any attempt to cut corners by re-using other dummy code''
__Note__ as part of this analysis I discovered some gap in the design
regarding timeout handling ⟶ Ticket #1417
building upon all the preparations done during the last days,
the compound of code connected to the `BufferProvide` can now
be migrated down into the vault:
- `BuffHandle`
- `BufferProvider`
- `BufferMetadata`
- `BufferProviderSetup`
- the simple standard implementations
- `DiagnosticBufferProvider`
- all directly related unit tests
Gradually, the header buffhandle.hpp became the minimal include
to use basic `BufferProvider` functionality, since `BuffHandle`
provides proxy functionality to lock and access a data buffer.
The whole compound connected to the `BufferProvider` has evolved
into a cohesive yet central connection hub related to buffer memory;
it should thus be extracted, moved down to »Vault« layer and used
4as a foundation for various high-level tasks. In preparation of
this step, relevant marker typedefs are re-arranged;
notably ''Stream Type'' now includes `buffhandle.hpp`
These are generic vehicles to verify computation structures.
Since they are in no way related to our models, they belong into »Vault« layer
Furthermore this reorganisation of test support components is prerequisite
to also move further generic facilities down into the Vault
This resolves a long-standing issue:
we could not use proper include paths for test-helper code that was defined alongside the tests.
Since some test-helper code must now be relocated into the »Vault«-Layer,
I took the opportunity to finally resolve this annoyance. The solution is two-fold
* introduce a dedicated test-support library
* remove the quirks that were necessary to support some plain-C tests
Together, this allows to raise the include-directory by one level,
and then to adjust all folder names so that they match the namespace
This is something I wanted to clean-up since a very long time indeed,
but always postponed it, since it requires a lot of nasty clean-up and disentangling....
Several parts had been stashed away into `src/lib/test`, which are not really related:
* the testrunner and testsuite builder
* generic helper utilities
* specialised test-support facilities, e.g. the EventLog
This changeset introduces a new top-level test folder and a namespace `::test`
Code from this folder is linked into a shared library `libtestsupport.so`,
which resides in the linkage hierarchy between »Vessel« and »Vault«,
yet is no longer part of the Lumiera application and only linked into the test-suite.
The next step with the `BufferProvider` and `OutputSlot` refactoring
will require to create new definition headers and to move those self-contained
services down into the »Vault« layer. In preparation, I decided deliberately to
abandon that old idea of a ''minimal bootstrap'' of arbitrary components in
favour of building the application as a coherent whole, with well-defined
boundaries and a backbone structure, marked as such explicitly by name
and arrangement of the code.
The changes were documented and applied to the integration branch first,
and now it's time to join that effort with the ongoing work on `BufferProvider`.
During the last months, a lot of gradual changes to policies and guidelines
had been finalised and documented. Moreover, recent clarifications regarding
memory management in the render engine will lead to a clearer definition
of the »Vault« Layer's role, with some ''engine code'' expected to move down.
On this occasion, I reworked relevant sections in the documentation to
confirm the role that now falls to central parts of the overall architecture.
Notably I decided to rename those parts related to the application backbone
as »Vessel« — to underpin the more active and defining role they assumed.
Explored further aspects of this new design;
as an aside, I tried to build a Concept to describe the policy,
which howerer turned out to be much more challenging than expected.
Lastly, I could now complete the prototype of `BufferProxyAdaptor`
While it is trivially simple to define some struct with callback functions,
such a definition turns out to be obnoxiously complex and hard to read in pracice,
thereby nullifying any benefit from the shorthand λ-based notation.
This kind of problem with the readability at client-side is quite common
when using callbacks for a policy-based design. To overcome this difficutly,
it would be necessary to define the policy and the callbacks in one
succinct builder expression. Unfortunately such a builder syntax
typically requires to write a lot of boilerplate template code.
After several failed experiments, I have found a bold new way to
exploit features of C++20 in conjunction with a macro, allowing
to generate a suitable data record directly form an C++ expression.
Now I have a working setup for a `BufferProxyAdaptor`.
Unfortunately this is still confusingly complex.
This is due to my intention to build a generic template,
that can be configured with a policy to invoke arbitrary callbacks,
whenever one of the passed-through buffers transitions through its lifecycle.
* we need to configure the `BufferProviderSetup` to use our specific implementation classes
* and the second one of these, our `BufferStore` implementation needs to invoke the callbacks
This closes a ''conceptual gap.''
There was a `lib::TestFrame::reseed()` function, which should be invoked
whenever a clean / deterministinc seed is required for computations involving
`TestFrame` data.
On occasion I noticed that most actual usages of `TestFrame`, which became quite common
lately, were missing this `reseed()` call, while the enclosing test
had indeed a proper Test::seedRand() call.
So my conclusion is: if even ''I myself'' miss that fine point frequently,
it is justified to automated this setup, even while `TestFrame` is right now
the only client of such a setup-facility.
There is a twist though.
`TestFrame` is part of the ''Engine'' test coverage, i.e. it is linked
into `libtest-core.so`. While the support for random numbers and reseeding
is obviously a very basic library feature, and linked into `liblumiera.so`
**Oh well**.
So now we have ''yet another'' static registration service.
I kept everything as simple as possible, which means, we continue
to use the ''common, global random generator'' from the library,
and there is no protection against some other code using those
library functions in a non-deterministic way.
`lib/random-reseed.cpp` is just a glorified hard-wired setup, beware!
This is an implementation experiment to validate the improvements
and internal clean-up. Previously, I was unable to implement such a
forwarding proxy buffer provider, as would be required for the
intended harmonisation of the `OutputSlot` interface with `BufferProvider`
With the reworked verison of the BufferProvider standard implementation
it seems straight-forward to...
* define a »Buffer Type« that marks any of the prepared buffers to pass-through
* use the resulting Buffer Descriptor together with a rigged storage backend,
that just returns the ''local tag'' as buffer adress
The first round of the [ticket:1410 BufferProvider clean-up] is roughly complete.
* a much cleaner diagnostic interface has been defined and implemented
* this allowed to modernise the `DiagnosticBufferProvider_test`
and the `BufferProviderProtocol_test`
* the implementation scheme has been split in two realms
* the old heap-memory based implementation has been re-arranged,
so that the aforementioned tests are **green** again.
Now is the point to draft a solution to get the interfaces of
`BufferProvider` and `OutputSlot` aligned, so that the latter
can expose its lifecycle through the former.
Today, I have analysed various aspects and developed a pathway
to get into that task, and next I will reshape the previously
failed `OutputProxyProvider_test` into a prototype to
validate this idea
* write down some first considerations regarding requirements
of the actual production-strength implementation of buffer management
* consider relevance of the part of the API that was retained
* write a new series of test cases to cover handling of embedded objects,
using the test support for tracking object creation
Some weeks ago, at the time where I started to dig into this topic,
I extended the BufferProviderProtocol_test to perform a buffer usage pattern
that mimics the usage in the real Render Engine, as implemented last year.
For this, I used the `TestFrame` and the chained-hash dummy computations.
Now, after reordering and clean-up of the `BufferProvider` implementation interfaces,
it can be showed with this computation scheme that
- all buffers are indeed allocated and handed out as expected
- the metadata handling is correct and points to the processed buffers
- the complete memory used in this test is uncorruped and holds correct numbers
It covers now roughly the same procedures as before,
yet the scheme how to do verification is quite different.
Furthermore, we allocate large random-sized buffers now
and implant `TestFrame` instances, and so also the verification
covers a significant storage area.
* can now replace old verification code by new diagnostic functions
* test case runs without failure
''UNRESOLVED:'' failure in verification of memory contents
In the end I refrained from applying most of the changes I had intended.
I am aware that `BufferProvider` is still quite a massive structure,
yet with the implementation refactorings done thus far, it seems
to be reasonably clear and self-contained now.
It would be possible to totally overthrow the grown structure and
expose only the two handles (BuffDescr and BuffHandle), together with
a factory function, but I cannot see how this would help to make the
code clearer and more approachable. Maybe it would make the usage
more obvious from the client's perspective — yet, to be honest,
there are not ''that much clients'' to justify such a move.
Furthermore, the implementation is far from finished: for both of
the back-end lines, a fully production-strength implementation
must be created, that is able to operate in a highly concurrent
environment; very likely some lock-free messaging and hand-over
between threads will become necessary, adding to the complexity.
Thus it seems better to note the ''design critique'', but to
postpone any further refactorings to a point where we can watch
a fully developed Render Engine under production load.
These were defined in a confusing and implementation-oriented way.
Since we have now mostly abstracted away the metadata handling from the
`BufferProvider` front-end, a more succinct definition was chosen
* a ''Descriptor'' is valid when the lookup succeeds
* a ''Handle'' is likewise valid when the underlying
state-Key can be resolved, and the memory address is given
Some aspects regarding the state remain unresolved though.
Ideally we'd want a `BuffHandle` to be valid preciselc when
it ''is in valid state to be accessed''
Now able to remove some obsoleted part of the old implementation.
The existing implementation of the ''Buffer Provider Protocol'' is
quite old and dated meanwhile; it was more like an explorative prototype
and mixed up implementation and test code.
At that time, notably the assumption was that the core render engine
will be implemented in plain-C and will push around `void*` all over the place.
The `BufferProvider` was meant as an adaptor, while questions of typing
were added as an afterthought. Especially some functionality to use
the `BuffHandle` in the way of a smart-ptr are completely obsoleted
by the reworked design of the Render Engine (2024).
Since the BufferStore implementation component is now supposed
to handle only the memory allocation, a complete end-to end data flow
must be ensured, so that allocation / deallocation functions get all
the relevant information. Notably, the `LocalTag` is now used as an
implementation-defined conduit for further (opaque) marker data.
Since all data passed over this API are simple values (even `LocalTag`
is effectively a 64bit integer), we can now avoid relying on output
parameters or similar side-effect tricks.
...just enough to be able to write the assertions in the first test case
(..and the test case fails still, due to the pending work on sorting out
the implementation APIs between `BufferStage` und `BufferStore`)
To push ahead with the refactoring,
I will build the test/diagnostics API completely anew,
and attach it solely to an instance of `DiagnosticBufferProvider`.
Unfortunately this ''leads us into chainsaw surgery...''
* it uncovers a host of conceptual contradictions
* actually, the `HeapMemProvider` was more an implementation
of diagnostic tracking, than it was an actual `BufferProvider`
implementation (hence the old name: `TrackingHeapBlockProivder`)
* and this lured me into using these tracking functions directly,
while the `DiagnosticBufferProvider` remained in unfinished state.
So basically I have now **disabled** all verification from the Unit-tests.
I'll have to rebuild them from scratch, thereby reproducing the logic/intention
- write some documentation
- identify some add-on API that must be retracted
- consider further how to disentangle ''diagnostics'' and ''implementation''
More or less forcibly squeeze all the existing implementation
into the new basic structure with two separate »pillars«
**Breakages**
- existing code expected the `BufferStore` to transition to ''locked'' state
- existing code expected `provideLockedBuffer(typeID)` to tail-call the
`buildHandle()` function of the enclosing `BufferProvider` base implementation
- various diagnostic usages bypassed the official APIs and called directly
into the implementation. This is no longer possible; rather, `DiagnosticBufferProvider`
needs to provide additional diagnostic API to suit those purposes
The outward interface has stood the test of time quite well.
But the back-side towards the implementation is a tangled mess,
presumably due to being developed from prototyping code: the
actual implementation is completely overgrown by diagnostics
and test instrumentation, since running in unit test setup
was the only actual usage of this code ever since.
My first attempt with decomposing into a decorator-chain failed.
Thus the new line of attack is to introduce two implementation
pillars right from top-level
- `BufferStage` : the state and metadata handling
- `BufferStore` : the storage backend
These are defined as nested interfaces, and the further refactoring
starts with subclassing everything, nested as it is.
The intention is to build the tracking functionality from scratch,
while wrapping the old implementation as delegate. Hopefully this allows
to reduce the complexity of the existing implementation step by step,
until I'm ''able to grasp what this implementation does''.
This old implementation works, but is highly problematic code.
The Git history shows that I had already problems to understand this code
while building the DiagnosticOutputSlot, one year later.
To my defense, in both cases (2011 and 2012), I had no clear idea yet
how the processing within the engine would work, and thus I considered
that Buffer Type registry a mere implementation detail, that can be
lumped into the »code behind«.
The prototype implementations of BuferProvider and OutputSlot are confusing;
in-depth analysis shows that this is related to intermingling use
of test instrumentation and features of a prototypical default.
To get anywhere with the existing code, it seems we'd need to split off
the test instrumentation (i.e. `DiagnosticBufferProvider`) altogether,
even if this causes duplication of implementation code. Many years ago,
when first implementing this prototype, I had no clear idea what the usage
in the Render Engine entails, yet I noticed that some basic buffer »tracking«
could be used as implementation primitive to simplify the prototyping work.
This might be one of the reasons, why the "code behind" in the protocol
implementations both in `BufferProvider` and `OutputSlot` became so strongly
tangled, so that I way unable to make both protocols orthogonally composable.
The implementation code for `DiagnosticOutputProvider` and `TrackingHeapBlockProvider`
look very confusing and tangled. It seems, instrumentation for tests
got mixed up with the effort to provide a simplistic prototype implementation.
What we need foremost here is the latter; most tests that import the
DiagnosticOutputProvider do not even use any instrumentation....
It seems evident that the the problems with the existing BufferProvider implementation
are rooted in a lack of understanding of the actual usage situation at that time(2011).
This caused a »hexagonal design«, i.e. something entirely driven by uint testing
for the sake of unit testing.
So to identify the unsuitable decisions, some degree of deductive work is necessary,
based on our much better understanding of the Render Engine.
A crucial challenge is that buffer management must perform well
within a massively concurrent environment, and do so without excessive latency.
This picks up the work for [ticket:1395 resolve integration of BufferProvider with DataSink (#1395)]:
Almost a year ago, before the long hiatus due to important general clean-up
and maintenance work, and the research regarding Video output —
a discrepancy was discovered between the »Buffer Provider Protocol«
and the »Output Slot Protocol«. This mismatch prevents us from building
a simple preliminary `BufferProvider` to be used for the engine integration
as part of the [ticket:1228 »Playback Vertical Slice«].
After re-establisihing the work context, I have read through the new
Render Node code built last year, and distilled the sequence of
buffer management operations that actually happen during a recursive
`pull()` call. With this kowledge, I am now able to fill in the
''standard case for rendering'' — which I was unable to formulate
during the first attempt of building the Render Engine in 2011.
__Next Steps__: Redesign of the standard implementation base of the
`BufferProvider` to prepare for a massively concurrent usage.
* Simplify some constructs from the Python-3 migration
* update copyright, since I did maintain the SCons build continuosly
* remove unused method from Buildhelper
Furthermore, the documentation page for our build system
https://lumiera.org/documentation/technical/build/SCons.html
has been reworked, to add a synopsis, some further background
information about the internal structure of SCons and about
the specific conventions and definitions used for Lumiera
...funny enough I never noticed this obvious mistake,
since I never install software directly into my system,
while the DEB-build does not use absolute paths...
Solution:
* SCons has this speical convention that a path prefixed with '#'
is resolved relative to the root of the build (where the SConstruct resides)
* now we apply this automatically to the two relevant settings
** INSTALLDIR
** TARGDIR
* but only (conditionally) if the configured path is relative, not absolute
As a consequence, most other hard-coded usages of the '#'-prefix can then be dropped
A long-standing unnecessary mutual dependency caused rebuild
of most tests in the standard target, because the ''valgrind suppression''
executable was classified as »tool«, while also depending on the libraies
with the test code.
...this bug was spotted as the test failed reproducibly
when built with `-fstack-protector-strong` — which adds
additional "canary" markers to some kinds of problematic
stack storage; this seemed to have the effect that now
the second test uses exactly the same location as the
preceding test, and thus finds valid data in the newly
created work buffers.
There is no reason for using uninitialised storage in this
test (I can recall that I wanted to build a generic helper
and intended to use that as ''virtual overlay'' over existing
memory — but that usage never took place, and the `struct Buffer`
is not a general-purpose tool now, but only made for this
specific test. Thus memory can be easily zero-initialised.
