The first round of the [ticket:1410 BufferProvider clean-up] is roughly complete.
* a much cleaner diagnostic interface has been defined and implemented
* this allowed to modernise the `DiagnosticBufferProvider_test`
and the `BufferProviderProtocol_test`
* the implementation scheme has been split in two realms
* the old heap-memory based implementation has been re-arranged,
so that the aforementioned tests are **green** again.
Now is the point to draft a solution to get the interfaces of
`BufferProvider` and `OutputSlot` aligned, so that the latter
can expose its lifecycle through the former.
Today, I have analysed various aspects and developed a pathway
to get into that task, and next I will reshape the previously
failed `OutputProxyProvider_test` into a prototype to
validate this idea
* can now replace old verification code by new diagnostic functions
* test case runs without failure
''UNRESOLVED:'' failure in verification of memory contents
...this turns out to be surprisingly difficult,
due to various subclass / protected visibility problems.
Not sure if the component design within `DiagnosticBufferProvider`
is sane yet...
This is clearly implementation code and can now be removed from the interfaces;
`SimpleBufferStateRegistry` can thus delegated to a private metadata registry
In the end I refrained from applying most of the changes I had intended.
I am aware that `BufferProvider` is still quite a massive structure,
yet with the implementation refactorings done thus far, it seems
to be reasonably clear and self-contained now.
It would be possible to totally overthrow the grown structure and
expose only the two handles (BuffDescr and BuffHandle), together with
a factory function, but I cannot see how this would help to make the
code clearer and more approachable. Maybe it would make the usage
more obvious from the client's perspective — yet, to be honest,
there are not ''that much clients'' to justify such a move.
Furthermore, the implementation is far from finished: for both of
the back-end lines, a fully production-strength implementation
must be created, that is able to operate in a highly concurrent
environment; very likely some lock-free messaging and hand-over
between threads will become necessary, adding to the complexity.
Thus it seems better to note the ''design critique'', but to
postpone any further refactorings to a point where we can watch
a fully developed Render Engine under production load.
doing so requires to trace a new call path from front to back,
into the `BufferMetadata` implementation: there is now an explicit check
that the buffer is in `LOCKED` state, but not yet `EMITTED` and als that
it was not yet `released()`
I think the additional effort is justified
* in the regular / currently intended usage in the Render Engine,
this predicate is not required, since compliance to the ''Buffer Provider Protocol''
as such is sufficient to ensure that
* but since all of this is getting more generic and polished now,
I could imagine future usages beyond the current confines; and then,
this semantics is what you'd expect from such a `bool()` check:
»I am allowed to use it«
These were defined in a confusing and implementation-oriented way.
Since we have now mostly abstracted away the metadata handling from the
`BufferProvider` front-end, a more succinct definition was chosen
* a ''Descriptor'' is valid when the lookup succeeds
* a ''Handle'' is likewise valid when the underlying
state-Key can be resolved, and the memory address is given
Some aspects regarding the state remain unresolved though.
Ideally we'd want a `BuffHandle` to be valid preciselc when
it ''is in valid state to be accessed''
...while the implementation provided herein is trivial,
it represents the entrance point into the process covered by the
''Buffer Provider Protocol''; thus I went again through all the
stages of the implementation, and confirmed my analyiss from 2024:
9a23aa773b
Today I'd even go so far as to call this a **crietrion of consistency**:
The path leading to the LOCK-transition must be reproducible.
The existing implementation, now re-cast into a new API,
ensures this by requiring an explicitly given Buffer Descriptor
at that point, and ''does not offer any one-step shortcut.''
__The point to note is__: the system used for implementing those type-IDs,
which is a tree of chained hashes, is deliberately open and extensible.
''In theory'' it would be possible thus to arrive at different hash-IDs
for the same criteria, if these are added gradually, in several steps,
as a refinement. This leeway is acceptable, as long as it can not happen
that two competing allocation processes want to lock the same
memory address, but tracked through different parent type-IDs.
The `BufferProvider` is both an interface and a facade, with
standard implementations to handle the state transitions and
further steps of the ''Buffer Provider Protocol''.
These implementations have gotten somewhat out of hand, since
they directly mixed memory allocation with very elaborate usage
of the `BufferMetadata`. The goal for this series of refactorings
is to introduce two ''implementation interfaces'' -- thereby
raising the abstraction level in this »code behind«
In the old version of `BufferProvider`, there was a single
backend implementation, which was assumed to call-back into
`buildHandle(typeID, storage, localTag)` after allocating memory.
This implies, the state transition to `LOCKED` happened there.
My next goal is to clarify and abstract also the state transition API,
and this refactoring helps a lot to see the underlying call structure.
Now able to remove some obsoleted part of the old implementation.
The existing implementation of the ''Buffer Provider Protocol'' is
quite old and dated meanwhile; it was more like an explorative prototype
and mixed up implementation and test code.
At that time, notably the assumption was that the core render engine
will be implemented in plain-C and will push around `void*` all over the place.
The `BufferProvider` was meant as an adaptor, while questions of typing
were added as an afterthought. Especially some functionality to use
the `BuffHandle` in the way of a smart-ptr are completely obsoleted
by the reworked design of the Render Engine (2024).
Since the BufferStore implementation component is now supposed
to handle only the memory allocation, a complete end-to end data flow
must be ensured, so that allocation / deallocation functions get all
the relevant information. Notably, the `LocalTag` is now used as an
implementation-defined conduit for further (opaque) marker data.
Since all data passed over this API are simple values (even `LocalTag`
is effectively a 64bit integer), we can now avoid relying on output
parameters or similar side-effect tricks.
This change largely resolves the problems with the storage API,
so that the implementation can now be split in two disjoint segments.
One operation, `mark_emitted()`, is still misplaced on that API, yet
must be retained until the refactoring on the metadata side is complete
(this latter refactoring can not be started sensibly, before the old
»tracking« implementation is cleaned up to care for allocation solely).
With this change, the existing implementation, which combines the
tracking functionality and the allocation, has been squeezed down
into the heap-memory implementation class, yet the tracking functionality
is already detached. The remaining code down there is a huge pile of mess,
that can hopefully be dismantled soon. But at least all of this code
passes the compiler again, and so it should be possible now to focus
on getting the `DiagnosticBufferProvider` up and running, thereby
providing a new clean re-implementation of the diagnostic functions.
This is an important step to prepare a split of the existing implementation APIs
into a type-management part and a memory-allocation part; generally speaking,
a memory allocator needs to know the allocation size, preferably also for
releasing memory. Up to now, this information could be retrieved locally,
with a metadata lookup through the hash-ID; now we ensure that the
size is always passed alongside through the implementation API.
Incidentally, it looks like we might be able to form a common allocation-descriptor,
to represent this internal API in a more concise way. Note further that
all these changes are considered not relevant for the performance,
since work buffers are large and media computation dominates.
More or less forcibly squeeze all the existing implementation
into the new basic structure with two separate »pillars«
**Breakages**
- existing code expected the `BufferStore` to transition to ''locked'' state
- existing code expected `provideLockedBuffer(typeID)` to tail-call the
`buildHandle()` function of the enclosing `BufferProvider` base implementation
- various diagnostic usages bypassed the official APIs and called directly
into the implementation. This is no longer possible; rather, `DiagnosticBufferProvider`
needs to provide additional diagnostic API to suit those purposes
As a consequence of preceding experiments..
* I'll stick to the structure of the BufferProvider front-end interface
* also the »code behind« will remain there; a purely abstract API has no merit here
* however, the ''setup'' of the actual implementation will move into a derived class
* this derived scope class, `BufferProviderSetup` will house two implementation classes
* these two, now named `Stage` and `Store` provide the base implementation
for the two "pillars" of the implementation
Admitted, this structure is now much more elaborate, but it's the only path forward
I'm currently able to come up in order to disentangle the existing implementation mess...
The outward interface has stood the test of time quite well.
But the back-side towards the implementation is a tangled mess,
presumably due to being developed from prototyping code: the
actual implementation is completely overgrown by diagnostics
and test instrumentation, since running in unit test setup
was the only actual usage of this code ever since.
My first attempt with decomposing into a decorator-chain failed.
Thus the new line of attack is to introduce two implementation
pillars right from top-level
- `BufferStage` : the state and metadata handling
- `BufferStore` : the storage backend
These are defined as nested interfaces, and the further refactoring
starts with subclassing everything, nested as it is.
...especially the extended attributes remain somewhat nebulous,
since non of the prospective usages are close to being implemented right now.
It seems, we'll get two distinct sources at construction time of the Node
* additional qualifiers from the Library plug-in
* internal flags or qualifiers provided by the `NodeBuilder`
Another related concern seems to be generation of cache-keys,
which however will ''consume'' the proc-hash generated by the ProcID,
but not change the ID itself; cache-key generation is a tricky subject
and was somewhat overlooked regarding the connection to the `BufferProvider`.
Opened a new ticket #1292 as reminder for this issue.
* Lumiera source code always was copyrighted by individual contributors
* there is no entity "Lumiera.org" which holds any copyrights
* Lumiera source code is provided under the GPL Version 2+
== Explanations ==
Lumiera as a whole is distributed under Copyleft, GNU General Public License Version 2 or above.
For this to become legally effective, the ''File COPYING in the root directory is sufficient.''
The licensing header in each file is not strictly necessary, yet considered good practice;
attaching a licence notice increases the likeliness that this information is retained
in case someone extracts individual code files. However, it is not by the presence of some
text, that legally binding licensing terms become effective; rather the fact matters that a
given piece of code was provably copyrighted and published under a license. Even reformatting
the code, renaming some variables or deleting parts of the code will not alter this legal
situation, but rather creates a derivative work, which is likewise covered by the GPL!
The most relevant information in the file header is the notice regarding the
time of the first individual copyright claim. By virtue of this initial copyright,
the first author is entitled to choose the terms of licensing. All further
modifications are permitted and covered by the License. The specific wording
or format of the copyright header is not legally relevant, as long as the
intention to publish under the GPL remains clear. The extended wording was
based on a recommendation by the FSF. It can be shortened, because the full terms
of the license are provided alongside the distribution, in the file COPYING.
We use the memory address to detect reference to ''the same language object.''
While primarily a testing tool, this predicate is also used in the
core application at places, especially to prevent self-assignment
and to handle custom allocations.
It turns out that actually we need two flavours for convenient usage
- `isSameObject` uses strict comparison of address and accepts only references
- `isSameAdr` can also accept pointers and even void*, but will dereference pointers
This leads to some further improvements of helper utilities related to memory addresses...
Code clean-up: mark all buffers with a dedicated tagging type
The point in question is: if we work the LocalTag into the type-hash,
could it be possible to miss an existing entry in the metadata registry?
This could cause two entries to be locked for a single buffer address,
leading to data corruption.
As far as I can see, in the current usage this would not happen,
but unfortunately this problem can not be ruled out, since the BufferProvider
API and protocol is designed to be open for various usage patterns.
However, the same potentially disastrous pattern could also materialise
when registering two different buffer types, and then locking each
for the same buffer location.
...this is a surprisingly tricky issue, since it undercuts the
generic and recursive implementation of buffer handling;
fortunately I've foreseen such demands may arise down the road
and I've reserved an »Local Key« (now renamed into `LocalTag`),
whose meaning is implementation defined and interpreted by
the specific `BufferProvider`
It became clear that a secondary system of connections must be added,
running top-down from a global model context, and thus contrary to the
regular orientation of the node network, which connects upwards from
predecessor to successor, in accordance with the pull principle.
If we accept this wiring as part of the primary structure, it can be
established immediately while building the nodes, thus adding a preconfigured
''pattern of Buffer Descriptors'' to each node, since there is no further
''moving part'' — beyond the wiring to the `BufferProvider`, which thus
becomes part of a global `ModelContext`
As an immediate consequence, the storage for this configuraion should
also be switched to `lib::Several` and handled similar to the primary
node wiring in the Builder...
Not entirely sure how to use the `emit()` call properly,
assuming that it means that data is complete in buffer,
but can still be read after that point
* at least for a simple, prototypical setup
* and actually shifting the onerous into the Level-1 builder \\
''(which is precisely the intention here)''
In the Lumiera code base, we use C-String constants as unique error-IDs.
Basically this allows to create new unique error IDs anywhere in the code.
However, definition of such IDs in arbitrary namespaces tends to create
slight confusion and ambiguities, while maintaining the proper use statements
requires some manual work.
Thus I introduce a new **standard scheme**
* Error-IDs for widespread use shall be defined _exclusively_ into `namespace lumiera::error`
* The shorthand-Macro `LERR_()` can now be used to simplify inclusion and referral
* (for local or single-usage errors, a local or even hidden definition is OK)
