turns out to be quite tricky, due to the overly convoluted implementation
of PolymorphicValue (which looks quite dated, generally speaking,
and could use a complete overhaul).
- pass the basic value type as template parameter through the Builder
- allow for the ability to define an initial value for the parameter
- invoke the protected constructor of PolymorphicValue, directly passing
a ''type tag'' to specify the implementation data package to be placed
into the embedded data buffer
- `Disposition` is the service-API used by the front-End container
- `ParamData<TY>` is a complete implementation with a inline-value buffer
- The front-End is based on the `PolymorphicValue` container from our support-Lib
Finally able to settle upon a layering for the implementation facilities
Provision:: represents the core idea of a parameter
Domain:: assumes the ability to handle typed values
Disposition:: complete implementation capabilities
BaseDomain:: establish type conversion functionality
TypedEntity:: provide a type and ID descriptor and scale configuration
ParamData:: full implementation based on an embedded value buffer
The original design (based on Alexandrescu) is more elegant,
but has the downside to bloat the VTable size due to the mix-in
arrangement used to generate the interface.
The ongoing design has revealed a tight constraint on element size
for the parameters; thus the change to a linearised inheritance chain.
The overall purpose remains the same: the ability to generate a
double-dispatch type conversion, so that the actual value type of the
parameter can be treated as an implicit implementation detail.
While the role of all the relevant interfaces has been determined,
it still remains difficult to establish a sensible relationship.
Which one is the most fundamental interface?
After a lot of mulling, it seems that the `Provision` interface
should be used as foundation, since it describes the very core
of the parameter concept developed here -- even while the actual
form of the operations must be ''somewhat cunningly reversed'',
so that the knowledge regarding the underlying type can be
kept an opaque implementation detail.
The `Domain` interface would then be layered on top, and describes
the capability to transfer values between buffers marked with different
domains (which effectively requires a double-dispatch in the general case).
Last week, I resumed the work that had been suspended
due to my focus on this year’s Froscon talk (on `io_uring`).
I have reconsidered my previous design ideas and sought to
arrive at a coherent design. Nevertheless, these are preliminary,
exploratory design studies that should be regarded as a basis
for future work rather....
Discovered through design-related code analysis
Knowledge-Technology(Claude) spotted discrepancies that led to these fixes.
Remark: the irregular ordering of check predicates stems from the way I developed the logic and bears no meaning.
Knowledge-Technology(Claude Opus 5) used
- to analyse and reassess the situation
- to discuss and weigh the findings and develop solutions
- to perform the selected mitigation and verify success
----
= Report by Claude =
A sweep with Clang-22 over all 597 translation units (driven from the newly
generated compilation database) showed 271 of them failing, yet with the
2502 diagnostics tracing back to very few root sites. This resolves those
where GCC is simply the more permissive of the two; the remaining clusters
are a matter of design and are left alone for now.
* `using Alias::OriginalName;` does *not* declare inherited constructors.
Per [namespace.udecl] a using-declarator names constructors only when its
terminal name equals the last component of the nested-name-specifier -- so
through an alias, the terminal name is read as the injected-class-name,
i.e. a type, and Clang then demands `typename`. Clang is right here.
Affects `hetero-data.hpp` (19 TUs) and `iter-source.hpp` (62 TUs); in the
latter the alias is renamed `_I` -> `Adapter` while we are at it.
Note `iter-source` as a whole awaits a design review, see #1125
* P1787R6 requires a template-argument-list after the `template` keyword.
`holder.template emplace (...)` in `opaque-holder.hpp` and two sites in
`node-builder.hpp` supply none, so the keyword has to go (83 TUs).
* A default template argument may be given only once per scope. The
out-of-class definition of `OutputSlot::AllocState` repeated the default
from its declaration in 'output-slot.hpp' -- note the friend-declaration
further down in the same header already got this right.
* `constexpr auto ROUGH_PRECISION = pow (10, -3)` is not a constant
expression by the standard; only GCC accepts it, through a builtin.
Hence `util::ipow()` in 'util-quant.hpp', an exponentiation with integral
exponent by square-and-multiply, as a counterpart to `ilog2()`. Exact for
representable base and result, and the static_assert documents that
`ipow(10.0,-3)` is bit-identical to `1e-3`. 193 TUs depended on this
single line.
Furthermore, two findings made while surveying:
* `lib/stat/statistic.hpp` held a second, dormant instance of the same
`pow` problem, in a `round<places>()` helper that nothing in Lumiera ever
instantiates -- which is why Clang never saw it. Since the helper is also
the naive and imprecise formulation, it is removed here; a note records
that it stays in use upstream in yoshimi-test, where this header comes
from, and that decimal rounding belongs to #1360 if ever wanted here.
* The alias `DataBuilder` was defined twice, identically, in
'weaving-pattern-builder.hpp' and in 'node-builder.hpp' which includes
the former. Both compilers tolerate the duplicate definition and only
Clang objects to the repeated default argument, which is why this went
unnoticed. The duplicate is dropped and its documentation moved to the
surviving definition -- together with a @todo, since `DataBuilder`,
`SizMark` and `UseHeapAlloc` are vocabulary of the node-builder DSL and
reside where they do by accident of include order.
Knowledge-Technology(Claude Opus 5) used
- to assess the situation and present implementation avenues
- for discussion while shaping the solution
- for generating code and validating results
Clang based tooling is used frequently for code analysis and navigation.
This requires however that the exact flags and invocation is captured for
each possible translation unit in the code base. That knowledge exists solely
in the build system, and the established way to hand it over is the
LLVM "JSON Compilation Database".
SCons has provided a `CompilationDatabase` builder since 4.0, and it fits
our setup well: it installs an emitter on the object builders and captures
each command line while the dependency graph is constructed. Hence nothing
is compiled -- generating takes about two seconds -- and the result always
describes the complete build graph as declared, rather than the target
actually requested on this invocation.
This feature is provided as ''opt-in'', either through the sticky option
`COMPILEDB=yes` or by direct invocation with `scons compiledb`
The `compile_commands.json` is generated into the build output `target/' dir
For configuration, a `.clangd` in project root instructs the language server
where to find this file.
__Note__ the ordering constraint recorded in `CompileDB.py`:
the emitter must be installed only after `Platform.configure()` has run,
or it also captures the `conftest_*` compilations of the configure checks.
That is why this is a plain helper invoked from the `SConstruct` -- similar to
Setup, Options, Platform -- and not a tool loaded from the `LumieraEnvironment`.
Both were added in 12/2008, at a time when the choice of SCons still had
to be defended, and neither was ever used in practice.
The MD5 based change detection of SCons, in concert with the declarative
build dependency tree alleviates the need for "cleaning paranoia" and makes
full rebuilds a rare event.
Beyond being unused, both tool integrations require to rewrite the `$CXX`
construction variable into "wrapper g++", which silently changes what any
tool reading the build commands concludes about the compiler in use. That
matters for the compilation database, which needs to be added next.
I took some work notes during that ongoing discussion in a separate ''scratchpad document.''
As part of the effort to create an overview digest, I also revisited these work notes
and extracted a complete listing of all mails, each marked with some key topics.
Furthermore I took some notes while interacting with claude and revising the generated text.
The type-independent data transfer is based on a double-dispatch,
which is implemented as visitor pattern; so the operation must start
from the source parameter's VTable and be ''deflected back'' to the receiver
This completes the base operations required to establish a »Domain« (data type)
Adding more test coverage to demonstrate usage patterns...
...turns out as better to code two distinct low-level operations,
albeit implemented by delegation to a single generic assignment implementation.
The visitor patter thus becomes relocated to the high-level interface,
allowing to assign one opaque parameter to another one
However then the problem is that we know ''our domain'' but not the ''other domain''...
... and thus this forces one direction to pass through a double dispatch
- systematic testing of boundary values for various numeric types
- verify even clamping of an uint-128 when targeting float
- verify signed / unsigned of different rank
- demonstrate the clamping now integrated into `extractAs()`
As an aside, move the `isNeg()` function (safe test for negative numbers)
into util.hpp and fix related bugs with unsigned there.
Signed and unsigned integral types have a partial overlap of definition domains,
yet due to the automatic promotion rules it is impossible to write
a domain clamping function directly, since `std::numeric_limits` returns
the domain boundaries in the given type, and `int(-1) < uint(0)`
evaluates to false, due to int being is promoted to uint.
The attempt to circumvent this problem here relies on a check if the
comparison crosses the zero point, and otherwise relies explicitly
on the promotion to the number with the larger domain.
- we need a generic operation to clamp to the smaller value domain
- furthermore we need a virtual function to apply a type specific limiter
__TODO__: this draft does not handle properly signed vs. unsigned types
...so this is not yet a visitor, rather the building blocks required
to be able to extract values and to assign and conform values
And my intention is to build something akin to a set of conversion rules
...after an extended break where I focused on learning IO_URING,
which is the topic of my upcoming talk at Froscon-2026;
(I built a prototype to read and display video with async IO)
Today ''I have re-read my design notes'' and the analysis seems to hold up well;
furthermore I solved the coding problem how to generate type handlers for a list of types.
While it is unfortunate that I seem to get deeper and deeper into that topic...
- it should be noted that data size matters most (pervasive use expected)
- the design is leaning towards multiple functional closures
- both together suggest to use a classical virtual interface, without much ado
- yet parameters must be copyable values, to allow for evaluation graphs and snapshots
- so this turns out as yet another use case for ''polymorphic values''
- furthermore, fluid usage requires flexible conversions between result values
- given the situation with numeric value types and strings, this implies ''to build a visitor''
🠲 ''putting that work aside for the moment, since I need to focus on my IO_URING research (for FrOSCon 26)''
In the current stage, a design-draft is required that is sufficiently adequate
to be used as code placeholder and to evolve into the full specification eventually.
- we need a single parameter container type with value semantics
- parameters must support a selection of basic value types, with conversions
- the value range can be limited, and a parameter can be associated
with a ''Scale'' (stepped, continuous, logarithmic)
- a parameter can be linked to a functional source
- parameters are (virtually) copyable
Parameters and Automation will certainly play a crucial role eventually.
They will form a network of relationships to cover all further concerns that can not directly be conceptualised as arrangement or processing.
A shift in the setup of the Job-Functor arguments necessitates to introduce a `Parameter` type — ''albeit just as placeholder''
As a follow-up to the decisions regarding Render Job arguments,
it should be noted that the `ProcessKey` in the `TurnoutSystem` becomes redundant;
we'd need some access to ''generic Parameters'' rather
...the plan is to wrap the `DataSink` and also to expose possible further control parameters here...
Rather difficult to get started, since mostly related to ''planned features''
This is the result of an requirement analysis,
based on the current (better) understanding of ''render job invocation''
* the `InvocationInstanceID` is @deprecated, because rather a data based hash is required
* instead, two free parameters are introduced, with a meaning defined by the specific `JobFunctor`
* notably the functor for rendering, the `RenderInvocation´ can use these for the exit node and an output context
**Note**: the C-structs for the `job` are no longer considered an interface (and will be remoulded into an internal detail of the dispatcher)
While the setup here in the test seems trivial,
this touches on some challenging questions of design:
Presumably the invocation of a Render Job will need some kind of
''Render Context'' — since the relation of the absolute nominal time
to some output frame number is strictly ''a concern of the render process''.
However, it seems prudent to keep that relation separate from the `RenderInvocation`,
so that access to all invocation parameters has to be supplied by the
Render Job Functor.
This change in the test setup now uses a `DiagnosticOutputSlot`
instead of directly allocating some buffer for output; the data
written into the `DataSink` can be investigated after-the-fact.
turns out that was not (yet) tested and did not even compile,
due to a missing cross-constructor to get from one data allocation
policy to another one backed by the same cluster.
- added test case to `NodeBuilder_test` to demonstrate basic usage
- use `AllocationCluster` in NodeOpera_test and verify that allocations took place
Works as expected, since it was directly adapted from NodeLink_test
The preceding changeset however caused a SEGFAULT,
since the initialisation order in the helper class `RenderPipeline`
was not correct, so that the node builder used a not-yet initialised
`AllocationCluster`....
In NodeLink_test, a processing pipeline was established that
works on `TestFrame` and is quite complex; here I use a slightly
simplified variant (featuring only a single port) to perform
the same kind of computations.
However, node memory allocations are managed by an `AllocationCluster`
The purpose of this test is to invoke a Render Node graph
using the full service environment as present in the Render Engine.
However, for sake of clarity, I do not want to pull up the Engine subsystem,
but rather use a local / mocked setup of all the services involved.
The node invocation is performed from within a worker thread,
to demonstrate integration with the thread-local memory pools.
The first simple test case builds and then pulls a source node.
...after having settled the problems related to buffer management
and the interplay between buffers and output sinks, the ''next goal''
is to drive ahead the integration between job planning and invocation...
The test suite is now **green** again.
The design rework and clean-up of unnecessary test-driven functionality
in the `DiagnosticBufferProvider` and `DiagnosticOutputSlot` required
to break some very old test-setup, using fake instances of the relevant
context facilities. This setup however was used by the newer family
of test support facilities, notable the `MockJob`, `MockSegmentation`
and `MockDispatcher`. These required a set of `DataSink` handles to be
present (even while they are not used currently).
This could be fixed easily now, relying on the simplified internal structure
of the `OutputSlot`: by re-arranging the types, the implied identity of the
`DataSink` handles could be made explicit, allowing for equality comparison.
__Remark__: even while `DataSink` with only a single abstracted function,
using function notation turned out to be rather confusing at usage site.
Naming this function as `lockFrame(frameNo)` clarifies its purpose nicely.
That call is effectively abstracted by means of the ''buffer type'',
relying on the VTable in the `BufferProvider` implementation.
These provide a faked context setup for unit testing.
I created them ''out of sheer desperation'' several years ago,
since it is impossible to verify the building blocks if you
don't know what the building itself will look like.
The recent design clean-up of OutputSlot has unfortunately
removed a bunch of »hexagonal hooks« — and now I seem to be unable
to hook into anything with my tests. Oh my!
- some tests are notoriously flaky
- increase resource limits for the testsuite to allow stronger stress tests
- provide a "quick" variant of BufferProviderLoad_test with lowered requirements
Furthermore, investigate test scaffolding in `DummyBuilderContext`,
which was broken due to the clean-up in `DiagnosticOutputSlot`.
It seems that only some placeholder instances are required here,
so that we might be successful with just instantiating the `UnimplementedConnection`
__TODO__: need to supply slightly more implementation into the latter,
so that we're able at least to ''create'' such a connection stub.
Integrate the work related to buffer management in the render engine.
* in-depth analysis and rework of the `BufferProvider` framework (see #1395)
* build a prototype for production-grade buffer allocation handling (closes: #1415)(see #1396)
* as an aside: Application code and namespace reorganisation
Within the reworked design of the `OutputSlot` a proxy implementation
of the `BufferProvider` API is used (class `OutputBufferProxy`).
After analysis of the invocation patterns, I concluded that it seems adequate
to attach the special-purpose metadata generated by this integration
to the generic system of thread-local replicated `BufferMetadata` storage.
However, since thread-local instances are effectively static(global),
I was concerned that unit-testing might be impeded by accumulation of
stale metadata in the store linked to the main thread. Thus I created
a special configuration variant to retain the previously used setup
for the sake of unit testing (which applies especially to use of the
`DiagnosticOutputSlot`). Using a `SimpleBufferStateRegistry` for in
that case has the benefit that simply a local instance can be used
in the test and all metadata will be discarded afterwards.
Furthermore the factory syntax for obtaining an `OutputSlot` was improved,
drawing on the recent changes to `lib::ScopedCollection`.
Also I looked through some of the documentation and updated
the UML sequence diagram to reflect `OutputSlot`'s improved internal structure
With this third test case, a fairly continuous processing
can be demonstrated, while also coordinating allocations
in a decentralised way.
The implementation behaves correct, and seemingly with
satisfactory performance -- while there is certainly
some room for improvement: with the processing pattern
in this test, a nominal computation load of 4 seconds
can be processed with 4 cores in ~ 1.3 seconds
Yet only in a simple way, by pressing with 100 threads....
This test runs for about 10 seconds, shows a massive allocation
of more than 1 GiB, but comes down clean, without deadlocks,
using all cores and returning all memory blocks at the end.
This test uncovered some missing safety checks,
and it highlighted (again) that this memory management scheme is
elaborate and potentially fragile: I had to deal with carried-on
state in the local metadata registry associated with the main thread
(in fact I did not ever want that registry to be used, but turns out
that the initialisation check, that runs in the constructor, already
pulls up an instance for the main thread, which carries stale state
after the test case is complete)
This is the meat of this test case...
- generate a ''random walk'' with buffer allocation requests
- attempt to make the load pattern nasty, since the goal of this test
is to verify correctness of the complex interplay between local pools
and central allocator.
''correct pattern was verified single-threaded, with print statements''
- provide mock instances of the relevant global services
- use a base config record for the worker thread(s)
- pregenerate ''Buffer Descriptors'' commonly used by the workers
- further access to the buffer manager should then be possible
by locking one of these descriptors...
The intention is to utilise this helper class, that was developed for the
scheduler stress testing; but this time, it should access the actual allocated
render buffers to generate the load.
This implies to use buffers of a significant size; it turned out that
the existing implementation was imprecise, insofar it always used full multiples
of the configured buffer size, which defeats the purpose of performing
a precisely calibrated number of calculation steps. As long as the buffer size
would remain comparatively small, this drift does not matter much.
Thus I changed the implementation now to ensure that exactly the calibrated
number of steps is performed, using a modulus to map it to the buffer memory.
Furthermore, I plan to use a random-walk pattern in the worker thread
to select from a collection of pre-established »buffer types«, similar
to what happens in the actual engine
To verify integration of this new engine-level facility,
it seems prudent to build a load test (without going into
extended stress- and performance testing).
For that we need
- random buffer demand
- a computational load that actually uses the buffer.
As part of the ''scheduler stress testing'', a self-calibrating
`ComputationalLoad` was developed, which even implements both a
merely computational load, but also a memory accessing computation.
This can be extended to optionally use an external buffer, instead
of allocating a precisely dimensioned memory block directly from heap.
Obviously this extension is ''dangerous'', since there is no way to
verify that the storage passed in can accommodated the storage required
based on the current calibration of the `ComputationalLoad`
As an aside: identified a timing check in `SchedulerService_test`
that showed the tendency to fail when invoked immediately after a build.
The mechanism is tuned now to engage rather quickly
whenever excess capacity builds up in a local allocator pool,
but when all buffers are used most of the time, they will be retained.
Can now demonstrate by integration test that capacity is sent back.
The goal is to develop a heuristic when to trigger clean-up of the local pool.
For developing this heuristic, I'll use a unit-test
- represent "allocations" by dummy buffer pointers
- use a queue and a stack to push those "allocations" around
- the "global pool" is just a queue with such dummy pointers
- next step is to define some λ-primitives
- finally, a λ-DSL to allow to perform "N operations of kind X"
This is a fall-back situation,
in case the allocation ''was not announced beforehand''
A direct locked global access is used then, which is verified by this test
...need an elaborate test setup, with the test worker thread
and the main thread proceeding in lockstep, otherwise the
effect would not be visible. Furthermore, the diagnostic
access for LocalMemPool can be re-used here, thereby
mapping requests onto the pool for the current thread.
This makes the test case pass, validating the theory.
Each `AllocReceiver` now integrates a `lib::LiveMark` instance,
and the callbacks are dispatched using the corresponding
`LiveMark::SafeHandle`, so that it is no longer possible
to dispatch allocation messages to a defunct receiver.
Furthermore, the `LiveMark` is explicitly disengaged prior
to unwinding a local pool, so that no further pending requests
can be handled ans dispatched concurrently. The local pool
sends back all the remaining allocations via async queue
to the `EngineBufferManager`
Not that such a shutdown can be triggered any time,
since an exception or failure might cause a worker thread
to unwind, thereby destroying the thread-local `LocalMemPool`
without further notice.
...even while this situation is unlikely, it needs to be considered:
A worker thread might terminate unexpectedly, due to an error;
all thread-local objects will be destroyed automatically — while the
`EngineBufferManager` might be processing further allocations requested
by that very thread, and dispatch them towards an already destroyed `LocalMemPool`
This turns out to be a nasty problem, since the whole setup was created
to allow the workers to operate independently from each other; any handshake
or registration/deregistration mechanism would require a lock of sorts,
thereby defeating the whole purpose of the setup.
The best solution I could find is to rely on a instance marker hash,
that can be checked with an atomic read prior to each dispatch of a
memory allocation. The destructor of the local pool would then be able
to ''disengage'', by invalidating that hash.
To evaluate this solution, I developed this mechanism
as a free-standing library component, called `LiveMark`
