Since the design established thus far calls for a fixed size container
and a size-constrained stack of virtual interfaces and implementation within,
it is clear that more diverse parts of the type and scale handling must be
forwarded into a second layer, maintained within a registry and keyed by ID.
Taking the prospective use into account, a two-level structure with a
fixed-size top-level index table and dynamically added extents seems to
cover all requirements, and allows for a lightweight, atomic-protected
read-access path.
The selection of the intermediary working type, used for exponential
and logarithmic functions when combining values from different scales,
was implemented too simplistic.
Following the example of the `NumberJoinType`, it seems preferrable
to spell out the decision logic explicitly. This allows to pick `long doubl`
as working type especially in cases where 64bit integral parameters are
involved, either as source or destination.
With these final tweaks, the `verify_valueJoining()` test is not complete,
presumably with path coverage of the conversion functions.
The Scale descriptor and operational logic is now defined to the point
that it can be integrated into the general Parameter implementation machinery.
Scrutinising the bounds of numeric precision.
As it turns out, most cases work as expected,
including the massive errors incurred by input-values given as float.
However: a logarithmic combination integral -> integral runs in loss of precision
So the selection logic for the intermediary working type is still overly simplistic!
The control logic for the various combinations of linear and logarithmic scales is nasty,
and the added path coverage indeed unveiled several bugs.
One of these issues, an unjustified limitation on the input value type of `asLogarithm`,
convinced me to refactor the conversion functions into free functions,
since they effectively ''always span the relation of two scales'' --
writing them as member function introduces an unnecessary asymetry.
There are some corner cases where the C++ number promotion rules can hurt us badly.
Devised a type selection logic to pick a suitable common numeric type explicitly,
using `std::common_type<T1,T2>` in most cases, yet handling explicitly the
case when the computation would wrap into unsigned; in such a mixed sign integral case,
we either use int64_t (when its precision suffices), or long double,
which can hold 64bit of mantissa without precision loss.
LIN -> LIN yet either additive or multiplicative
LOG -> LIN
LIN -> LOG
and different kinds of logarithms
all computations using a common intermediary numeric type,
preferably a suitably sized floating-point
The `join()`-operation allows to combine a value pulled from automation
to be combined with a local ''base value'' stored in the parameter.
When both the source and the target parameter use the same Scale,
this operation is simple; it becomes more tricky once they don't --
because then linear / logarithmic and type conversions must be handled.
I now decided to provide a generic implementation in the Scale (descriptor),
while the enclosing parameter shall handle the dispatch when connecting
to a opaque source with another Scale. And this generic implementation
relies on the same technique as was used for the parameter type conversions,
thus it should be factored out into a library header.
As with all those C++ base types, unfortunately they are giving you
only rough size guarantees. For Lumiera, I defined short hand names
some time ago, notably because the explicit long X and short X qualifiers
do not work with ''constructor syntax'' for direct conversions.
But I choose the name `f128` at that time -- which might raise ill-guided
expectations, since the `long double` is not that long on relevant linux 64bit
platforms; it can just hold 64bit of mantissa at least however.
As replacement, I'll use `lflp` now, because it has also 4 letters,
looks somewhat weird (and thus catches attention) and points into the
right direction. I considered `lfloat` as alternative, but that looks
too much like `float`, visually...
finally I was able to come up with a rather concise test
that works with float, double and long double; especially
the aliasing due to rounding could be demonstrated, and
the precision limit is made plausible.
Interestingly, `std::fabs()` really works precise enough
so that a very large positive domain can be implemented
and tested with exact match
The floating-point variant turns out to be difficult to test,
since it is never ''actually exact'' -- due to the nature of floating-points.
We can not expect reasonable behaviour at the domain boundaries,
even while in some cases the computation works (and in others it
goes infinite and is sanitised to `minVal`).
The limit for rejection is itself rather a heuristic;
thus it is difficult to show "wrong" behaviour beyond the limit,
while the limit itself is typically already so large that it is
hard to handle in test code -- at least when expecting visible results.
The implementation should cover all integral data types,
both signed and signed, over the entire value domain.
Verify that by testing the bounds and the complex computation path
These two helpers ''just happen to work'' in the full domain, and also for unsigned types.
I can not recall that I actually was aware of that, but it's nice.
Thus add some test cases to cover the bounds of the domain
Scales are defined as descriptor records and will be used as const.
Besides some information functions, a Scale
* allows to conform a value to be compliant with the Scale's constraints
* can ''join'' and thus apply a ''feed'' on top of a ''base value'';
this is the foundation for applying an automation function, while
handling multiplicative and logarithmic cases properly
It is clear by now that a `Scale` will act as a refinement of a full-domain value type.
However, in order to develop the definition and builder notation for parameters,
a much more detailed understanding of scales is necessary -- including the question
where to draw a line between the complex structured `ParamSet` and a simple
`Parameter` that is conceived to be number-value-like.
...yet the actual problem remains that the value type of the
prototype remains a runtime-dynamic hidden implementation trait.
How can we remould the scale hidden within a parameter, while
in the regular setup, the scale is implemented as an immutable
entry placed into a registry?
Overall, this is a questionable endeavour...
- I want a framework that can be extended flexibly
- I am concerned about the performance penalty of that flexibility
What makes this situation problematic is the fact that I am deep in
prototyping work here, and have not the slightest idea as to what would be
the actual usage context. So I am lacking a point of reference, yet still
I am engaged into a very tricky (premature) optimisation...
Anyway. I have now found a way to integrate an optimised access code-path
in a safe way, while retaining the general case with flexible type conversion:
- whenever a BaseDomain<X> is used, a corresponding type-ID is marked into the ID-Record
- this allows when it is possible to side-step any type conversion,
- in which case we can also forcibly side-step the virtual dispatch,
and invoke directly the value-accessors of the default implementation,
reaching into the embedded value buffer.
Can now express the automatic conversion between all allowed `BaseTypes`
by invoking the `TypeHandler` interface directly. The interfaces were streamlined
and storage size is managed automatically at compile time.
`PolymorphicValue` incurs some quite confusing requirements due to
the clumsy way how the instance handling and copy support must be injected.
As pointed out in #1197, with modern C++, this could be much simplified.
An overhaul of `PolymorphicValue` seems urgent thus -- in the meanwhile
we need to introduce more flexibiltiy with base-class template parameters,
so that we can inject the CopySupport interface at usage site of the
complete implementation class chain. Furthermore, the size of the VTable
is added automatically to the storage requirements in `PolymorphicValue`,
so we must subtract it when specifying the payload size by a compile-time
expression based on `sizeof(ParamData<double>)`
With these tweaks, we can now keep the limit of 4 «slots» (32 bytes)
and still reserve 64bit for ID and type flags.
The solution from yesterday works, but is needlessly indirect.
However, if we want to circumvent ''any indirection'' in the
simple standard case, then it turns out to be necessary to
have some kind of ''flag store'' already in the `Disposition`
(the implementation service interface). Because otherwise
it would be impossible to trigger the short-circuit access
without going through some kind of indirection.
Need to define a generic ID-Record with some flexible extension space
that can be used by subclasses; and notably we need to generate some
base-type-IDs on a static level -- which should be possible, since all
allowed BaseTypes are defined as Type-Sequence.
Add value access implementation of the Provision interface
by direct cast and access to the embedded value in `ParamData<VAL>`;
obviously this is an indirect dispatch, based on the actual type `VAL`
The test demonstrates the basic machinery built thus far
- have an embedded container with inline storage
- dispatch using type information encoded into the VTable
- `Disposition` is the service-API used by the front-End container
- `ParamData<TY>` is a complete implementation with a inline-value buffer
- The front-End is based on the `PolymorphicValue` container from our support-Lib
Finally able to settle upon a layering for the implementation facilities
Provision:: represents the core idea of a parameter
Domain:: assumes the ability to handle typed values
Disposition:: complete implementation capabilities
BaseDomain:: establish type conversion functionality
TypedEntity:: provide a type and ID descriptor and scale configuration
ParamData:: full implementation based on an embedded value buffer
The original design (based on Alexandrescu) is more elegant,
but has the downside to bloat the VTable size due to the mix-in
arrangement used to generate the interface.
The ongoing design has revealed a tight constraint on element size
for the parameters; thus the change to a linearised inheritance chain.
The overall purpose remains the same: the ability to generate a
double-dispatch type conversion, so that the actual value type of the
parameter can be treated as an implicit implementation detail.
While the role of all the relevant interfaces has been determined,
it still remains difficult to establish a sensible relationship.
Which one is the most fundamental interface?
After a lot of mulling, it seems that the `Provision` interface
should be used as foundation, since it describes the very core
of the parameter concept developed here -- even while the actual
form of the operations must be ''somewhat cunningly reversed'',
so that the knowledge regarding the underlying type can be
kept an opaque implementation detail.
The `Domain` interface would then be layered on top, and describes
the capability to transfer values between buffers marked with different
domains (which effectively requires a double-dispatch in the general case).
Last week, I resumed the work that had been suspended
due to my focus on this year’s Froscon talk (on `io_uring`).
I have reconsidered my previous design ideas and sought to
arrive at a coherent design. Nevertheless, these are preliminary,
exploratory design studies that should be regarded as a basis
for future work rather....
The type-independent data transfer is based on a double-dispatch,
which is implemented as visitor pattern; so the operation must start
from the source parameter's VTable and be ''deflected back'' to the receiver
This completes the base operations required to establish a »Domain« (data type)
Adding more test coverage to demonstrate usage patterns...
...turns out as better to code two distinct low-level operations,
albeit implemented by delegation to a single generic assignment implementation.
The visitor patter thus becomes relocated to the high-level interface,
allowing to assign one opaque parameter to another one
However then the problem is that we know ''our domain'' but not the ''other domain''...
... and thus this forces one direction to pass through a double dispatch
- systematic testing of boundary values for various numeric types
- verify even clamping of an uint-128 when targeting float
- verify signed / unsigned of different rank
- demonstrate the clamping now integrated into `extractAs()`
As an aside, move the `isNeg()` function (safe test for negative numbers)
into util.hpp and fix related bugs with unsigned there.
Signed and unsigned integral types have a partial overlap of definition domains,
yet due to the automatic promotion rules it is impossible to write
a domain clamping function directly, since `std::numeric_limits` returns
the domain boundaries in the given type, and `int(-1) < uint(0)`
evaluates to false, due to int being is promoted to uint.
The attempt to circumvent this problem here relies on a check if the
comparison crosses the zero point, and otherwise relies explicitly
on the promotion to the number with the larger domain.
- we need a generic operation to clamp to the smaller value domain
- furthermore we need a virtual function to apply a type specific limiter
__TODO__: this draft does not handle properly signed vs. unsigned types
...so this is not yet a visitor, rather the building blocks required
to be able to extract values and to assign and conform values
And my intention is to build something akin to a set of conversion rules
...after an extended break where I focused on learning IO_URING,
which is the topic of my upcoming talk at Froscon-2026;
(I built a prototype to read and display video with async IO)
Today ''I have re-read my design notes'' and the analysis seems to hold up well;
furthermore I solved the coding problem how to generate type handlers for a list of types.
While it is unfortunate that I seem to get deeper and deeper into that topic...
- it should be noted that data size matters most (pervasive use expected)
- the design is leaning towards multiple functional closures
- both together suggest to use a classical virtual interface, without much ado
- yet parameters must be copyable values, to allow for evaluation graphs and snapshots
- so this turns out as yet another use case for ''polymorphic values''
- furthermore, fluid usage requires flexible conversions between result values
- given the situation with numeric value types and strings, this implies ''to build a visitor''
🠲 ''putting that work aside for the moment, since I need to focus on my IO_URING research (for FrOSCon 26)''
In the current stage, a design-draft is required that is sufficiently adequate
to be used as code placeholder and to evolve into the full specification eventually.
- we need a single parameter container type with value semantics
- parameters must support a selection of basic value types, with conversions
- the value range can be limited, and a parameter can be associated
with a ''Scale'' (stepped, continuous, logarithmic)
- a parameter can be linked to a functional source
- parameters are (virtually) copyable
Parameters and Automation will certainly play a crucial role eventually.
They will form a network of relationships to cover all further concerns that can not directly be conceptualised as arrangement or processing.
A shift in the setup of the Job-Functor arguments necessitates to introduce a `Parameter` type — ''albeit just as placeholder''
- some tests are notoriously flaky
- increase resource limits for the testsuite to allow stronger stress tests
- provide a "quick" variant of BufferProviderLoad_test with lowered requirements
Furthermore, investigate test scaffolding in `DummyBuilderContext`,
which was broken due to the clean-up in `DiagnosticOutputSlot`.
It seems that only some placeholder instances are required here,
so that we might be successful with just instantiating the `UnimplementedConnection`
__TODO__: need to supply slightly more implementation into the latter,
so that we're able at least to ''create'' such a connection stub.
...even while this situation is unlikely, it needs to be considered:
A worker thread might terminate unexpectedly, due to an error;
all thread-local objects will be destroyed automatically — while the
`EngineBufferManager` might be processing further allocations requested
by that very thread, and dispatch them towards an already destroyed `LocalMemPool`
This turns out to be a nasty problem, since the whole setup was created
to allow the workers to operate independently from each other; any handshake
or registration/deregistration mechanism would require a lock of sorts,
thereby defeating the whole purpose of the setup.
The best solution I could find is to rely on a instance marker hash,
that can be checked with an atomic read prior to each dispatch of a
memory allocation. The destructor of the local pool would then be able
to ''disengage'', by invalidating that hash.
To evaluate this solution, I developed this mechanism
as a free-standing library component, called `LiveMark`
This is a collection with unique-ownership,
and the ability to create all content objects in ''RAII style''
from the constructor call, using a ''population functor''.
It is often used to maintain a collection of resource handles,
and especially to start a series of threads (for testing).
With the variadic λ-captures from C++20, it is possible to rewrite
the most frequently used population functors to be easier to use.
Notably we can now bind constructor arguments, that will be passed
to each of the created elements — a prominent example being the
thread function when launching several threads. Furthermore,
I added a constructor variant that consumes a Lumiera Iterator.
Adding this kind of generic variants is possible now with C++20,
since overload resolution takes concepts into account.
Well, finally.
Lumiera Iterators were conceived as a concept from the very beginning,
and this idea dates back even before C++11. Furthermore, over time some
''trait templates'' have been developed, to detect if some template argument
might match the known pattern of functions associated with a Lumiera Iterator.
Meanwhile this concept is used pervasively in the whole code base, and also
the traits templates proved to be reliable. It therefore seems that the time is ripe
to gradually introduce a genuine **concept** `lum_iter`
I'll take a defensive approach here, since any mistake would lead to
a massive breakage of compilation. The new concept is defined simply
by delegating to to the tried-and-tested trait template, which is
also renamed `can_IterForEach` ⟼ `can_LumieraIter`
One immediate benefit is that the compiler is now able to select
overloads based on that concept, which allows to drop the tricky
selection of variants based on the SFINAE technique and `enable_if`.
For validation, one very heavily used case of that pattern was switched
in `lib/format-util.hpp', which notably lies in the code-path used by `util::join`
Since thread_local implies a static setup, we need a way how to
build a new service instance. Usually this would be a default construction,
yet there might be services with special init requirements.
Using an explicit template parameter for this setup configuration
allows to make the primary template parameter just the service itself,
which makes the default usage much clearer.
- Use a ''monostate'' as access point to thread-local service instances
- rely on a configuration-context to setup the actual service
- investigate runtime overhead using a multithreaded microbenchmark
This is a design study to find out what would be required
to build an opaque function wrapper, similar to `std::function`,
yet without using any heap memory ever.
Such a building block would allow for some interesting options
for a well structured design of elaborate data structures.
This changeset provides an implementation draft.
__Two important findings to note__:
* it is impossible to get away with just a single function pointer
* because at the very least we need to invoke the destructor of the
embedded delegate. Thus something akin to a "manager" is required.
* this implies that the component would not be so efficient as I hoped,
since it requires two pointers and requires to emit code for each
type of delegate.
* by making the component **NonCopyable**, I avoided the complexities
of virtual copy support. In case the latter is desired, it would be
much simpler to switch to a classical OO approach with a VTable
and use our `PolymorphicValue` wrapper to handle copy and cloning.
This commit is the aggregated result of extended research
that is only loosely related to current development topics.
One theme was the construction of nested data structures using tuples,
which faces some difficulties due to tuple manipulations being limited
almost entirely to compile-time processing. However, the barrier between
compile-time and runtime can be overcome with the help of a dispatcher
table -- which leads to the secondary problem to handle the diverse
tuple elements with a common, generic functor. Such a functor can be
constructed, if and only if those element types are ''reconcilable''
The STDLIB provides two metafunctions, `std::common_type` and
`std::common_refernce`, which however pose some challenges for the
task at hand, since they include additional processing. In the end,
a custom version of the ''basic type reconciliation'' was implemented
and tested, and then used as foundation to build a dispatcher table
for runtime-indexed access and iteration over tuples.
As an aside, several deeply hidden bugs in our library code were resolved,
and the tuple-helpers, tuple string rendering and associated utilities
were regrouped and modernised.
As result of this research work, the following components are provided:
- meta/common-result
- meta/fun-trampoline
- meta/tuple-index-adaptor
The detailed analysis of corner cases indicated a lot of hidden dangers
when stretching the `IterExplorer` design beyond its limits.
By deliberate choice, this design was kept open so that advanced cohesive
implementation structures can be built. It is even possible ''and tolerated''
to rely on ''references'' to some degree -- assuming however that the
referred memory location remains stable.
As it turned out, these basic assumptions can be broken by a combination
of read-ahead processing (as is necessary for grouping and filtering),
together with adaptor objects embedded as value, especially when
the resulting pipeline object is then moved or copied during evaluation.
While my general attitude towards such extremely clever implementation
constructs is "don't shoot yourself into your foot unless you need to" --
it seems prudent to clean-up some crappy code from the first attempts
towards an iterator pipeline -- notably some of the »Itertools«.
This expands the processing scheme developed thus far, so that
an arbitrary nested iterable can be handled, as long as it can be
adapted by the generic mechanisms provided by IterExplorer.
Notably there is now a type pattern match to distinguish various cases,
even spanning the ''rather esoteric'' possibility that an underlying
iterator produces STL collections by-value.
By examining these special cases, the limitations of the `IteratorExplorer`
design were highlighted, and a number of tricky bugs could be fixed.
