and yes ... this revealed a **long standing bug**
The `Filter::pullFilter()` invocation in the ctor may produce dangling refs,
whenever an underlying source-iterator generates a reference that points
into the iterator itself.
The reason is: due to the »onion shell« design of the iterator pipeline,
we are bound to move a source iterator into the next layer constructor.
With this minor change, the internal result-tuple may now also hold references,
in case a source iterator exposes a reference (which is in fact the standard case).
Under the right circumstances, source-manipulation through the iterator becomes possible.
Moreover, the optimiser should now be able to elide the result-value tuple in many cases.
and access the iterator internals directly instead.
Obviously this is an advanced and possibly dangerous feature, and only possible
when no additional transformer functions are interspersed; moreover this prompted
a review of some long standing type definitions to more precisely reflect the intention.
Note: most deliberately, the Transformer element in IterExplorer must expose a reference type,
and capture the results into an internal ItemWrapper. This is the only way we can support arbitrary functions.
Indeed the solution worked out yesterday could be extracted and turned generic.
Some in-depth testing is necessary though, and possibly some qualifications to allow pass-through of references...
Moreover, last days I started collecting notes regarding problem solving patterns,
which I tend to use frequently, but which might not be obvious and thus can easily
be forgotten. In fact, I had encountered several cases, where I did invent some
roughly similar solution repeatedly, having forgotten about already settled matters.
Hopefully the habit of collecting notes and hints at a central location serves to remedy
Basically I am sick of writing for-loops in those cases
where the actual iteration is based on one or several data sources,
and I just need some damn index counter. Nothing against for-loops
in general — they have their valid uses — sometimes a for-loop is KISS
But in these typical cases, an iterator-based solution would be a
one-liner, when also exploiting the structured bindings of C++17
''I must admit that I want this for a loooooong time —''
...but always got intimidated again when thinking through the fine points.
Basically it „should be dead simple“ — as they say
Well — — it ''is'' simple, after getting the nasty aspects of tuple binding
and reference data types out of the way. Yesterday, while writing those
`TestFrame` test cases (which are again an example where you want to iterate
over two word sequences simultaneously and just compare them), I noticed that
last year I learned about the `std::apply`-to-fold-expression trick, and
that this solution pattern could be adapted to construct a tuple directly,
thereby circumventing most of the problems related to ''perfect forwarding''
So now we have a new util function `mapEach` (defined in `tuple-helper.hpp`)
and I have learned how to make this application completely generic.
As a second step, I implemented a proof-of-concept in `IterZip_test`,
which indeed was not really challenging, because the `IterExplorer`
is so very sophisticated by now and handles most cases with transparent
type-driven adaptors. A lot of work went into `IterExplorer` over the years,
and this pays off now.
The solution works as follows:
* apply the `lib::explore()` constructor function to the varargs
* package the resulting `IterExplorer` instantiations into a tuple
* build a »state core« implementation which just lifts out the three
iterator primitives onto this ''product type'' (i.e. the tuple)
* wrap it in yet another `IterExplorer`
* add a transformer function on top to extract a value-tuple for each ''yield'
As expected, works out-of-the-box, with all conceivable variants and wild
mixes of iterators, const, pointers, references, you name it....
PS: I changed the rendering of unsigned types in diagnostic output
to use the short notation, e.g. `uint` instead of `unsigned int`.
This dramatically improves the legibility of verification strings.
I changed the rendering of unsigned types in diagnostic output
to use the short notation, e.g. `uint` instead of `unsigned int`.
This dramatically improves the legibility of verification strings.
Moreover, I took the opportunigy to look through the existing page
with codeing style guides to explicitly write down some conventions
formed over years of usage.
I did not just »make up« those light heartedly, rather these conventions
are the result of a craftsman's ''attentive observation and self-reflection.''
* based on reproducible data in `TestFrame`
* using Murmur64A hash-chaining to »mark« with a parameter
This emulates the simplest case of 1:1 processing and can also be applied ''in-place''
Change data layout to place a metadata record ''behind the'' payload data,
and add a checksum to allow for validating dummy calculations and also
detect data corruption on data modified after initial generation.
By virtue of a marker data word, the presence of a valid metadata record can be confirmed.
* Lumiera source code always was copyrighted by individual contributors
* there is no entity "Lumiera.org" which holds any copyrights
* Lumiera source code is provided under the GPL Version 2+
== Explanations ==
Lumiera as a whole is distributed under Copyleft, GNU General Public License Version 2 or above.
For this to become legally effective, the ''File COPYING in the root directory is sufficient.''
The licensing header in each file is not strictly necessary, yet considered good practice;
attaching a licence notice increases the likeliness that this information is retained
in case someone extracts individual code files. However, it is not by the presence of some
text, that legally binding licensing terms become effective; rather the fact matters that a
given piece of code was provably copyrighted and published under a license. Even reformatting
the code, renaming some variables or deleting parts of the code will not alter this legal
situation, but rather creates a derivative work, which is likewise covered by the GPL!
The most relevant information in the file header is the notice regarding the
time of the first individual copyright claim. By virtue of this initial copyright,
the first author is entitled to choose the terms of licensing. All further
modifications are permitted and covered by the License. The specific wording
or format of the copyright header is not legally relevant, as long as the
intention to publish under the GPL remains clear. The extended wording was
based on a recommendation by the FSF. It can be shortened, because the full terms
of the license are provided alongside the distribution, in the file COPYING.
⚠ __This is a problematic decision__
It temporarily **breaks compatibility with 32bit** until this issue is resolved.
== Explanation ==
Lumiera relies on a mix of the Standard library and Lib-Boost for calculation of hash values.
Before C++11, the Standard did not support and hashtable implementation; meanwhile, we
got several hash based containers in the STL and a framework for hashes,
which unfortunately is incomplete and cumbersome to use.
The C++ Committee has spend endless discussions and was not able to settle
on a convincing solution without major drawbacks regarding one aspect or the other.
This situation is problematic, since Lumiera relies heavily on the technique
of building stable systematic identifiers based on chained hash values.
It is thus essential to use a strong, reliable and portable hash function.
But unfortunately...
* the standard-fallback solution is known to be weak.
* Lib-Boost automatically uses stronger implementations for 64bit systems
* this implies that Hash-Values **are non-portable**
As the Lumiera project currently has no developer time to expend on such a
difficult and deep topic of fundamental research, today I decided to go down
the path of least resistance and **effectively abandon any system
that can not compile and use the 64bit `hash_combine` implementation.
This changeset extracts code from Lib-Boost 1.67 and adds a static assertion
to **break compilation** on non-64bit-platforms (whatever this means)
After augmenting our `lib/random.hpp` abstraction framework to add the necessary flexibility,
a common seeding scheme was ''built into the Test-Runner.''
* all tests relying on some kind of randomness should invoke `seedRand()`
* this draws a seed from the `entropyGen` — which is also documented in the log
* individual tests can now be launched with `--seed` to force a dedicated seed
* moreover, tests should build a coherent structure of linked generators,
especially when running concurrently. The existing tests were adapted accordingly
All usages of `rand()` in the code base were investigated and replaced
by suitable calls to our abstraction framework; the code base is thus
isolated from the actual implementation, simplifying further adaptation.
Last summer, I already identified a problmatic aspect
which could cause the Scheduler to fall idle without further notice:
5b62438eb4
Basically this situation should raise a **Scheduler-Emergency**,
but the only location where it can be easily detected is way down
in the implementation and has currently no clean way of signalling.
Moreover, how to handle a Scheduler-Emergency is likewise an open
question, an will in turn require even more cross-cutting notifications
and trigger actions somewhere at Render-Engine top-level.
By marking the location where this problem could be detected,
inadvertently I broke the SchedulerCommutator_test, which of course
must execute precisely this logic and check for the proper result.
Yet the problem as such is tricky and possibly far-reaching;
notably when processing long-running render jobs will reliably trigger
this situation — unless we establish additional dedicated control-logic
especially to cope with long-running jobs (opened #1382 for this topic)
__Bottom line__: we are far from addressing any of these issues right now,
and thus I reduced that failure to a warning message, so that at least
`SchedulerCommutator_test` passes again (it's not actually a defect there)
- SchedulerStress_test simply takes too long to complete (~4 min)
and is thus aborted by the testrunner. Add a switch to allow for
a quick smoke test.
- SchedulerCommutator_test aborts due to an unresolved design problem,
which I marked as failure
- add some convenience methods for passing arguments to tests
We use the memory address to detect reference to ''the same language object.''
While primarily a testing tool, this predicate is also used in the
core application at places, especially to prevent self-assignment
and to handle custom allocations.
It turns out that actually we need two flavours for convenient usage
- `isSameObject` uses strict comparison of address and accepts only references
- `isSameAdr` can also accept pointers and even void*, but will dereference pointers
This leads to some further improvements of helper utilities related to memory addresses...
Problems in `Rational_test` were caused by `#include' reorderings regarding ''rational'' and ''intgral'' numbers.
The actual root cause is the fact that `FSecs` is only a typedef,
which prevents us from providing a string conversion for rational numbers without ambiguity
* most usages are drop-in replacements
* occasionally the other convenience functions can be used
* verify call-paths from core code to identify usages
* ensure reseeding for all tests involving some kind of randomness...
__Note__: some tests were not yet converted,
since their usage of randomness is actually not thread-safe.
This problem existed previously, since also `rand()` is not thread safe,
albeit in most cases it is possible to ignore this problem, as
''garbled internal state'' is also somehow „random“
As it turns out, by far margin we mostly use rand() to generate
test values within a limited interval, using the ''modulo trick''
and thus excluding the upper bound.
Looking into the implementation of the distributions in the
libStdC++ shows that ''constructing'' a distribution on-the-fly
is cheap and boils down to checking and then storing the bounds;
so basically there is no need to keep ''cached distribution objects''
around, because for all practical purposes these behave like free functions
What is required occasionally is a non-zero HashValue, and sometimes
an interval of floating-point number or a normal distribution seem useful.
Providing these as free-standing convenience functions,
implicitly accessing the default PRNG.
* add new option to the commandline option parser
* pass this as std::optional to the test-suite constructor
* use this value optionally to inject a fixed value on re-seeding
* provide diagnostic output to show the actual seed value used
this seems to be the ''classical problem situation''
where a »clean« Dependency-Injection would require to waste storage
for a pointer to the same global resource in each and every distinct test class.
Since the Test-Suite is effectively global — even more so now due to
the reliance on "the" global `RandomSequencer` (PNRG) — we'll have to
bite the bullet and access a global static variable hidden behind teh scenes.
...to the base-class of all tests
* `seedRand()` shall be invoked by every test using randomisation
* it will draw a new seed for the implicit default-PRNG
* it will document this seed value
* but when a seed was given via cmdline, it will inject that instead
* `makeRandGen()` will create a new dedicated generator instance,
attached (by seeding) to the current default-PRNG
It is not clear yet how to pass the actual `SeedNucleus`, which
for obvious reasons must be maintained by the `test::Suite`
Using random or pseudo-random numbers as input for tests
can be a very effective tool to spot unintended behaviour in
corner cases, and also helps writing more principled test verifications.
However, investigating failures in randomised tests can be challenging.
A well-proven solution is to exploit the **determinism** of pseudo-random-numbers
by documenting a randomly generated seed, that can be re-injected for investigation.
Up to now, most tests rely on the old library function `rand()`, while
at some places already the C++ standard framework for random number generation
is used, packaged into a custom wrapper. Adding adequate support for
documented seed values seems to be easy to achieve, after switching
existing usages of `rand()` to a suitable drop-in replacement.
After some consideration, I decided ''against'' wiring random generator instances
explicitly, while allowing to do so on occasion, when necessary. Thus
the planned seeding mechanism will rather re-seed a ''implicit default''
generator, which could then be used to construct explicit generator instances
when required (e.g. for multithreaded tests)
As a starting point, this changeset replaces the `randomise()` API call
by a direct access to the ''reseeding functionality'' exposed by the
C++ framework and all default generators. Since we already provide a
dedicated static instance of the plattform entropy source, re-randomisation
can be achieved by seeding from there.
NOTE: there was extended debate in the net, questioning the viability
of the `std::random_seq` -- these arguments, while valid from a theoretical
point of view, seem rather moot when placed into a practical context,
where even 2^32 different generation-paths(cycles) are more than enough
to provide sufficient diffusion of results (unless the goal is really to
engage into Monte-Carlo simulations for scientific research or large model
simulations).
Notable most of the more catchy reprovals raised by Melissa O'Neill
have been refuted by experts of the field, even while being still propagated
at various places in the net, often combined with promoting PCG-Random.
Originally, this helper was called `IterIndex`, thereby following a
common naming scheme of iteration-related facilities in Lumiera, e.g.
* `IterAdapter`
* `IterExplorer`
* `IterSource`
However, I myself was not able to recall this name, and found myself
now for the second time unable to find this piece of code, even while
still able to recall vaguely that I had written something of this kind.
(and unable to find it by a text search for "index", for obvious reasons)
So, on a second thought, the original name is confusing: we do not create
an index of / for iterators; rather we are iterating an index. So this
is what it should be called...
showing the Node-symbol and a reduced rendering of
either the predecessor or a collection of source nodes.
For this we need functionality to traverse the node graph depth-first
and collect all leaf nodes (which are the source nodes without predecessor);
such can be implemented with the help of the expandAll() functionality
of `lib::IterExplorer`. In addition we need to collect, sort and deduplicate
all the source-node specs; since this is a common requirement, a new
convenience builder was added to `lib::IterExplorer`
...taking into account the prospecive usage context
where the builder expressions will be invoked from within
a media-library plug-in, using std::string_view to pass
the symbolic information seems like a good fit, because
the given spec will typically be assembled from some
building blocks, and thus in itself not be literal data.
...as follow-up to yesterday's decisions
- each Port will just feature a (stable) reference to a ProcID record
- which is deduplicated and likewise refers to deduplicated symbolic tags
- and further spec and hash values are computed on-demand by this entity
__Note__: all functionality belonging to the ''Builder'' can be assumed to run **non-concurrent**
Building a precise Frame Cache is a tough job, and is doomed to fail
when attempting to tie cache invalidation to state changes. The only
viable path is to create a system of systematic tagging of processing
steps, and use this as foundation for chained hash values, linked
in accordance to the actual processing structure.
This is complicated by the secondary concern of maintaining memory efficacy
for the render node model, which can be expected to grow to massive scale.
And even while this invocation can not be fully devised right now,
an attempt can be made to build a foundation that is not outright
wasteful, by detaching the logical information from the specific
weaving pattern used for implementation, and by minimising the
representation in memory and computing the compound information
on-demand....
Requirement analysis indicates that a »Node ID« is rather tangential
to the core operation of calculating media; the only infliction point
seems to be the generation of ''systematic cache keys.''
A spec — especially for the `Turnout` however is very relevant for
diagnostics, error reporting and unit testing. So we are in the
difficult situation where rather elaborate functionality is
required only for a secondary concern, and moreover the
node data structure imposes a critical memory leverage.
The immediate next goal is to verify properties of render nodes
generated by the builder framework; two kinds of validations
can be distinguished
* structural aspects of the wiring
* the fact that processing functionality is invoked in proper order
Looking into the structural aspects brings about the necessity
to identify the actual processing function bound into some functor.
Some recapitulation of goals and requirements revealed, that this
can not be a merely technical identity record — because the intention
is to base the ''cache key'' on chained processing node identities,
so that the key is stable as long as the user-visible results will be
equivalent. And while structural data can be aggregated, at the
core this information must be provided by the scheme embedded
into the domain ontology, which is tasked with invoking the
builder in order to implement a ''specific processing-asset''
Review the achievements from the last days and map out the further path
for test-driven build-up of a render-node network and invocation.
Notably ''several layers of prototyping'' are in the works now;
it is important to understand the purpose of each such round of
prototyping and to draw the necessary conclusions after closing out.
The next topic to investigate relates to the ''identity'' of nodes and
ports within nodes; this entails to generate a ''symbolic spec'' that
can be verified and used as base for a systematic hash-ID and cache-key...
Since it would in fact be possible to access and write beyond the configured storage,
simply by using the builder API without considering consistency,
it seems advisable to use explicit runtime checks here, instead of
only assertions, and to throw an exception when violating bounds.
Moreover, unsuccessfully attempted to better arrange the functionality
between PortBuilder and WeavingBuilder; seemingly we have an rather tight
coupling here, and also the expectations regarding the processing function
seem to be too tight (but that's the reason why it's an prototype...)
...which then also allow to fill in the missing parts for the
default 1:1 wiring scheme, which connects each »input slot«
of the processing function with the corresponding ''lead node''
The intention is to offer an automatic 1:1 association
between the »input parameter slots« of the processing function
and the ''lead nodes,'' thereby always using the same default
port, corresponding to the current port number under construction.
Unfortunately, the preceding refactoring removed the information
necessary for a simple implementation, as the port array is now
built up late, in the final build() function...
The next step is to round out the first prototypical implementation,
which requires access to ''lead node ports'' and thereby generally
places focus on the interplay of ''data builders'' within the ongoing
build process. While the prototype still uses the fall-back to simple
heap allocation, notably the intended usage will require to wire-through
the connection to a single `AllocationCluster`. This poses some
challenge, since further ''data builders'' will be added step-wise,
implying that this wiring can not be completed at construction time.
Thus it seems indicated to slightly open-up the internal allocator
policy base template used by `lib::SeveralBuilder` to allow for some
kind of ''cross building'' based on a shared compatible base allocator
type, so that the allocation policy wiring can be passed-on from an
existing `SeveralBuilder`
- the chaining constructor is picked reliably when the
slicing is done by a direct static_cast
- the function definition can be passed reliably in all cases
after it has been ''decayed,'' which is done here simply by
taking it by-value. This is adequate, since the function
definition must be copied / inlined for each invocation.
With these fixes, the simplest test case now for the first time
**runs through without failure**
This change allows to disentangle the usages of `lib::SeveralBuilder`,
so that at any time during the build process only a single instance is
actively populated, all in one row — and thus the required storage can
either be pre-allocated, or dynamically extended and shrinked (when
filling elements into the last `SeveralBuilder` currently activated)
By packaging into a λ-closure, the building of the actual `Port`
implementation objects (≙ `Turnout` instances) is delayed until the
very end of the build process, and then unloaded into yet another
`lib::Several` in one strike. Temporarily, those building functor
objects are „hidden“ in the current stack frame, as a new `NodeBuilder`
instance is dropped off with an adapted type parameter (embedding the
λ-type produced by the last nested `PortBuilder` invocation, while
inheriting from previous ones.
However, defining a special constructor to cause this »chaining«
poses some challenge (regarding overload resolution). Moreover,
since the actual processing function shall be embedded directly
(as opposed to wrapping it into a `std::function`), further problems
can arise when this function is given as a ''function reference''
Conduct in-depth analysis to handle a secondary, implementation-related
(and frankly quite challenging) concern regarding the placement of node
and port connectivity data in memory. The intention is for the low-level
model to use a custom data structure based on `lib::Several`, allowing for
flexible and compact arrangement of the connectivity descriptors within
tiled memory blocks, which can then later be discarded in bulk, whenever
a segment of the render graph is superseded. Yet since the generated
descriptors are heterogeneous and, due to virtual functions, can not be
trivially copied, the corresponding placement invocations on the
data builder API must not be mixed, but rather given in ordered strikes
and preceded by a dimensioning call to pre-reserve a bulk of storage
However, doing so directly would jeopardise the open and flexible nature
of the node builder API, thereby creating a dangerous coupling between
the implementation levels of the node graph and of prospective library
wrapper plug-ins in charge of controlling details of the graph layout.
The solution devised here entails a functional helper data structure
created temporarily within the builder API stack frames; the detailed
and local type information provided from within the library plug-in
can thereby be embedded into opaque builder functors, allowing to
delay the actual data generation up until the final builder step,
at which point the complete number and size requirements of
connectivity data is known and can be used for dimensioning.
This investigation was set off by a warning regarding an
unused argument in `SeveralBuilder`, using `AllocationPolicy::moveElem()`
This warning is correct and easy to fix, but (luckily) it brought my
attention to the fact that a `SeveralBuilder<Port>` can not grow dynamically,
which is somewhat mitigated by the default policy to pre-allocate several
elements, which would work to some degree but waste a lot of memory.
This points to a deeper problem with the implementation pattern used for
all those Builders: they create their product by-value, which must then
be moved into the intended target location.
And doing so is **extremely dangerous**, given that our very goal is to
build a complex data structure internally connected by direct references
and ideally also allocated with a high degree of memory locality.
Unfortunately I do not see any favourable alternative yet;
Ideally all products should be `NonCopyable` — but then, the builder
implementation scheme would become even more complicated and less intuitive
and additionally the client code would need to pre-declare the number of
expected Leads and Ports (not clear if this is even feasible)
...and as expected, this turns up quite some inconsistencies,
especially regarding usage of the »buffer types«.
Basically, the `PortBuilder` is responsible for the high-level functionality
and thus must ensure the nested `WiringBuilder` is addressed and parameterised
properly to connect all »slots« of the processing function.
- can use a helper function in the WiringBuilder to fill in connections
- but the actual buffer types passed over these connectinos are totally
unchecked at that level, and can not see yet how this danger can be
mitigated one level above, where the PortBuilder is used.
- it is still unclear what a »buffer type« actually means; it could
be the pointer type, but it could also imply a class or struct type
to be emplaced into the buffer, which is a special extension to the
`BufferProvider` protocol, yet seems to be used here rather to transport
specific data types required by the actual media handling library (e.g. FFmpeg)
__Analysis__: what kind of verifications are sensible to employ
to cover building, wiring and invocation of render nodes?
Notably, a test should cover requirements and observable functionality,
while ''avoiding direct hard coupling to implementation internals...''
__Draft__: the most simple node builder invocation conceivable...
* decision how to provide a default service for tests
while also allow for configuration of more specific services
* as starting point for the prototype: use the `TrackingHeapBlockProvider`
(simply because this is the only implementation available and tested)
Prototyping and analysis revealed that some aspects of the render node wiring
refers to effectively global services and can thus be taken out of the picture
by relying on classical ''Dependency Injection''
Consequently, `EngineCtx` needs a default implementation, which brings up
a simplistic fall-back version of those services in support for prototyping.
Moreover, dedicated lifecycle functionality must be provided to bring up
and shut down the actual service instances intended for operational use.
...need to pass a binding for the actual processing function
in a way that it acts as a ''prototype'' — since the `Feed`,
i.e. the ''Invocation Adapter'' must be generated for each
invocation anew within the current stack frame
(so to avoid spurious heap allocations)
...seems that the former is well suited to serve as detail builder
used internally by the latter to provide a simplified standard adaptation
for a given processing function.
The integration can be achieved to layer a specialised detail builder class
on top, which can be entered only by specifying the concrete function or lambda
to wrap for the processing; the further builder-API-functions to control
the wiring in detail become thus only accessible after the function type
is known; this allows to place the detail builder as member into the
enclosing port builder and thus to allocate everything within the current
stack frame invoking the builder chain.
...after having determined the several levels of prototyping
currently employed, an important step ahead could be achieved
by analysing the intended and implied usage context of this
builder scheme, while still assuming the simplifications
related to prototyping.
It can be assumed that
* the Level-2 builder object is ''somehow provided''
* the invocation happens from within a media-handling lib-plugin
* alongside with the desired `ProcAsset` spec, an `ExpectationContext`
will be provided, allowing to pass-through additional semantic tags
The implementation in the lib-plugin is then able to draw from specific
knowledge related to the **Domain Ontology** for ''especially for this library''
and provide the necessary wrappers and parameter mapping information.
⟹ the **Level-2**-builder should thus expose an API to
* set up a straight forward mapping, based on a given wrapper functor
to delegate to the actual library invocation
* allow optionally to override some of the input connections
* alternatively allow to use a complete `InvocationAdapter`,
including a `FeedManifold`, as provided directly by the library-plugin
...by relying on DI for some effectively global services, notably
the cache provider, the API for building and wiring render nodes
can be simplified to cover only the actual node connectivity
Doing so directly seems to be a better solution than to inject an OutputBufferProvider;
the latter will still be needed, yet will not be part of the regular weaving pattern,
but used directly at top-level to obtain the output `BuffHandle`, which is then
passed to the `Port::weave()` call
...still not convinced that this is a good design,
since it seems to subvert the general design to treat one special case.
However, I can't see a good way to address this special case directly
There might be one specific output result buffer at top level
for each invocation, which must be delivered into a prepared
output sink. This amounts to one special case, cross-cutting
an otherwise completely generic data flow scheme.
After considering several solutions, it seems most straight-forward
to configure a specific `OutputBufferProvider` to serve as a proxy for
the `OutputSlot` / `DataSink` provided at top-level to the Render-Job.
As an asside, this analyis reveals that the result-slot number does
not belong into the `FeedManifold`, which is dynamic (on the stack);
rather, it's a fixed value configured as part of the `WeavingPattern`
Code clean-up: mark all buffers with a dedicated tagging type
The point in question is: if we work the LocalTag into the type-hash,
could it be possible to miss an existing entry in the metadata registry?
This could cause two entries to be locked for a single buffer address,
leading to data corruption.
As far as I can see, in the current usage this would not happen,
but unfortunately this problem can not be ruled out, since the BufferProvider
API and protocol is designed to be open for various usage patterns.
However, the same potentially disastrous pattern could also materialise
when registering two different buffer types, and then locking each
for the same buffer location.
...this seems to be a tricky aspect; we use hash-chaining to create
derived entries, which may cause the identity of an entry to depend
on the order of specialisation. Looked through the possible code paths,
but these seem to be quite complicated; I see the lurking danger of
creating a second entry (with a different hash), and then in worst case
even locking/unlocking a given buffer twice....
...this is a surprisingly tricky issue, since it undercuts the
generic and recursive implementation of buffer handling;
fortunately I've foreseen such demands may arise down the road
and I've reserved an »Local Key« (now renamed into `LocalTag`),
whose meaning is implementation defined and interpreted by
the specific `BufferProvider`
Requirement analysis shows that the ''actual buffer provider'' to use
constitutes yet another independent degree of freedom, which conceivably
must be handled by the Builder internals rather than by the Domain Ontology.
Thus the simple solution to use a `BuffDescr` to mark the type must be augmented
to also allow configuration of the underlying `BufferProvider`, which generates
the descriptor and can later be invoked with this descriptor to ''lock an actual Buffer.''
In some cases, setup of the buffer types could even be more complicated and require
access to the actual (runtime) invocaton context; such extreme cases however
could be rendered as an extension of the scheme established here,
by storing the (up to now transient) constructor functors persistently.
Which leads to the decision not to care for those extremely complicated
corner cases right now, and thus to construct all buffer descriptors
in the `build()` call
...still fighting to find a suitable API to define
how inputs and outputs are connected and mapped to function parameters.
The solution drafted here uses the reshaped `DataBuilder` (≙`lib::SeveralBuilder`)
to add up connections for each »slot«, disregarding the possibility of permutations.
Similar to `NodeBuilder`, a policy template is used to pass down the setup
for an actual custom allocator.
After applying all the preceding refactorings, it turns out that
the `DataBuilder` defined here ''is essentially `lib::SeveralBuilder`'',
only with a different arrangement of the type parameters, due to the
specific usage context here.
It is thus possible to replace all the interim / helper / rebinding templates
by simple templated typedefs. The only tangible difference is that for
usage in the Builder, a ''selector policy'' is passed as a simple type argument,
which in practice wires the concrete allocator information down into each
sub-builder created during the ongoing construction of a node structure.
redefine the policy for `lib::SeveralBuilder` to be a template-template parameter.
In fact it should have been this way from start, yet defining this kind of
very elaborate code bottom-up lets you sometime miss the wood for the trees
So to restate: `lib::SeveralBuilder` takes a ''policy template,''
which then in turn will be instantiated with the same types `I` (interface)
and `E` (element type) used on `SeveralBuilder` itself. Obviously, there can be
further types involved and thus additional type parameters may be necessary,
notably the ''Allocator'' — yet these are better injected when ''defining''
the policy template itself.
The default binding for this policy template is defined as `allo::HeapOwn`,
which causes the builder to allocate the storage extents through the standard
heap allocator, and for the created `lib::Several` to take full ownership of
embedded objects, invoking their destructors when falling out of scope.
As a direct consequence of the insights regarding Dependency-Injection,
a ''Builder Toolkit'' is required, which can be used to adapt various
kinds of ''Weaving Patterns'' — since obviously it is not possible to
settle down on a single Pattern, and thus several ''families of builders''
will emerge, one for each ''line of construction'' for ''Weaving Patterns''.
To stress this point, what I am coding here is a prototype, aimed at
being used as part of a **Test Domain Ontology** — and other Domain Ontologies
(e.g. für FFmpeg) will certainly require other construction schemes
for their Weaving Patterns. So this is an open field, and can not be
settled once and for all.
This immediately leads to another, rather technical problem:
If we're about to work with ''delegate Builders,'' then also
a way to pass-down the allocator configuration is required.
We had settled on a preliminary solution with the helper `DataBuilder`,
yet this solution looks like it defines how `lib::SeveralBuilder`
should be used in most of the cases. So there is now a conflict
between the existing definition scheme for `lib::SeveralBuilder`,
which was achieved in a bottom-up way, and a slightly different
definition scheme ''as it should be''
Starting to attack this latter detail problem, as a first step,
the definition of `DataBuilder` can be simplified by collapsing
it with the `lib::allo::SetupSeveral`
It became clear that a secondary system of connections must be added,
running top-down from a global model context, and thus contrary to the
regular orientation of the node network, which connects upwards from
predecessor to successor, in accordance with the pull principle.
If we accept this wiring as part of the primary structure, it can be
established immediately while building the nodes, thus adding a preconfigured
''pattern of Buffer Descriptors'' to each node, since there is no further
''moving part'' — beyond the wiring to the `BufferProvider`, which thus
becomes part of a global `ModelContext`
As an immediate consequence, the storage for this configuraion should
also be switched to `lib::Several` and handled similar to the primary
node wiring in the Builder...
It seems we need a `WeavingPattern`-Builder, which obviously
must be rather flexible, since those patterns are to be composed
from several layers, which should be extensible within a given ''Domain Ontology''
So this seems to lead to a builder-DSL which creates »**onion layers**«
of builders, with the ability to extend and specialise the type on each layer.
''As it will be quite challenging to get this into usable shape,
it seems best to approach this step by step through prototyping...''
Not entirely sure how to use the `emit()` call properly,
assuming that it means that data is complete in buffer,
but can still be read after that point
* at least for a simple, prototypical setup
* and actually shifting the onerous into the Level-1 builder \\
''(which is precisely the intention here)''
The deeper problem is that we must not engage into any premature decisions
regarding the structure or layout of the actual processing function invocation.
Thus attempting to create a kind of »firewall« of sorts, by connecting
the building blocks strictly through template parameter and preferably
figuring out any detailed knowledge locally, through ''compile-time introspection...''
...even the initial effort to stub its operation turns into a
challenge, since honestly there is near nothing we can assume safely,
without sliding into uncovered provisions regarding the ''Domain Ontology''
- it is clear that this adaptor will be a ''Concept''
- yet it must in some way access the `FeedManifold` and also control additional storage
- a rather obvious solution is to layer it ''on top'' of the manifold
...which brings about various (preliminary) decisions regarding
Metadata storage in the `Turnout`-object, which acts as a guidance
and specification for the actual invocation for this specific node.
As starting point, I choose the ''KISS'' solution of embedding some
blocks of `UninitialisedStorage` directly into the `Turnout`; obviously
these blocks must be oversized, since we can not effort emitting a
dedicated template instance for each different count of input / output
feeds. Moreover, these data buffers are assumed to be filled with
valid objects by the builder ''(this is a lurking danger)''
...turns out that the intended structure is still too fine grained
and explicit and many operational steps can be collapsed into a single
virtual scope, wherein they can be deemed implementation detail...
...so the solution is to build up the working data as `lib::SeveralBuilder`;
however, a more concise notation can be achieved with a suitably configured
wrapping subclass; together with the cross-builder trick, this allows
to write the allocation configuration in a clearly libelled way,
while the field definition and the builder constructor hides the
complexities of picking up the extension point and passing on the
wiring to the allocator instance.
...using the same pattern here as was successful also for the underlying lib::SeveralBuilder;
even if it may seem logically backwards, it just reads much better and
is more understandable, and has the added benefit of providing a dedicated
definition scope, which can be kept separate from the constructor definition
of the actual builder
{{{
prepareNode()
.withAllocator<XYZ>()
.addLead(predecessor)
.build()
}}}
...turns out to be surprisingly tricky, since the nested
lib::SeveralBuilder instances require parametrisation by a
''policy template,'' which in turn relies on the actual allocator.
And we want to provide the allocator as a constructor parameter,
including the ability to pick up a custom specialisation for
some specific allocator (notably AllocationCluster requires
to hook into this kind of extension point, to be able to
employ its dedicated API for dynamic allocation adjustment)
* conduct analysis regarding allocator handling in the Builder
* turns out we'll have to keep around two different allocators while building
* ⟹ establish the goal to confine usage of the Node allocator to the lower Levels
* consequently must open up the `lib::SeveralBuilder` to be usable
as an intermediary data structure, while building up the target data
* in the initial design, the `SeveralBuilder` was kept opaque, since
contents can be expected to be re-located frequently and thus exposing
elements and taking references could be dangerous — yet this is also
true for `std::vector` however, so people are assumed to know
when they want to shoot themselves into their own foot
...especially what is necessary to represent at this level and what information
is implicit; notably there will be an implicit default wiring, but we allow
for case-by-case deviations
The Builder will have to perform several passes, gradually refining
the model into the low-level Render Node network. Right now, some
guesses regarding the last steps of this process are possible,
thus defining the lowest level of a model builder structure
* Level-3 : mapping data flow paths
* Level-2 : detailed configuration of data buffer passing
* Level-1 : build the actual parameter structures for invocation
In the current »Vertical Slice« we're able to fully define Level-1
and maybe Level-2
To escape a possible deadlock in analysis, I resort to developing
some kind of free-wheeling presupposition how the **Builder** could
be implemented — a centrepiece of the Lumiera architecture envisioned
thus far — which ''unfortunately'' can only be planned and developed
in a more solid way ''after'' the current »Vertical Slice« is completed.
Thus I find myself in the uncomfortable situation of having to work towards
a core piece, which can not yet be built, since it relies heavily on
the very structures to be built...
...the complexity of details is a nightmare
...still fighting to grasp a generic structure allowing to ''fold down''
the details into the specific ''domain ontologies'' for the media libraries
...and this line of analysis brings us deep into the ''Buffer Provider''
concept developed in 2012 — which appears to be very well to the point
and stands the test of time.
Adding some ''variadic arguments'' at the right place surprisingly leads
to an ''extension point'' — which in turn directly taps into the
still quite uncharted territory interfacing to a **Domain Ontology**;
the latter is assumed to define how to deal with entities and relationships
defined by some media handling library like e.g. FFmpeg.
So what we're set to do here is actually ''ontology mapping....''
The immediate next step is to build some render nodes directly
in a test setting, without using any kind of ''node factory.''
Getting ahead with this task requires to identify the constituents
to be represented on the first code layer for the reworked code
(here ''first layer'' means any part that are ''not'' supplied
by generic, templated building blocks).
Notably we need to build a descriptor for the `FeedManifold` —
which in turn implies we have to decide on some fundamental aspects
of handling buffers in the render process.
To allow rework of the `ProcNode` connectivity, a lot of presumably obsoleted
draft code from 2011 has to be detached, to be able to keep it in-tree
for further reference (until the rework and refactoring is settled).
As outlined in #1367, the integration effort requires some rework
of existing code, which will be driven ahead by the `NodeLinkage_test`
* redefine Node Connectivity
* build simple `ProcNode` directly in scope
* create an `TurnoutSystem` instance
* perform a ''dummy Node-Invocation''
As a replacement for the `RefArray` a new generic container
has been implemented and tested, in interplay with `AllocationCluster`
* the front-end container `lib::Several<I>` exposes only a reference
to the ''interface type'' `I`, while hiding any storage details
* data can only be populated through the `lib::SeveralBuilder`
* a lot of flexibility is allowed for the actual element data types
* element storage is maintained in a storage extent, managed through
a custom allocator (defaulting to `std::allocator` ⟹ heap storage)
The `SeveralBuilder` employs the same tactic as `std::vector`,
by over-allocating a reserve buffer, which grows in exponential
increments, to amortise better the costs of re-allocation.
This tactic does not play well with space limited allocators
like `AllocationCluster` however; it is thus necessary to provide
an extension point where the actuall allocator's limitation can be
queried, allowing to use what is available as reserve, but not more.
With these adaptations, a full usage cycle backed by `AllocationCluster`
can be demonstrated, including variations of dynamic allocation adjustment.
...identified as part of bug investigation
* make clear that reserve() prepares for an absolute capacity
* clarify that, to the contrary, ensureStorageCapaciy() means the delta
Moreover, it turns out that the assertion regarding storage limits
triggers frequently while writing the test code; so we can conclude
that the `AllocationCluster` interface lures into allocating without
previous check. Consequently, this check now throws a runtime exception.
As an aside, the size limitation should be accessible on the interface,
similar to `std::vector::max_size()`
By means of the extension point, which produces a dedicated policy
for use with `AllocationCluster`, it becomes possible to use the
specialised API to adjust the latest allocation in the cluster.
When this is not actually usable, the policy will fall back
on the standard implementation (which is wasteful when
applied to `AllocationCluster`, since memory for the
obsoleted, smaller blocks not de-allocated then...
- decided to allow creating empty lib::Several;
no need to be overly rigid in this point,
since it is move-assignable anyway...
- populate with enough elements to provoke several reallocations
with copying over the existing elements
- precisely calculate and verify the expected allocation size
- verify the use-count due to dedicated allocator instances
being embedded into both the builder and hidden in the deleter
- move-assign data
- all checksums go to zero at end
The setup for `ArrayBucket` is special, insofar it shell de-allocate itself,
which creates the danger of re-entrant calls, or to the contrary, the danger
to invoke this clean-up function without actually invoking the destructor.
These problems become relevant once the destructor function itself is statefull,
as is the case when embedding a non-trivial, instance bound allocator
to be used for the clean-up work. Using the new `lib::TrackingAllocator`
highlighted this potential problem, since the allocator maintains a use-count.
Thus I decided to move the »destruction mechanics« one level down into
a dedicated and well encapsulated base class; invoking ArrayBucket's destructor
thereby becomes the only way to trigger the clean-up, and even ElementFactory::destroy()
can now safely check if the destructor was already invoked, and otherwise
re-invoke itself through this embedded destructor function. Moreover,
as an additional safety measure, the actual destructor function is now
moved into the local stack frame of the object's destructor call, removing
any possibility for the de-allocation to interfere with the destructor
invokation itself
part of the observed deviation stems form bugs in logging and checksum calculation;
but there seems to be a real problem hidden in the allocator usage of the
new component, since the use-cnt of the handle does not drop to zero
While there might be the possibility to use the magic of the standard library,
it seems prudent rather to handle this insidious problem explicitly,
to make clear what is going on here.
To allow for such explicit alignment handling, I have now changed the
scheme of the storage definition; the actual buffer now starts ''behind''
the `ArrayBucket<I>` object, which thereby becomes a metadata managing header.
__To summarise the problem__: since we are maintaining a dynamically sized buffer,
and since we do not want to expose the actual element type through the
front-end object, we're necessarily bound to perform a raw-memory allocation.
This is denoted in bytes, and thus the allocator can no longer manage
the proper alignment automatically. Rather, we get a storage buffer with
just ''some accidental'' alignment, and we must care to request a sufficient
overhead to be able to shift the actual storage area forward to the next
proper alignment boundary. Obviously this also implies that we must
store this individual padding adjustment somewhere in the metadata,
in order to be able to report the correct size of the block later
on de-allocation.
The solution implemented thus far turns out to be not sufficient
for ''over-aligned-data'', as the raw-allocator can not perform the
''magic work'' because we're exposing only `std::byte` data.
This adaptor works in concert with the generic allocator
building blocks (prospective ''Concepts'') and automatically
registers a either static or dynamic back-link to the factory
for clean-up.
Use this wrapper fore more in-depth test of the new `TrackingAllocator`
and verify proper behaviour through the `EventLog`
- create two vectors, attached to the `TrackingAllocator`
- emplace Tracker-Objects
- move an object to the other vector
- destroy the containers
🠲 Event-Log looks plausible!
- use a meta-registry of pools
- retrieve and manage the `MemoryPool` instances by shared_ptr, with a weak registry entry
- use a hastable for the allocations, keyed by the allocated memory address
- ability to verify a hash-checksum
- ability to watch number of allocations and allotted bytes
- using either a common global pool or a separate dedicated pool
- log all operations into a common `EventLog` instance
- front-end adaptors for use as C++ custom allocator
...these features are now used quite regularly,
and so a dedicated documentation test seems indicated.
Actually my intention is to add a tracking allocator to these test helpers
(and then to use that to verify the custom allocator usage of `lib::Several`)
Phew... this was a tough one — and not sure yet if this even remotely works...
Anyway, the `lib::SeveralBuilder` is already prepared for collaboration with a
custom allocator, since it delegates all memory handling through a base policy,
which in turn relies on std::allocator_traits.
The challenge however is to find a way...
* to make this clear and easy to use
* to expose an extension point for specific tweaks
* and to make all this work without excessive header cross dependencies
This is a low-level interface to allow changing the size of
the currently latest allocation in `AllocationCluster`; a client
aware of this capability can perform a real »in-place re-alloc«,
assuming the very specific usage constraints can be met.
`lib::Several<X>` will use this feature when attached to an
`AllocationCluster`; with this special setup, an previously
unknown number of non-copyable objects can be built without
wasting any storage, as long as the storage reserve in the
current extent of the `AllocationCluster` is sufficient.
...use some pointer arithmetic for this test to verify
some important cases of object placement empirically.
Note: there is possibly a very special problematic case
when ''over aligned objects'' are not placed in accordance
to their alignment requirements. Fixing this problem would
be non-trivial, and thus I have only left a note in #1204
- spread change now retains the nominal element reserve
- `capacity()` and `capReserve()` now exposed on the builder API
- factor out the handling check safety functions
- rewrite the `resize()` builder function to be more generic
__Test now covers__ example with trivial data type, which can
indeed be resized and allows to grow buffer on-the fly without
requiring any knowledge of the actual type (due to using `memmove`)
building on the preceding analysis, we can now demonstrate that
the container is initially able to grow, but looses this capability
after accepting one element of unknown subclass type...
`lib::Several` is designed to be highly adaptable, allowing for
several quite distinct usage styles. On the downside, this requires
to perform some checks at runtime only, since the ability to handle
some element depends on specific circumstances.
This is a notable difference to `std::vector`, which is simply not capable
of handling ''non-copyable'' types, even if given an up-front memory reservation.
The last test case provided with the previous changeset did not trigger
an exception, but closer investigation revealed that this is correct,
since in this specific situation the container can accept this object type,
thereby just loosing the ability to move-relocate further objects.
A slightly re-arranged test scenario can be used to demonstrate this fine point.
- the test-dummy objects need a `noexcept` move ctor
- **bug** here: need an explicit check to prevent other types
than the known element type from ''sneaking in''
The `SeveralBuilder` is very flexible with respect to added elements,
but it will investigate the provided type information and reject any
further build operation that can not be carried out safely.
...turns out that we must ensure to pass a plain "object" type
to the standard allocator framework (no const, no references).
Here, ''object in C++ terminology'' means a scalar or record type,
but no functor, no references and no void,
Consider what (not) to support.
Notably I decided ''not to support'' moving out of an iterator,
since doing so would contradict the fundamental assumptions of
the »Lumiera Forward Iterator« Concept.
Start verifying some variations of element placement,
still focussing on the simple cases
Parts of the decision logic for element handling was packaged
as separate »strategy« class — but this turned out to be neither
a real abstraction, nor configurable in any way. Thus it is better
to simplify the structure and turn these type predicates into simple
private member functions of the SeveralBuilder itself
Elements maintained within the storage should be placed such
as to comply with their alignment requirements; the element spacing
thus must be increased to be a multiple of the given type's alignment.
This solution works in most common cases, where the alignement is
not larger as the platform's bus width (typically 64bit); but for
''over-aligned types'' this scheme may still generate wrong object
start positions (a completely correct solution would require to
add a fixed offset to the beginning of the storage array and also
to capture the alignment requirements during population and to
re-check for each new type.
...and the nice thing is, the recently built `IterIndex` iteration wrapper
covers this functionality right away, simply because `lib::Several`
is a generic container with subscript operator.
...passes the simplest unit test
* create a Several<int>
* populate from `std::initializer_list`
* random-access to elements
''next step would be to implement iteration''
After some fruitless attempts, I settled for using std::function directly,
in order to establish a working baseline of this (tremendously complicated)
allocation logic. Storing a std::function in the ArrayBucket is certainly
wasteful (it costs 4 »slots« of memory), but has the upside that
it handles all those tricky corner cases magically; notably
the functor can be stored completely inline in the most relevant
case where the allocator is a monostate; moreover we bind a lambda,
which can be optimised very effectively, so that in the simplest case
there will be only the single indirection through the ''invoker''.
This **completes the code path for a simple usage cycle**
🠲 ''and hooray ... the test crashes with a double-free''
- ensure the ''deleter function'' is invoked
- care for proper ''deleter'' setup in case of exception while copying
- need to »lock-in« on one specific kind of ''destructor invocation scheme,''
since we do not keep track of individual concrete element types
Parts of this logic were first coded down in the `realloc` template method,
where it did not really belong; thus reintegrate similar logic one level above,
in the SeveralBuilder::adjustStorage(). Moreover, for performance reasons,
always start with an initial chunk, similar to what `std::vector` does...
since this is meant as a policy implementation, reduce it to the bare operation;
the actual container storage handling logic shall be implemented in the container
and based on those primitive and configurable base operations
...still fighting to get the design of the `AllocationPolicy`
settled to work well with `AllocationCluster` while also allowing
to handle data types which are (not) trivially copyable.
This changeset attempts to turn the logic round: now we capture
an ''move exclusion flag'' and otherwise allow the Policy to
decide on its own, based on the ''element type''
- verifies if new element can just fit in
- otherwise ensure the storage adjustments are basically possible
- throw exception in case the new element can not be accommodated
- else request possible storage adjustments
- and finally let the allocator place the new element
Draft skeleton of the logic for element creation.
This turns out to be a rather challenging piece of code,
since we have to rely on logical reasoning about properties
of the element types in order to decide if and how these
elements can be emplaced, including the possibility to
re-allocate and move existing data to a new location.
- if we know the exact element type, we can handle any
copyable or movable object
- however, if the container is filled with a mixture of types,
we can not re-allocate or grow dynamically, unless all data
is trivially copyable (and can thus be handled through memmove)
- moreover we must ensure the ability to invoke the proper destructor
In-depth analysis of storage management revealed a misconception
with respect to possible storage optimisations, requiring more
metadata fields to handle all corner cases correctly.
It seems prudent to avoid any but the most obvious optimisations
and wait for real-world usage for a better understanding of the
prevalent access patterns. However, in preparation for any future
optimisations, all access coordination and storage metadata is
now relocated into the `ArrayBucket`, and thus resides within the
managed allocation, allowing for localised layout optimisations.
To place this into context: the expected prevalent use case is
for the »Render Nodes Network«, which relies on `AllocationCluster`
for storage management; most nodes will have only a single predecessor
or successor, leading to a large number of lib::Several intsances
populated with a single data element. In such a scenario, it is
indeed rather wasteful to allocate four »slot« of metadata for
each container instance; even more so since most of this
metadata is not even required in such a scenario.
...which basically ''seems doable'' now, yet turns up several unsolved problems
- need a way to handle excess storage for the raw allocation
- generally should relocate all metadata into the ArrayBucket
- mismatch at various APIs; must re-think where to pass size explicitly
- unclear yet how and where to pass the actual element type to create
...turns out to be rather challenging, due to the far reaching requirements
* the default case (heap allocation) ''must work out-of-the box''
* optionally a C++ standard conformant `Allocator` can be adapted
* which works correct even in case this allocator is ''not a monostate''
* **essential requirement** is to pass an `AllocationCluster` reference directly
* need a ''generic extension point'' to adapt to similar elaborate custom schemes
__Note__: especially we want to create a direct collaboration between the allocation policy and the underlying allocator to allow support for a dedicate ''realloc operation''
- code spelled out as intended, according to generic scheme
- can now encode the »unmanaged« case directly as `null`-deleter,
because in all other cases a deleter function is mandatory now
- add default constructor to `ArrayBucket`, detailing the default spread
even while at first sight only a ''deleter instance'' is required,
it seems prudent to rearrange the code in accordance to the prospective
Allocator / Object Factory concept, and rather try to incorporate
the specifics of the memory layout into this generic view, thereby
abstracting the actual allocator away.
This can be achieved by using a standard-allocator for `std::byte`
as the base allocator and treat each individual element allocator
as a specialised cross-allocator (assuming that this cross adaptation
is actually trivial in almost all cases)
The fundamental decision is that we want to have a single generic front-end,
meaning that we must jump dynamically into a configured deleter function.
And on top of that comes the additional requirement that ''some allocators''
are in fact tied to a specific instance, while other allocators are monostate.
However, we can distinguish both by probing if the allocator can be default constructed,
and if a default constructed allocator is equivalent to the currently used alloctor instance.
If this test fails, we must indeed maintain a single allocator instance,
and (to avoid overengineering for this rather special use case) we will
place this allocator instance into heap memory then, with a self-cleanup mechanism
On the other hand, all monostate allocators can be handled through static trapolines.
- the basic decision is to implement ''realloc'' similar to `std::vector`
- however the situation is complicated by the desire to allow arbitrary element types
- ⟹ must build a strategy based on the properties of the target type
- the completely dynamic growth is only possibly for trivially-movable types
- can introduce a dedicated ''element type'' though, and store a trampolin handler
- create by forwarding allocator arguments to policy
- builder-Op to append from iterator
- decide to collapse the ArrayBucket class, since
access is going through unsafe pointer arithmetic anyway
- favour dynamic polymorphism
- use additional memory for management data alongside the element allocation
- encode a flag and a deleter pointer to enable ownership of the allocation
- inherit base container privately into builder, so the build ends with a slice
Some decisions
- use a single template with policy base
- population via separate builder class
- implemented similar to vector (start/end)
- but able to hold larger (subclass) objects
- basically works out-of-the-box now
- the hard wired fixed Extent size is a serious limitation
- however, this is not the intended primary use, rather complementary
...this is an important detail: quite commonly, a custom allocator
is actually implemented as monostate, to avoid bloating every client container
with a backlink pointer; by inheriting the `StdFactory` adapter from the
allocator, the empty-base optimisation can be exploited.
In the standard case thus LinkedElements is the same size as a single
pointer, which is already exploited at several places in the code base.
Notably `AllocationCluster` uses a »virtual overlay« to dress-up the
position pointer as `LinkedElements`, allowing to delegate most of the
administration and memory management to existing and verified code.
With this adjustments, `LinkedElements` pass the tests again
and the rework of `AllocationCluster` is considered complete.
This is the first validation of the new design:
the policy to take ownership can be reimplemented simply
by delegating to the adaptor for a C++ standard allocator
The following structure can be expected, after __switching to C++20__
* Concept **Allocator** deals with the bare memory allocation
* Concept **Factory** handles object creation and disposal by delegation
* Concept **Handle** is a ready-made functor for dependency-injection
Right now, an implementation of the ''prospective Factory Concept''
can be provided, by delegating through `std::allocator_traits` to a given
`std::allocator` or compatible object
By default, LinkedElements uses a policy OwningHeapAllocated;
while retaining this interface, this policy should be recast
to rely on a standard compliant allocator, with a default
fallback to `std::allocator<T>`
This way, a single policy would serve all the cases where
objects are actually owned and managed by `LinkedElements`,
and most special policies would be redundant.
This turns out to be quite tedious and technical however,
since the newer standard mandates to use std::allocator_traits
as front-end, and moreover the standard allocators are always
tied to one specific target type, while `LinkedElements` is
deliberately used to maintain a polymorphic sequence.
since the calculation to find the current block start
has been recast as a private method, it is now possible to
calculate the allocation statistics without mutating the pos pointer.
To enable such usages, add a wrapper for `LinkedElements` to expose
an element-pointer temporarily as a immutable `LinkedElements` list,
allowing to iterate or use subscript and size information functions
...what I've implemented yesterday is effectively the same functionality
as provided automatically by the C++ object system when using a virtual destructor.
Thus a much cleaner solution is to turn `Destructor` into a interface
and let C++ do all the hard work.
Verified in test: works as intended
This is the first draft, implementing the invocation explicitly
through a trampoline function. While it seems to work,
the formulation can probably be simplified....
These diagnostics helpers must rely on low-level trickery,
since the implementation strives at avoiding unnecessary storage overhead.
Since `AllocationCluster` is move-only (for good reasons) and `StorageManager`
can not be constructed independently, a »backdoor« is created by
forced cast, relying on the known memory layout
- rather accept hard-wired limits than making the implementation excessively generic
- by exploiting the layout, the administrative overhead can be reduced significantly
- the trick with the "virtual managment overlay" allows to hand-off most of the
clean-up work to C++ destructor invocation
- it is important to verify these low-level arrangements explicitly by unit-test
* this is pure old-style low-level trickery
* using a layout trick, the `AllocationCluster`
can be operated with the bare minimum of overhead
* this trick relies on the memory layout of `lib::LinkedElements`
...due to the decision to use a much simpler allocation scheme
to increase probability for actual savings, after switching the API
and removing all trading related aspects, a lot of further code is obsoleted
Notably this raises the difficult question,
whether to ensure **invocation of destructors**.
Not invoking dtors ''breaks one of the most fundamental contracts''
of the C++ language — yet the infrastructure to invoke dtors in such
a heterogeneous cluster of allocations creates a hugely significant
overhead and is bound to poison the caches (objects to be deallocated
typically sit in cold memory pages).
What makes this decision especially daunting is the fact that the
low-level-Model can be expected to be one of the largest systemic
data structures (letting aside the media buffers).
I am leaning towards a compromise: turn down this decision
towards the user of the `AllocationCluster`
After some analysis, it became clear that the existing code for
`AllocationCluster` (while in itself valid) will likely miss the point
for the expected usage in the low-level Model: most segments of the
model will be rather small, and thus there is not enough potential for
amortisation when using such a per-type and per-segment scheme;
a rather simplistic linear allocator will be sufficient.
On the other hand, with the current C++ standard it is easy to provide
a complient allocator implementation for STL containers, and thus the
interface should be retro-fitted accordingly.
At the time of the initial design attempts, I naively created a
classic interface to describe an fixed container allocated ''elsewhere.''
Meanwhile the C++ language has evolved and this whole idea looks
much more as if it could be a ''Concept'' (C++20). Moreover, having
several implementations of such a container interface is deemed inadequate,
since it would necessitate ''at least two indirections'' — while
going the Concept + Template route would allow to work without any
indirection, given our current understanding that the `ProcNode` itself
is ''not an interface'' — rather a building block.
- the starting point is the idea to build a dedicated ''turnout system''
- `StateAdapter`, `BuffTable` ⟶ `FeedManifold` and _Invocation_ will be fused
- actually, the `TurnoutSystem` will be ''pulled'' and orchestrate the invocation
- the structure is assumed to be recursive
The essence of the Node-Invocation, as developed 2009 / 2011 remains intact,
yet it will be organised along a clearer structure
Within the existing body of code, there are two unfinished attempts
towards building a node invocation and management of data buffers.
The first attempt was entirely driven from the angle of invoking a
processing function, while the second one draws from a wider scope
and can be considered the solution to build upon regarding data buffers
in general. However, the results of the first approach are well suited
for their specific purpose, so both solutions will be combined.
Thus the arrangement of data feeds going in and out of the render node
shall be renamed into `BuffTable` -> `FeedManifold`
...which seems to be basically fine thus far
...beyond some renaming and rearranging
''it turns out that the final, crucial links,
necessary to tie all together, are yet to be developed''
the `BreakingPoint` tool conducts a binary search to find the ''stress factor''
where a given schedule breaks. There are some known deviations related to the
measurement setup, which unfortunately impact the interpretation of the
''stress factor'' scale. Earlier, an attempt was made, to watch those factors
empirically and work a ''form factor'' into the ''effective stress factor''
used to guide this measurement method.
Closer investigation with extended and elastic load patters now revealed
a strong tendency of the Scheduler to scale down the work resources when not
fully loaded. This may be mistaken by the above mentioned adjustments as a sign
of a structural limiation of the possible concurrency.
Thus, as a mitigation, those adjustments are now only performed at the
beginning of the measurement series, and also only when the stress factor
is high (implying that the scheduler is actually overloaded and thus has
no incentive for scaling down).
These observations indicate that the »Breaking Point« search must be taken
with a grain of salt: Especially when the test load does ''not'' contain
a high degree of inter dependencies, it will be ''stretched elastically''
rather than outright broken. And under such circumstances, this measurement
actually gauges the Scheduler's ability to comply to an established
load and computation goal.
...well — more of a logical contradiction, not so much a bug.
The underlying problematic situation arises when meanwhile the
Extent storage has been expanded, and especially the active slots
are in »wrapped state«. In this case, the newly allocated extents
must be rotated in, which invalidates existing index numbers.
This problem was amended by exploting a chaching mechanism, allowing
to re-attach and validate an index position still stored in an old
iterator; especially this can happen when attempting to attach a
follow-up dependency onto a job planned earlier, but not yet scheduled.
The problem here was an assertion failure, which was triggered with a
high probability; the fix for the problem detailed above used the yield()
function, while it actually was only interested in retrieving the
Extent's address to probe if the extent matches an known storage location.
The solution is to provide a dedicated function for this check, which
can then skip the sanity check (because in this case we do not want
to use the Extent, and thus can touch obsoleted/inactive Extents
without problem)
In the end, I decided that it ''is to early to decide anything'' in this respect...
The actual situation encountered is a **Catch-22**:
* in its current form, the »Tick« handler detects compulsory jobs beyond deadline
* since such a Job ''must not be touched anymore,'' there is no way scheduling can proceed
* so this would constitute a ''Scheduler Emergency''
All fine — just the »Tick« handler ''itself is a compulsory job'' — and being a job, it can well be driven beyond its deadline. In fact this situation was encountered as part of stress testing.
Several mitigations or real solutions are conceivable, but in the end,
too little is known yet regarding the integration of the scheduler within the Engine
Thus I'll marked the problematic location and opened #1362
Investigate the behaviour over a wider range of job loads,
job count and worker pool sizes. Seemingly the processing
can not fully utilise the available worker pool capacity.
By inspection of trace-dumps, one impeding mechanism could
be identified: the »stickiness« of the contention mitigation.
Whenever a worker encounters repeated contention, it steps up
and adds more and more wait cycles to remove pressure from the
schedule coordination. As such this is fine and prevents further
degradation of performance by repeated atomic synchronisation.
However, this throttling was kept up needlessly after further
successful work-pulls. Since job times of several milliseconds
can be expected on average in media processing, such a long
retention would spread a performance degradation over a duration
of several frames. Thus, the scheme for step-down was changed
to decrease the throttling by a power series rather than just
documenting the level.
Use the statistic functions imported recently from Yoshimi-test
to compute a linear regression model as immediate test result.
Combining several measurement series, this allows to draw conclusions
about some generic traits and limitations of the scheduler.
Visual tweaks specific to this measurement setup
* include a numeric representation of the regression line
* include descriptive axis labels
* improve the key names to clarify their meaning
* heuristic code for the x-ticks
Package these customisations as a helper function into the measurement tool
After a lot of further tinkering, seemingly arriving at a
somewhat satisfactory solution for the layout and arrangement of
test definitions and especially the table for measurement series.
While the complete setup remains fragile indeed, and complexity is more
hidden than reduced — the pragmatic compromise established yesterday
at least allows to reduce the amount of boilerplate in the test or
measurement setup to make the actual specifics stand out clearly.
----
As an aside, the usage of the `DataFile` type imported from Yoshimi-test
recently was re-shaped more towards a generic handling of tabular data with
CSV storage option; thus renaming the type now into `DataTable`.
Persistent storage is now just one option, while another usage pattern
compounds observation data into table rows, which are then directly
rendered into a CSV string, e.g. for visualisation as Gnuplot graph.
Rework the existing tool to capture the measurement series
into the newly integrated CSV-based data storage, allowing
to turn the results into a Gnuplot-visualisation.
...which is added automatically whenever additional data columns are present
Result can only be verified visually
* the upper diagram should show the first fibonacci points
* a (correct) linear regression line should be overlayed in red
* below, a secondary diagram should appear, with aligned axis
* the row "one" in this diagram should be shown as impulses
* the further rows "two" and "three" should be drawn as
green points, using the secondary Y-axis (values 100-250)
* Gnuplot can handle missing data points
The idea is to build the Layout-branching into the generated Gnuplot script,
based on the number of data columns detected. If there is at least one further
data column, then the "mulitplot" layout will be used to feature this
additional data in a secondary diagram below with aligned axis;
if more than one additional data column is present, all further
visualisation will draw points, using the secondary Y-axis
Moreover, Gnuplot can calculate the linear regresssion line itself,
and the drawing will then be done using an `arrow` command,
defining a function regLine(x) based on the linear model.
- `forElse` belongs to the metaprogramming utils
- have a CSVLine, which is a string with custom appending mechanism
- this in turn allows CSVData to accept arbitrary sized tuples,
by rendering them into CSVLine
the metafunction `is_basically<X>` performed only an equality match,
while, given it's current usage, it should also include a subtype-interface-match.
This changes especially the `is_StringLike<S>` metafunction,
both on const references and on classes built on top of string
or string_view.
Whenever a class defines a single-arg templated constructor,
there is danger to shadow the auto-generated copy operations,
leading to insidious failures.
Some months ago, I did the ''obvious'' and added a tiny helper,
allowing to mask out the dangerous case when the ''single argument''
is actually the class itself (meaning, it is a copy invocation and
not meant to go through this templated ctor...
As this already turned out as tremendously helpful, I now extended
this helper to also cover cases where the problematic constructor
accepts variadic arguments, which is quite common with builder-helpers
The intention is to create a library of convenient building blocks;
providing a visualisation should be as simple as invoking a free function
with CSV data, yet with the ability to tweak some lables or display
variations if desired.
This can be achieved by..
* having a series of ready-made standard visualisations
* expose a function call for each, accepting a data-context builder
* provide secondary convenience shortcuts, which add some of the expected bindings
* notably a shortcut is provided to take the data as CSV-string
* augmented by a wrapper/builder to allow defining data points inline
Basically GenNode and the enclosed record were designed to be
immutable — yet some valid usage patterns emerged where gradually
building structure seems adequate — which can be accomodated by
entering a ''mutation mode'' explicitly through the Rec::Mutator.
Over time, a builder usage style came in widespread usage, especially
when building test data structures. There seems to be no deeper reason
preventing the Mutator from being ''moved'' — notwithstanding the fact
that using such a ''movable builder'' can be dangerous, especially
when digressing from the strict »fluent inline builder« usage style
and storing the mutator into a variable.
For populating the data context for a text template instantiation,
we have now a valid case where it seems helpful to partially populate
the context and then move it further down into an implementation
function, which does the bulk of the work.
Deliberately keep it unstructured and add dedicated functions
for each new emerging use case; hopefully some commen usage scheme
will emerge over time.
* Data is to be handed in as an iterator over CSV-strings.
* will have to find out about additional parametrisation on a case-by-case base
The default visuals of gnuplot are simple,
yet tend to look cluttered and are not well suited for our purpose
We need the following presentation
* a scatter diagram with a regression line
* additionally a secondary diagram stacked below, with aligned axis
Thus 🠲 R-T-F-M
* The [http://gnuplot.info/ Gnuplot docu] is exhaustive, yet hard to get into
* Helpful was this collection of [http://gnuplotting.org/ example solutions for scientific plots]
* and — Stackoverflow...
A minimalist `TextTemplate` engine is available for in-project use.
* supports only the bare minimum of features (no programming language)
* substitution of `${placeholder}` by key-name data access
* conditional section `${if key}...${end if}`
* iteration over a data sequence
* other then most solutions available as library,
this implementation does **not require** a specific data type,
nor does it invent a dynamic object system or JSON backend;
rather, a generic ''Data Source Adapter'' is used, which can
be specialised to access any kind of ''structured data''
* the following `DataSource` specialisations are provided
* `std::map<string,string>`
* Lumiera »External Tree Description« (based on `GenNode`)
* a string-based spec for testing
This extension is required to use GenNode as data source for text-template instantiation.
I am aware that such a function could counter the design intent for GenNode,
because it could be (ab)used to "just get the damn value" and then
parse back the results...
...turns out challenging, since our intention here
is borderline to the intended design of the Lumiera ETD.
It ''should work'' though, when combined with a Variant-visitor...
Document existing data binding logic and investigate in detail
what must be done to enable a similar binding backed by Lumiera's ETD structures.
This analysis highlights some tricky aspects, which can be accommodated by
slight adjustments and generalisations in the `TextTemplate` implementation
* `GenNode` is not structured string data, rather binary data
* thus exposing a std::string_view is not adequate, requiring to
pick up the result type from the actual data binding
* moreover, to allow for arbitrary nested scopes, a back-pointer
to the parent scope must be maintained, which requires stable memory locations.
This can best be solved within the InstanceCore itself, which manages
the actual hierarchy of data source references.
* the existing code happens already to fulfil this requirement, but
for sake of clarity, handling of such a nested scope is now extracted
into a dedicated operation, to highlight the guaranteed memory layout.
...hoped to keep it simple, but this is inevitable, since we
want to provide a CSV list as value within a list of key=value
bindings, and all packaged into a simple string for easy testing.
Thus the parsing RegExp just needs two branches for simple and quoted vals
We use a DataSrc<DAT> template to access the actual data to be substituted.
However, when applying the Text-Template, we need to pick the right
specialisation, based on the type of the actual data provided.
Here we face several challenges:
* Class-Template-Argument-Deduction starts from the *primary* template's constructors.
Without that, the compiler will only try the copy constructor and will
never see the constructors of partial specialisations.
This can be fixed by providing a ''dummy constructor''.
* The specifics of how to provide a custom CTAD deduction guide
for a **nested template** are not well documented. I have found
several bug reports, and seemingly one of these bugs failed my
my various attempts. Moreover it is ''not clear if such a deduction
guide can even be given outside of the class definition scope.''
For the intended usage pattern this would be crucial, since users
are expected to provide further specialisations of the DataSrc-template
* Thus I resorted to the ''old school solution,'' which is to use
a ''free builder function'' as an extension point. Thus users could
provide further overloads for the `buildDataSrc()` function.
* Unfortunately, SFINAE-Tricks are way more limited for function overload.
Thus it seems impossible to have a generic and more specialised cases,
unless all special cases are disjoint.
Thus the solution is far from perfect, ''yet for the current situation it seems
sufficient'' (and C++20 Concepts will greatly help to resolve this kind of problems)
...implemented by simply parsing the string into key=value pairs,
which are then stored into a shared map. The actual data binding
implementation can thus be inherited from the existing Map-binding
While they were detected just fine, thy were passed-through
unaltered, which subverts the purpose of such an escape,
which is to allow for the tag syntax to be present in the
processed, substituted document (e.g. when generating a
shell script)
thus `\${escaped}` becomes `${escaped}`
...using a ''special protocol'' to represent iterative data sequences
* use an Index-Key with a CSV list of element prefixes
* synthesise key-prefixes for each data element
* perform lookup with the decorated key first
This allows to somehow ''emulate'' nested associations within a single, flat Map.
Obviously this is more like a proof-of-concept; actually the Map-databinding
is meant to handle the simple cases, where just placeholders are to be substituted.
The logic structures are much more relevant when binding to structural data,
most notably to the Lumiera _External Tree Description_ format, which is
used for model data and inter-layer communication.
- the basic interpretation of Action-tokens is already in place
- add the interpretation of conditional and looping constructs
- this includes helpers for
* reset to another Action-token index
* recursive interpretation of the next token
* handling of nested loop evaluation context
In order to make this implementation compile, also the skeleton
of the Map-string-string data binding must be completed, including
a draft how to handle nested keys in a simple map
Sting-view is tricky, since it deliberately does not define a
conversion operator; rather, string has an explicit constructor.
This design was chosen on purpose, since creating a string will
„materialise“ the string-view, which could have severe performance
ramifications when done automatically.
Regarding Lumiera's string-conversion tooling, it seems indicated
thus to add std::string_view explicitly as a known conversion path,
even while this conversion does not happen implicitly.
playing the »fence post problem« the other way round
and abandoning the ''pull processing'' in favour of direct manipulation
leads to much clearer formulation of the code-generation logic
...turns out the ''pipeline design'' is not a good fit for the
Action compilation, since the compiler needs to refer to previous Actions;
better to let the compiler ''build'' the `ActionSeq`
...implemented as »custom processing layer« within a
demand-driven parsing pipeline, with the ability to
inject additional Action-tokens to represent the intermittent
constant text between tags; special handling to expose one
constant postfix after the last active tag.
MatchSeq was imported recently from the Yoshimi-testsuite,
as supporting helper for the CSV table component.
Actually this is just a thin wrapper on top of std::regex_iterator,
which in turn has properties and behaviour very similar to Lumiera's
»Forward Iterator« concept (in fact, it was a source of inspiration to
generalise such a pattern).
So this is an obvious round out and cleanup, as it requires just some
minor additions and adjustments to allow processing a sequence of matches
through a for-loop or some elaborate pipelining setup.
The way I've written this helper template, as a byproduct
it is also possible to maintain the back-refrence to the container
through a smart-ptr. In this case, the iterator-handle also manages
the ownership automatically.
...mostly we want the usual convenient handling pattern for iterators,
but with the proviso actually to perform an access by subscript,
and the ability to re-set to another current index
* establish the feature set to provide
* choose scheme for runtime representation
* break down analysis to individual parsing and execution steps
* conclude which actions to conduct and the necessary data
* derive the abstract binding API required
Conducted an extended investigation regarding text templating
and the library solutions available and still maintained today.
The conclusion is
* there are some mature and widely used solutions available for C++
* all of these are considered a mismatch for the task at hand,
which is to generate Gnuplot scripts for test data visualisation
Points of contention
* all solutions offer a massive feature set, oriented towards web content generation
* all solutions provide their own structured data type or custom property-tree framework
**Decision** 🠲 better to write a minimalistic templating engine from scratch rather
In the Lumiera code base, we use C-String constants as unique error-IDs.
Basically this allows to create new unique error IDs anywhere in the code.
However, definition of such IDs in arbitrary namespaces tends to create
slight confusion and ambiguities, while maintaining the proper use statements
requires some manual work.
Thus I introduce a new **standard scheme**
* Error-IDs for widespread use shall be defined _exclusively_ into `namespace lumiera::error`
* The shorthand-Macro `LERR_()` can now be used to simplify inclusion and referral
* (for local or single-usage errors, a local or even hidden definition is OK)
reduce footprint of lib/util.hpp
(Note: it is not possible to forward-declare std::string here)
define the shorthand "cStr()" in lib/symbol.hpp
reorder relevant includes to ensure std::hash is "hijacked" first
In the Lumiera code base, a convenient string conversion is used
an many places, and is also ''magically'' integrated into the usual
C++ style output with `<<` operators.
However, there is a ''gotcha'' — in the ''rare cases'' when we
actually want to use the C++ input/output framework to copy stream
data from an input source into an output sink, obviously we do not want
the input source to be »string converted«....
showDecimal -> decimal10 (maximal precision to survive round-trip through decimal representation=
showComplete -> max_decimal10 (enough decimal places to capture each possible distinct floating-point value)
Use these new functions to rewrite the format4csv() helper
...this uncovered one inconsistency: when directly adding values
into one of the embedded data vectors, the inconsistent size
was allowed to persist even when adding / removing lines.
This is in contradiction to the behavior for the CSV dump,
which uses index positions from the front of all vectors uniformely.
Thus changed the behaviour of adding a new row, so that it now
caps all vectors to a common size
also added function to clear the table
verify also that clean-up happens in case of exceptions thrown;
as an aside, add Macro to check for ''any'' exception and match
on something in the message (as opposed to just a Lumiera Exception)
...using the same method for sake of uniformity
Also move the permissions helpers to the file.hpp support functions
and setup a separate unit test for these
Inspired by https://stackoverflow.com/a/58454949
Verified behaviour of fs::create_directory
--> it returns true only if it ''indeed could create'' a new directory
--> it returns false if the directory exists already
--> it throws when some other obstacle shows up
As an aside: the Header include/limits.h could be cleaned up,
and it is used solely from C++ code, thus could be typed, namespaced etc.
Since this is a much more complicated topic,
for now I decided to establish two instances through global variables:
* a sequence seeded with a fixed starting value
* another sequence seeded from a true entropy source
What we actually need however is some kind of execution framework
to define points of random-seeding and to capture seed values for
reproducible tests.
Relying on random numbers for verification and measurements is known to be problematic.
At some point we are bound to control the seed values -- and in the actual
application usage we want to record sequence seeding in the event log.
Some initial thoughts regarding this intricate topic.
* a low-ceremony drop-in replacement for rand() is required
* we want the ability to pick-up and control each and every usage eventually
* however, some usages explicitly require true randomness
* the ability to use separate streams of random-number generation is desirable
Yesterday I decided to include some facilities I have written in 2022
for the Yoshimi-Testsuite. The intention is to use these as-is, and just
to adapt them stylistically to the Lumiera code base.
However — at least some basic documentation in the form of
very basic unit-tests can be considered »acceptance criteria«
- reformat in Lumieara-GNU style
- use the Lumiera exceptions
- use Lumiera format-string frontend
- use lib/util
NOTE: I am the original author of the code introduced here,
and thus I can re-license it under GPL 2+
[http://yoshimi.sourceforge.net/ Yoshimi] is a software sound synthesizer,
derived from `ZynAddSubFx` and developed by an OpenSource community.
The Repository [https://github.com/Ichthyostega/yoshimi-test/ Yoshimi-test]
is used by the Yoshimi developers to maintain a suite of automated
acceptance tests for the Yoshimi application.
This task involves watching execution times to detect long-term performance trends,
which in turn requires to maintain time-series data in CSV files and to perfrom some
simple statistic calculations, including linear regression. Requiring any external
statistics package as dependency was not deemed adequate for such a simple task,
and thus a set of self-contained helper functions was created as a byproduct.
This task attaches an excerpt of the Yoshimi-test history with those helpers.
- better use a Test-Chain-Load without any dependencies
- schedule all at once
- employ instrumentation
- use the inner »overall time« as dependent result variable
The timing results now show an almost perfect linear dependency.
Also the inner overall time seems to omit the setup and tear-down time.
But other observed values (notably the avgConcurrency) do not line up
In binary search, in order to establish the invariant initially,
a loop is necessary, since a single step might not be sufficient.
Moreover, the ongoing adjustments jeopardise detection of the
statistical breaking point condition, by causing a negative delta
due to gradually approaching the point of convergence -- leading
to an ongoing search in a region beyond the actual breaking point.
Relying on the new instrumentation facility, the actually effective
concurrency and cumulative run time of the test jobs can be established.
These can now be cast into a form-factor to represent actual excess expenses
in relation to the theoretical model.
By allowing to adjust the adapted schedule by this form factor,
it can be made to reflect more closely the actual empiric load,
hopefully leading to a more realistic effect of the stress-factor
and thus results better suited to conclude on generic behaviour.
- supplement the pre-dimensioning for data capture; without that,
sporadic memory corruption indeed happens (as expected, since
concurrent re-allocation of the vector with an entry for each
thread is not threadsafe, and this test shows much contention)
- add a top-level logging for better diagnostics of errors
emanating from the test run
Basically users are free to place the measurement calls to their liking.
This implies that bracketed measurement intervals can be defined overlapping
even within a single thread, thereby accounting the overlapping time interval
several times. However, for the time spent per thread, only actual thread
activity should be counted, disregarding overlaps. Thus introduce a
new aggregate, ''active time'', which is the sum of all thread times.
As an aside, do not need explicit randomness for the simple two-thread
test case — timings are random anyway...
+ bugfix for out-of-bounds access
...since we've established already an integration over the event timeline,
it is just one simple further step to determine the concurrency level
on each individual segment of the timeline. Based on this attribution
- the averaged concurrenty within the observation range can be computed as weighted mean
- moreover we can account for the precise cumulated time spent at each concurrency level
...using a simplistic allocation of next-slot based on initialisation
of a thread_local storage. This implies that this helper can not be
reset or reused, and that there can not be multiple or long-lived instances.
Keep-it-simple for now...
...to sort out the interpretation of measurement results,
the actual duration and concurrency of ComputationLoad invocations
should be recorded, allowing to draw conclusions regarding the
Scheduler's performance as opposed to further system and thread
management effects due to concurrent operation under pressure.
After an extended break due to "real life issues"....
Pick up the investigation, with the goal to ascertain a valid definition
and understanding of all test parameters. A first step is to establish
a baseline ''without using a computational load''; this might be some kind
of base overhead of the scheduler.
However -- the way the test scaffolding was built, it is difficult to
create a feedback loop for the statistical test setup with binary search,
since it is not really clear how the single control parameter of the test algorithm,
the so called "stress factor", shall be interpreted and how it can be
combined with a base load.
An extended series of tests, while watching the observed value patterns qualitatively,
seems to corroborate the former results, indicating that the base expense
in my test setup (using a debug build) is at ~200µs / Node / core.
Yet the difficulty to interpret this result and arrive at a logical and generic model
prevents me from translating this into a measurement scheme, which can
be executed independently from a specific test setup and hardware
This is just another (obvious) degree of freedom, which could be
interesting to explore in stress testing, while probably not of much
relevance in practice (if a job is expected to become runable earlier,
in can as well be just scheduled earlier).
Some experimentation shows that the timing measurements exhibit more
fluctuations, but also slightly better times when pressure is low, which
is pretty much what I'd expect. When raising pressure, the average
times converge towards the same time range as observed with time bound
propagation.
Note that enabling this variation requires to wire a boolean switch
over various layers of abstraction; arguably this is an unnecessary
complexity and could be retracted once the »experimentation phase«
is over.
This completes the preparation of a Scheduler Stress-Test setup.
...watching those dumps on the example Graph with excessive dependencies
made blatantly clear that we're dispatching a lot of unnecessary jobs,
since the actual continuation is /always/ triggered by the dependency-NOTIFY.
Before the rework of NOTIFY-Handling, this was rather obscured, but now,
since the NOTIFY trigger itself is also dispatched by the Scheduler,
it ''must be this job'' which actually continues the calculation, since
the main job ''can not pass the gate'' before the dependency notification
arrives.
Thus I've now added a variation to the test setup where all these duplicate
jobs are simply omitted. And, as expected, the computation runs faster
and with less signs of contention. Together with the other additional
parameter (the base expense) we might now actually be able to narrow down
on the observation of a ''expense socket'', which can then be
attributed to something like an ''inherent scheduler overhead''
While the idea with capturing observation values is nice,
it definitively does not belong into a library impl of the
search algorithm, because this is usage specific and grossly
complicates the invocation.
Rather, observation data can be captured by side-effect
from the probe-λ holding the actual measurement run.
...based on the adapted time-factor sequence
implemented yesterday in TestChainLoad itself
- in this case, the TimeBase from the computation load is used as level speed
- this »base beat« is then modulated by the timing factor sequence
- working in an additional stress factor to press the schedule uniformly
- actual start time will be added as offset once the actual test commences
...so IterExplorer got yet another processing layer,
which uses the grouping mechanics developed yesterday,
but is freely configurable through λ-Functions.
At actual usage sit in TestChainLoad, now only the actual
aggregation computation must be supplied, and follow-up computations
can now be chained up easily as further transformation layers.
In-depth investigation and reasoning highlighted another problem,
which could lead to memory corruption in rare cases; in the end
I found a solution by caching the ''address'' of the current Epoch
and re-validating this address on each Epoch-overflow.
After some difficulties getting any reliable measurement for a Release-build,
it turned out that this solution even ''improves performance by 22%''
Remark-1: the static blockFlow::Config prevents simple measurements by
just recompiling one translation unit; it is necessary to build the
relevant parts of Vault-layer with optimisation to get reliable numbers
Remark-2: performing a full non-DEBUG build highlighted two missing
header-inclusions to allow for the necessary template specialisations.
...discovered by during investigation of latest Scheduler failures.
The root of the problems is that block overflow can potentially trigger
expansion of the allocation pool. Under some circumstances, this on-the fly
allocation requires a rotation of index slots, thereby invalidating
existing iterators.
While such behaviour is not uncommon with storage data structures (see std::vector),
in this case it turns out problematic because due to performance considerations,
a usage pattern emerged which exploits re-using existing storage »Slots« with known
deadline. This optimisation seems to have significant leverage on the
planning jobs, which happen to allocated and arrange a whole strike of
Activities with similar deadlines.
One of these problem situations can easily be fixed, since it is triggered
through the iterator itself, using a delegate function to request a storage expansion,
at which point the iterator is able to re-link and fix its internal index.
This solution also has no tangible performance implications in optimised code.
Unfortunately there remains one obscure corner case where such an pool expansion
could also have invalidated other iterators, which are then used later to
attach dependency relations; even a partial fix for that problem seems
to cause considerable performance cost of about -14% in optimised code.
This amounts to a rather massive refactoring, prompted by the enduring problems
observed when pressing the scheduler. All the various glitches and (fixed) crashes
are related to the way how planning-jobs enter the schedule items,
which is also closely tied to the difficulties getting the locking
for planning-jobs correct.
The solution pursued hereby is to reorder the main avenues into the
scheduler implementation. There is now a streamlined main entrance,
which **always** enqueues only, allowing to omit most checks and
coordination. On the other hand, the complete coordination and dispatch
of the work capacity is now shifted down into the SchedulerCommutator,
thereby linking all coordination and access control close together
into a single implementation facility.
If this works out as intended
- several repeated checks on the Grooming-Token could be omitted (performance)
- the planning-job would no longer be able to loose / drop the Token,
thereby running enforcedly single-threaded (as was the original intention)
- since all planning effectively originates from planning-jobs, this
would allow to omit many safety barriers and complexities at the
scheduler entrance avenue, since now all entries just go into the queue.
WIP: tests pass compiler, but must be adapted / reworked
...whenever the planning falls behind schedule, it can happen that
the planner-worker immediately dispatches its own jobs; while the calculation
is broken anyway in this situation, especially this call scheme leads to
dropping the Grooming-Token prior to the calculation dispatched directly.
Since the dependency relation can only be established after creating
both predecessor and successor schedules, the corresponding allocation
of the NOTIFY-Activity is not protected against concurrent access,
which probably leads to the assertion failure due to corruption of
the allocator's internal data structures...
...causing the system to freeze due to excess memory allocation.
Fortunately it turned out this was not an error in the Scheduler core
or memory manager, but rather a sloppiness in the test scaffolding.
However, this incident highlights that the memory manager lacks some
sanity checks to prevent outright nonsensical allocation requests.
Moreover it became clear again that the allocation happens ''already before''
entering the Scheduler — and thus the existing sanity check comes too late.
Now I've used the same reasoning also for additional checks in the allocator,
limiting the Epoch increment to 3000 and the total memory allocation to 8GiB
Talking of Gibitbytes...
indeed we could use a shorthand notation for that purpose...
The scheduler implementation uses a randomised redistribution of
work capacity, taking into account the current ''scale'' of next pending event.
While this works surprisingly well overall, sometimes, in very tight and dense scheules
the workers seem to be spread somewhat too arbitrarily. Thus, if the scheduler
is working through a zone with several events as close as 1ms, often it takes
up to 3ms for another worker to show up.
With this change, the scattering range in the ''near zone'' (50µs ... 5ms)
is made dynamic, and now flexibly depends on current head time.
The closer the next event, the more tightly focussed will be the
capacity redistribution, if capacity becomes available just some 100µs
ahead of next demand, it is no longer „sent away“, but rather relocated
by roughly the same distance behind the next event.
while my basic assessment is still that contention will not play a significant
role given the expected real world usage scenario — when testing with
tighter schedule and rather short jobs (500µs), some phases of massive contention
can be observed, leading to significant slow-down of the test.
The major problem seems to be that extended phases of contention will
effectively cause several workers to remain in an active spinning-loop for
multiple microseconds, while also permanently reading the atomic lock.
Thus an adaptive scheme is introduced: after some repeated contention events,
workers now throttle down by themselves, with polling delays increased
with exponential stepping up to 2ms. This turns out to be surprisingly
effective and completely removes any observed delays in the test setup.
...turns out to be a secondary problem (but must be fixed non the less).
Since the planning-job no longer drops the token now, the workers
have to wait; since they are waiting actively and contending on the token,
a significant slowdown can happen.
Sometimes the planning job gets behind its own scheduler and thus
enters dispatch, in which case it drops the GoomingToken, causing
an Assertion failure on return.
The **actual problem** however is the slowdown due to active spinning
Turns out that we need to implemented fine grained and explicit handling logic
to ensure that Activity planning only ever happens protected by the Grooming-Token.
This is in accordance to the original design, which dictates that all management tasks
must be done in »management mode«, which can only be entered by a single thread at a time.
The underlying assumption is that the effort for management work is dwarfed in comparison
to any media calculation work.
However, in
5c6354882d
...I discovered an insidious border condition, an in an attempt to fix it,
I broke that fundamental assumpton. The problem arises from the fact that we
do want to expose a *public API* of the Scheduler. Even while this is only used
to ''seed'' a calculation stream, because any further planning- and management work
will be performed by the workers themselves (this is a design decision, we do not
employ a "scheduler thread")
Anyway, since the Scheduler API ''is'' public, ''someone from the outside'' could
invoke those functions, and — unaware of any Scheduler internals — will
automatically acquire the Grooming-Token, yet never release it,
leading to deadlock.
So we need a dedicated solution, which is hereby implemented as a
scoped guard: in the standard case, the caller is a management-job and
thus already holds the token (and nothing must be done). But in the
rare case of an »outsider«, this guard now ''transparently'' acquires
the token (possibly with a blocking wait) and ''drops it when leaving scope''
In the course of the last refactorings, a slight change in processing
order was introduced, which turned out to improve parallelisation considerably.
- Some further implementation logic can be relegated into the ActivationEvent
- the handling of start times now also incldues a check for sake of symmetry
- document the semantics change: λ-post no longer dispatches directly
...this feature seems to be no longer necessary now;
leaving the actual implementation in-code for the time being,
but removed it from the public access API.
The last round of refactorings yielded significant improvements
- parallelisation now works as expected
- processing progresses closer to the schedule
- run time was reduced
The processing load for this test is tuned in a way to overload the
scheduler massively at the end -- the result must be correct non the less.
There was one notable glitch with an assertion failure from the memory manager.
Hopefully I can reproduce this by pressing and overloading the Scheduler more...
The rework from yesterday turned out to be effective ... unfortunately
a bit to much: since now late follow-up notifications take precedence,
a single worker tends to process the complete chain depth-first, because
the first chain will be followed and processed, even before the worker
was able to post the tasks for the other branches. Thus this single
worker is the only one to get a chance to proceed.
After some consideration, I am now leaning towards a fundamental change,
instead of just fixing some unfavourable behaviour pattern: while the
language semantics remains the same, the scheduler should no longer
directly dispatch into the next chain **from λ-post**. That is, whenever
a POST / NOTIFY is issued from the Activity-chain, the scheduler goes
through prioritisation.
This has further ramifications: we do not need a self-inhibition mechanism
any more (since now NOTIFY picks up the schedule time of the target).
With these changes, processing seems to proceed more smoothly,
albeit still with lots of contention on the Grooming token,
at least in the example structure tested here.
While the recent refactoring...
206c67cc
...was a step into the right direction, it pushed too hard,
overlooking the requirement to protect the scheduler contents
and thus all of the Activity-chains against concurrent modification.
Moreover, the recent solution still seems not quite orthogonal.
Thus the handling of notifications was thoroughly reworked:
- the explicit "double-dispatch" was removed, since actual usage
of the language indicates that we only need notifications to
Gate (and Hook), but not to any other conceivable Activity.
- thus it seems unnecessary to turn "notification" into some kind
of secondary work mode. Rather, it is folded as special case
into the regular dispatch.
This leads to new processing rules:
- a POST goes into λ-post (obviously... that's its meaning)
- a NOTIFY now passes its *target* into λ-post
- λ-post invokes ''dispatch''
- and **dispatching a Gate now implies to notify the Gate**
This greatly simplifies the »state machine« in the Activity-Language,
but also incurs some limitations (which seems adequate, since it is
now clear that we do not ''schedule'' or ''dispatch'' arbitrary
Activities — rather we'll do this only with POST and NOTIFY,
and all further processing happens by passing activation
along the chain, without involving the Scheduler)
use a feature of the Activity-Language prepared for this purpose:
self-Inhibition of the Chain. This prevents a prerequisite-NOTIFY
to trigger a complete chain of available tasks, before these tasks
have actually reached their nominal scheduling time.
This has the effect to align the computations much more strictly
with the defined schedule
The main (test) thread is kept in a blocking wait until the
planned schedule is completed. If however the schedule overruns,
the wake-up job could just be triggered prematurely.
This can easily be prevented by adding a dependency from the last
computation job to the wake-up job. If the computation somehow
flounders, the SAFETY_TIMEOUT (5s) will eventually raise
an exception to let the test fail cleanly (shutting down
the Scheduler automatically)
...it seems impossible to solve this conundrum other than by
opening a path to override a contextual deadline setting from
within the core Activity-Language logic.
This will be used in two cases
- when processing a explicitly coded POST (using deadline from the POST)
- after successfully opening a Gate by NOTIFY (using deadline from Gate)
All other cases can now supply Time::NEVER, thereby indicating that
the processing layer shall use contextual information (intersection
of the time intervals)
...this is an interesting test failure, which highlights inconsistencies
with handling of deadlines when processing follow-up from NOTIFY-triggers
There was also some fuzziness related to the ''meaning'' of λ-post,
leading to at least one superfluous POST invocation for each propagation;
fixing this does not solve the problem yet removes unnecessary overhead
and lock-contention
this bug was there since the first draft, yet was covered
by another bug with the start-up logic.
And this latter one was fixed recently...
fa8622805
As a result, even when the COMPUTATION_CAPACITY is set to 0
still a single worker boots up (which should not be the case)
Solution: we do not need to "safeguard" against rounding errors,
since this is an internal implementation function, it is assumed
that the caller knows about its limitations...
This partially reverts commit 72f11549e6.
"Chain-Load: Scheduler instrumentation for observation"
Hint: revert this changeset to re-introduce the print statements for diagnostic
..initial gauging is a tricky subject,
since existing computer's performance spans a wide scale
Allowing
- pre calibration -98% .. +190%
- single run ±20%
- benchmark ±5%
...which can be deliberately attached (or not attached) to the
individual node invocation functor, allowing to study the effect
of actual load vs. zero-load and worker contention
...during development of the Chain-Load, it became clear that we'll often
need a collection of small trees rather than one huge graph. Thus a rule
for pruning nodes and finishing graphs was added. This has the consequence
that there might now be several exit nodes scattered all over the graph;
we still want one single global hash value to verify computations,
thus those exit hashes must now be picked up from the nodes and
combined into a single value.
All existing hash values hard coded into tests must be updated
This is a trick to get much better scheduling and timing guesses.
Instead of targeting a specific level, rather a fixed number of nodes
is processed in each chunk, yet still always processing complete levels.
The final level number to expect can be retrieved from the chain-load graph.
With this refactoring, we can now schedule a wake-up job precisely
after the expected completion of the last level
Invent a special JobFunctor...
- can be created / bound from a λ
- self-manages its storage on the heap
- can be invoked once, then discards itself
Intention is to pass such one-time actions to the Scheduler
to cause some ad-hoc transitions tied to curren circumstances;
a notable example will be the callback after load-test completion.
In the first draft version, a blocked Gate was handled by
»polling« the Gate regularly by scheduling a re-invocation
repeatedly into the future (by a stepping defined through
ExecutionCtx::getWaitDelay()).
Yet the further development of the Activity-Language indicates
that the ''Notification mechanism'' is sufficient to handle all
foreseeable aspects of dependency management. Consequently this
''Gate poling is no longer necessary,'' since on Notification
the Gate is automatically checked and the activation impulse
is immediately passed on; thus the re-scheduled check would
never get an opportunity actually to trigger the Gate; such
an active polling would only be necessary if the count down
latch in the Gate is changed by "external forces".
Moreover, the first Scheduler integration tests with TestChainLoad
indicate that the rescheduled polling can create a considerable
additional load when longer dependency chains miss one early
prerequisite, and this additional load (albeit processed
comparatively fast by the Scheduler) will be shifted along
needlessly for quite some time, until all of the activities
from the failed chain have passed their deadline. And what
is even more concerning, these useless checks have a tendency
to miss-focus the capacity management, as it seems there is
much work to do in a near horizon, which in fact may not be
the case altogether.
Thus the Gate implementation is now *changed to just SKIP*
when blocked. This helped to drastically improve the behaviour
of the Scheduler immediately after start-up -- further observation
indicated another adjustment: the first Tick-duty-cycle is now
shortened, because (after the additional "noise" from gate-rescheduling
was removed), the newly scaled-up work capacity has the tendency
to focus in the time horizon directly behind the first jobs added
to the timeline, which typically is now the first »Tick«.
ð¡ this leads to a recommendation, to arrange the first job-planning
chunk in such a way that the first actual work jobs appear in the area
between 5ms and 10ms after triggering the Scheduler start-up.Scheduler¡
Introducing a fixed pre-delay on each new Calc-Streem seemed like an obvious remedy,
yet on closer investigation it turned out that the start-up logic as such was contradictory,
which was only uncovered by some rather special schedule patterns.
After fixing the logic deficiencies, Scheduler starts up as intended
and the probabilistic capacity-control seems to work as designed.
Thus no need to introduce an artificial delay at begin, even while
this implies that typically the first round of job-planning will be
performed synchronous, in the invoking thread (which may be surprising,
but is completely within the limits of the architecture; we do not
employ specifically configured threads and planning should be done
in short chunks, thus the first chunk can well be done by the caller)
The first complete integration test with Chain-Load
highlighted some difficulties with the overall load regulation:
- it works well in the standard case (but is possibly to eager to scale up)
- the scale-up sometimes needs several cycles to get "off the ground"
- when the first job is dispatched immediately instead of going
through the queue, the scheduler fails to boot up
- prime diagnostics with the first time invocation
- print timings relative to this first invocation
- DUMP output to watch the crucial scheduling operations
... so this (finally) is the missing cornerstone
... traverse the calculation graph and generate render jobs
... provide a chunk-wise pre-planning of the next batch
... use a future to block the (test) thread until completed
- decided to abstract the scheduler invocations as λ
- so this functor contains the bare loop logic
Investigation regarding hash-framework:
It turns out that boost::hash uses a different hash_combine,
than what we have extracted/duplicated in lib/hash-value.hpp
(either this was a mistake, or boost::hash did use this weaker
function at that time and supplied a dedicated 64bit implementation later)
Anyway, should use boost::hash for the time being
maybe also fix the duplicated impl in lib/hash-value.hpp
- use a dedicated context "dropped off" the TestChainLoad instance
- encode the node-idx into the InvocationInstanceID
- build an invocation- and a planning-job-functor
- let planning progress over an lib::UninitialisedStorage array
- plant the ActivityTerm instances into that array as Scheduling progresses
Introduced as remedy for a long standing sloppiness:
Using a `char[]` together with `reinterpret_cast` in storage management helpers
bears danger of placing objects with wrong alignment; moreover, there are increasing
risks that modern code optimisers miss the ''backdoor access'' and might apply too
aggressive rewritings.
With C++17, there is a standard conformant way to express such a usage scheme.
* `lib::UninitialisedStorage` can now be used in a situation (e.g. as in `ExtentFamily`)
where a complete block of storage is allocated once and then subsequently used
to plant objects one by one
* moreover, I went over the code base and adapted the most relevant usages of
''placement-new into buffer'' to also include the `std::launder()` marker
... special rule to generate a fixed expansion on each seed
... consecutive reductions join everything back into one chain
... can counterbalance expansions and reductions
...as it turns out, the solution embraced first was the cleanest way
to handle dynamic configuration of parameters; just it did not work
at that time, due to the reference binding problem in the Lambdas.
Meanwhile, the latter has been resolved by relying on the LazyInit
mechanism. Thus it is now possible to abandon the manipulation by
side effect and rather require the dynamic rule to return a
''pristine instance''.
With these adjustments, it is now possible to install a rule
which expands only for some kinds of nodes; this is used here
to crate a starting point for a **reduction rule** to kick in.
It seams indicated to verify the generated connectivity
and the hash calculation and recalculation explicitly
at least for one example topology; choosing a topology
comprised of several sub-graphs, to also verify the
propagation of seed values to further start-nodes.
In order to avoid addressing nodes directly by index number,
those sub-graphs can be processed by ''grouping of nodes'';
all parts are congruent because topology is determined by
the node hashes and thus a regular pattern can be exploited.
To allow for easy processing of groups, I have developed a
simplistic grouping device within the IterExplorer framework.
- with the new pruning option, start-Nodes can now be anywhere
- introduce predicates to detect start-Nodes and exit-Nodes
- ensure each new seed node gets the global seed on graph construction
- provide functionality to re-propagate a seed and clear hashes
- provide functionality to recalculate the hashes over the graph
up to now, random values were completely determined by the
Node's hash, leading to completely symmetrical topology.
This is fine, but sometimes additional randomness is desirable,
while still keeping everything deterministic; the obvious solution
is to make the results optionally dependent on the invocation order,
which is simply to achieve with an additional state field. After some
tinkering, I decided to use the most simplistic solution, which is
just a multiplication with the state.
...so this was yet another digression, caused by the desire
somehow to salvage this problematic component design. Using a
DSL token fluently, while internally maintaining a complex and
totally open function based configuration is a bit of a stretch.
For context: I've engaged into writing a `LazyInit` helper component,
to resolve the inner contradiction between DSL use of `RandomDraw`
(implying value semantics) and the design of a processing pipeline,
which quite naturally leads to binding by reference into the enclosing
implementation.
In most cases, this change (to lazy on-demand initialisation) should be
transparent for the complete implementation code in `RandomDraw` -- with
one notable exception: when configuring an elaborate pipeline, especially
with dynamic changes of the probability profile during the simulation run,
then then obviously there is the desire to use the existing processing
pipeline from the reconfiguration function (in fact it would be quite
hard to explain why and where this should be avoided). `LazyInit` breaks
this usage scenario, since -- at the time the reconfiguration runs --
now the object is not initialised at all, but holds a »Trojan« functor,
which will trigger initialisation eventually.
After some headaches and grievances (why am I engaging into such an
elaborate solution for such an accidental and marginal topic...),
unfortunately it occurred to me that even this problem can be fixed,
with yet some further "minimal" adjustments to the scheme: the LazyInit
mechanism ''just needs to ensure'' that the init-functor ''sees the
same environment as in eager init'' -- that is, it must clear out the
»Trojan« first, and it ''could apply any previous pending init function''
fist. That is, with just a minimal change, we possibly build a chain
of init functors now, and apply them in given order, so each one
sees the state the previous one created -- as if this was just
direct eager object manipulation...
...this is a more realistic demo example, which mimics
some of the patterns present in RandomDraw. The test also
uses lambdas linking to the actual storage location, so that
the invocation would crash on a copy; LazyInit was invented
to safeguard against this, while still allowing leeway
during the initialisation phase in a DSL.
...oh my.
This is getting messy. I am way into danger territory now....
I've made a nifty cool design with automatically adapted functors;
yet at the end of the day, this does not bode well with a DSL usage,
where objects appear to be simple values from a users point of view.
- Helper function to find out of two objects are located
"close to each other" -- which can be used as heuristics
to distinguish heap vs. stack storage
- further investigation shows that libstdc++ applies the
small-object optimisation for functor up to »two slots«
in size -- but only if the copy-ctor is trivial. Thus
a lambda capturing a shared_ptr by value will *always*
be maintained in heap storage (and LazyInit must be
redesigned accordingly)...
- the verify_inlineStorage() unit test will now trigger
if some implementation does not apply small-object optimisation
under these minimal assumptions
