- `forElse` belongs to the metaprogramming utils
- have a CSVLine, which is a string with custom appending mechanism
- this in turn allows CSVData to accept arbitrary sized tuples,
by rendering them into CSVLine
Whenever a class defines a single-arg templated constructor,
there is danger to shadow the auto-generated copy operations,
leading to insidious failures.
Some months ago, I did the ''obvious'' and added a tiny helper,
allowing to mask out the dangerous case when the ''single argument''
is actually the class itself (meaning, it is a copy invocation and
not meant to go through this templated ctor...
As this already turned out as tremendously helpful, I now extended
this helper to also cover cases where the problematic constructor
accepts variadic arguments, which is quite common with builder-helpers
The intention is to create a library of convenient building blocks;
providing a visualisation should be as simple as invoking a free function
with CSV data, yet with the ability to tweak some lables or display
variations if desired.
This can be achieved by..
* having a series of ready-made standard visualisations
* expose a function call for each, accepting a data-context builder
* provide secondary convenience shortcuts, which add some of the expected bindings
* notably a shortcut is provided to take the data as CSV-string
* augmented by a wrapper/builder to allow defining data points inline
Deliberately keep it unstructured and add dedicated functions
for each new emerging use case; hopefully some commen usage scheme
will emerge over time.
* Data is to be handed in as an iterator over CSV-strings.
* will have to find out about additional parametrisation on a case-by-case base
A minimalist `TextTemplate` engine is available for in-project use.
* supports only the bare minimum of features (no programming language)
* substitution of `${placeholder}` by key-name data access
* conditional section `${if key}...${end if}`
* iteration over a data sequence
* other then most solutions available as library,
this implementation does **not require** a specific data type,
nor does it invent a dynamic object system or JSON backend;
rather, a generic ''Data Source Adapter'' is used, which can
be specialised to access any kind of ''structured data''
* the following `DataSource` specialisations are provided
* `std::map<string,string>`
* Lumiera »External Tree Description« (based on `GenNode`)
* a string-based spec for testing
This extension is required to use GenNode as data source for text-template instantiation.
I am aware that such a function could counter the design intent for GenNode,
because it could be (ab)used to "just get the damn value" and then
parse back the results...
...turns out challenging, since our intention here
is borderline to the intended design of the Lumiera ETD.
It ''should work'' though, when combined with a Variant-visitor...
Document existing data binding logic and investigate in detail
what must be done to enable a similar binding backed by Lumiera's ETD structures.
This analysis highlights some tricky aspects, which can be accommodated by
slight adjustments and generalisations in the `TextTemplate` implementation
* `GenNode` is not structured string data, rather binary data
* thus exposing a std::string_view is not adequate, requiring to
pick up the result type from the actual data binding
* moreover, to allow for arbitrary nested scopes, a back-pointer
to the parent scope must be maintained, which requires stable memory locations.
This can best be solved within the InstanceCore itself, which manages
the actual hierarchy of data source references.
* the existing code happens already to fulfil this requirement, but
for sake of clarity, handling of such a nested scope is now extracted
into a dedicated operation, to highlight the guaranteed memory layout.
...hoped to keep it simple, but this is inevitable, since we
want to provide a CSV list as value within a list of key=value
bindings, and all packaged into a simple string for easy testing.
Thus the parsing RegExp just needs two branches for simple and quoted vals
...implemented by simply parsing the string into key=value pairs,
which are then stored into a shared map. The actual data binding
implementation can thus be inherited from the existing Map-binding
While they were detected just fine, thy were passed-through
unaltered, which subverts the purpose of such an escape,
which is to allow for the tag syntax to be present in the
processed, substituted document (e.g. when generating a
shell script)
thus `\${escaped}` becomes `${escaped}`
...turns out the ''pipeline design'' is not a good fit for the
Action compilation, since the compiler needs to refer to previous Actions;
better to let the compiler ''build'' the `ActionSeq`
...implemented as »custom processing layer« within a
demand-driven parsing pipeline, with the ability to
inject additional Action-tokens to represent the intermittent
constant text between tags; special handling to expose one
constant postfix after the last active tag.
MatchSeq was imported recently from the Yoshimi-testsuite,
as supporting helper for the CSV table component.
Actually this is just a thin wrapper on top of std::regex_iterator,
which in turn has properties and behaviour very similar to Lumiera's
»Forward Iterator« concept (in fact, it was a source of inspiration to
generalise such a pattern).
So this is an obvious round out and cleanup, as it requires just some
minor additions and adjustments to allow processing a sequence of matches
through a for-loop or some elaborate pipelining setup.
The way I've written this helper template, as a byproduct
it is also possible to maintain the back-refrence to the container
through a smart-ptr. In this case, the iterator-handle also manages
the ownership automatically.
...mostly we want the usual convenient handling pattern for iterators,
but with the proviso actually to perform an access by subscript,
and the ability to re-set to another current index
* establish the feature set to provide
* choose scheme for runtime representation
* break down analysis to individual parsing and execution steps
* conclude which actions to conduct and the necessary data
* derive the abstract binding API required
Conducted an extended investigation regarding text templating
and the library solutions available and still maintained today.
The conclusion is
* there are some mature and widely used solutions available for C++
* all of these are considered a mismatch for the task at hand,
which is to generate Gnuplot scripts for test data visualisation
Points of contention
* all solutions offer a massive feature set, oriented towards web content generation
* all solutions provide their own structured data type or custom property-tree framework
**Decision** 🠲 better to write a minimalistic templating engine from scratch rather
In the Lumiera code base, we use C-String constants as unique error-IDs.
Basically this allows to create new unique error IDs anywhere in the code.
However, definition of such IDs in arbitrary namespaces tends to create
slight confusion and ambiguities, while maintaining the proper use statements
requires some manual work.
Thus I introduce a new **standard scheme**
* Error-IDs for widespread use shall be defined _exclusively_ into `namespace lumiera::error`
* The shorthand-Macro `LERR_()` can now be used to simplify inclusion and referral
* (for local or single-usage errors, a local or even hidden definition is OK)
reduce footprint of lib/util.hpp
(Note: it is not possible to forward-declare std::string here)
define the shorthand "cStr()" in lib/symbol.hpp
reorder relevant includes to ensure std::hash is "hijacked" first
showDecimal -> decimal10 (maximal precision to survive round-trip through decimal representation=
showComplete -> max_decimal10 (enough decimal places to capture each possible distinct floating-point value)
Use these new functions to rewrite the format4csv() helper
...this uncovered one inconsistency: when directly adding values
into one of the embedded data vectors, the inconsistent size
was allowed to persist even when adding / removing lines.
This is in contradiction to the behavior for the CSV dump,
which uses index positions from the front of all vectors uniformely.
Thus changed the behaviour of adding a new row, so that it now
caps all vectors to a common size
also added function to clear the table
verify also that clean-up happens in case of exceptions thrown;
as an aside, add Macro to check for ''any'' exception and match
on something in the message (as opposed to just a Lumiera Exception)
...using the same method for sake of uniformity
Also move the permissions helpers to the file.hpp support functions
and setup a separate unit test for these
Inspired by https://stackoverflow.com/a/58454949
Verified behaviour of fs::create_directory
--> it returns true only if it ''indeed could create'' a new directory
--> it returns false if the directory exists already
--> it throws when some other obstacle shows up
As an aside: the Header include/limits.h could be cleaned up,
and it is used solely from C++ code, thus could be typed, namespaced etc.
Since this is a much more complicated topic,
for now I decided to establish two instances through global variables:
* a sequence seeded with a fixed starting value
* another sequence seeded from a true entropy source
What we actually need however is some kind of execution framework
to define points of random-seeding and to capture seed values for
reproducible tests.
Relying on random numbers for verification and measurements is known to be problematic.
At some point we are bound to control the seed values -- and in the actual
application usage we want to record sequence seeding in the event log.
Some initial thoughts regarding this intricate topic.
* a low-ceremony drop-in replacement for rand() is required
* we want the ability to pick-up and control each and every usage eventually
* however, some usages explicitly require true randomness
* the ability to use separate streams of random-number generation is desirable
Yesterday I decided to include some facilities I have written in 2022
for the Yoshimi-Testsuite. The intention is to use these as-is, and just
to adapt them stylistically to the Lumiera code base.
However — at least some basic documentation in the form of
very basic unit-tests can be considered »acceptance criteria«
Basically users are free to place the measurement calls to their liking.
This implies that bracketed measurement intervals can be defined overlapping
even within a single thread, thereby accounting the overlapping time interval
several times. However, for the time spent per thread, only actual thread
activity should be counted, disregarding overlaps. Thus introduce a
new aggregate, ''active time'', which is the sum of all thread times.
As an aside, do not need explicit randomness for the simple two-thread
test case — timings are random anyway...
+ bugfix for out-of-bounds access
...since we've established already an integration over the event timeline,
it is just one simple further step to determine the concurrency level
on each individual segment of the timeline. Based on this attribution
- the averaged concurrenty within the observation range can be computed as weighted mean
- moreover we can account for the precise cumulated time spent at each concurrency level
...using a simplistic allocation of next-slot based on initialisation
of a thread_local storage. This implies that this helper can not be
reset or reused, and that there can not be multiple or long-lived instances.
Keep-it-simple for now...
...to sort out the interpretation of measurement results,
the actual duration and concurrency of ComputationLoad invocations
should be recorded, allowing to draw conclusions regarding the
Scheduler's performance as opposed to further system and thread
management effects due to concurrent operation under pressure.
...so IterExplorer got yet another processing layer,
which uses the grouping mechanics developed yesterday,
but is freely configurable through λ-Functions.
At actual usage sit in TestChainLoad, now only the actual
aggregation computation must be supplied, and follow-up computations
can now be chained up easily as further transformation layers.
...during development of the Chain-Load, it became clear that we'll often
need a collection of small trees rather than one huge graph. Thus a rule
for pruning nodes and finishing graphs was added. This has the consequence
that there might now be several exit nodes scattered all over the graph;
we still want one single global hash value to verify computations,
thus those exit hashes must now be picked up from the nodes and
combined into a single value.
All existing hash values hard coded into tests must be updated
- use a dedicated context "dropped off" the TestChainLoad instance
- encode the node-idx into the InvocationInstanceID
- build an invocation- and a planning-job-functor
- let planning progress over an lib::UninitialisedStorage array
- plant the ActivityTerm instances into that array as Scheduling progresses
... special rule to generate a fixed expansion on each seed
... consecutive reductions join everything back into one chain
... can counterbalance expansions and reductions
...as it turns out, the solution embraced first was the cleanest way
to handle dynamic configuration of parameters; just it did not work
at that time, due to the reference binding problem in the Lambdas.
Meanwhile, the latter has been resolved by relying on the LazyInit
mechanism. Thus it is now possible to abandon the manipulation by
side effect and rather require the dynamic rule to return a
''pristine instance''.
With these adjustments, it is now possible to install a rule
which expands only for some kinds of nodes; this is used here
to crate a starting point for a **reduction rule** to kick in.
It seams indicated to verify the generated connectivity
and the hash calculation and recalculation explicitly
at least for one example topology; choosing a topology
comprised of several sub-graphs, to also verify the
propagation of seed values to further start-nodes.
In order to avoid addressing nodes directly by index number,
those sub-graphs can be processed by ''grouping of nodes'';
all parts are congruent because topology is determined by
the node hashes and thus a regular pattern can be exploited.
To allow for easy processing of groups, I have developed a
simplistic grouping device within the IterExplorer framework.
up to now, random values were completely determined by the
Node's hash, leading to completely symmetrical topology.
This is fine, but sometimes additional randomness is desirable,
while still keeping everything deterministic; the obvious solution
is to make the results optionally dependent on the invocation order,
which is simply to achieve with an additional state field. After some
tinkering, I decided to use the most simplistic solution, which is
just a multiplication with the state.
...so this was yet another digression, caused by the desire
somehow to salvage this problematic component design. Using a
DSL token fluently, while internally maintaining a complex and
totally open function based configuration is a bit of a stretch.
...this is a more realistic demo example, which mimics
some of the patterns present in RandomDraw. The test also
uses lambdas linking to the actual storage location, so that
the invocation would crash on a copy; LazyInit was invented
to safeguard against this, while still allowing leeway
during the initialisation phase in a DSL.
- Helper function to find out of two objects are located
"close to each other" -- which can be used as heuristics
to distinguish heap vs. stack storage
- further investigation shows that libstdc++ applies the
small-object optimisation for functor up to »two slots«
in size -- but only if the copy-ctor is trivial. Thus
a lambda capturing a shared_ptr by value will *always*
be maintained in heap storage (and LazyInit must be
redesigned accordingly)...
- the verify_inlineStorage() unit test will now trigger
if some implementation does not apply small-object optimisation
under these minimal assumptions
...which is crucial for the solution pursued at the moment;
std::function is known to apply a small-object optimisation,
yet unfortunately there are no guarantees by the C++ standard
(it is only mandated that std::function handles a bare function
pointer without overhead)
Other people have investigated that behaviour already,
indicating that at least one additional »slot« of data
can be handled with embedded storage in all known implementations
(while libstdc++ seemingly imposes the strongest limitations)
https://stackoverflow.com/a/77202545/444796
This experiment in the unit-test shows that for my setup
(libstdc++ and GCC-8) only a lambda capturing a single pointer
is handled entirely embedded into the std::function; already
a lambda capturing a shared-ptr leads to overflow into heap
the RandomDraw rules developed last days are meant to be used
with user-provided λ-adapters; employing these in a context
of a DSL runs danger of producing dangling references.
Attempting to resolve this fundamental problem through
late-initialisation, and then locking the component into
a fixed memory location prior to actual usage. Driven by
the goal of a self-contained component, some advanced
trickery is required -- which again indicates better
to write a library component with adequate test coverage.
...now using the reworked partial-application helper...
...bind to *this and then recursively re-invoke the adaptation process
...need also to copy-capture the previously existing mapping-function
first test seems to work now
Investigation in test setup reveals that the intended solution
for dynamic configuration of the RandomDraw can not possibly work.
The reason is: the processing function binds back into the object instance.
This implies that RandomDraw must be *non-copyable*.
So we have to go full circle.
We need a way to pass the current instance to the configuration function.
And the most obvious and clear way would be to pass it as function argument.
Which however requires to *partially apply* this function.
So -- again -- we have to resort to one of the functor utilities
written several years ago; and while doing so, we must modernise
these tools further, to support perfect forwarding and binding
of reference arguments.
- strive at complete branch coverage for the mapping function
- decide that the neutral value can deliberately lie outside
the value range, in which case the probability setting
controls the number of _value_ result incidents vs
neutral value result incidents.
- introduce a third path to define this case clearly
- implement the range setting Builder-API functions
- absorb boundrary and illegal cases
For sake of simplicity, since this whole exercise is a byproduct,
the mapping calculations are done in doubles. To get even distribution
of values and a good randomisation, it is thus necessary to break
down the size_t hash value in a first step (size_t can be 64bit
and random numbers would be subject to rounding errors otherwise)
The choice of this quantiser is tricky; it must be a power of two
to guarantee even distribution, and if chosen to close to the grid
of the result values, with lower probabilities we'd fail to cover
some of the possible result values. If chosen to large, then
of course we'd run danger of producing correlated numbers on
consecutive picks.
Attempting to use 4 bits of headroom above the log-2 of the
required value range. For example, 10-step values would use
a quantiser of 128, which looks like a good compromise.
The following tests will show how good this choice holds up.
The first step was to allow setting a minimum value,
which in theory could also be negative (at no point is the
code actually limited to unsigned values; this is rather
the default in practice).
But reconsidering this extensions, then you'd also want
the "neutral value" to be handled properly. Within context,
this means that the *probability* controls when values other
than the neutral value are produced; especially with p = 1.0
the neutral value shall not be produced at all
...since the Policy class now defines the function signature,
we can no longer assume that "input" is size_t. Rather, all
invocations must rely on the generic adaptaion scheme.
Getting this correct turns out rather tricky again;
best to rely on a generic function-composition.
Indeed I programmed such a helper several years ago,
with the caveat that at that time we used C++03 and
could not perfect-forward arguments. Today this problem
can be solved much more succinct using generic Lambdas.
to define this as a generic library component,
any reference to the actual data source moust be extracted
from the body of the implementation and supplied later
at usage site. In the actual case at hand the source
for randomness would be the node hash, and that is
absolutely an internal implementation detail.
The idea is to use some source of randomness to pick a
limited parameter value with controllable probability.
While the core of the implementation is nothing more
than some simple numeric adjustments, these turn out
to be rather intricate and obscure; the desire to
package these technicalities into a component
however necessitates to make invocations
at usage site self explanatory.
...using a pre-established example as starting point
It seems that building up this kind of generator code
from a set of free functions in a secluded namespace
is the way most suitable to the nature of the C++ language
Use a simple destructor-trick to set up a concise notation
for temporarily manipulating a value for testing.
The manipulation will automatically be undone
when leaving scope
...to bring it more in line with all the other calls dealing with Activity*
...allows also to harmonise the ActivityLang::dispatchChain()
...and to compose the calls in Scheduler directly
NOTE: there is a twist: our string-formatting helper did not render
custom string conversions for objects passed as pointer. This was a
long standing problem, caused by ambiguous templates overloads;
now I've attempted to solve it one level more down, in util::StringConv.
This solution may turn out brittle, since we need to exclude any direct
string conversion, most notably the ones for C-Strings (const char*)
In case this solution turns out unsustainable, please feel free
to revert this API change, and return to passing Activity& in λ-post,
because in the end this is cosmetics.
As follow-up to the rework of thread-handling, likewise also
the implementation base for locking was switched over from direct
usage of POSIX primitives to the portable wrappers available in
the C++ standard library. All usages have been reviewed and
modernised to prefer λ-functions where possible.
With this series of changes, the old threadpool implementation
and a lot of further low-level support facilities are not used
any more and can be dismantled. Due to the integration efforts
spurred by the »Playback Vertical Slice«, several questions of
architecture could be decided over the last months. The design
of the Scheduler and Engine turned out different than previously
anticipated; notably the Scheduler now covers a wider array of
functionality, including some asynchronous messaging. This has
ramifications for the organisation of work tasks and threads,
and leads to a more deterministic memory management. Resource
management will be done on a higher level, partially superseding
some of the concepts from the early phase of the Lumiera project.
This is Step-2 : change the API towards application
Notably all invocation variants to support member functions
or a reference to bool flags are retracted, since today a
λ-binding directly at usage site tends to be more readable.
The function names are harmonised with the C++ standard and
emergency shutdown in the Subsystem-Runner is rationalised.
The old thread-wrapper test is repurposed to demonstrate
the effectiveness of monitor based locking.
After the fundamental switch from POSIX to the C++14 wrappers
the existing implementation of the Monitor can now be drastically condensed,
removing several layers of indirection. Moreover, all signatures
shall be changed to blend in with the names and patterns established
by the C++ standard.
This is Step-1 : consolidate the Implementation.
(to ensure correctness, the existing API towards application code was retained)
While not directly related to the thread handling framework,
it seems indicated to clean-up this part of the application alongside.
For »everyday« locking concerns, an Object Monitor abstraction was built
several years ago and together with the thread-wrapper, both at that time
based on direct usage of POSIX. This changeset does a mere literal
replacement of the POSIX calls with the corresponding C++ wrappers
on the lowest level. The resulting code is needlessly indirect, yet
at API-level this change is totally a drop-in replacment.
This, and the GUI thread prompted an further round of
design extensions and rework of the thread-wrapper.
Especially there is now support for self-managed threads,
which can be launched and operate completely detached from the
context used to start them. This resolves an occasional SEGFAULT
at shutdown. An alternative (admittedly much simpler) solution
would have been to create a fixed context in a static global
variable and to attach a regular thread wrapper from there,
managed through unique_ptr.
It seems obvious that the new solution is preferable,
since all the tricky technicalities are encapsulated now.
Add a complete demonstration for a setup akin to what we use
for the Session thread: a threaded component which manages itself
but also exposes an external interface, which is opened/closed alongside
...extract and improve the tuple-rewriting function
...improve instance tracking test dummy objects
...complete test coverage and verify proper memory handling
After quite some detours, with this take I'm finally able to
provide a stringent design to embody all the variants of thread start
encountered in practice in the Lumiera code base.
Especially the *self-managed* thread is now represented as a special-case
of a lifecycle-hook, and can be embodied into a builder front-end,
able to work with any client-provided thread-wrapper subclass.
to cover the identified use-cases a wide variety of functors
must be accepted and adapted appropriately. A special twist arises
from the fact that the complete thread-wrapper component stack works
without RTTI; a derived class can not access the thread-wrapper internals
while the policy component to handle those hooks can not directly downcast
to some derived user provided class. But obviously at usage site it
can be expected to access both realms from such a callback.
The solution is to detect the argument type of the given functor
and to build a two step path for a safe static cast.
...after resolving the fundamental design problems,
a policy mix-in can be defined now for a thread that deletes
its own wrapper at the end of the thread-function.
Such a setup would allow for »fire-and-forget« threads, but with
wrapper and ensuring safe allocations. The prominent use case
for such a setup would be the GUI-Thread.
Concept study of the intended solution successful.
Can now transparently embed any conceivable functor
and an arbitrary argument sequence into a launcher-λ
Materialising into a std::tuple<decay_t<TYPES...>> did the trick.
Considering a solution to shift the actual launch of the new thread
from the initialiser list into the ctor body, to circumvent the possible
"undefined behaviour". This would also be prerequisite for defining
a self-managed variant of the thread-wrapper.
Alternative / Plan.B would be to abandon the idea of a self-contained
"thread" building block, instead relying on precise setup in the usage
context -- however, not willing to yield yet, since that would be exactly
what I wanted to avoid: having technicalities of thread start, argument
handover and failure detection intermingled with the business code.
On a close look, the wrapper design as pursued here
turns out to be prone to insidious data race problems.
This was true also for the existing solution, but becomes
more clear due to the precise definitions from the C++ standard.
This is a confusing situation, because these races typically do not
materialise in practice; due to the latency of the OS scheduler the
new thread starts invoking user code at least 100µs after the Wrapper
object is fully constructed (typically more like 500µs, which is a lot)
The standard case (lib::Thread) in its current form is correct, but borderline
to undefined behaviour, and any initialisation of members in a derived class
would be off limits (the thread-wrapper should not be used as baseclass,
rather as member)
...while reworking the application code, it became clear that
actually there are two further quite distinct variants of usage.
And while these could be implemented with some trickery based on
the Thread-wrapper defined thus far, it seems prudent better to
establish a safely confined explicit setup for these cases:
- a fire-and-forget-thread, which manages its own memory autonomously
- a thread with explicit lifecycle, with detectable not-running state
...these were already written envisionaging he new API,
so it's more or less a drop-in replacement.
- cant use vector anymore, since thread objects are move-only
- use ScopedCollection instead, which also has the benefit of
allocating the requires space up-front. Allow to deduce the
type parameter of the placed elements
... which became apparent after switching to the new Thread-wrapper implementation
... the reason is a bug in the Thread-Monitor (which will also be reworked soon)
