Yet another chainsaw massacre.
One of the most obnoxious annoyances with C++ metaprogramming
is the need to insert `typename` and `template` qualifiers into
most definitions, to help the compiler to cope with the syntax,
which is not context-free.
The recent standards adds several clarifications, so that most
of these qualifiers are redundant now, at least at places where
it is unambiguously clear that only a type can be given.
GCC already supports most of these relaxing rules
(Clang unfortunately lags way behind with support of newer language features...)
The Lumiera »Reference Platform« is now upgraded to Debian/Buster, which provides GCC-14 and Clang-20.
Thus the compiler support for C++20 language features seems solid enough, and C++23,
while still in ''experimental stage'' can be seen as a complement and addendum.
This changeset
* upgrades the compile switches for the build system
* provides all the necessary adjustments to keep the code base compilable
Notable changes:
* λ-capture by value now requires explicit qualification how to handle `this`
* comparison operators are now handled transparently by the core language,
largely obsoleting boost::operators. This change incurs several changes
to implicit handling rules and causes lots of ambiguities — which typically
pinpoint some long standing design issues, especially related to MObjects
and the ''time entities''. Most tweaks done here can be ''considered preliminary''
* unfortunately the upgraded standard ''fails'' to handle **tuple-like** entities
in a satisfactory way — rather an ''exposition-only'' concept is introduced,
which applies solely to some containers from the STL, thereby breaking some
very crucial code in the render entities, which was built upon the notion of
''tuple-like'' entities and the ''tuple protocol''. The solution is to
abandon the STL in this respect and **provide an alternative implementation**
of the `apply` function and related elements.
This resolves an intricate problem related to metaprogramming with
variadic templates and function signatures. Due to exceptional complexity,
a direct solution was blocked for several years, and required a better
organisation of the support code involved; several workarounds were
developed, gradually leading to a transition path, which could now
be completed in an focused clean-up effort over the last week.
Metaprogramming with sequences of types is organised into three layers:
- simple tasks can be solved with the standard facilities of the language,
using pattern match with variadic template specialisations
- the ''type-sequence'' construct `Types<T...>` takes the centre stage
for the explicit definition of collections of types; it can be re-bound
to other variadic templates and supports simple direct manipulation
- for more elaborate and advanced processing tasks, a ''Loki-style type list''
can be obtained from a type-sequence, allowing to perform recursive
list processing task with a technique similar to LISP.
after all the relevant library components do support both kinds of
type sequences transparently, any usages in core code can now be
switched over to the new, variadic type sequences.
Attempting to reduce the remaining pre-C++11 workarounds before upgrade to C++20...
As a first step: rename the old type-sequence implementation into `TyOLD`
to make it clearly distinguishable; a new variadic implementation `TySeq`
was already introduced as partial workaround, and the next steps
will be to switch over essential parts of the type-sequence library.
Investigated this topic again...
* these were initially created before C++11
* at that time, ''non-copyable'' objects were not common place
* but we embraced that concept already, and thus had quite some pain
when attempting to use such objects in STL containers
* with C++11 and ''move semantics'' these problems basically evaporated
* most usages were already upgraded and resolved
* another use case is to handle a state variable, which is based on
an immutable entity (like Time entities); `ItemWrapper` can be used
as a remedy in such a situation
The `FixedFrameQuantiser` relied on three functions from the raw-time handling library.
Since this (and NTSC drop-frame) are the only usages, these functions
can be relocated into the implemntation translation unit `lib/time/quantiser.cpp`
On closer inspection, this reveals some room for improvements:
Instead of relying on raw-computation functions written in C,
we could rather revert the dependency and express these computations
in terms of our Time-entities, which are written in C++, are much more
systematic and provide consistency checks and protection against numeric
overflow, all integrated with linear arithmetic and concise notation.
After performing these rearrangements,
most of the functions can be collapsed into ''almost nothing''.
This was taken as opportunity to re-check and improve the remaining
implementation core of the `FixedFrameQuantiser` -- the handling of
extreme corner cases can be much improved, now representing the
"grid-local time" as `Offset`, which doubles the possible value range.
The reworked unit test shows that, with this change, now the limitation
happens prior to quantisation, meaning that we always get a grid-aligned
result, even in the most extreme corner cases.
...extract these functions and the associated test
from the low-level C time handling library and
document them with a dedicated C++ header and test.
''This is unfinished work'' —
the extracted functions provide only the low level computations;
actually, a specialised time quantisation or time code would be required.
------------
Note though,
after extracting these functions, the rest of the plain-C test
can be removed, since equivalent functionality is covered in
much more detail by the tests of the C++ time handling framework.
Notably this allows to get rid of the direct component accessor functions.
------------
__Remark__: the base implementation of many time conversion functions
and especially NTSC drop-frame was provided by Stefan Kangas
See:
6a44134833
While these function may seem superficially plausible,
I more and more come to the conclusion that offering such
function as ''basic building blocks'' is in itself an
ill-guided approach to handling of time entities.
Time is neither „just a number“ — nor does it „contain“ hours, minutes and seconds.
It is possible to ''represent'' it through a **time-code**, which incurs
a quantisation step and implies a reference grid.
Thus Lumiera ''should not offer'' a »basic time handling library«.
Doing so would be just an invitation to bypass proper time handling
and avoid the use of more demanding but also more adequate mental concepts.
So the next step will be to remove functions not deemed adequate, and
better directly inline the respective modulus based computations.
Other functions can be integrated into the respective implementation
translation units for time quantisation and timecode representation.
Indeed — this change set is kind of sad.
Because I still admire the design of the GAVL library,
and would love to use it for processing of raw video.
However, up to now, we never got to the point of actually
doing so. For the future, I am not sure if there remains
room to rely on lib-GAVL, since FFmpeg roughly covers
a similar ground (and a lot beyond that). And providing
a plug-in for FFmpeg is unavoidable, practically speaking.
So I still retain the nominal dependency on lib-GAVL
in the Build system (since it is still packaged in Debian).
But it is pointless to rely on this library just for an
external type-def `gavl_time_t`. We owe much to this
inspiration, but it can be expected that we'll wrap
these raw time-values into a dedicated marker type
soon, and we certainly won't be exposing any C-style
interface for time calculations in future, since
we do not want anyone to side-step the Lumiera
time handling framework in favour of working
„just with plain numbers“
NOTE: lib-GAVL hompage has moved to Github:
https://github.com/bplaum/gavl
- remove obsolete configuration settings
- walk through all settings according to the documentation
https://www.doxygen.nl/manual/config.html
- now try to use the new feature to rely on Clang for C++ parsing
- walk through the doxygen-warnings.txt and fix some obvious misspellings
and structural problems in the documentation comments.
With Debian-Trixie, we are now using Doxygen 1.9.8 —
which produces massively better results in various fine points.
However, there are still problems with automatic cross links,
especially from implementation to the corresponding test classes.
During Render Node invocation, automation parameter data must be maintained.
For the simple standard path, this just implies to store the ''absolute nominal Time''
directly in the invoking stack frame and let some parameter adaptors do the translation.
However, it is conceivable to have much more elaborate translation functions,
and thus we must be prepared to handle an arbitrary number of parameter slots,
where each slot has arbitrary storage requirements.
The conclusion is to start with an intrusive linked list of overflow buckets.
* Lumiera source code always was copyrighted by individual contributors
* there is no entity "Lumiera.org" which holds any copyrights
* Lumiera source code is provided under the GPL Version 2+
== Explanations ==
Lumiera as a whole is distributed under Copyleft, GNU General Public License Version 2 or above.
For this to become legally effective, the ''File COPYING in the root directory is sufficient.''
The licensing header in each file is not strictly necessary, yet considered good practice;
attaching a licence notice increases the likeliness that this information is retained
in case someone extracts individual code files. However, it is not by the presence of some
text, that legally binding licensing terms become effective; rather the fact matters that a
given piece of code was provably copyrighted and published under a license. Even reformatting
the code, renaming some variables or deleting parts of the code will not alter this legal
situation, but rather creates a derivative work, which is likewise covered by the GPL!
The most relevant information in the file header is the notice regarding the
time of the first individual copyright claim. By virtue of this initial copyright,
the first author is entitled to choose the terms of licensing. All further
modifications are permitted and covered by the License. The specific wording
or format of the copyright header is not legally relevant, as long as the
intention to publish under the GPL remains clear. The extended wording was
based on a recommendation by the FSF. It can be shortened, because the full terms
of the license are provided alongside the distribution, in the file COPYING.
After augmenting our `lib/random.hpp` abstraction framework to add the necessary flexibility,
a common seeding scheme was ''built into the Test-Runner.''
* all tests relying on some kind of randomness should invoke `seedRand()`
* this draws a seed from the `entropyGen` — which is also documented in the log
* individual tests can now be launched with `--seed` to force a dedicated seed
* moreover, tests should build a coherent structure of linked generators,
especially when running concurrently. The existing tests were adapted accordingly
All usages of `rand()` in the code base were investigated and replaced
by suitable calls to our abstraction framework; the code base is thus
isolated from the actual implementation, simplifying further adaptation.
- SchedulerStress_test simply takes too long to complete (~4 min)
and is thus aborted by the testrunner. Add a switch to allow for
a quick smoke test.
- SchedulerCommutator_test aborts due to an unresolved design problem,
which I marked as failure
- add some convenience methods for passing arguments to tests
We use the memory address to detect reference to ''the same language object.''
While primarily a testing tool, this predicate is also used in the
core application at places, especially to prevent self-assignment
and to handle custom allocations.
It turns out that actually we need two flavours for convenient usage
- `isSameObject` uses strict comparison of address and accepts only references
- `isSameAdr` can also accept pointers and even void*, but will dereference pointers
This leads to some further improvements of helper utilities related to memory addresses...
* most usages are drop-in replacements
* occasionally the other convenience functions can be used
* verify call-paths from core code to identify usages
* ensure reseeding for all tests involving some kind of randomness...
__Note__: some tests were not yet converted,
since their usage of randomness is actually not thread-safe.
This problem existed previously, since also `rand()` is not thread safe,
albeit in most cases it is possible to ignore this problem, as
''garbled internal state'' is also somehow „random“
In the Lumiera code base, we use C-String constants as unique error-IDs.
Basically this allows to create new unique error IDs anywhere in the code.
However, definition of such IDs in arbitrary namespaces tends to create
slight confusion and ambiguities, while maintaining the proper use statements
requires some manual work.
Thus I introduce a new **standard scheme**
* Error-IDs for widespread use shall be defined _exclusively_ into `namespace lumiera::error`
* The shorthand-Macro `LERR_()` can now be used to simplify inclusion and referral
* (for local or single-usage errors, a local or even hidden definition is OK)
reduce footprint of lib/util.hpp
(Note: it is not possible to forward-declare std::string here)
define the shorthand "cStr()" in lib/symbol.hpp
reorder relevant includes to ensure std::hash is "hijacked" first
- use a dedicated context "dropped off" the TestChainLoad instance
- encode the node-idx into the InvocationInstanceID
- build an invocation- and a planning-job-functor
- let planning progress over an lib::UninitialisedStorage array
- plant the ActivityTerm instances into that array as Scheduling progresses
...refine the handling of FrameRates close to the definition bounds
...implement the actual rule to scale allocator capacity on announcement
...hook up into the seedCalcStream() with a default of +25FPS
+ test coverage
...whenever a new CalcStream is seeded, it would be prudent
not only to step up the WorkForce (which is already implemented),
but also to provide a hint to the BlockFlow allocator regarding
the expected calculation density.
Such a hint would allow to set a more ample »epoch« spacing,
thereby avoiding to drive the allocator into overload first.
The allocator will cope anyway and re-balance in a matter of
about 2 seconds, but avoiding this kind of control oscillations
altogether will lead to better performance at calculation start.
This is Step-2 : change the API towards application
Notably all invocation variants to support member functions
or a reference to bool flags are retracted, since today a
λ-binding directly at usage site tends to be more readable.
The function names are harmonised with the C++ standard and
emergency shutdown in the Subsystem-Runner is rationalised.
The old thread-wrapper test is repurposed to demonstrate
the effectiveness of monitor based locking.
While not directly related to the thread handling framework,
it seems indicated to clean-up this part of the application alongside.
For »everyday« locking concerns, an Object Monitor abstraction was built
several years ago and together with the thread-wrapper, both at that time
based on direct usage of POSIX. This changeset does a mere literal
replacement of the POSIX calls with the corresponding C++ wrappers
on the lowest level. The resulting code is needlessly indirect, yet
at API-level this change is totally a drop-in replacment.
The existing TypedCounter_test was excessively clever and convoluted,
yet failed to test the critical elements systematically. Indeed, two
bugs were hidden in synchronisation and instance access.
- build a new concurrent test from scratch, now using the threadBenchmark
function for the actual concurrent execution and just invoked a
random selected access to the counter repeatedly from a large number
of threads.
- rework the TypedContext and counter to use Atomics where applicable;
measurements indicate however that this has only negligible impact
on the amortised invocation times, which are around 60ns for single-threaded
access, yet can increase by factor 100 due to contention.
While it would be straight forward from an implementation POV
to just expose both variants on the API (as the C++ standard does),
it seems prudent to enforce the distinction, and to highlight the
auto-detaching behaviour as the preferred standard case.
Creating worker threads just for one computation and joining the results
seemed like a good idea 30 years ago; today we prefer Futures or asynchronous
messaging to achieve similar results in a robust and performant way.
ThreadJoinable can come in handy however for writing unit tests, were
the controlling master thread has to wait prior to perform verification.
So the old design seems well advised in this respect and will be retained
* using a simplified preliminary implementation of hash chaining (see #1293)
* simplistic implementation of hashing for time values (half-rotation)
* for now just hashing the time into the upper part of the LUID
Maybe we can even live with that implementation for some time,
depending on how important uniform distribution of hash values is
for proper usage of the frame cache.
Needless to say, various further fine points need more consideration,
especially questions of portability (32bit anyone?). Moreover, since
frame times are typically quantised, the search space for the hashed
time values is drastically reduced; conceivably we should rather
research and implement a good hash function for 128bit and then combine
all information into a single hash key....
Note: changing behaviour of TimeSpan to possibly flip start and end,
and also to use Offset as Offset and then re-orient,
since this seems the least surprising behaviour.
These changes carry over into changed default and limiting
on ZoomWindow constructor and various mutators, and most
notably shifting the time span always into allowed domain.
...building on these Library changes, plus the safe-add function
developed some days ago, it is now possible to mark a large displacement
as `time::Offset`, and apply this to yield any valid time position,
even extreme negative values
The APIs for time quantisation were drafted in an early stage of the project
and then never followed-up. Especially Grid::gridAlign has no
real-world usage yet, and is only massaged in some tests.
When looking at QuantiserBasics_test, I was puzzled and led astray,
since this function suggests to materialise a continuous time into
a quantised time -- which it doesn't (there is another dedicated
function Quantiser::materialise() to that end); so, without engaging
into the discussion if this function is of any use, I'll hereby
choose a name better reflecting what it does.
This is a deep refactoring to allow to represent the distance
between all valid time points as a time::Offset or time::Duration.
By design this is possible, since Time::MAX was defined as 1/30 of
the maximum value technically representable as int64_t. However,
introducing a different limiter for offsets and durations turns
out difficult, due to the inconsistencies in the exiting hierarchy
of temporal entities. Which in turn seems to stem from the unfortunate
decision to make time entities immutable, see #1261
Since the limiter is hard wired into the `time::TimeValue` constructor,
we are forced to create a "backdoor" of sorts, to pass up values
with different limiting from child classes. This would not be so
much of a problem if calculations weren't forced to go through `TimeVar`,
which does not distinguish between time points and time durations.
This solution rearranges all checks to be performed now by time::Offset,
while time::Duration will only take the absolute value at construction,
based on the fact that there is no valid construction path to yield
a duration which does not go through an offset first.
Later, when we're ready to sort out the implementation base of time values
(see #1258), this design issue should be revisited
- either we'll allow derived classes explicitly to invoke the limiter functions
- or we may be able to have an automatic conversion path from clearly
marked base implementation types, in which case we wouldn't use the
buildRaw_() and _raw() "backdoor" functions any more...
When drafting the time handling framework some years ago,
I foresaw the possible danger of mixing up numbers relating
to fractional seconds, with other plain numbers intended as
frame counts or as micro ticks. Thus I deliberately picked
an incompatible integer type for FSecs = boost::rational<long>
However, using long is problematic in itself, since its actual
bit length is not fixed, and especially on 32bit platforms long
is quite surprisingly defined to be the same as int.
However, meanwhile, using the new C++ features, I have blocked
pretty much any possible implicit conversion path, requiring
explicit conversions in the relevant ctor invocations. So,
after weighting in the alternatives, FSecs is now defined
as boost::rational<int64_t>.
Mostly, std::regexp can be used as a drop-in replacement.
Note: unfortunately ECMA regexps do not support lookbehind assertions.
This lookbehind is necesary here because we want to allow parsing values
from strings with additional content, which means we need explicitly to
exclude mismatches due to invalid syntax.
We can work around that issue like "either line start, or *not* one of these characters.
Alternatively we could consider to make the match more rigid,
i.e we would require the string to conain *only* the timecode spec to be parsed.
effectively we rely in the micro tick timescale promoted by libGAVL,
but it seems indicated to introduce our own constant definition.
And also clarify some comments and tests.
(this changeset does not change any values or functionality)
...which showed up under high system load.
The initialisation of the member variables for the check sum
could be delayed while the corresponding thread was already running
the (trivial) implementation turned out to be correct as written,
but it was (again) damn challenging to get the mulithreaded chaotic
test fixture and especially the lambda captures to work correct.
