Testcase: A simple Sementation with a single and bounded Segment
As aside, figured out how to unpack an iterator such as to
tie a fixed number of references through a structural binding:
auto const& [s1,s2,s3] = seqTuple<3> (mockSegs.eachSeg());
...now able to build a mock segmentation which issues dummy jobs,
and is wired such as to verify the right job is invoked for each segment.
And this allows to build and verify the Dispatcher,
without being able to invoke actual render jobs yet.
There are 12 distinct cases regarding the orientation of two intervals;
The Segmentation::splitSplice() operation shall insert a new Segment
and adjust / truncate / expand / split / delete existing segments
such as to retain the *Invariant* (seamless segmentation covering
the complete time axis)
- how to pass-in a specification given as GenNode
- now this might be translated into a MockJobTicket allocated in the MockSegmentation
Unimplemented: actually build the Segment with suitable start/end time
right now we're lacking a complete working implementation of render node invocation,
and thus the Dispatcher implementation can only be verified with the help
of mocked jobs. However, at least a preliminary implementation of tagging the
invocation instance is available, and thus we're able to verify that
a given job instance indeed belongs to and is "backed" by a specific JobTicket.
This is prerequisite for building up a (likewise mocked) Fixture datastructure,
and this in turn was meant to form the basis for attacking an actual Scheduler
implementation, followed by a real render node invocation.
- can now create a Job from JobTicket::NIL
- on invocation this Job will to nothing
Only when the first real output backend is implemented,
we can decide if this simplistic implementation is enough,
or if an empty output must be explicitly generated...
* using a simplified preliminary implementation of hash chaining (see #1293)
* simplistic implementation of hashing for time values (half-rotation)
* for now just hashing the time into the upper part of the LUID
Maybe we can even live with that implementation for some time,
depending on how important uniform distribution of hash values is
for proper usage of the frame cache.
Needless to say, various further fine points need more consideration,
especially questions of portability (32bit anyone?). Moreover, since
frame times are typically quantised, the search space for the hashed
time values is drastically reduced; conceivably we should rather
research and implement a good hash function for 128bit and then combine
all information into a single hash key....
...using the MockJobTicket setup as point of reference,
since the actual invocation of render nodes will only be drafted
later in this "Vertical Slice" integration effort...
- introduce a JobTicket::NOP (null-object pattern)
- assuming that the function splitSplice() will retain complete coverage allways
Remark:
`Fixture::getPlaylistForRender()` is a leftover from the very early implementation drafts.
This function was more or less based on the way Cinelerra works; it is clear by now
that Lumiera can not possibly work this way, given that we'll build a low-level model
and dispatch precompiled render jobs....
The Fixture and the low-level model backbone deserve a distinct namespace on their own.
Since it's built by the Builder from the Session contents, and also used by the frame dispatch,
we can expect dependence on some types from Steam-Layer, and thus this namespace
needs to reside in Steam-Layer rather, while the actual low-level Model
might become part of Vault-Layer, creating a hierarchy of data structures.
(Remark: likely also the session related namespaces will need a reorganisation)
The idea is to escape a "design deadlock" by using a test-driven prototype
implementation of the data structure to back a further development
of the Dispatcher and Scheduler implementation, which then can be used
to gradually elaborate and switch over to an actual implementation
data structure
...requires a first attempt towards defining a `JobTiket`.
This turns out quite tricky, due to using those `LinkedElements`
(intrusive single linked list), which requires all added records
actually to live elsewhere. Since we want to use a custom allocator
later (the `AllocationCluster`), this boils down to allocating those
records only when about to construct the `JobTicket` itself.
What makes matters even worse: at the moment we use a separate spec
per Media channel (maybe these specs can be collapsed later non).
And thus we need to pass a collection -- or better an iterator
with raw specs, which in turn must reveal yet another nested
sequence for the prerequisite `JobTickets`.
Anyhow, now we're able at least to create an empty `JobTicket`,
backed by a dummy `JobFunctor`....
Looks like we'll actually retain and use this low-level solution
in cases where we just can not afford heap allocations but need
to keep polymorphic objects close to one another in memory.
Since single linked lists are filled by prepending, it is rather
common to need the reversed order of elements for traversal,
which can be achieved in linear time.
And while we're here, we can modernise the templated emplacement functions
- build the reworked Job-planning pipeline more or less from scratch
- back that with mocked `Dispatcher` and `JobTicket`
- then transfer this into a `RenderDrive`, which can be tested as well
- could continue then to a `CalcStream` integration test....
- decision: the Monad-style iteration framework will be abandoned
- the job-planning will be recast in terms of the iter-tree-explorer
- job-planning and frame dispatch will be disentangled
- the Scheduler will deliberately offer a high-level interface
- on this high-level, Scheduler will support dependency management
- the low-level implementation of the Scheduler will be based on Activity verbs
The APIs for time quantisation were drafted in an early stage of the project
and then never followed-up. Especially Grid::gridAlign has no
real-world usage yet, and is only massaged in some tests.
When looking at QuantiserBasics_test, I was puzzled and led astray,
since this function suggests to materialise a continuous time into
a quantised time -- which it doesn't (there is another dedicated
function Quantiser::materialise() to that end); so, without engaging
into the discussion if this function is of any use, I'll hereby
choose a name better reflecting what it does.
...in an attempt to clarify why numerous cross links are not generated.
In the end, this attempt was not very successful, yet I could find some breadcrumbs...
- file comments generally seem to have a problem with auto link generation;
only fully qualified names seem to work reliably
- cross links to entities within a namespace do not work,
if the corresponding namespace is not documented in Doxygen
- documentation for entities within anonymous namespaces
must be explicitly enabled. Of course this makes only sense
for detailed documentation (but we do generate detailed
documentation here, including implementation notes)
- and the notorious problem: each file needs a valid @file comment
- the hierarchy of Markdown headings must be consistent within each
documentation section. This entails also to individual documented
entities. Basically, there must be a level-one heading (prefix "#"),
otherwise all headings will just disappear...
- sometimes the doc/devel/doxygen-warnings.txt gives further clues
the reason for the failure, as it turned out,
is that 'noexcept' is part of the function signature since C++17
And, since typically a STL container has const and non-const variants
of the begin() and end() function, the match to a member function pointer
became ambuguous, when probing with a signature without 'noexcept'
However, we deliberately want to support "any STL container like" types,
and this IMHO should include types with a possibly throwing iterator.
The rationale is, sometimes we want to expose some element *generator*
behind a container-like interface.
At this point I did an investigation if we can emulate something
in the way of a Concept -- i.e. rather than checking for the presence
of some functions on the interface, better try to cover the necessary
behaviour, like in a type class.
Unfortunately, while doable, this turns out to become quite technical;
and this highlights why the C++20 concepts are such an important addition
to the language.
So for the time being, we'll amend the existing solution
and look ahead to C++20
as it turns out, "almost" the whole codebase compiles in C++17 mode.
with the exception of two metaprogramming-related problems:
- our "duck detector" for STL containers does not trigger anymore
- the Metafunction to dissect Function sigantures (meta::_Fun) flounders
When drafting the time handling framework some years ago,
I foresaw the possible danger of mixing up numbers relating
to fractional seconds, with other plain numbers intended as
frame counts or as micro ticks. Thus I deliberately picked
an incompatible integer type for FSecs = boost::rational<long>
However, using long is problematic in itself, since its actual
bit length is not fixed, and especially on 32bit platforms long
is quite surprisingly defined to be the same as int.
However, meanwhile, using the new C++ features, I have blocked
pretty much any possible implicit conversion path, requiring
explicit conversions in the relevant ctor invocations. So,
after weighting in the alternatives, FSecs is now defined
as boost::rational<int64_t>.
This was prompted by a test failing under Boost-1.65 (--> see #294)
When reviewed now, the whole idea of testing Steam-Layer Commands for
equivalence feels a bit sketchy.
Just the comparison for the command ''identity'' alone seems sufficient,
i.e. the test if a command-ID is associated with the same backend-handle
and thus the same functor binding.
- we got occasional hangups when waiting for disabled state
- the builder was not triggered properly, sometimes redundant, sometimes without timeout
As it turned out, the loop control logic is more like a state machine,
and the state variables need to be separated from the external influenced variables.
As a consequence, the inChange_ variable was not calculated properly when disabled in a race,
and then the loop went into infinite wait state, without propagating this to
the externally waiting client, which caused the deadlock
basically we can pick just any convention here, and so we should pick the convention in a way
that makes most sense informally, for a *human reader*. But what we previously did, was to pick
the condition such as to make it simple in some situations for the programmer....
With the predictable result: even with the disappointingly small number of usages we have up to now,
we got that condition backwards several times.
OK, so from now on!!!
Time::NEVER == Time::MAX, because "never" is as far as possible into the future
- most notably the NOBUG logging flags have been renamed now
- but for the configuration, I'll stick to "GUI" for now,
since "Stage" would be bewildering for an occasional user
- in a similar vein, most documentation continues to refer to the GUI
...it should have been explicit from start, since there is no point
in converting an EntryID into a plain flat string without further notice
this became evident, when the compiler picked the string overload on
MakeRec().genNode(specialID)
...which is in compliance to the rules, since string is a direct match,
while BareEntryID would be an (slicing) upcast. However, obviously we
want the BareEntryID here, and not an implicit string conversion,
thereby discarding the special hash value hidden within the ID
As it turns out, using the functional-notation form conversion
with *parentheses* will fall back on a C-style (wild, re-interpret) cast
when the target type is *not* a class. As in the case in question here, where
it is a const& to a class. To the contrary, using *curly braces* will always
attempt to go through a constructor, and thus fail as expected, when there is
no conversion path available.
I wasn't aware of that pitfall. I noticed it since the recently introduced
class TimelineGui lacked a conversion operator to BareEntryID const& and just
happily used the TimelineGui object itself and did a reinterpret_cast into BareEntryID
I am fully aware this change has some far reaching ramifications.
Effectively I am hereby abandoning the goal of a highly modularised Lumiera,
where every major component is mapped over the Interface-System. This was
always a goal I accepted only reluctantly, and my now years of experience
confirm my reservation: it will cost us lots of efforts just for the
sake of being "sexy".
- state-of-the-art implementation of access with Double Checked Locking + Atomics
- improved design for configuration of dependencies. Now at the provider, not the consumer
- support for exposing services with a lifecycle through the lib::Depend<SRV> front-end
IterSource should be refactored to have an iteration control API similar to IterStateWrapper.
This would resolve the need to pass that pos-pointer over the abstraction barrier,
which is the root cause for all the problems and complexities incurred here
after extended analysis, it turned out to be a "placeholder concept"
and introduces an indirection, which can be removed altogether
- simple command invocation happens at gui::model::Tangible
- it is based on the command (definition) ID
- instance management happens automatically and transparently
- the extended case of context-bound commands will be treated later,
and is entirely self-contained
due to the refactorings, the instance was moved out prior to checking for
bound arguments. This is ammended now, albeit at the price of passing an
additional flagn and some tricky boolean conditions
in accordance to the design changes concluded yesterday.
- in the standard cases we now check the global registry first
- automatically create anonymous clone copy from global commands
- reorganise code internally to use common tail implementation
as it turns out, we can always trigger commands right away,
the moment all arguments are known. Thus it is sufficient to
send a single argument binding message, which allows us to
get rid of a lot or ugly complexities (payload visitor).
It seems more adequate to push the somewhat intricate mechanics
for the "fall back" onto generic commands down into the implementation
level of CommandInstanceManager. The point is, we know the standard
usage situation is to rely on the instance manager, and thus we want
to avoid redundant table lookups, only to support the rare case of
fallback to global commands. The latter is currently used only from
unit-tests, but might in future also be used by scripts.
Due to thread safety considerations, I have refrained from handing
out a direct reference to the command token sitting in the registry,
even while not doing so incurs a small runtime penalty (accessing
the shared ref-count for creating a copy of the smart-handle).
This is the typical situation where you'd be tempted to sacrifice
sanity for the sake of an imaginary performance benefit, which
in fact is dwarfed by all the machinery of UI-Bus and argument
passing via GenNode.
The instance manager opens (creates) a new instance by cloning
from the prototype. Unless this instance is dispatched, it does not
allow to open a further instance (for the same instanceID). But of course
it allows to open a different instance from the same prototype
This changeset fixes a huge pile of problems, as indicated in the
error log of the Doxygen run after merging all the recent Doxygen improvements
unfortunately, auto-linking does still not work at various places.
There is no clear indication what might be the problem.
Possibly the rather unstable Sqlite support in this Doxygen version
is the cause. Anyway, needs to be investigated further.
this is indeed a change of concept.
A 'command instance' can not be found through the official
Command front-end anymore, since we do not create a registration.
This allows us to avoid decorating command IDs with running counters
Up to now, these dummy functions where used by various unit tests
directly, by creating command definitions within the test fixture.
But since it is foreseeable that we'll need dummy commands for various
further unit tests, it seems adequate to setup a global static registration
with the newly created system of command registrations for these dummies.
this is a prerequisite for command instance management:
We have now an (almost) complete framework for writing actual
command definitions in practice, which will be registered automatically.
This could be complemented (future work) by a script in the build process
to regenerate proc/cmd.hpp based on the IDs of those automatic definitions.
as it stands, this does not work, since lambdas are passed by-value,
while function references can only be passed by explicit reference,
otherwise they'll degrade to a function pointer. And std::function
requires a plain function signature as type argument, not the type
of a function pointer (which doesn't mean you can't construct a
std::function from a FP, indeed there is an explicit overload for
that).
The point in question is how to manage these definitions in practice,
since we're about to create a huge lot of them eventually. The solution
attempted here is heavily inspired by the boost-test framework
this bit of Sed magic relies on the fact that we happen to write
the almost correct class name of a test into the header comment.
HOWTO:
for F in $(find tests -type f \( -name '*.cpp' \) -exec egrep -q '§§TODO§§' {} \; -print);
do sed -r -i -e'
2 {h;x;s/\s+(.+)\(Test\).*$/\\ref \1_test/;x};
/§§TODO§§/ {s/§§TODO§§//;G;s/\n//}'
$F;
done
Doxygen will only process files with a @file documentation comment.
Up to now, none of our test code has such a comment, preventing the
cross-links to unit tests from working.
This is unfortunate, since unit tests, and even the code comments there,
can be considered as the most useful form of technical documentation.
Thus I'll start an initiative to fill in those missing comments automatically
Writing and debugging such tests is always an interesting challenge...
Fortunately this exercise didn't unveil any problem in the newly written
code, only some insidious problems in the test fixture itself. Which
again highlights the necessity, that each *command instance* needs
to be an independent clone from the original *command prototype*,
since argument binding messages and trigger messages can appear
in arbitrary order.
the intention is to cover more of the full invocation path,
without running all of the application infrastructure. So this
second test cases simulates how messages are handled in CoreService,
where the CommandHandler (visitor) actually invokes the SessionCommand
facade
command processing against the session is not yet implemented,
so to allow for unit testing, we magically recognise all commands
starting with "test." and invoke them directly within the Dispatcher.
With this addition, the basic functionality of the dispatcher works now
need also to start and stop the interface registry,
since by policy we do not run the application framework itself
for execution of the test suite; thus if some test actually needs
an application service, it must be started/stopped manually
...since the session loop will be notified on any change via the
interface, adding a command will activate the loop, and the builder
timeout is handled separately via the dirty state. So there is no
need to spin around the loop in idle state.
As a aside, timeout waiting on a condition variable can be intentional
and should thus not be logged as an error automatically. It is up to the
calling context to decide if a timeout constitutes an exceptional situation.
It is always a trade-off performance vs. readability.
Sometimes a single-threaded implementation of self-contained logic
is preferable to a slightly more performant yet obscure implementation
based on our threadpool and scheduler.
...this means to turn Looper into a state machine.
Yet it seems more feasible, since the DispatcherLoop has a nice
checkpoint after each iteration through the while loop, and we'd
keep that whole builder-dirty business completely confined within
the Looper (with a little help of the DispatcherLoop)
Let's see if the state transition logic can actually be implemented
based just on such a checkpoint....?
reason is, only files with a @file comment will be processed
with further documentation commands. For this reason, our Doxygen
documentation is lacking a lot of entries.
HOWTO:
find src -type f \( -name '*.cpp' -or -name '*.hpp' \) -not -exec egrep -q '\*.+@file' {} \; -print -exec sed -i -r -e'\_\*/_,$ { 1,+0 a\
\
\
/** @file §§§\
** TODO §§§\
*/
}' {} \;
While in general it is fine to clean-up any entity IDs
to be US-ASCII alphanumerics (plus some allowed interpunction),
the GenNodes and also keys in object-bindings for diff are
considerd internal interfaces, assuming that any passed
ID symbol is already sanitised and checked. So the
sanitise operation can be skipped. This changeset
adds the same option directly to lib::EntryID,
allowing to create an EntryID that matches
a similar GenNode's (hash) ID.
the functionality as such is already covered,
but it seems important enough to warrant a dedicated test.
incidentally, Duration still lacked a default ctor.
Time values are default constructible, yet immutable.
Completely removed the nested hierarchy, where
the top-level implementation forwarded to yet another
sub-implementation of the same interface. Rather, this
sub-implementation (OpClosure) is now a mere implementation
detail class without VTable, and without half-baked
re-implementation of the CmdClosure interface. And the
state-switch from unbound to bound arguments is now
implemented as a plain-flat boolean flag, instead of
hiding it in the VTable.
To make this possible, without having to rewrite lots of
tests, I've created a clone of StorageHolder as a
"proof-of-concept" dummy implementation, for the sole
purpose of writing test fixtures. This one behaves
similar to the real-world thing, but cares only
for closing the command operation and omits all
the gory details of memento capturing and undo.
Seems this was part of the confusion when looking at
the inheritance graph: Names where almost reversed
to the meaning. the ArgumentHolder was *not* the
argument holder, but the top level closure. And
the class "Closure" was not "the" Closure, but
just the argument holder. ;-)
this was a classical example of a muddled and messed-up design,
driven just by the fact that I wanted to "spare" some functions,
with the net effect of writing more functions, plus a proxy class
plus create a lot of confusion for the reader.
This was easy to resolve though, once I resorted to the
general adivice to make public interface methods final,
make the extension ponts protected and never
to chain two extension points
First part is to define the steps (the protocol) at the
model element level, which gets a command prepared and invoked.
Test fails still, because there is no actual argument binding
invoked in the TestNexus
we deleted an object on the heap,
and afterwards re-accessed the memory through the
dangling pointer to verify the deletion actually happened.
This works most of the time, unless the memory manager decides
to map that page differently -- in which case we just hit
random memory contents.
A better idea is thus to place this TestFrame object
into a statically allocated buffer and invoke the dtor
explicitly. This allows us to conduct the test reliably.
- remove unnecessary includes
- expunge all remaining usages of boost::format
- able to leave out the expliti string(elm) in output
- drop various operator<<, since we're now picking up
custom string conversions automatically
- delete diagnostics headers, which are now largely superfluous
- use newer helper functions occasionally
I didn't blindly change any usage of <iostream> though;
sometimes, just using the output streams right away
seems adequate.
...and learned a lot about the new type_traits on the way.
As it seems, it is not possible to get a clean error message
when passing an "object" with no custom string conversion;
instead, some overload for an rvalue-ostream kicks in.
probably I'll go for shoing a type string in these cases
now we use boost::format through our own front-end util::_Fmt
solely, which both helps to reduce compilation time and code size,
and gives us a direct string conversion, which automatically
uses any custom operator string() available on arguments.
While desirable as such, I did this conversion now, since
it allows us to get rid of boost::str, which in turn helps
to drill down any remaning uses of our own util::str
Note: not fixing all relevant warnings.
Especially, the "-Woverloaded-virtual" of Clang defeats the whole purpose
of generated generic interfaces. For example, our Variant type is instantiated
with a list of types the variant can hold. Through metaprogramming, this
instantiation generates also an embedded Visitor interface, which has
virtual 'handle(TY)' functions for all the types in question
The client now may implement, or even partially implement this Visitor,
to retrieve specific data out of given Variant instance with unknown conent.
To complain that some other virtual overload is now shaddowed is besides the point,
so we might consider to disable this warning altogether
the object VTable is typically emitted when the compiler
encounters the first non-static non-inline function of
the class or a derived class.
Sometimes this happens within the wrong library and so
the compiler needs a nudge to emit those infrastructure functions.
But in most cases this works out of the box and need no further
magic incanctations, which might have a downside.
Especially because also a non-inline dtor does incur a call overhead,
whereas an inline dtor can be trivially elided.
this was introduced into namespace mobject and spread from there.
Since the habit is to use more specific typedefs like PClip,
it is preferrable to spell out the full namespace
In Lumiera, "Tracks" are not what you'd expect from
conventional video editing software. They are a mere
grouping devide, and are also used to implement the
"media bins" and tool palettes.
But having "folders" on the timeline would be likewise
confusing, as would be to have a "branch" or "tree".
To get out of that dilemma, we chose an understandable
but deliberately somewhat strange name: "Fork"
It was common understanding on the Mailinglist that we
should handle this renaming in a tuned-down and discrete
way: The UI will continue to show "Tracks" for a familiar
sight and "Bins" in the Asset section. But Lumiera developers
will be nudged to accomodate by renaming the entity in
source code accordingly
This switches the Lumiera UI from GTK-2 to GTK-3
Unfortunately, this move breaks two crucial features, which have been
disabled for now: the display of video and our custom timeline widget.
Since both of these require some reworking, which in fact has already
started, we prefer to do the library and framework switch right away.
previously this operation was named 'attach', which an be confused
with attching an object to this location. Indeed, the session interface
even offers such an attach function. By renaming the focus moving
operation into QueryFocus::shift(Scope), this ambiguity is resolved
This is the first step towards a generic backbone to connect
any GUI elements to the session within Proc-Layer.
It is based on a spefic understanding of Model-View-Controller,
which turns the Model-Controller interactions into messages.
Uniform sequence at start of source files
- copyright claim
- license
- file comment
- header guard
- lumiera includes
- library / system includes
Lumiera uses Brittish spelling. Add an according note to the styleguide.
In Clang, static object fields are initialised from top to bottom,
but before any other variables in anoymous namespaces. To the contrary,
GCC evaluates *any* initialisation expression in the translation
unit together from top to bottom. Thus, in the clang generated
code, in two cases the static initialisation could use a not yet
constructed local lib::_Fmt formatter object.
c++11 uses another hashtable implementation.
This uncovered some poorly written tests, which relied on
objects being returned in a specific order. As far as poissible,
we're using generic query functions now to get our test objects.
But these tests still rely on a specifically crafted test index content,
which as such is acceptable IMHO. The only remaining problem is
that we check the order of generated output in some tests, and this
order is still implementation dependent.
In the November developer meeting, Christian and I agreed that
it's best to remove that offending LUID specifications altogether.
Those embedded LUIDs where one of the issues blocking the transition to C++11
to make them stand out more prominently, some entity comments
where started with a line of starts. Unfortunately, doxygen
(and javadoc) only recogise comments which are started exactly
with /**
This caused quite some comments to be ignored by doxygen.
Credits to Hendrik Boom for spotting this problem!
A workaround is to end the line of stars with *//**
lib::Depend<TY> works as drop-in replacement for lib::Singleton<TY>
This changeset removes the convoluted special cases like
SingletonSub and MockInjector.
Clang is more insistent when it comes to enforcing 'protected' visibility.
Since in this case the basic design can be considered sane and optimal, the
only (and obvious) solution is to nest the PIMPL into a default base class
for implementation; this mirrors the structure of the interface.
Compilation with Clang 3.0 (which is available in Debian/stable) fails,
mostly due to some scoping and naming inconsistencies which weren't detected
by GCC. At some instances, Clang seems to have problems to figure out a
perfectly valid type definition; these can be resolved by more explicit
typing (which is preferrable anyway)
using our util::_Fmt front-end helps to reduce the code size,
since all usages rely on a single inclusion of boost::format
including boost::format via header can cause quite some code bloat
NOTE: partial solution, still some further includes to reorganise
this draft fills in the structure how to get from an invocation
of the engine service to the starting of actual CalcStream instances.
Basically the EngineService implementation is repsonsile to
instruct the Segmentation to provide a suitable Dispatcher.
