...need to pass a binding for the actual processing function
in a way that it acts as a ''prototype'' — since the `Feed`,
i.e. the ''Invocation Adapter'' must be generated for each
invocation anew within the current stack frame
(so to avoid spurious heap allocations)
...seems that the former is well suited to serve as detail builder
used internally by the latter to provide a simplified standard adaptation
for a given processing function.
The integration can be achieved to layer a specialised detail builder class
on top, which can be entered only by specifying the concrete function or lambda
to wrap for the processing; the further builder-API-functions to control
the wiring in detail become thus only accessible after the function type
is known; this allows to place the detail builder as member into the
enclosing port builder and thus to allocate everything within the current
stack frame invoking the builder chain.
...after having determined the several levels of prototyping
currently employed, an important step ahead could be achieved
by analysing the intended and implied usage context of this
builder scheme, while still assuming the simplifications
related to prototyping.
It can be assumed that
* the Level-2 builder object is ''somehow provided''
* the invocation happens from within a media-handling lib-plugin
* alongside with the desired `ProcAsset` spec, an `ExpectationContext`
will be provided, allowing to pass-through additional semantic tags
The implementation in the lib-plugin is then able to draw from specific
knowledge related to the **Domain Ontology** for ''especially for this library''
and provide the necessary wrappers and parameter mapping information.
⟹ the **Level-2**-builder should thus expose an API to
* set up a straight forward mapping, based on a given wrapper functor
to delegate to the actual library invocation
* allow optionally to override some of the input connections
* alternatively allow to use a complete `InvocationAdapter`,
including a `FeedManifold`, as provided directly by the library-plugin
...by relying on DI for some effectively global services, notably
the cache provider, the API for building and wiring render nodes
can be simplified to cover only the actual node connectivity
Doing so directly seems to be a better solution than to inject an OutputBufferProvider;
the latter will still be needed, yet will not be part of the regular weaving pattern,
but used directly at top-level to obtain the output `BuffHandle`, which is then
passed to the `Port::weave()` call
...still not convinced that this is a good design,
since it seems to subvert the general design to treat one special case.
However, I can't see a good way to address this special case directly
There might be one specific output result buffer at top level
for each invocation, which must be delivered into a prepared
output sink. This amounts to one special case, cross-cutting
an otherwise completely generic data flow scheme.
After considering several solutions, it seems most straight-forward
to configure a specific `OutputBufferProvider` to serve as a proxy for
the `OutputSlot` / `DataSink` provided at top-level to the Render-Job.
As an asside, this analyis reveals that the result-slot number does
not belong into the `FeedManifold`, which is dynamic (on the stack);
rather, it's a fixed value configured as part of the `WeavingPattern`
Code clean-up: mark all buffers with a dedicated tagging type
The point in question is: if we work the LocalTag into the type-hash,
could it be possible to miss an existing entry in the metadata registry?
This could cause two entries to be locked for a single buffer address,
leading to data corruption.
As far as I can see, in the current usage this would not happen,
but unfortunately this problem can not be ruled out, since the BufferProvider
API and protocol is designed to be open for various usage patterns.
However, the same potentially disastrous pattern could also materialise
when registering two different buffer types, and then locking each
for the same buffer location.
...this seems to be a tricky aspect; we use hash-chaining to create
derived entries, which may cause the identity of an entry to depend
on the order of specialisation. Looked through the possible code paths,
but these seem to be quite complicated; I see the lurking danger of
creating a second entry (with a different hash), and then in worst case
even locking/unlocking a given buffer twice....
...this is a surprisingly tricky issue, since it undercuts the
generic and recursive implementation of buffer handling;
fortunately I've foreseen such demands may arise down the road
and I've reserved an »Local Key« (now renamed into `LocalTag`),
whose meaning is implementation defined and interpreted by
the specific `BufferProvider`
Requirement analysis shows that the ''actual buffer provider'' to use
constitutes yet another independent degree of freedom, which conceivably
must be handled by the Builder internals rather than by the Domain Ontology.
Thus the simple solution to use a `BuffDescr` to mark the type must be augmented
to also allow configuration of the underlying `BufferProvider`, which generates
the descriptor and can later be invoked with this descriptor to ''lock an actual Buffer.''
In some cases, setup of the buffer types could even be more complicated and require
access to the actual (runtime) invocaton context; such extreme cases however
could be rendered as an extension of the scheme established here,
by storing the (up to now transient) constructor functors persistently.
Which leads to the decision not to care for those extremely complicated
corner cases right now, and thus to construct all buffer descriptors
in the `build()` call
...still fighting to find a suitable API to define
how inputs and outputs are connected and mapped to function parameters.
The solution drafted here uses the reshaped `DataBuilder` (≙`lib::SeveralBuilder`)
to add up connections for each »slot«, disregarding the possibility of permutations.
Similar to `NodeBuilder`, a policy template is used to pass down the setup
for an actual custom allocator.
After applying all the preceding refactorings, it turns out that
the `DataBuilder` defined here ''is essentially `lib::SeveralBuilder`'',
only with a different arrangement of the type parameters, due to the
specific usage context here.
It is thus possible to replace all the interim / helper / rebinding templates
by simple templated typedefs. The only tangible difference is that for
usage in the Builder, a ''selector policy'' is passed as a simple type argument,
which in practice wires the concrete allocator information down into each
sub-builder created during the ongoing construction of a node structure.
redefine the policy for `lib::SeveralBuilder` to be a template-template parameter.
In fact it should have been this way from start, yet defining this kind of
very elaborate code bottom-up lets you sometime miss the wood for the trees
So to restate: `lib::SeveralBuilder` takes a ''policy template,''
which then in turn will be instantiated with the same types `I` (interface)
and `E` (element type) used on `SeveralBuilder` itself. Obviously, there can be
further types involved and thus additional type parameters may be necessary,
notably the ''Allocator'' — yet these are better injected when ''defining''
the policy template itself.
The default binding for this policy template is defined as `allo::HeapOwn`,
which causes the builder to allocate the storage extents through the standard
heap allocator, and for the created `lib::Several` to take full ownership of
embedded objects, invoking their destructors when falling out of scope.
As a direct consequence of the insights regarding Dependency-Injection,
a ''Builder Toolkit'' is required, which can be used to adapt various
kinds of ''Weaving Patterns'' — since obviously it is not possible to
settle down on a single Pattern, and thus several ''families of builders''
will emerge, one for each ''line of construction'' for ''Weaving Patterns''.
To stress this point, what I am coding here is a prototype, aimed at
being used as part of a **Test Domain Ontology** — and other Domain Ontologies
(e.g. für FFmpeg) will certainly require other construction schemes
for their Weaving Patterns. So this is an open field, and can not be
settled once and for all.
This immediately leads to another, rather technical problem:
If we're about to work with ''delegate Builders,'' then also
a way to pass-down the allocator configuration is required.
We had settled on a preliminary solution with the helper `DataBuilder`,
yet this solution looks like it defines how `lib::SeveralBuilder`
should be used in most of the cases. So there is now a conflict
between the existing definition scheme for `lib::SeveralBuilder`,
which was achieved in a bottom-up way, and a slightly different
definition scheme ''as it should be''
Starting to attack this latter detail problem, as a first step,
the definition of `DataBuilder` can be simplified by collapsing
it with the `lib::allo::SetupSeveral`
It became clear that a secondary system of connections must be added,
running top-down from a global model context, and thus contrary to the
regular orientation of the node network, which connects upwards from
predecessor to successor, in accordance with the pull principle.
If we accept this wiring as part of the primary structure, it can be
established immediately while building the nodes, thus adding a preconfigured
''pattern of Buffer Descriptors'' to each node, since there is no further
''moving part'' — beyond the wiring to the `BufferProvider`, which thus
becomes part of a global `ModelContext`
As an immediate consequence, the storage for this configuraion should
also be switched to `lib::Several` and handled similar to the primary
node wiring in the Builder...
It seems we need a `WeavingPattern`-Builder, which obviously
must be rather flexible, since those patterns are to be composed
from several layers, which should be extensible within a given ''Domain Ontology''
So this seems to lead to a builder-DSL which creates »**onion layers**«
of builders, with the ability to extend and specialise the type on each layer.
''As it will be quite challenging to get this into usable shape,
it seems best to approach this step by step through prototyping...''
Not entirely sure how to use the `emit()` call properly,
assuming that it means that data is complete in buffer,
but can still be read after that point
* at least for a simple, prototypical setup
* and actually shifting the onerous into the Level-1 builder \\
''(which is precisely the intention here)''
The deeper problem is that we must not engage into any premature decisions
regarding the structure or layout of the actual processing function invocation.
Thus attempting to create a kind of »firewall« of sorts, by connecting
the building blocks strictly through template parameter and preferably
figuring out any detailed knowledge locally, through ''compile-time introspection...''
...even the initial effort to stub its operation turns into a
challenge, since honestly there is near nothing we can assume safely,
without sliding into uncovered provisions regarding the ''Domain Ontology''
- it is clear that this adaptor will be a ''Concept''
- yet it must in some way access the `FeedManifold` and also control additional storage
- a rather obvious solution is to layer it ''on top'' of the manifold
...which brings about various (preliminary) decisions regarding
Metadata storage in the `Turnout`-object, which acts as a guidance
and specification for the actual invocation for this specific node.
As starting point, I choose the ''KISS'' solution of embedding some
blocks of `UninitialisedStorage` directly into the `Turnout`; obviously
these blocks must be oversized, since we can not effort emitting a
dedicated template instance for each different count of input / output
feeds. Moreover, these data buffers are assumed to be filled with
valid objects by the builder ''(this is a lurking danger)''
...turns out that the intended structure is still too fine grained
and explicit and many operational steps can be collapsed into a single
virtual scope, wherein they can be deemed implementation detail...
...so the solution is to build up the working data as `lib::SeveralBuilder`;
however, a more concise notation can be achieved with a suitably configured
wrapping subclass; together with the cross-builder trick, this allows
to write the allocation configuration in a clearly libelled way,
while the field definition and the builder constructor hides the
complexities of picking up the extension point and passing on the
wiring to the allocator instance.
...using the same pattern here as was successful also for the underlying lib::SeveralBuilder;
even if it may seem logically backwards, it just reads much better and
is more understandable, and has the added benefit of providing a dedicated
definition scope, which can be kept separate from the constructor definition
of the actual builder
{{{
prepareNode()
.withAllocator<XYZ>()
.addLead(predecessor)
.build()
}}}
...turns out to be surprisingly tricky, since the nested
lib::SeveralBuilder instances require parametrisation by a
''policy template,'' which in turn relies on the actual allocator.
And we want to provide the allocator as a constructor parameter,
including the ability to pick up a custom specialisation for
some specific allocator (notably AllocationCluster requires
to hook into this kind of extension point, to be able to
employ its dedicated API for dynamic allocation adjustment)
...especially what is necessary to represent at this level and what information
is implicit; notably there will be an implicit default wiring, but we allow
for case-by-case deviations
The Builder will have to perform several passes, gradually refining
the model into the low-level Render Node network. Right now, some
guesses regarding the last steps of this process are possible,
thus defining the lowest level of a model builder structure
* Level-3 : mapping data flow paths
* Level-2 : detailed configuration of data buffer passing
* Level-1 : build the actual parameter structures for invocation
In the current »Vertical Slice« we're able to fully define Level-1
and maybe Level-2
To escape a possible deadlock in analysis, I resort to developing
some kind of free-wheeling presupposition how the **Builder** could
be implemented — a centrepiece of the Lumiera architecture envisioned
thus far — which ''unfortunately'' can only be planned and developed
in a more solid way ''after'' the current »Vertical Slice« is completed.
Thus I find myself in the uncomfortable situation of having to work towards
a core piece, which can not yet be built, since it relies heavily on
the very structures to be built...
...the complexity of details is a nightmare
...still fighting to grasp a generic structure allowing to ''fold down''
the details into the specific ''domain ontologies'' for the media libraries
...and this line of analysis brings us deep into the ''Buffer Provider''
concept developed in 2012 — which appears to be very well to the point
and stands the test of time.
Adding some ''variadic arguments'' at the right place surprisingly leads
to an ''extension point'' — which in turn directly taps into the
still quite uncharted territory interfacing to a **Domain Ontology**;
the latter is assumed to define how to deal with entities and relationships
defined by some media handling library like e.g. FFmpeg.
So what we're set to do here is actually ''ontology mapping....''
The immediate next step is to build some render nodes directly
in a test setting, without using any kind of ''node factory.''
Getting ahead with this task requires to identify the constituents
to be represented on the first code layer for the reworked code
(here ''first layer'' means any part that are ''not'' supplied
by generic, templated building blocks).
Notably we need to build a descriptor for the `FeedManifold` —
which in turn implies we have to decide on some fundamental aspects
of handling buffers in the render process.
To allow rework of the `ProcNode` connectivity, a lot of presumably obsoleted
draft code from 2011 has to be detached, to be able to keep it in-tree
for further reference (until the rework and refactoring is settled).
As outlined in #1367, the integration effort requires some rework
of existing code, which will be driven ahead by the `NodeLinkage_test`
* redefine Node Connectivity
* build simple `ProcNode` directly in scope
* create an `TurnoutSystem` instance
* perform a ''dummy Node-Invocation''
- the starting point is the idea to build a dedicated ''turnout system''
- `StateAdapter`, `BuffTable` ⟶ `FeedManifold` and _Invocation_ will be fused
- actually, the `TurnoutSystem` will be ''pulled'' and orchestrate the invocation
- the structure is assumed to be recursive
The essence of the Node-Invocation, as developed 2009 / 2011 remains intact,
yet it will be organised along a clearer structure
Within the existing body of code, there are two unfinished attempts
towards building a node invocation and management of data buffers.
The first attempt was entirely driven from the angle of invoking a
processing function, while the second one draws from a wider scope
and can be considered the solution to build upon regarding data buffers
in general. However, the results of the first approach are well suited
for their specific purpose, so both solutions will be combined.
Thus the arrangement of data feeds going in and out of the render node
shall be renamed into `BuffTable` -> `FeedManifold`
...which seems to be basically fine thus far
...beyond some renaming and rearranging
''it turns out that the final, crucial links,
necessary to tie all together, are yet to be developed''
In the Lumiera code base, we use C-String constants as unique error-IDs.
Basically this allows to create new unique error IDs anywhere in the code.
However, definition of such IDs in arbitrary namespaces tends to create
slight confusion and ambiguities, while maintaining the proper use statements
requires some manual work.
Thus I introduce a new **standard scheme**
* Error-IDs for widespread use shall be defined _exclusively_ into `namespace lumiera::error`
* The shorthand-Macro `LERR_()` can now be used to simplify inclusion and referral
* (for local or single-usage errors, a local or even hidden definition is OK)
Introduced as remedy for a long standing sloppiness:
Using a `char[]` together with `reinterpret_cast` in storage management helpers
bears danger of placing objects with wrong alignment; moreover, there are increasing
risks that modern code optimisers miss the ''backdoor access'' and might apply too
aggressive rewritings.
With C++17, there is a standard conformant way to express such a usage scheme.
* `lib::UninitialisedStorage` can now be used in a situation (e.g. as in `ExtentFamily`)
where a complete block of storage is allocated once and then subsequently used
to plant objects one by one
* moreover, I went over the code base and adapted the most relevant usages of
''placement-new into buffer'' to also include the `std::launder()` marker
doing so would contradict the fundamental architecture,
all kinds of failures and timeouts need to be handled within
Scheduler-Layer-2 rather.
Jobs are never aborted, nor do they need to know if and when they are invoked
The second design from 2017, based on a pipeline builder,
is now renamed `TreeExplorer` ⟼ `IterExplorer` and uses
the memorable entrance point `lib::explore(<seq>)`
✔
after completing the recent clean-up and refactoring work,
the monad based framework for recursive tree expansion
can be abandoned and retracted.
This approach from functional programming leads to code,
which is ''cool to write'' yet ''hard to understand.''
A second design attempt was based on the pipeline and decorator pattern
and integrates the monadic expansion as a special case, used here to
discover the prerequisites for a render job. This turned out to be
more effective and prolific and became standard for several exploring
and backtracking algorithms in Lumiera.
- allow to configure the expected job runtime in the test spec
- remove link to EngineConfig and hard-wire the engine latency for now
... extended integration testing reveals two further bugs ;-)
... document deadline calculation
This finishes the last series of refactorings; the basic concept
remains the same, but in the initial version we arranged the expander
function in the pipeline to maintain a Tuple (parent, child) for the
JobTickets. Unfortunately this turned out to be insufficient, since
JobTicket is effectively const and responsible for a complete Sement,
so there is no room to memorise a Deadline for the parent dependency.
This leads to the better idea to link the JobPlanning aggregators
themselves by parent-child references, which is possible since the
whole dependency chain actually sits in the stack embedded into the
Expander (in the pipeline)
...in the hope that the Optimiser is able to elide those references entirely,
when (as is here the case) they point into another field of a larger object compound
...as a preparation for solving a logical problem with the Planning-Pipeline;
it can not quite work as intended just by passing down the pair of
current ticket and dependent ticket, since we have to calculate a chained
calculation of job deadlines, leading up to the root ticket for a frame.
My solution idea is to create the JobPlanning earlier in the pipeline,
already *before* the expansion of prerequisites, and rather to integrate
the representation of the dependency relation direcly into JobPlanning
...using hard coded values instead of observation of actual runtimes,
but at least the calculation scheme (now relocated from TimeAnchor to JobPlanning)
should be a reasonable starting point.
TODO: test fails...
- collect list of entities to be picked up from the dispatcher-pipeline
- as it turns out: there is no sensible use for the realTimeDeadline in
in the FrameCoord record ==> remove it
The initial implementation effort for Player and Job-Planning
has been reviewed and largely reworked, and some parts are now
obsoleted by the reworked alternative and can be disabled.
The basic idea will be retained though: JobPlanning is a
data aggregator and performs the final step of creating a Job
- fix a bug in the MockDispatcher, when duplicating the ExitNodes.
A vector-ctor with curly braces will be interpreted as std::initializer_list
- add visualisation of the contents appearing at the end of the pipeline
*** something still broken here, increments don't happen as expected
`steam/engine/mock-dispatcher.hpp |cpp` now integrates this
''complete mock setup for render jobs and frame dispatching.''
The exising `DummyJob` has been slightly adapted and renamed
to `MockJob` and is tightly integrated with the other mocks.
The implementation of a `MockDispatcher` necessitated to change
the use of `MockJobTicket`. The initial attempts used a complete
mock implementation, but this approach turned out not to be viable.
Instead — based on the ideas developed for the mock setup —
now the prospective real implementation of `JobTicket` is available
and will be used by the mock setup too. Instead of a synthetic spec,
now a setup of recursively connected `ExitNode`(s) is used; the latter
seems to develop into some kind of Facade for the render node network.
Based on this mock setup, we can now demonstrate the (mostly) complete
Job-Planning pipeline, starting from a segmentation up to render jobs,
and verify proper connectivity and job invocation.
✔
- has to be prepared / supported by the RenderEnvironmentClosure
- actual translation happens when building the Dispatcher-Pipeline
- implementation delegate through
virtual size_t Dispatcher::resolveModelPort (ModelPort)
The existing implementation of the Player from 2012~2015 inclduded
an additional differentiation by media channel (for multichannel media)
and would build a separate CalcStream for each channel.
The in-depth analysis conducted for the ongoing »Vertical Slice« effort
revealed that this differentiation is besides the point and would never
be materialised: Since -- by definition -- all media processing has
to be done by the engine, also the generation of the final output format
including any channel multiplexing will happen in render nodes.
The only exception would be when only a single channel of multichannel
media is extracted -- yet this case would then translate into a
dedicated ModelPort.
Based on this reasoning, a lot of complexity (and some contradictions)
within the JobTicket implementation can be removed -- together with
some further leftovers of the fist attempt to build JobTickets always
from a Mock specification (we now use construction by the Segment,
based on an ExitNode, which is the expected actual implementation
for production setup)
...by defining a new scheme for access to custom allocators
...and then passing a reference to such an accessor into the
JobTicket ctor, thereby allowing the ticket istelf recursively
to place further JobTicket instances into the allocation space
--> success, test passes (finally)
Up to now, a draft/mock implementation was used, relying on a »spec tuple«,
which was fabricated by MockJobTicket. But with the introduction of
NodeGraphAttachment, the MockSequence now generates a nested ExitNode structure,
and thus the JobTicket will be created through the "real" ctor, and
no longer via MockJobTicket.
Thus it is possible to skip this whole interspersed »spec tuple«,
since ExitNode *is* already this aggregated / abstracted Spec
PROBLEM: can not implement Spec-generation, since
- we must use a λ for internal allocation of JobTickets
- but recursive type inference is not possible
Will thus need to abandon the Spec-Tuple and relocate this
traversal-and-generation code into JobTicket itself
Use another unit-test (FixtureSegment_test) to guide and cover
the transition from the existing fake-implementation to the
actual implementation, where the JobTicket will be generated
on-demand, from a NodeGraphAttachment
It turns out that the real (not mocked) implementation of JobTicket creation
is already required now for this planned (mock)Dispatcher setup;
moreover, this real implementation turns out to be almost identical
to the mock implementation written recently -- just nested structure
of prerequiste JobTickets need to be changed into a similar structur
of ExitNodes
-- as an aside: rearrange various tests to be more in-line
with the envisioned architecture of playback and engine
...this opens up yet another difficult question and a host of new problems
- how are prerequisites detected or arranged by the Builder
- how are prerequisites represented?
- what is an ExitNode in terms of implementation? A subclass of ProcNode?
- how will the actual implementation of JobTicket creation (on-demand) work?
- how to adapt the Mock implementation, while retaining the Specification
for Segments and prerequisites?
..this is now the third attempt, and it seems this one leads to a
clean solution for the type rebinding problem, while also allowing
to unit-test each step in isolation.
The idea is to layer a *templated* builder class on top,
but to slice it away in each step, re-assemble the pipeline
and decorate a new builder instance on top. The net result
is a tightly interconnected processing pipeline without
any spurious interspersed leftovers from the builder,
while all intermediate steps are fully operational
and can thus be unit-tested...
...still very rough edged...
...based on the idea to have a pair(Dependent,Dependency) and to shift these on each level of expansion
PROBLEMS:
- what to use as root level?
- can not handle JobTicket const& in transform-iterator (assignement operator)
...it turns out that we actually do not need to wrap TreeExplorer
on the builder types, because basically there is only a single active
builder type, and the complete processing pipeline can be assembled
in a single terminal function.
The type rebinding problem can thus be solved just by a simple
marker struct, which inherits from a template parameter
...hard to tackle...
The idea is to wrap the TreeExplorer builder, so that our specific
builder functions can delegated to the (inherited) generic builder functions
and would just need to supply some cleverly bound lambdas. However,
resulting types are recursive, which does not play nice with type inference,
and working around that problem leads to capturing a self reference,
which at time of invocation is already invalidated (due to moving the
whole pipeline into the final storage)
...which leads to the next daunting problems:
- we need some mocked ModelPort and DataSink placeholders
- we need a way how to inherit from a partial TreeExplorer pipeline
several years ago, it seemed like a good idea to incorporate
the link between nominal time and wall-clock time into a dedicated
anchor point, which also regulates the continued frame planning.
But it turned out that such a design mixes up several concepts
and introduces confusion regarding the meaning of "real time"
- latency can not be reasonably defined for a whole planning chunk
- skipping or sliding due to missed deadlines can not reasonably handled
within such an abstract entity; it must be handled rather at the
level of a playback process
- linking the frame grid generation directly to a planning chunk
undercuts the possible abstraction of a planning pipeline
...which is build a »Job planning pipeline« step by step
in a test setup, and then factor that out as RenderDrive,
to supersede the existing CalcPlanContinuation and get
rid of the Monads this way...
Challenges
- there is a inconsistency with channel usage
- need to establish a way how to transport the output-Sink into the JobFunctor
- need a way to propagate the current frame number to the next planning chunk
Last testcase: add deeply nested Prerequisites.
Turns out that the allocator must be able to handle
re-entrant allocations, which std::deque can not fulfil.
Thus using std::list here for the Mock implementation.
In the end, the real allocations will be done by our custom
allocator (AllocationCluster), which can be arranged easily
to support re-entrant allocation calls (since the whole point
is to just place those objects into a pre-allocated large block
and only de-allocate them later in one sway. Thus the allocator
does not need to wait for the object constructor to finish, which
trivially allows for re-entrant calls)
To complete the mock setup, the next step would be to extend the GenNode-based spec langage
to allow defining prerequisite Mock-JobTickets. Setting this up seems rather straight forward --
however, defining a simple testcase to cover this extension runs into surprisingly tricky problems..
- for one, the singleValIterator from Itertools has serious difficulties handling references
- but even more surprising, it seems impossible to make the "prerequisites iterator"
fit into the Tree-Explorer framework (which I intend to use as replacement
for the monadic approach)
after some extended analysis of generic types and template instances,
it seems that not TreeExplorer as such is the primary problem, but rather
there is a conceptual mismatch somewhere deep down in Itertools or Iter-Adapter
By reasoning and analysis I conclude that the differentiation into
multiple channels is likely misplaced in JobTicket; it belongs ratther
into the Segment and should provide a suitable JobTicket for each ModelPort
Handling of prerequisites also needs to be reshaped entirely after
switching to a pipeline builder for the Job-planning pipeline; as
preliminary access point, just add an iterator over the immediate
prerequisites, thereby shifting the exploration mechanism entirely
out of the JobTicket implementation
Testcase: A simple Sementation with a single and bounded Segment
As aside, figured out how to unpack an iterator such as to
tie a fixed number of references through a structural binding:
auto const& [s1,s2,s3] = seqTuple<3> (mockSegs.eachSeg());
right now we're lacking a complete working implementation of render node invocation,
and thus the Dispatcher implementation can only be verified with the help
of mocked jobs. However, at least a preliminary implementation of tagging the
invocation instance is available, and thus we're able to verify that
a given job instance indeed belongs to and is "backed" by a specific JobTicket.
This is prerequisite for building up a (likewise mocked) Fixture datastructure,
and this in turn was meant to form the basis for attacking an actual Scheduler
implementation, followed by a real render node invocation.
- can now create a Job from JobTicket::NIL
- on invocation this Job will to nothing
Only when the first real output backend is implemented,
we can decide if this simplistic implementation is enough,
or if an empty output must be explicitly generated...
* using a simplified preliminary implementation of hash chaining (see #1293)
* simplistic implementation of hashing for time values (half-rotation)
* for now just hashing the time into the upper part of the LUID
Maybe we can even live with that implementation for some time,
depending on how important uniform distribution of hash values is
for proper usage of the frame cache.
Needless to say, various further fine points need more consideration,
especially questions of portability (32bit anyone?). Moreover, since
frame times are typically quantised, the search space for the hashed
time values is drastically reduced; conceivably we should rather
research and implement a good hash function for 128bit and then combine
all information into a single hash key....
...using the MockJobTicket setup as point of reference,
since the actual invocation of render nodes will only be drafted
later in this "Vertical Slice" integration effort...
- introduce a JobTicket::NOP (null-object pattern)
- assuming that the function splitSplice() will retain complete coverage allways
Remark:
`Fixture::getPlaylistForRender()` is a leftover from the very early implementation drafts.
This function was more or less based on the way Cinelerra works; it is clear by now
that Lumiera can not possibly work this way, given that we'll build a low-level model
and dispatch precompiled render jobs....
...requires a first attempt towards defining a `JobTiket`.
This turns out quite tricky, due to using those `LinkedElements`
(intrusive single linked list), which requires all added records
actually to live elsewhere. Since we want to use a custom allocator
later (the `AllocationCluster`), this boils down to allocating those
records only when about to construct the `JobTicket` itself.
What makes matters even worse: at the moment we use a separate spec
per Media channel (maybe these specs can be collapsed later non).
And thus we need to pass a collection -- or better an iterator
with raw specs, which in turn must reveal yet another nested
sequence for the prerequisite `JobTickets`.
Anyhow, now we're able at least to create an empty `JobTicket`,
backed by a dummy `JobFunctor`....
- build the reworked Job-planning pipeline more or less from scratch
- back that with mocked `Dispatcher` and `JobTicket`
- then transfer this into a `RenderDrive`, which can be tested as well
- could continue then to a `CalcStream` integration test....
- introduce a new entity: RenderDrive
- it supersedes the CalcPlanCalculation, but is managed by CalcStream
- moreover, the RenderDrive will house a IterTreeExplorer-Pipeline
- define the concerns and relationships more clearly (see Drawing)
- prerequisite to disentangle the Job-planning "mechanics"
- decision: the Monad-style iteration framework will be abandoned
- the job-planning will be recast in terms of the iter-tree-explorer
- job-planning and frame dispatch will be disentangled
- the Scheduler will deliberately offer a high-level interface
- on this high-level, Scheduler will support dependency management
- the low-level implementation of the Scheduler will be based on Activity verbs
...in an attempt to clarify why numerous cross links are not generated.
In the end, this attempt was not very successful, yet I could find some breadcrumbs...
- file comments generally seem to have a problem with auto link generation;
only fully qualified names seem to work reliably
- cross links to entities within a namespace do not work,
if the corresponding namespace is not documented in Doxygen
- documentation for entities within anonymous namespaces
must be explicitly enabled. Of course this makes only sense
for detailed documentation (but we do generate detailed
documentation here, including implementation notes)
- and the notorious problem: each file needs a valid @file comment
- the hierarchy of Markdown headings must be consistent within each
documentation section. This entails also to individual documented
entities. Basically, there must be a level-one heading (prefix "#"),
otherwise all headings will just disappear...
- sometimes the doc/devel/doxygen-warnings.txt gives further clues
