benn/lumiera_documentation
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

BufferProvider as a frontend to buffer management

Browse source
This commit is contained in:
Fischlurch 2011-07-09 02:17:37 +02:00
parent 7e3054df18
commit a199bff92b
5 changed files with 110 additions and 19 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
src/proc/engine/buffhandle.cpp → src/proc/engine/buffer-provider.cpp
View file

@ -1,5 +1,5 @@
/*
BuffHandle - Buffer handling support for the render engine
BufferProvider - Abstraction for Buffer management during playback/render
Copyright (C) Lumiera.org
2008, Hermann Vosseler <Ichthyostega@web.de>
@ -21,11 +21,14 @@
* *****************************************************/
#include "proc/engine/buffhandle.hpp"
#include "proc/engine/buffer-provider.hpp"
namespace engine {
BufferProvider::~BufferProvider() { }
/** */

72
src/proc/engine/buffer-provider.hpp Normal file
View file

@ -0,0 +1,72 @@
/*
BUFFER-PROVIDER.hpp - Abstraction for Buffer management during playback/render
Copyright (C) Lumiera.org
2011, Hermann Vosseler <Ichthyostega@web.de>
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 2 of
the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
*/
/** @file buffer-provider.hpp
** Abstraction to represent buffer management and lifecycle within the render engine.
** It turns out that -- throughout the render engine implementation -- we never need
** direct access to the buffers holding media data. Buffers are just some entity to be \em managed,
** i.e. "allocated", "locked" and "released"; the actual meaning of these operations is an implementatino detail.
** The code within the render engine just pushes around BufferHandle objects, which act as a front-end,
** being created by and linked to a BufferProvider implementation. There is no need to manage the lifecycle
** of buffers automatically, because the use of buffers is embedded into the render calculation cycle,
** which follows a rather strict protocol anyway. Relying on the capabilities of the scheduler,
** the sequence of individual jobs in the engine ensures...
** - that the availability of a buffer was ensured prior to planning a job ("buffer allocation")
** - that a buffer handle was obtained ("locked") prior to any operation requiring a buffer
** - that buffers are marked as free ("released") after doing the actual calculations.
**
** @see state.hpp
** @see output-slot.hpp
*/
#ifndef PROC_ENGINE_BUFFR_PROVIDER_H
#define PROC_ENGINE_BUFFR_PROVIDER_H
#include "lib/error.hpp"
#include "proc/engine/buffhandle.hpp"
#include <boost/noncopyable.hpp>
namespace engine {
/**
* Handle for a buffer for processing data, abstracting away the actual implementation.
* The real buffer pointer can be retrieved by dereferencing this smart-handle class.
*
* @todo as of 6/2011 buffer management within the engine is still a bit vague
*/
class BufferProvider
: boost::noncopyable
{
public:
virtual ~BufferProvider(); ///< this is an interface
};
} // namespace engine
#endif

21
src/proc/engine/buffhandle.hpp
View file

@ -24,12 +24,17 @@
** Various bits needed to support the buffer management within the render nodes.
** When pulling data from predecessor nodes and calculating new data, each render node
** needs several input and output buffers. These may be allocated and provided by several
** different "buffer providers" (for example the frame cache). For accessing those buffers,
** the node needs to keep a table of buffer pointers, and for releasing the buffers later
** on, we need some handles. The usage pattern of those buffer pointer tables is stack-like,
** thus it makes sense to utilise a single large buffer pointer array per pull() calldown
** sequence and dynamically claim small chunks for each node.
**
** different "buffer providers" (for example the frame cache). Typically, the real buffers
** will be passed as parameters to the actual job instance when scheduled, drawing on the
** results of prerequisite jobs. Yet the actual job implementation remains agnostic with
** respect to the way actual buffers are provided; the invocation just pushes BuffHandle
** objects around. The actual render function gets an array of C-pointers to the actual
** buffers, and for accessing those buffers, the node needs to keep a table of buffer
** pointers, and for releasing the buffers later on, we utilise the buffer handles.
** The usage pattern of those buffer pointer tables is stack-like, thus the actual
** implementation utilises a single large buffer pointer array per pull() call
** sequence and dynamically claims small chunks for each node.
**
** @see nodewiring-def.hpp
** @see nodeoperation.hpp
** @see bufftable.hpp storage for the buffer table
@ -105,11 +110,11 @@ namespace engine {
/**
* Buffer Type information.
* Given a BufferDescriptor, it is possible to allocate a buffer
* of suitable size and type by using State::allocateBuffer().
* of suitable size and type by using BufferProvider::allocateBuffer().
*/
struct BufferDescriptor
{
lumiera::StreamType& sType_;
long typeToken_;
};

2
src/proc/play/output-slot.hpp
View file

@ -39,7 +39,7 @@
#include "lib/error.hpp"
#include "lib/handle.hpp"
#include "lib/time/timevalue.hpp"
#include "proc/engine/buffhandle.hpp"
#include "proc/engine/buffer-provider.hpp"
#include "lib/iter-source.hpp"
//#include "lib/sync.hpp"

27
wiki/renderengine.html
View file

@ -1077,6 +1077,17 @@ Please note the shortcomings and logical contradictions in the solution currentl
* The current design rather places the implementation according to the roles of the involved entities, which causes some ping-pong on the implementation level. Especially the ScopeLocator singleton can be accessed multiple times. This is the usual clarity vs. performance tradeoff. Scope resolution is assumed rather to be //not performance critical.//
</pre>
</div>
<div title="BufferProvider" modifier="Ichthyostega" modified="201107090002" created="201107082330" tags="Rendering spec draft" changecount="5">
<pre>It turns out that -- throughout the render engine implementation -- we never need direct access to the buffers holding media data. Buffers are just some entity to be //managed,// i.e. &quot;allocated&quot;, &quot;locked&quot; and &quot;released&quot;; the //actual meaning of these operations can be left to the implementation.// The code within the render engine just pushes around ''buffer handles'', which act as a front-end, being created by and linked to a buffer provider implementation. There is no need to manage the lifecycle of buffers automatically, because the use of buffers is embedded into the render calculation cycle, which follows a rather strict protocol anyway. Relying on the [[capabilities of the scheduler|SchedulerRequirements]], the sequence of individual jobs in the engine ensures...
* that the availability of a buffer was ensured prior to planning a job (&quot;buffer allocation&quot;)
* that a buffer handle was obtained (&quot;locked&quot;) prior to any operation requiring a buffer
* that buffers are marked as free (&quot;released&quot;) after doing the actual calculations.
__see also__
&amp;rarr; OutputSlot relying on a buffer provider to deal with frame output buffers
&amp;rarr; RenderMechanics for details on the buffer management within the node invocation for a single render step
</pre>
</div>
<div title="BuildFixture" modifier="Ichthyostega" modified="201011290504" created="201011282003" tags="Builder spec operational" changecount="32">
<pre>//Building the fixture is actually at the core of the [[builder's operation|Builder]]//
{{red{WIP as of 11/10}}} &amp;rarr; see also the [[planning page|PlanningBuildFixture]]
@ -3313,7 +3324,7 @@ Thus the mapping is a copyable value object, based on a associative array. It ma
First and foremost, mapping can be seen as a //functional abstraction.// As it's used at implementation level, encapsulation of detail types in't the primary concern, so it's a candidate for generic programming: For each of those use cases outlined above, a distinct mapping type is created by instantiating the {{{OutputMapping&lt;DEF&gt;}}} template with a specifically tailored definition context ({{{DEF}}}), which takes on the role of a strategy. Individual instances of this concrete mapping type may be default created and copied freely. This instantiation process includes picking up the concrete result type and building a functor object for resolving on the fly. Thus, in the way typical for generic programming, the more involved special details are moved out of sight, while being still in scope for the purpose of inlining. But there //is// a concern better to be encapsulated and concealed at the usage site, namely accessing the rules system. Thus mapping leads itself to the frequently used implementation pattern where there is a generic frontend as header, calling into opaque functions embedded within a separate compilation unit.
</pre>
</div>
<div title="OutputSlot" modifier="Ichthyostega" modified="201107080131" created="201106162339" tags="def Concepts Player spec" changecount="23">
<div title="OutputSlot" modifier="Ichthyostega" modified="201107082305" created="201106162339" tags="def Concepts Player spec" changecount="24">
<pre>Within the Lumiera player and output subsystem, actually sending data to an external output requires to allocate an ''output slot''
This is the central metaphor for the organisation of actual (system level) outputs; using this concept allows to separate and abstract the data calculation and the organisation of playback and rendering from the specifics of the actual output sink. Actual output possibilities can be added and removed dynamically from various components (backend, GUI), all using the same resolution and mapping mechanisms (&amp;rarr; OutputManagement)
@ -3357,12 +3368,12 @@ Solving this problem through //generic programming// -- i.e coding both cases ef
:extend and adapt the protocol such to make both models similar; concentrate all differences //within a separate buffer provider.//
!!!discussion
the generic approach looks as it's becoming rather convoluted in practice. We'd need to hand over additional parameters to the factory, which passes them through to the actual job implementation created. And there would be a coupling between slot and job (the slot is aware it's going to be used by a job, and even provides the implementation). Obviously, a benefit is that the actual code path executed within the job is without indirections, and all written down in a single location. Another benefit is the possibility to extend this approach to cover further buffer handling models -- it doesn't pose any requirements on the structure of the buffer handling.
If we accept to retrieve the buffer(s) via an indirection, which we kind of do anyway //within the render node implementation// -- the unified model looks more like a clean solution. It's more like doing away with some local optimisations possible if we handle the models explicitly, so it's not much of a loss, given that the majority of the processing time will be spent within the inner pixel calculation loops for frame processing anyway. When following this approach, the buffer provider becomes a third, independent partner, and the slot cooperates tightly with this buffer provider, while the client (processing node) still just talks to the slot. Basically, this unified solution is like extending the shared buffer model to both cases.
If we accept to retrieve the buffer(s) via an indirection, which we kind of do anyway //within the render node implementation// -- the unified model looks more like a clean solution. It's more like doing away with some local optimisations possible if we handle the models explicitly, so it's not much of a loss, given that the majority of the processing time will be spent within the inner pixel calculation loops for frame processing anyway. When following this approach, the BufferProvider becomes a third, independent partner, and the slot cooperates tightly with this buffer provider, while the client (processing node) still just talks to the slot. Basically, this unified solution is like extending the shared buffer model to both cases.
&amp;rArr; conclusion: go for the unified approach!
!!!unified data exchange cycle
The nominal time of a frame to be delivered is used as an ID throughout that cycle
# within a defined time window prior to delivery, the client can retrieve the buffer from the ''buffer provider''.
# within a defined time window prior to delivery, the client can ''allocate and retrieve the buffer'' from the BufferProvider.
# the client has to ''emit'' within a (short) time window prior to deadline
# now the slot gets exclusive access to the buffer for output, signalling the buffer release to the buffer provider when done.
@ -4893,7 +4904,7 @@ At first sight the link between asset and clip-MO is a simple logical relation b
[img[Entities comprising the Render Engine|uml/fig128389.png]]
</pre>
</div>
<div title="RenderImplDetails" modifier="Ichthyostega" modified="200810140245" created="200806220211" tags="Rendering impl img" changecount="22">
<div title="RenderImplDetails" modifier="Ichthyostega" modified="201107082336" created="200806220211" tags="Rendering impl img" changecount="23">
<pre>Below are some notes regarding details of the actual implementation of the render process and processing node operation. In the description of the [[render node operation protocol|NodeOperationProtocol]] and the [[mechanics of the render process|RenderMechanics]], these details were left out deliberately.
!Layered structure of State
@ -4923,7 +4934,7 @@ We //do employ//&amp;nbsp; some virtual calls for the buffer management in order
* only output buffers are allocated. It is //never necessary//&amp;nbsp; to allocate input buffers!
* buffers are to be allocated as late as possible, typically just before invoking {{{process()}}}
* buffers are allways allocated by calling to the preceeding StateAdapter in the callstack (&quot;parent stae&quot;), because of the possibility of writing the result to cache.
* {{{pull()}}} returns a handle for the single output requested by this call. Using this ID, the caller may retrieve the actual buffer holding the result from the &quot;current state&quot; StaeProxy.
* {{{pull()}}} returns a handle for the single output requested by this call. Using this ID, the caller may retrieve the actual buffer holding the result from the &quot;current state&quot; StateProxy.
* any other buffers filled with results in the course of the same {{{process()}}} call can be released immediately before returning from the {{{pull()}}}
* similar, and input buffers are to be released immediately after the {{{process()}}} call, but before returing from this {{{pull()}}}
* buffers are allways released by calling to the &quot;current state&quot; (which is a StateProxy), providing the buffer-ID to be released
@ -5497,11 +5508,11 @@ if (oldText.indexOf(&quot;SplashScreen&quot;)==-1)
<div title="StateAdapter" modifier="Ichthyostega" modified="200807132344" created="200806261912" tags="Rendering impl def" changecount="10">
<pre>A small (in terms of storage) and specifically configured StateProxy object which is created on the stack for each individual {{{pull()}}} call. It is part of the invocation state of such a call and participates in the buffer management. Thus, in a calldown sequence of {{{pull()}}} calls we get a corresponding sequence of &quot;parent&quot; states. At each level, the &amp;rarr; WiringDescriptor of the respective node defines a Strategy how the call is passed on.</pre>
</div>
<div title="StateProxy" modifier="Ichthyostega" modified="200806010404" created="200706220352" tags="def" changecount="4">
<div title="StateProxy" modifier="Ichthyostega" modified="201107082334" created="200706220352" tags="def" changecount="5">
<pre>An Object representing a //Render Process// and containing associated state information.
* it is created in the Controller subsystem while initiating the BuildProcess
* it is created in the Player subsystem while initiating the RenderProcess
* it is passed on to the generated Render Engine, which in turn passes it down to the individual Processors
* moreover, it contains methods to communicate with other state relevant parts of the system, thereby shielding the rendering code from any complexities of Thread communication if necessary. (thus the name Proxy)
* moreover, it contains methods to communicate with other state relevant parts of the system, thereby shielding the rendering code from any complexities of state synchronisation and management if necessary. (thus the name Proxy)
* in a future version, it may also encapsulate the communication in a distributed render farm
</pre>
</div>