remove the superfluous TiddlyWikis

..after integrating all still relevant asciidoced content
into the main website.

doc/design/backend/Scheduler.txt

@@ -0,0 +1,25 @@
+The Scheduler
+-------------
+:Author: CehTeh
+:Date: 6/2007
+
+//MENU: label Scheduler
+
+Scheduling is done with two priority queues, one for high priority jobs and one for low priority jobs.
+These priority queues are ordered by absolute time values plus some job specific identified.
+
+There are following (non exhaustive) kinds of jobs:
+
+* started job
+* job to be canceled
+* unscheduled job
+* dependency providing jobs
+
+Jobs implement a kind of future. We try hard to avoid any blocking waits.
+The Job scheduler runs singlethreaded. Its only task is to schedule and delegate jobs to worker threads,
+by itself it will never do any extensive processing.
+
+Each job has an pre configured behaviour for the case of failure or deadline miss.
+Any canceling and expireing jobs gets noted in *Statistics* to adjust performance and timings
+for optimal performance and I/O throughput.
+

doc/design/backend/index.txt

@@ -1,5 +1,114 @@
 Design Documents: Backend
 =========================
 
-Eventually, this will have design documentation for the Backend.
+What follows is a summary regarding the design of Lumiera's *Data Handling Backend*
+
+This is the foundation layer responsible for any high performance or high volume
+data access. Within Lumiera, ther are two main kinds how data is handled:
+
+* The Session and the object models manipulated through the GUI is kept in memory.
+  It is backed by a _storage backend,_ which provides database like storage and
+  especially logging, replaying and ``Undo'' of all ongoing modifications..
+* Media data is handled _frame wise_ -- as described below.
+
+The backend uses *memory mapping* to make data available to the program.
+This is somewhat different to the more common open/read/write/close file access,
+while giving superior performance and much better memory utilization.
+The data backend must be capable to handle more data than will fit into the memory
+or even address space on 32 bit architectures. Moreover, a project might access more files
+than the OS can handle at a any time, thus the for _Files used by the Backend,_ it needs a
+*FilehandleCache* to manage file handle dynamically.
+
+Which parts of a file are actually mapped to physical RAM is managed by the kernel;
+it keeps a *FileMapCache* to manage the *FileMaps* we've set up.
+In the End, the application itself only requests *Data Frames* from the Backend.
+
+To minimize latency and optimize CPU utilization we have a *Prefetch thread* which operates
+a *Scheduler* to render and cache frames which are _expected to be consumed soon_. The intention
+is to manage the rendering _just in time_.
+
+The prefetcher keeps *Statistics* for optimizing performance.
+
+
+Accessing Files
+---------------
+
++FileDescriptor+ is the superclass of all possible filetypes, it has a weak reference to a 
++FileHandle+ which is managed in within the +FilehandleCache+. On creation,  only the existence
+(when reading) or access for write for new files are checked. The +FileDescriptor+ stores some
+generic metadata about the underlying file and intended use. But the actual opening is done on demand.
+
+The _content of files is memory mapped_ into the process address space.
+This is managed by +FileMap+ entries and a +FileMapCache+.
+
+File Handles
+~~~~~~~~~~~~
+A +FilehandleCache+ serves to store a finite maximum number of +FileHandles+ as a MRU list.
+FileHandles are managed by the +FilehandleCache+; basically they are just storing the underlying OS file
+handles and managed in a lazy/weak way, (re)opened when needed and aging in the cache when not needed,
+since the amount of open file handles is limited aged ones will be closed and reused when the system
+needs to open another file.
+
+File Mapping
+~~~~~~~~~~~~
+The +FileMapCache+ keeps a list of +FileMaps+, which are currently not in use and subject of aging.
+Each +FileMap+ object contains many +Frames+. The actual layout depends on the type of the File.
+Mappings need to be _page aligned_ while Frames can be anywhere within a file and dynamically sized.
+
+All established ++FileMap++s are managed together in a central +FileMapCache+.
+Actually, +FileMap+ objects are transparent to the application. The upper layers will just
+request Frames by position and size. Thus, the +File+ entities associate a filename with the underlying
+low level File Descriptor and access
+
+Frames
+~~~~~~
++Frames+ are the smallest datablocks handled by the Backend. The application tells the Backend to make
+Files available and from then on just requests Frames. Actually, those Frames are (references to) blocks
+of continuous memory. They can be anything depending on the usage of the File (Video frames, encoder frames,
+blocks of sound samples). Frames are referenced by a smart-pointer like object which manages the lifetime
+and caching behavior.
+
+Each frame referece can be in one out of three states:
+
+readonly::
+  the backing +FileMap+ is checked out from the aging list, frames can be read
+  
+readwrite::
+  the backing +FileMap+ is checked out from the aging list, frames can be read and written
+  
+weak::
+  the +FileMap+ object is checked back into the aging list, the frame can't be accessed but we can
+  try to transform a weak reference into a readonly or readwrite reference
+
+
+Frames can be addressed uniquely whenever a frame is not available. The backend can't serve a cached
+version of the frame, a (probably recursive) rendering request will be issued.
+
+Prefetching
+~~~~~~~~~~~
+There are 2 important points when we want to access data with low latency:
+
+. Since we handle much more data than it will fit into most computers RAM.
+  The data which is backed in files has to be paged in and available when needed.
+  The +Prefetch+ Thread manages page hinting to the kernel (posix_madvise()..)
+. Intermediate Frames must eventually be rendered to the cache.
+  The Backend will send +Renderjobs+ to the +Scheduler+.
+
+Whenever something queries a +Frame+ from the backend it provides hints about what it is doing.
+These hints contain:
+
+* Timing constraints
+ - When will the +Frame+ be needed
+ - could we drop the request if it won't be available (rendered) in-time
+* Priority of this job (as soon as possible, or just in time?)
+* action (Playing forward, playing backward, tweaking, playback speed, recursive rendering of dependent frames)
+
+.Notes
+* The Backend will try to render related frames in groups.
+* This means that following frames are scheduled with lower priority.
+* Whenever the program really requests them the priority will be adjusted.
+ 
+ 
+-> more about link:Scheduler.html[the Scheduling of calculation jobs]
+
 

doc/design/plugins/PluginBrainstorm.txt

@@ -3,12 +3,81 @@ Plugin Brainstorm
 :Author: Christian Thäter
 :Date: 2008-09-15
 
-Raw Version
------------
+Lumiera will use a very simple and language neutral plugin system. The focus is on easy and independent distribution of plugins and small specific interfaces. Ultimate flexibility is of second concern.
+
+.Concept
+Plugins are just shared libraries which offer well defined Interfaces.
+A Plugin may offer more than one interface and may in turn request/use interfaces
+from other Plugins or from the main application.
 
 
-[cpp]
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Interface Definition
+--------------------
+
+Interfaces are declared in header files. They use some tool macros to give a convenient definition language.
+Basically, Interfaces are fixed -- with the exception that new functions may be added.
+link:PluginVersioning.html[Plugin Version Management] should stay out of the view most of the time.
+
+Plugin interfaces are simple C structs with some metadata at the beginning and function prototypes
+added at the end. With some macros we can map simple functions to versioned interfaces.
+Compiled plugins will stay compatible even if the interface is extended, while sourcecode need maintenance.
+
+An interface needs a name and a version. They define a block where the actual function prototypes can be added.
+New prototypes have to be added at the end, existing prototypes must never be changed.
+Each function prototype must be given with its different parts:
+
+- return type
+- name
+- arguments list
+- version.
+
+
+.Example
+[source,c]
+-----------------------------------------
+LUMIERA_INTERFACE(foo, 1,
+	LUMIERA_IPROTO(void, bar, (void)),
+	LUMIERA_IPROTO(int, baz, (int i))
+);
+
+LUMIERA_INTERFACE(foo, 2,
+	LUMIERA_IPROTO(void, bar, (void)),
+	LUMIERA_IPROTO(int, baz, (float i))
+);
+-----------------------------------------
+
+Note that the version 2 interface _changed the parameter from int to float_ for the 'baz' function.
+
+The interface/plugin framework will expand the above definitions into:
+
+[source,c]
+-----------------------------------------
+struct lumiera_interface_foo_1
+{
+	struct lumiera_interface interface_header_;
+	void (*bar) (void);
+	int (*baz) (int i);
+};
+
+struct lumiera_interface_foo_2
+{
+	struct lumiera_interface interface_header_;
+	void (*bar) (void);
+	int (*baz) (float i);
+};
+-----------------------------------------
+
+
+Implementation of Interfaces
+----------------------------
+Interfaces can be implemented either in core code or through plugins.
+In each case, such an _instantiation_ of an interface means that actual functions are mapped
+to the corresponding slots in the interface structure.
+
+
+.Implementing an interface
+[source,c]
+-----------------------------------------
 LUMIERA_INTERFACE_DECLARE (interface_descriptor, 0,
                            /* The following slots are some human-readable descriptions of certain properties */
                            LUMIERA_INTERFACE_SLOT (const char*, name, (LumieraInterface)),
@@ -18,4 +87,21 @@ LUMIERA_INTERFACE_DECLARE (interface_descriptor, 0,
                            LUMIERA_INTERFACE_SLOT (const char*, license, (LumieraInterface))
                            /* TODO add more things here, dependencies, provisions etc */
                            );
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+-----------------------------------------
+
+
+Lumiera Plugin API
+-------------------
+The Lumiera Interface/Plugin framework provides some functions to manage Plugins.
+Actually a user requests interfaces. The libraries which implement Plugins are managed transparently.
+Interfaces are exported as instances and are not necessary singleton. This means that a single Plugin
+can export the same interface type several times under different names. The naming rules for interfaces
+need to be defined elsewhere.
+
+loading and opening a Plugin
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Plugins are looked up in `$LUMIERA_PLUGIN_PATH`, which is a colon separated list of directories,
+and then in a specific ``Lumiera plugin dir'', where standard plugins get installed alongside
+with the Application
+
+

doc/design/plugins/PluginVersioning.txt

@@ -0,0 +1,44 @@
+ PluginVersioningCases
+======================
+:Author: MichaelPloujnikov
+:Date Created: 200707121127
+:Date Changed: 200707160404
+:Count Changes: 46
+
+//MENU: label Plugin Version
+
+
+Compatibility matrix
+--------------------
+
+.Source compatibility
+[grid="rows"]
+`100`200`200~~~~
+*~CALLER~ \ ^CALLEE^**,OLD^**^,NEW^**^
+OLD,works,works but a recent interface definition must be available
+NEW,works,works
+~~~~
+
+.Binary compatibility
+[grid="rows"]
+`100`200`200~~~~
+*~CALLER~ \ ^CALLEE^**,OLD^**^,NEW^**^
+OLD,works,works
+NEW,caller gets "revision not sufficient" at runtime and should implement fallbacks,works
+~~~~
+
+^*^) CALLER is the user of an interface, CALLEE is the interface provider (usually a plugin)
+
+^**^) OLD means an initial revision, NEW means some later revision of an interface
+
+Observations
+------------
+
+Compiling a newer Plugin for some older main application release has some quirks (interface definitions are intended to be shipped with the main application). This should be rarely the case.
+
+When compiling, older Plugins should be updated to new interface revisions.
+Caller should provide a fallback to older interface revisions for binary compatibility.
+
+Generally, sources should just be properly maintained and updated to use the most recent interfaces revision.
+
+For binary compatibility everything will work well, provided that the caller kept proper fallback functionality for older interface revisions. Plugins which are independently distributed (packaged) in binary form don't need to be updated with every new main application release and just work.

doc/technical/build/Dependencies.txt

@@ -0,0 +1,90 @@
+Dependencies
+------------
+:Author: CehTeh
+:Date: 3/2008
+
+
+Lumiera is written for GNU/Linux. We try to make the best out of modern system programming techniques
+to reach the best possible performance. Lumiera shall scale with the provided Hardware,
+the more RAM and the more/faster CPU's you have the better.
+Nevertheless lower end 32bit machines are supported too.
+
+Secondary targets will be other free operating systems which offer a decent Posix API. +
+Porting to other more or less similar platforms will be possible, if -- by coincidence -- 
+Someone(TM) helps with porting.
+
+Having said that -- for the time being, the core team won't spend much effort on porting.
+
+Platform
+--------
+We work and test on PC hardware, 32 and 64 bit. It is intended that Lumiera supports
+other platforms running run GNU/Linux.
+Lumiera expects a 'standard' desktop installation running a Xserver.
+
+Graphics::
+  There are no special requirements for the graphic system.
+  Hardware accelleration will likely be added later through extensions,
+  but will remain strictyl optional. (For now we'll watch the evolution
+  in that area and might revisit that topic when there are more compelling
+  and widely supported solutions available)
+
+Disks::
+  Video editing requires decent disk speed, so it is suggested to use a
+  fast/big array of disks configured as raid.
+
+Special Hardware::
+  Sopport for special hardware would be possible, but depends on certain conditions
++
+* we need access / donations for the hardware
+* specs and APIs must be open.
+* someone to do the actual interfacing and support needs to join the team
+
+
+Languages and Tools
+-------------------
+
+* C / C++
+
+ - a C99 / C++98 compatible compiler
+ - GCC 4.4 or better is fine. Basically we try to use just the stock language.
+   On rare occasions, we _did_ use some GCC extensions, but there would be workarounds,
+   should this become a problem.
+ - std::tr1 extensions for C++ (smart-ptrs, hashtables, function objects)
+ 
+* BOOST (listed below are the DEBIAN package names)
+ - libboost-dev (at least 1.40)
+ - libboost-program-options-dev
+ - libboost-program-options-dev
+ - libboost-filesystem-dev
+ - libboost-regex-dev
+
+* Script languages
+ - Python (2.5) for build scripts
+ - bash (some test scripts use bash specific extensions)
+ - Lua is planned to become a general glue and binding language
+
+Build Tools
+~~~~~~~~~~~
+
+* Git
+* SCons
+* pkg-config
+* Doxygen
+
+We maintain a Debian package (debhelper, CDBS, git-buildpackage)
+
+Libraries
+~~~~~~~~~
+
+* BOOST
+* NoBug
+* http://gmerlin.sourceforge.net/gavl.html[GAVL] (for raw media support) 
+* for the GUI: gtkmm-2.4 gdl-1.0 libglibmm-2.4 cairomm-1.0 xv
+ - libgtkmm-2.4-dev
+ - libcairomm-1.0-dev
+ - libgdl-1-dev
+ - libglibmm-2.4-dev, requiring glib2.0 and gthread-2.0
+ - libxv-dev
+ - librsvg-2.0 and librsvg2-dev for rendering Icons
+
+

doc/technical/build/index.txt

@@ -7,9 +7,8 @@ As work progresses, we will add more information on the Lumiera build system.
 
 build -- continuous integration -- packaging
 
-* SCons
-* Autotools
-* Dependencies
+* link:SCons.html[Buildsystem]
+* link:Dependencies.html[Dependencies]
 * link:BuildDroneDraft.html[»Builddrone« concept from 2008]
 * Packaging: link:LumieraDebianPackage.html[Debian] RPM
 * Lumiera link:../infra/debianDepot.html/[debian depot]